sdxl emoji

Introduction

The SDXL-EMOJI model by fofr is a fine-tuned version of the stable-diffusion model, adapted for generating emoji-style images. It leverages Apple emojis for its training and is compatible with various modern machine learning frameworks, including Hugging Face's Diffusers library.

Architecture

The SDXL-EMOJI model utilizes the Stable Diffusion architecture with additional training from LoRA (Low-Rank Adaptation) techniques. The base model used is stabilityai/stable-diffusion-xl-base-1.0, and it incorporates Pivotal Tuning with Dreambooth and Textual Inversion for improved performance in emoji generation.

Training

The model was trained by fine-tuning on existing emoji datasets using Pivotal Tuning, which combines Dreambooth LoRA with Textual Inversion. This approach allows the model to learn new concepts and representations specific to emoji design.

Guide: Running Locally

Steps to Run

1. Install Required Packages:

   pip install replicate diffusers transformers accelerate safetensors huggingface_hub
   

2. Clone Necessary Repositories:

   git clone https://github.com/replicate/cog-sdxl cog_sdxl
   

3. Inference with Replicate API:

   • Obtain a Replicate API token.
   • Run the following Python script:

     import replicate
     
     output = replicate.run(
         "sdxl-emoji@sha256:dee76b5afde21b0f01ed7925f0665b7e879c50ee718c5f78a9d38e04d523cc5e",
         input={"prompt": "A TOK emoji of a man"}
     )
     print(output)
     

4. Inference with Diffusers:

   • Use the DiffusionPipeline from Hugging Face:

     import torch
     from huggingface_hub import hf_hub_download
     from diffusers import DiffusionPipeline
     from cog_sdxl.dataset_and_utils import TokenEmbeddingsHandler
     
     pipe = DiffusionPipeline.from_pretrained(
             "stabilityai/stable-diffusion-xl-base-1.0",
             torch_dtype=torch.float16,
             variant="fp16",
     ).to("cuda")
     
     pipe.load_lora_weights("fofr/sdxl-emoji", weight_name="lora.safetensors")
     
     embedding_path = hf_hub_download(repo_id="fofr/sdxl-emoji", filename="embeddings.pti", repo_type="model")
     embhandler = TokenEmbeddingsHandler(pipe.text_encoder, pipe.tokenizer)
     embhandler.load_embeddings(embedding_path)
     prompt="A <s0><s1> emoji of a man"
     images = pipe(prompt, cross_attention_kwargs={"scale": 0.8}).images
     #your output image
     images[0]
     

Suggested Cloud GPUs

For optimal performance, it is recommended to use cloud-based GPUs such as those offered by AWS (Amazon Web Services), GCP (Google Cloud Platform), or Azure.

License

The SDXL-EMOJI model is released under the CreativeML Open RAIL-M license, which allows for a wide range of uses while ensuring ethical guidelines are followed.