sdxl_chinese_ink_lora

Introduction

The SDXL_CHINESE_INK_LORA model is a finetuned version of the Stable Diffusion XL model, specifically designed for creating images in the style of contemporary Chinese ink paintings.

Architecture

This model utilizes the Stable Diffusion XL base model, enhanced by the LCM-LORA adapter for faster inference and the Chinese Ink LORA for stylistic rendering.

Training

The model is trained using the stabilityai/stable-diffusion-xl-base-1.0 as the base, with additional adapters for style rendering. The inference process leverages LCM-LORA to accelerate sampling.

Guide: Running Locally

1. Install Required Libraries:

   Update pip and install necessary libraries:

   pip install --upgrade pip
   pip install --upgrade diffusers transformers accelerate peft
   pip install matplotlib
   

2. Load the Model:

   Load the base model and adapters:

   import torch
   from diffusers import DiffusionPipeline, LCMScheduler
   import matplotlib.pyplot as plt
   
   pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",
                                            variant="fp16",
                                            torch_dtype=torch.float16
                                            ).to("cuda")
   
   pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
   pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl", adapter_name="lcm")
   pipe.load_lora_weights("ming-yang/sdxl_chinese_ink_lora", adapter_name="Chinese Ink")
   pipe.set_adapters(["lcm", "Chinese Ink"], adapter_weights=[1.0, 0.8])
   

3. Generate Images:

   Use the trigger word "Chinese Ink" to generate images:

   prompts = ["Chinese Ink, mona lisa picture, 8k", "mona lisa, 8k"]
   generator = torch.manual_seed(1)
   images = [pipe(prompt, num_inference_steps=8, guidance_scale=1, generator=generator).images[0] for prompt in prompts]
   
   fig, axs = plt.subplots(1, 2, figsize=(40, 20))
   axs[0].imshow(images[0])
   axs[0].axis('off')
   axs[1].imshow(images[1])
   axs[1].axis('off')
   plt.show()
   

4. Cloud GPUs:

   Consider using cloud-based GPUs such as AWS EC2, Google Cloud, or Azure for efficient execution.

License

This model is released under the creativeml-openrail-m license. Please ensure compliance with the license terms when using the model.