anime painter

Introduction

Anime-Painter, developed by xinsir, is a ControlNet model designed to transform simple sketches into high-quality anime illustrations. It supports various line widths and styles, allowing users without drawing skills to create anime-style images using prompts and tags. The model demonstrates improved performance compared to previous iterations, offering high aesthetic scores and enhanced prompt-following capabilities.

Architecture

Anime-Painter utilizes the ControlNet-SDXL architecture, finetuned from the stable-diffusion-xl-base-1.0 model by StabilityAI. It integrates complex data augmentation techniques and a vast, filtered dataset of anime images to achieve superior results in generating visually appealing anime illustrations.

Training

The model was trained using a large dataset of anime images, filtered for high visual quality. Training involved complex tricks to enhance performance, particularly in aesthetic scoring and prompt adherence. The model was evaluated against popular anime images and demonstrated improved aesthetic scores and control abilities.

Guide: Running Locally

1. Installation and Setup:

   • Install the necessary libraries, including diffusers, torch, PIL, numpy, and opencv-python.
   • Ensure you have access to a compatible GPU for efficient processing.

2. Model Initialization:

   • Load the ControlNetModel and StableDiffusionXLControlNetPipeline from the diffusers library.
   • Initialize the model and associated components with pre-trained weights.

3. Image Preparation:

   • Prepare a sketch or use a pre-defined image to generate a scribble using the HED detector.
   • Resize the image to a resolution of 1024x1024 for optimal performance.

4. Image Generation:

   • Define prompts and negative prompts for image generation.
   • Use the pipeline to generate images based on the provided sketches and prompts.

5. Output:

   • Save the generated images in a high-quality format such as PNG.

Cloud GPUs: Consider using cloud-based GPU services like AWS, Google Cloud, or Azure for enhanced performance.

License

Anime-Painter is released under the Apache-2.0 license, allowing for wide usage and modification while maintaining attributions.