Sketch Paint

Introduction

The Sketch-Paint model, created by STRANGERZONEHF, is a text-to-image generation tool designed to transform textual descriptions into artistic images. It utilizes advanced diffusion techniques to produce high-quality and diverse visuals based on given prompts.

Architecture

The model is built using the Diffusers library and incorporates LoRA (Low-Rank Adaptation) to enhance its capability of generating detailed images. The base model for Sketch-Paint is black-forest-labs/FLUX.1-dev, which provides a robust foundation for the image generation process. The model employs specific image processing parameters such as a constant learning rate scheduler and an AdamW optimizer to fine-tune its outputs.

Training

The model was trained with a total of 19 images using parameters like:

• LR Scheduler: constant
• Optimizer: AdamW
• Network Dimensions: 64
• Epochs: 17

The training involved detailed multi-resolution noise discounting and specific inference steps, recommended between 30–35, for optimal results.

Guide: Running Locally

1. Setup Environment:

   • Ensure you have torch and the Diffusers library installed.
   • Set up your environment to support CUDA for GPU acceleration.

2. Load Model:

   import torch
   from pipelines import DiffusionPipeline
   
   base_model = "black-forest-labs/FLUX.1-dev"
   pipe = DiffusionPipeline.from_pretrained(base_model, torch_dtype=torch.bfloat16)
   
   lora_repo = "strangerzonehf/Sketch-paint"
   pipe.load_lora_weights(lora_repo)
   
   device = torch.device("cuda")
   pipe.to(device)
   

3. Generate Images:

   • Use the trigger word "Sketch paint" to initiate the image generation process.
   • Adjust the dimensions and aspect ratio based on your requirements, with recommendations being 1280 x 832 (3:2) for best results.

4. Cloud GPUs:

   • For enhanced performance, consider using cloud GPU services such as AWS EC2, Google Cloud, or Azure.

License

The Sketch-Paint model is distributed under the CreativeML Open RAIL-M license, allowing for its use in various applications while adhering to the specified conditions.