Flux.1 dev Controlnet Upscaler

Introduction

The FLUX.1-DEV-ControlNet-Upscaler is a model developed by the Jasper research team for enhancing low-resolution images. It is designed for use with the diffusers library and is part of the ControlNet suite, focusing on image-to-image super-resolution tasks.

Architecture

The model leverages the FluxControlNetModel and FluxControlNetPipeline within the diffusers library to handle image upscaling tasks. It integrates ControlNet for conditioning and controlling the image generation process, allowing for precise adjustments in super-resolution tasks.

Training

The model was trained using a synthetic data degradation scheme. This approach involves artificially degrading real-life images with various techniques such as Gaussian and Poisson noise, blurring, and JPEG compression. The training methodology is inspired by the work of Wang et al. ("Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data"), which focuses on preparing models to handle real-world image degradation scenarios.

Guide: Running Locally

To run the FLUX.1-DEV-ControlNet-Upscaler model locally, follow these steps:

1. Install Required Libraries: Ensure you have Python and the diffusers library installed.
2. Load the Model:

   import torch
   from diffusers.utils import load_image
   from diffusers import FluxControlNetModel
   from diffusers.pipelines import FluxControlNetPipeline
   
   # Load pipeline
   controlnet = FluxControlNetModel.from_pretrained(
     "jasperai/Flux.1-dev-Controlnet-Upscaler",
     torch_dtype=torch.bfloat16
   )
   pipe = FluxControlNetPipeline.from_pretrained(
     "black-forest-labs/FLUX.1-dev",
     controlnet=controlnet,
     torch_dtype=torch.bfloat16
   )
   pipe.to("cuda")
   

3. Prepare Input Image:

   # Load a control image
   control_image = load_image(
     "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Upscaler/resolve/main/examples/input.jpg"
   )
   w, h = control_image.size
   control_image = control_image.resize((w * 4, h * 4))
   

4. Run the Model:

   image = pipe(
       prompt="", 
       control_image=control_image,
       controlnet_conditioning_scale=0.6,
       num_inference_steps=28, 
       guidance_scale=3.5,
       height=control_image.size[1],
       width=control_image.size[0]
   ).images[0]
   

For optimal performance, it is recommended to use a cloud GPU service such as AWS, Google Cloud, or Azure.

License

The model is licensed under the Flux.1-dev non-commercial license. More details can be found in the license document.