flux ip adapter

Introduction

This repository provides an IP-Adapter checkpoint for the FLUX.1-dev model by Black Forest Labs. It is designed to facilitate text-to-image generation, leveraging Stable Diffusion, and ControlNet technologies. The model is available for use in ComfyUI, a user-friendly interface for working with AI models.

Architecture

The IP-Adapter is trained to handle image resolutions of both 512x512 and 1024x1024. Initially, it is trained on 512x512 resolution for 50,000 steps and on 1024x1024 resolution for 25,000 steps. This dual-resolution capability allows for versatile image generation, making it suitable for various applications within ComfyUI.

Training

The model is in active development, with new checkpoints being released regularly. The initial v1 version is ready for use, and users are encouraged to stay updated with the latest releases. The training process involves fine-tuning for different resolutions to ensure high-quality output.

Guide: Running Locally

To run the IP-Adapter locally, follow these steps:

1. Clone the Repository:

   • Navigate to ComfyUI/custom_nodes and clone the repository using:

     git clone https://github.com/XLabs-AI/x-flux-comfyui.git
     

   • Ensure the path is ComfyUI/custom_nodes/x-flux-comfyui/*.

2. Setup:

   • Navigate to the cloned directory:

     cd ComfyUI/custom_nodes/x-flux-comfyui/
     

   • Run the setup script:

     python setup.py
     

3. Update:

   • Regularly update the repository with:

     git pull
     

4. Download Required Models:

   • Obtain the Clip-L model from OpenAI's Clip-VIT Large and place it in ComfyUI/models/clip_vision/*.
   • Download the IP-Adapter from Hugging Face and place it in ComfyUI/models/xlabs/ipadapters/*.

5. Utilize the Model:

   • Use the Flux Load IPAdapter and Apply Flux IPAdapter nodes within ComfyUI, selecting the appropriate CLIP model to generate images.

6. Example Workflows:

   • Example workflows are available in the workflows folder of the repository.

For improved performance and efficiency, consider using cloud GPUs, such as those offered by AWS, Google Cloud, or Azure.

License

The IP-Adapter's weights are distributed under the FLUX.1 Non-Commercial License. For more details, refer to the license documentation.