flux ip adapter v2

Introduction

FLUX-IP-ADAPTER-V2 is an image-to-image model designed by XLABS AI, utilizing the FLUX.1-dev model as its base. It is designed for use with ComfyUI and supports image generation workflows.

Architecture

The IP Adapter employs a resolution of 512x512 for 150k training steps and 1024x1024 for 350k steps. It maintains the aspect ratio during training, enhancing its usability for image-to-image tasks. The model integrates with the diffusers library and supports an English language interface.

Training

The model was trained using datasets such as CaptionEmporium/coyo-hd-11m-llavanext and CortexLM/midjourney-v6. The training process involved maintaining aspect ratios and varying resolutions to optimize the image generation capabilities.

Guide: Running Locally

To run the model locally, follow these steps:

1. Clone the Repository:
   Navigate to the ComfyUI/custom_nodes directory and clone the repository x-flux-comfyui. Ensure the file path is ComfyUI/custom_nodes/x-flux-comfyui/*.

2. Setup Environment:
   In the cloned directory, execute python setup.py to set up the environment.

3. Update Repository:
   Use git pull to update or reinstall as necessary.

4. Download Necessary Models:
   Download the Clip-L model.safetensors from OpenAI's VIT CLIP large model and place it in ComfyUI/models/clip_vision/*.
   Obtain the IPAdapter from Hugging Face and place it in ComfyUI/models/xlabs/ipadapters/*.

5. Run Inference:
   Use the Flux Load IPAdapter and Apply Flux IPAdapter nodes, select the appropriate CLIP model, and start generating images.

6. Testing:
   If results are suboptimal, adjust the IP strength settings and review example workflows provided in the repository.

For optimal performance, consider using cloud GPUs from providers like AWS, Google Cloud, or Azure.

License

The model is distributed under the FLUX.1-dev Non-Commercial License. More details can be found here.