AnimateDiff-Lightning

Introduction

AnimateDiff-Lightning is a lightning-fast text-to-video generation model developed by ByteDance. It is capable of generating videos more than ten times faster than the original AnimateDiff model. The model leverages cross-model diffusion distillation and is available for research purposes. More details can be found in the research paper titled "AnimateDiff-Lightning: Cross-Model Diffusion Distillation."

Architecture

The model is distilled from AnimateDiff SD1.5 v2 and provides checkpoints for 1-step, 2-step, 4-step, and 8-step models. The 2-step, 4-step, and 8-step models offer high-quality video generation, while the 1-step model is intended for research purposes. The model performs best when used with stylized base models, including both realistic and anime/cartoon styles.

Training

AnimateDiff-Lightning employs a diffusion distillation process. It is optimized for rapid video generation, supporting various base models to enhance output quality. Users are encouraged to experiment with different settings, such as using Motion LoRAs with a strength of 0.7 to 0.8 to avoid watermarks.

Guide: Running Locally

Basic Steps

1. Set Up Environment: Ensure you have a CUDA-compatible device and necessary libraries installed, such as PyTorch and Diffusers.
2. Download Checkpoints: Obtain the AnimateDiff-Lightning checkpoints and a preferred base model.
3. Run Pipeline: Utilize the AnimateDiffPipeline to generate videos from text prompts.
4. Export and Adjust: Export the resulting frames to GIF or other formats and adjust settings like inference steps for optimal results.

Suggested Cloud GPUs

To efficiently run AnimateDiff-Lightning, consider using cloud GPU services such as AWS EC2 with NVIDIA GPUs, Google Cloud's AI Platform, or Microsoft Azure's GPU instances.

License

AnimateDiff-Lightning is released under the creativeml-openrail-m license. Users are encouraged to refer to the license terms for usage guidelines and compliance.