ltx video 0.9 vae finetune

Introduction

The LTX Video 0.9 VAE Fine-Tune is a modified version of the original LTX Video 0.9 VAE model aimed at reducing checkerboard artifacts. These artifacts are a common issue in the original model due to its architecture. The finetuning focuses mainly on the decoder to preserve the latent space.

Architecture

The model architecture involves strided convolutions in the encoder and pixel shuffle upscaling in the decoder, which contribute to the checkerboard artifacts. The finetuning process includes two versions: one with only the decoder finetuned and another with both the encoder and decoder finetuned.

Training

Training primarily targeted the decoder to avoid altering the latent space. Some limited training of the encoder was also done, with the decoder frozen, to further reduce artifacts. This approach was partially successful, reducing but not completely eliminating the artifacts.

Guide: Running Locally

1. Clone the Repository: Obtain the model files from the Hugging Face repository.
2. Install Dependencies: Ensure all required libraries and dependencies are installed.
3. Load the Model: Initialize the finetuned model version of your choice.
4. Inference: Run the model on your video data to test its performance.

For optimal performance, consider using cloud GPUs such as those offered by AWS, Google Cloud Platform, or Azure.

License

The model is under the OpenRAIL license, which is unchanged from the original. The release of the training code and a commercially permissive license is pending from Lightricks.