vit gpt2 image captioning

Introduction

The ViT-GPT2 Image Captioning model, developed by NLPConnect, is designed to generate descriptive text captions for images. This model leverages the Vision Transformer (ViT) and GPT-2 architectures to achieve image-to-text conversion, making it suitable for tasks like image captioning.

Architecture

ViT-GPT2 Image Captioning model is a combination of ViT as the vision encoder and GPT-2 as the language model decoder. The model processes input images to generate captions, utilizing transformers for both visual and textual data processing. The architecture is implemented in PyTorch.

Training

The model was initially trained by @ydshieh in Flax, with a PyTorch version provided for easier integration with existing PyTorch workflows. The training data and procedures adhere to the requirements for efficient image captioning tasks, utilizing datasets like COCO for training purposes.

Guide: Running Locally

1. Install Dependencies: Ensure that transformers, torch, and PIL are installed in your Python environment.
2. Load the Model: Use the VisionEncoderDecoderModel, ViTImageProcessor, and AutoTokenizer from the transformers library to load the model and preprocessing tools.
3. Set Up Device: Determine if a CUDA-enabled GPU is available, and set the model to use it for faster inference.
4. Run Inference: Use the provided predict_step function to generate captions for your images. Alternatively, use the pipeline method for a more streamlined approach.

Consider utilizing cloud GPU providers such as AWS, GCP, or Azure for enhanced computational resources, especially for large-scale image processing tasks.

License

The ViT-GPT2 Image Captioning model is licensed under the Apache-2.0 License, allowing users to freely use, modify, and distribute the software while adhering to the license terms.