rag token base

Introduction

RAG-Token is a model from the paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis et al. It consists of a question encoder, retriever, and generator. The model is designed for knowledge-intensive tasks and leverages retrieval-augmented generation.

Architecture

RAG-Token consists of:

• Question Encoder: Can be any model compatible with AutoModel.
• Retriever: Should be a RagRetriever instance.
• Generator: Can be any model compatible with AutoModelForSeq2SeqLM.

The model uses a non-finetuned version and is designed to convert all capital input letters to lowercase.

Training

To fine-tune the RAG-Token model:

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
retriever = RagRetriever.from_pretrained("facebook/rag-token-base")
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-base", retriever=retriever)

input_dict = tokenizer.prepare_seq2seq_batch("who holds the record in 100m freestyle", "michael phelps", return_tensors="pt") 

outputs = model(input_dict["input_ids"], labels=input_dict["labels"])

loss = outputs.loss

# train on loss

Note that optimal results are obtained by using the full retriever with config.index_name="legacy" and config.use_dummy_dataset=False.

Guide: Running Locally

1. Installation: Ensure you have the transformers library installed.
2. Load the Model:

   from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
   
   tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
   retriever = RagRetriever.from_pretrained("facebook/rag-token-base")
   model = RagTokenForGeneration.from_pretrained("facebook/rag-token-base", retriever=retriever)
   

3. Run Inference:
   • Prepare your input and run the model to obtain outputs.
4. Cloud GPUs: For better performance, consider using cloud GPUs like AWS EC2, Google Cloud, or Azure to handle computation-heavy tasks.

License

This model is licensed under the Apache 2.0 License.