byt5 small

Introduction
ByT5-Small is a tokenizer-free version of Google's T5 model, designed to work directly on raw UTF-8 byte sequences. It follows the architecture of MT5 and is pre-trained on the multilingual C4 (mC4) dataset without any supervised training. This model is particularly effective for handling noisy text data.

Architecture
ByT5 operates on byte sequences using a standard Transformer architecture, requiring minimal modifications. It is capable of processing text in multiple languages directly from raw bytes, making it robust to noise and effective for tasks sensitive to spelling and pronunciation. The model is pre-trained with a focus on byte-level processing to compete with token-level counterparts in terms of performance and efficiency.

Training
ByT5 was pre-trained using mC4 with an average span-mask of 20 UTF-8 characters. It requires fine-tuning for specific downstream tasks. The training methodology emphasizes the model's robustness to noise and its capability to handle different languages without a tokenizer.

Guide: Running Locally
To run ByT5-Small locally, follow these steps:

1. Install Transformers Library:
   Ensure you have the transformers library installed. You can install it using pip:

   pip install transformers
   

2. Load the Model:
   Use the following Python code to load the model:

   from transformers import T5ForConditionalGeneration, AutoTokenizer
   import torch
   
   model = T5ForConditionalGeneration.from_pretrained('google/byt5-small')
   tokenizer = AutoTokenizer.from_pretrained('google/byt5-small')
   

3. Prepare Inputs:
   Encode your input text with UTF-8 encoding, and adjust for special tokens:

   input_ids = torch.tensor([list("Your text here".encode("utf-8"))]) + 3
   

4. Inference or Training:
   For batched inference or training, use the tokenizer to handle padding:

   model_inputs = tokenizer(["Text1", "Text2"], padding="longest", return_tensors="pt")
   

5. Suggestion for Cloud GPUs:
   Consider using cloud services like AWS, Google Cloud, or Azure for access to powerful GPUs that can accelerate training and inference.

License
ByT5-Small is released under the Apache-2.0 License, allowing for broad usage and modification with attribution.