T5-Efficient-Tiny: Deep-Narrow Version

Introduction

T5-Efficient-Tiny is a compact version of Google's T5 model, optimized with a deep-narrow architecture for improved performance in NLP tasks. This model was introduced in the paper "Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers."

Architecture

The T5-Efficient-Tiny model follows a deep-narrow configuration, which means it has a greater depth (number of transformer blocks) relative to its width (other dimensions). This design choice improves efficiency and performance. The model comprises 15.58 million parameters, requiring approximately 62.32 MB in full precision or 31.16 MB in half precision.

Training

Pre-Training

The model was pretrained using span-based masked language modeling on the C4 dataset for 524,288 steps.

Fine-Tuning

This model is a pretrained checkpoint and requires fine-tuning for specific tasks. Fine-tuning examples are available for:

• PyTorch: Summarization, Question Answering, Text Classification
• TensorFlow: Summarization, Text Classification
• JAX/Flax: Summarization, Text Classification

Guide: Running Locally

1. Environment Setup: Install the required libraries, such as transformers and torch or tensorflow.
2. Download the Model: Obtain the T5-Efficient-Tiny model from Hugging Face's model hub.
3. Fine-Tuning: Adapt existing fine-tuning scripts as needed for your specific task and framework.
4. Inference: Use the model for text generation tasks, ensuring your input data is preprocessed appropriately.

For optimal performance, consider using cloud GPUs like AWS, Google Cloud, or Azure.

License

T5-Efficient-Tiny is licensed under the Apache 2.0 License, allowing for broad use and modification.