rst topic classification 11b

Introduction

The RST-Topic-Classification-11B model is part of the reStructured Pre-training (RST) paradigm, which unifies 26 types of signals from diverse data sources into a monolithic model. It achieves superior performance across various NLP tasks and examinations, notably outperforming competitors like GPT-3 in certain scenarios.

Architecture

The RST models, including the RST-Topic-Classification-11B, consist of 11 billion parameters. They are designed to handle multiple NLP tasks through data-centric pre-training and JSON-based storage mechanisms, enhancing data accessibility and efficiency.

Training

The models are trained using signals from 10 data sources, such as WordNet, Wikipedia, and arXiv. The training process emphasizes data storage and retrieval, allowing for efficient fine-tuning across different downstream tasks.

Model Variants

• rst-all-11b: General-purpose model for all applications.
• rst-fact-retrieval-11b: Knowledge-intensive tasks.
• rst-summarization-11b: Summarization tasks.
• rst-temporal-reasoning-11b: Temporal reasoning tasks.
• rst-information-extraction-11b: Information extraction tasks.
• rst-intent-detection-11b: Intent prediction tasks.
• rst-topic-classification-11b: General text classification.
• rst-word-sense-disambiguation-11b: Word sense disambiguation.
• rst-natural-language-inference-11b: Natural language inference.
• rst-sentiment-classification-11b: Sentiment analysis.
• rst-gaokao-rc-11b: Multiple-choice question answering.
• rst-gaokao-cloze-11b: Cloze tests.
• rst-gaokao-writing-11b: Essay writing and grammar correction.

Guide: Running Locally

To run the RST models locally, follow these steps:

1. Install Libraries:

   • Ensure you have Python and pip installed. Use pip to install the Transformers library:

     pip install transformers
     

2. Load the Model:

   from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
   
   tokenizer = AutoTokenizer.from_pretrained("XLab/rst-all-11b")
   model = AutoModelForSeq2SeqLM.from_pretrained("XLab/rst-all-11b")
   

3. Prepare Input and Generate Output:

   inputs = tokenizer.encode("TEXT: your input text here", return_tensors="pt")
   outputs = model.generate(inputs)
   print(tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
   

4. Cloud GPUs:

   • For enhanced performance, consider using cloud GPUs available through platforms such as AWS, Google Cloud, or Azure.

License

The RST models are released under the Academic Free License v3.0 (AFL-3.0).