rugpt3large_based_on_gpt2

Introduction

The RUGPT3LARGE_BASED_ON_GPT2 is a language model specifically designed for the Russian language. It utilizes advanced transformer architecture for text generation tasks, leveraging the capabilities of both PyTorch and the Transformers library. This model has been developed and maintained by the SberDevices team.

Architecture

The model is based on the GPT-2 architecture, which is a transformer-based model widely used for generating human-like text. It has been adapted for the Russian language, using a large corpus for pretraining. The architecture allows for a sequence length of up to 2048 tokens, making it suitable for a variety of text generation tasks.

Training

The model was pretrained on 80 billion tokens for three epochs with a sequence length of 1024 tokens and then finetuned for an additional epoch with a sequence length of 2048 tokens. The training process involved 128 GPUs for the initial phase and 16 GPUs for finetuning, totaling approximately 14 days of training. The final model achieved a perplexity score of 13.6 on the test set, indicating its proficiency in generating coherent text.

Guide: Running Locally

To run the RUGPT3LARGE_BASED_ON_GPT2 model locally, follow these steps:

1. Install Required Packages: Ensure you have Python installed, then set up a virtual environment and install the transformers and torch libraries.

   pip install transformers torch
   

2. Load the Model: Use the Transformers library to load the model and tokenizer.

   from transformers import GPT2LMHeadModel, GPT2Tokenizer
   
   model_name = "ai-forever/rugpt3large_based_on_gpt2"
   model = GPT2LMHeadModel.from_pretrained(model_name)
   tokenizer = GPT2Tokenizer.from_pretrained(model_name)
   

3. Generate Text: Use the model to generate text by providing a prompt.

   input_text = "Ваш текст здесь"
   input_ids = tokenizer.encode(input_text, return_tensors='pt')
   output = model.generate(input_ids, max_length=200, num_return_sequences=1)
   print(tokenizer.decode(output[0], skip_special_tokens=True))
   

For optimal performance, especially with larger models, consider using cloud GPUs from providers like AWS, Google Cloud, or Azure.

License

The RUGPT3LARGE_BASED_ON_GPT2 model follows the licensing agreements set by its authors and maintaining organization. Users should refer to the official documentation or contact the developers for specific licensing details.