Reformer Model: Crime and Punishment

Introduction

The Reformer model is a language model specifically trained on the novel "Crime and Punishment" by Fyodor Dostoevsky. This model is capable of generating text based on the novel's style and content.

Architecture

The Reformer model was trained using a ReformerLM configuration. It utilizes small sub-word units to improve text generation efficiency and quality. The model was initially trained using the Flax framework and later converted to Hugging Face's PyTorch format as ReformerModelWithLMHead.

Training

The training data for the model comes from a dataset containing approximately 0.5 million tokens sourced from the text of "Crime and Punishment." The training process was conducted using a Colab notebook provided by the authors of the Reformer model, which can be accessed here.

Guide: Running Locally

1. Installation: Ensure you have Python and PyTorch installed. Install the Hugging Face Transformers library:

   pip install transformers
   

2. Loading the Model:

   from transformers import ReformerModelWithLMHead, ReformerTokenizer
   
   model = ReformerModelWithLMHead.from_pretrained("google/reformer-crime-and-punishment")
   tok = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
   

3. Generate Text:

   input_text = "A few months later"
   input_ids = tok.encode(input_text, return_tensors="pt")
   output = model.generate(input_ids, do_sample=True, temperature=0.7, max_length=100)
   generated_text = tok.decode(output[0])
   print(generated_text)
   

4. GPU Recommendations: For better performance, especially when generating large texts, consider using cloud GPUs such as those offered by Google Cloud or AWS.

License

The model and its code are available under the licenses specified by Hugging Face and the original authors of the Reformer model. Please refer to their documentation for specific licensing details.