Rwkv In Qwen2.5 7 B base

Introduction

The RWKVInQwen2.5-7B-BASE model is a transformation of the Qwen2.5-7B's checkpoint to the RWKV-7 architecture. This model has been trained for one day on a single server equipped with 8xA800 GPUs. Although it delivers satisfactory conversational performance, it lacks proficiency in mathematical tasks. Future improvements are planned by incorporating a more varied dataset.

Architecture

The model utilizes the RWKV architecture, which is noted for its scalability and efficiency. This architecture is designed to handle large-scale language processing tasks, making it a suitable choice for converting the Qwen2.5-7B checkpoints.

Training

Training was conducted on a server with 8xA800 GPUs over a single day. While the model performs well in generating conversational text, its current training did not emphasize mathematical capabilities, which limits its performance in math-related tasks.

Guide: Running Locally

To run the model locally, follow these steps:

1. Clone the Repository: Ensure you have the necessary code locally by cloning the repository.
2. Install Dependencies: Make sure all required libraries and dependencies are installed.
3. Configure Settings: Modify the configuration YAML file to point to your local checkpoint path.
4. Run the Model: Use the command below to start the chat interface:

   python tests/test_chat_cli.py --config_file config.yaml --is_hybrid --num_gpus 1
   

For optimal performance, consider using cloud GPUs such as those available from AWS, Google Cloud, or Azure.

License

The model and its code are distributed under the terms of the Apache License 2.0, allowing users to freely use, modify, and distribute the software, provided they comply with the terms of the license.