M O A Spec Llama 3 8 B Instruct

Introduction

The MOASpec-Llama-3-8B-Instruct is a checkpoint developed from the research paper "Mixture of Attentions for Speculative Decoding." This work introduces a new architecture that improves the inference speed of large language models (LLMs).

Architecture

The model is built on the "Mixture of Attentions" framework, which is designed to optimize speculative decoding. This technique enhances the efficiency of LLM inference, making it faster and more effective.

Training

The MOASpec-Llama-3-8B-Instruct model is based on the Meta-Llama-3-8B architecture, incorporating a mixture of attentions to fine-tune its parameters. The base model consists of 8 billion parameters, while the MOA Spec parameters are 0.25 billion.

Guide: Running Locally

1. Clone the repository from GitHub: git clone https://github.com/huawei-noah/HEBO/tree/mixture-of-attentions/
2. Install necessary dependencies.
3. Load the model using the Hugging Face Transformers library.
4. Run inference tasks as needed.

For optimal performance, it is recommended to use cloud GPUs, such as those provided by AWS, Google Cloud, or Azure.

License

This project is licensed under the MIT License. For more information, refer to the LICENSE file. Note that this open-source project is not officially supported by Huawei.