Deep Seek V3 A W Q

Introduction

The DeepSeek-V3-AWQ is a quantized version of the DeepSeek V3 chat model, which is optimized for text generation. This model addresses overflow issues by modifying the model code to use 4-bit precision.

Architecture

The model uses the Transformers library and is designed for text generation tasks. It supports both English and Chinese languages and is built on the DeepSeek V3 base model. The architecture includes custom code to enhance performance with 4-bit precision and uses the AWQ method for quantization.

Training

The model has been tested on vLLM with an 8x H100 setup. During testing, it achieved an inference speed of 5 tokens per second with a batch size of 1 and short prompts.

Guide: Running Locally

1. Clone the Model Repository: Download the model files from the Hugging Face repository.
2. Install Dependencies: Ensure you have the Transformers library and other necessary dependencies installed.
3. Download the Model: Use the provided links to download the model files.
4. Run the Model: Utilize the model for text generation tasks in your local environment.

For optimal performance, using cloud GPUs such as the NVIDIA H100 is recommended.

License

The DeepSeek-V3-AWQ model is licensed under the MIT License, allowing for flexible use and modification.