Open LLM List
Deep Seek V2.5
deepseek-aiDeepSeek-V2.5 Documentation
Introduction
DeepSeek-V2.5 is an advanced model integrating the capabilities of DeepSeek-V2-Chat and DeepSeek-Coder-V2-Instruct. It enhances general and coding abilities, aligning better with human preferences. The model is optimized for writing and instruction following, surpassing its predecessors in several benchmarks such as AlpacaEval 2.0, ArenaHard, and HumanEval python.
Architecture
DeepSeek-V2.5 combines the features of its predecessors to provide improved general and coding abilities. The enhancements allow it to perform better in benchmarks and align more closely with human preferences.
Training
The model has been fine-tuned and optimized to enhance its performance in various tasks, including writing and instruction following. It shows significant improvements in metrics compared to previous versions.
Guide: Running Locally
To run DeepSeek-V2.5 locally, you need 80GB*8 GPUs to utilize it in BF16 format for inference.
Inference with Hugging Face's Transformers
- Install Hugging Face Transformers.
- Import necessary libraries and load the model using the
AutoTokenizerandAutoModelForCausalLM. - Set
max_memoryand configure the model for inference. - Use the tokenized input to generate outputs.
Inference with vLLM (Recommended)
- Merge the specified Pull Request into your vLLM codebase.
- Load the model using
AutoTokenizerand vLLM. - Prepare sampling parameters and input messages.
- Generate outputs using the LLM.
Additional Features
- Function Calling: Allows the model to call external tools.
- JSON Output: Ensures the model generates valid JSON objects.
- FIM Completion: Completes content between provided prefix and suffix.
Cloud GPUs
For optimal performance, consider using cloud GPU providers like AWS, Google Cloud, or Azure.
License
The code repository is licensed under the MIT License. Usage of DeepSeek-V2 Base/Chat models is subject to a specific Model License. Commercial use is supported, and further licensing details can be found in the Model License documentation.
