Ministral 8 B Instruct 2410 G G U F

Introduction

The Ministral-8B-Instruct-2410 Language Model is an instruct fine-tuned model designed for on-device computing and at-the-edge use cases. It is released under the Mistral Research License and offers significant performance improvements over existing models of similar size.

Architecture

The Ministral-8B model is a dense transformer with 8,019,808,256 parameters. It has 36 layers, 32 heads, a hidden dimension of 12,288, and a vocabulary size of 131,072. The model utilizes a 128k context window with interleaved sliding-window attention and supports function calling.

Training

Ministral-8B is trained with a focus on multilingual and code data, using a V3-Tekken tokenizer. It outperforms other models in various benchmarks, including knowledge, commonsense, code, math, and multilingual tasks.

Guide: Running Locally

Basic Steps

1. Install Dependencies:

   • For vLLM:

     pip install --upgrade vllm
     pip install --upgrade mistral_common
     

   • For Mistral Inference:

     pip install mistral_inference --upgrade
     

2. Set Up Model:

   • Download necessary files using the huggingface_hub:

     from huggingface_hub import snapshot_download
     from pathlib import Path
     
     mistral_models_path = Path.home().joinpath('mistral_models', '8B-Instruct')
     mistral_models_path.mkdir(parents=True, exist_ok=True)
     
     snapshot_download(repo_id="mistralai/Ministral-8B-Instruct-2410", allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"], local_dir=mistral_models_path)
     

3. Run the Model:

   • Use the model with vLLM for offline inference or set up a server for client-server communication.

Cloud GPUs

For optimal performance, especially with large context sizes, consider using cloud GPUs with at least 24 GB of RAM.

License

The Ministral-8B-Instruct-2410 is distributed under the Mistral Research License. This license allows non-commercial research use, and any commercial utilization requires a separate license from Mistral AI. For detailed information on the terms, refer to the Mistral Research License.