M N 12 B Mag Mell R1 G G U F

Introduction

MN-12B-MAG-MELL-R1-GGUF is a quantized version of the MN-12B-Mag-Mell-R1 model, designed for efficient text generation using the GGUF framework. This model utilizes llama.cpp for quantization and offers various quantization types to accommodate different hardware capabilities.

Architecture

The model is based on the MN-12B-Mag-Mell-R1 architecture, utilizing llama.cpp's imatrix quantization technique. It includes multiple quantization formats (e.g., Q8_0, Q6_K_L) that balance between quality and resource requirements, making it suitable for varying levels of RAM and VRAM availability.

Training

The model was originally trained by inflatebot and then quantized using llama.cpp release b4381, with the imatrix option. The quantization process involved optimizing the model for different use cases and hardware configurations, focusing on balancing performance and resource usage.

Guide: Running Locally

1. Install Hugging Face CLI:

   pip install -U "huggingface_hub[cli]"
   

2. Download the Model:

   • Use the CLI to download a specific quantization file:

     huggingface-cli download bartowski/MN-12B-Mag-Mell-R1-GGUF --include "MN-12B-Mag-Mell-R1-Q4_K_M.gguf" --local-dir ./
     

   • For models larger than 50GB, download all parts:

     huggingface-cli download bartowski/MN-12B-Mag-Mell-R1-GGUF --include "MN-12B-Mag-Mell-R1-Q8_0/*" --local-dir ./
     

3. Run the Model:

   • Use LM Studio or other compatible environments to execute the model with your chosen quantization.

4. Hardware Recommendations:

   • For optimal performance, use a system with sufficient RAM/VRAM. Consider cloud GPUs from providers like AWS or Google Cloud for resource-intensive tasks.

License

The specific licensing terms for MN-12B-MAG-MELL-R1-GGUF are not stated in the provided documentation. Please refer to the Hugging Face model repository or contact the model's author for detailed licensing information.