magnum v4 12b gguf

Introduction
The Magnum-V4-12B-GGUF model is part of a series designed to emulate the prose quality of the Claude 3 models, specifically Sonnet and Opus. This model is fine-tuned on top of mistralai/Mistral-Nemo-Instruct-2407 using various datasets. It supports English language text generation and is compatible with conversational applications.

Architecture
The model is based on the AutoModelForCausalLM architecture and uses the AutoTokenizer. It integrates various plugins, including the LigerPlugin, to enhance performance with features like fused linear cross-entropy and RoPE. The model leverages datasets for training without system prompts to improve its capabilities in generating natural language responses.

Training
The training process involved two epochs using 8xH100 GPUs, provided by Recursal AI and Featherless AI. Axolotl was utilized for fine-tuning, enabling the model to benefit from full-parameter adjustments. The learning rate was set to 0.00001, with a cosine scheduler and gradient checkpointing enabled for efficiency.

Guide: Running Locally

1. Preparation: Ensure Python and the required libraries are installed. Clone the Magnum-V4-12B-GGUF repository.
2. Environment Setup: Install dependencies using a package manager like pip.
3. Model Download: Download the model weights from Hugging Face.
4. Execution: Use scripts to load and run the model for text generation tasks.
5. Hardware Requirements: A GPU is recommended for optimal performance. Consider using cloud GPUs like NVIDIA's H100 via platforms such as AWS or Google Cloud.

License
The Magnum-V4-12B-GGUF model is licensed under the Apache 2.0 License, allowing for broad use, modification, and distribution with proper attribution.