Nemotron Mini 4 B Instruct G G U F

Introduction

Nemotron-Mini-4B-Instruct-GGUF is a quantized version of the NeMo-based Nemotron-Mini-4B-Instruct model, designed for text generation tasks. Quantization is performed using llama.cpp to optimize for different hardware and performance needs.

Architecture

The model utilizes LLAMACPP IMATRIX quantizations and offers a variety of quantization formats like Q8_0, Q6_K_L, Q5_K_S, and more. This provides flexibility for different hardware configurations, especially focusing on ARM and GPU optimizations.

Training

The original model by Nvidia was optimized using the imatrix option and a specific dataset. Quantization involves converting the weights into more compact formats without significantly losing model performance, enabling efficient deployment.

Guide: Running Locally

1. Install Hugging Face CLI:
   Ensure you have huggingface-cli installed with:

   pip install -U "huggingface_hub[cli]"
   

2. Download the Model:
   Use the CLI to download the specific quantization format you need. For example:

   huggingface-cli download bartowski/Nemotron-Mini-4B-Instruct-GGUF --include "Nemotron-Mini-4B-Instruct-Q4_K_M.gguf" --local-dir ./
   

3. Choose the Correct File:

   • Identify the quantization format based on your hardware capacity (RAM, VRAM).
   • For ARM processors, consider using Q4_0_X_X formats for better speed.
   • For GPU, ensure the model fits within VRAM for optimal performance.

4. Run the Model:
   You can execute the model using LM Studio or similar platforms supporting NeMo models.

Cloud GPUs: Consider using cloud services like AWS or Google Cloud with powerful GPUs to handle larger models and faster inference times.

License

The model is distributed under the NVIDIA Community Model License. For full license terms, refer to the license document.