NemoMix-Unleashed-12B-GGUF

Introduction

NemoMix-Unleashed-12B-GGUF is a quantized version of the NemoMix-Unleashed-12B model for text generation. It utilizes the llama.cpp library for quantization and supports various quantization formats, optimizing for different hardware configurations and performance needs.

Architecture

The model is based on the original NemoMix-Unleashed-12B and utilizes the transformers library for text generation. The quantization process uses the llama.cpp library, specifically the imatrix option, to produce various quantization levels that enable efficient use in different environments.

Training

No specific training details are provided in the current documentation. The model is derived from the original NemoMix-Unleashed-12B, and the quantization process is carried out using datasets and tools mentioned in the credits.

Guide: Running Locally

To run NemoMix-Unleashed-12B-GGUF locally, follow these steps:

1. Install the Hugging Face CLI:

   pip install -U "huggingface_hub[cli]"
   

2. Download the Model:
   Use the Hugging Face CLI to download the desired quantized model file. For example:

   huggingface-cli download bartowski/NemoMix-Unleashed-12B-GGUF --include "NemoMix-Unleashed-12B-Q4_K_M.gguf" --local-dir ./
   

3. Determine Your Resources:

   • Assess your available RAM and VRAM.
   • Choose a model file size slightly smaller than your available VRAM for optimal performance.

4. Select a Quantization Type:

   • K-quants (e.g., Q5_K_M) are generally recommended for ease of use.
   • I-quants (e.g., IQ3_M) are suited for lower resource environments but may perform slower on some hardware.

5. Run the Model:
   Utilize an inference engine compatible with your hardware (e.g., LM Studio) to execute the model.

Cloud GPUs

Consider using cloud GPUs for running larger models or for faster performance. Providers like AWS, Google Cloud, and Azure offer scalable GPU instances suitable for model inference.

License

The model and its associated files are available under licenses specified by the original author and Hugging Face. Please refer to the specific license terms on the model's Hugging Face page for compliance and usage restrictions.