Falcon3 10 B Instruct abliterated G G U F

Introduction

Falcon3-10B-Instruct-abliterated-GGUF is a text generation model designed to work across four languages: English, French, Spanish, and Portuguese. It uses the GGUF library and is configured to produce uncensored, conversational outputs. The model is licensed under the Falcon-LLM license.

Architecture

The model employs LLAMACPP IMATRIX quantizations and uses llama.cpp for quantization processes. Quantization types range from high-quality F16 to lower-quality IQ2_S, providing a variety of options based on hardware capabilities and desired performance.

Training

The original model used for training is Falcon3-10B-Instruct-abliterated, available on Hugging Face. The quantization process utilizes the imatrix option with a specific dataset curated for this purpose, allowing for different levels of model compression and performance optimization.

Guide: Running Locally

1. Install Hugging Face CLI:

   pip install -U "huggingface_hub[cli]"
   

2. Download a specific model file:

   huggingface-cli download bartowski/Falcon3-10B-Instruct-abliterated-GGUF --include "Falcon3-10B-Instruct-abliterated-Q4_K_M.gguf" --local-dir ./
   

3. Download multiple files (if model > 50GB):

   huggingface-cli download bartowski/Falcon3-10B-Instruct-abliterated-GGUF --include "Falcon3-10B-Instruct-abliterated-Q8_0/*" --local-dir ./
   

Cloud GPUs: To maximize performance, consider using cloud GPU services to handle larger models, ensuring your chosen quantization fits within your available VRAM.

License

The model is licensed under the Falcon-LLM license. For more details, refer to the Falcon-LLM Terms and Conditions.