Falcon3 3 B Instruct abliterated G G U F

Introduction

Falcon3-3B-Instruct-abliterated-GGUF is a versatile language model designed for text generation. It is available in GGUF format and supports four languages: English, French, Spanish, and Portuguese. The model is known for its "abliterated" and "uncensored" characteristics, making it suitable for various conversational applications.

Architecture

The model uses the Falcon3 architecture and employs various quantization methods using llama.cpp for optimized performance. These quantizations include various configurations such as Q8_0, Q6_K_L, and others that balance quality and resource usage.

Training

The model is based on the original Falcon3-3B-Instruct-abliterated by huihui-ai and quantized using the imatrix option from a custom dataset. The quantizations were performed using llama.cpp release b4381, and the model can be run in LM Studio.

Guide: Running Locally

1. Install the huggingface-cli:

   pip install -U "huggingface_hub[cli]"
   

2. Download a specific file:

   huggingface-cli download bartowski/Falcon3-3B-Instruct-abliterated-GGUF --include "Falcon3-3B-Instruct-abliterated-Q4_K_M.gguf" --local-dir ./
   

3. For large models (over 50GB), download them in parts:

   huggingface-cli download bartowski/Falcon3-3B-Instruct-abliterated-GGUF --include "Falcon3-3B-Instruct-abliterated-Q8_0/*" --local-dir ./
   

4. Utilize cloud GPUs for faster computation and handling larger models, especially when VRAM is a limitation. Services like AWS, Azure, or Google Cloud provide options to run such models efficiently.

License

The model is distributed under the Falcon LLM License. For more details, refer to the Falcon LLM Terms and Conditions.