Open LLM List
Llama 3.3 70 B Instruct abliterated G G U F
bartowskiIntroduction
The Llama-3.3-70B-Instruct-abliterated-GGUF is a large language model designed for text generation, supporting multiple languages. It is based on the Llama-3 architecture and offers quantized versions for efficient deployment.
Architecture
The model supports eight languages including English, French, Italian, Portuguese, Hindi, Spanish, Thai, and German. It utilizes the GGUF library and offers various quantization levels to optimize performance across different hardware configurations.
Training
Quantization was performed using the llama.cpp framework, utilizing the imatrix option for dataset calibration. The original model is derived from the huihui-ai/Llama-3.3-70B-Instruct-abliterated base model.
Guide: Running Locally
- Install Hugging Face CLI:
pip install -U "huggingface_hub[cli]" - Download Model Files:
- Use the following command to download the desired quantized file:
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-abliterated-GGUF --include "Llama-3.3-70B-Instruct-abliterated-Q4_K_M.gguf" --local-dir ./ - For models larger than 50GB, ensure all parts are downloaded:
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-abliterated-GGUF --include "Llama-3.3-70B-Instruct-abliterated-Q8_0/*" --local-dir ./
- Use the following command to download the desired quantized file:
- Select the Right Quantization:
- Determine your system's available RAM and VRAM to choose the appropriate quant. Aim for a model file size 1-2GB smaller than your total VRAM for maximum speed.
- For maximum quality, combine system RAM and VRAM capacities, then select a quant accordingly.
- Run Locally:
- Use LM Studio or similar environments to execute the model.
Cloud GPUs: Consider using cloud services with GPUs such as AWS, Google Cloud, or Azure for handling large models efficiently.
License
The Llama-3.3-70B-Instruct-abliterated-GGUF model is distributed under the Llama 3.3 Community License Agreement. Usage requires adherence to the Acceptable Use Policy and may involve additional commercial terms for large-scale deployments. The model and its derivatives must include proper attribution and comply with legal regulations.
