Llama Deepsync 3 B G G U F

Introduction

Llama-Deepsync-3B-GGUF by QuantFactory is a quantized version of the Llama-3.2-3B-Instruct model, specifically designed for text generation tasks that require deep reasoning, logical structuring, and problem-solving. It excels in applications like education, programming, and creative writing, offering robust natural language processing capabilities.

Architecture

The Llama 3.2 model is an auto-regressive language model utilizing an optimized transformer architecture. It employs supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The model supports generating long texts and has multilingual capabilities across 29 languages.

Training

Llama-Deepsync-3B has been fine-tuned to significantly improve in areas such as coding, mathematics, instruction following, and generating structured outputs like JSON. It supports long-context scenarios up to 128K tokens and can generate up to 8K tokens in a single output.

Guide: Running Locally

To run the Llama-Deepsync-3B model, you will need:

1. Install Transformers: Ensure you have transformers version 4.43.0 or later. Update via:

   pip install --upgrade transformers
   

2. Set Up the Model: Use the Transformers library to load and run the model:

   import torch
   from transformers import pipeline
   
   model_id = "prithivMLmods/Llama-Deepsync-3B"
   pipe = pipeline(
       "text-generation",
       model=model_id,
       torch_dtype=torch.bfloat16,
       device_map="auto",
   )
   

3. Cloud GPU Recommendation: For efficient execution, consider using cloud GPUs like those available from AWS, Google Cloud, or Azure.

4. Interact with the Model: Use the pipeline to generate responses:

   messages = [{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"}]
   outputs = pipe(messages, max_new_tokens=256)
   print(outputs[0]["generated_text"][-1])
   

For further details and advanced usage, refer to huggingface-llama-recipes.

License

The model is licensed under the CreativeML OpenRail-M license, allowing use with specific conditions.