Qwen2.5 3 B Instruct

Introduction

Qwen2.5 is part of the latest series of Qwen large language models, featuring a range of models from 0.5 to 72 billion parameters. It offers enhanced capabilities in coding, mathematics, instruction following, long text generation, structured data understanding, and multilingual support. The instruction-tuned 3B Qwen2.5 model showcases improvements over its predecessors in these areas and supports a wide range of languages and features.

Architecture

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training
• Architecture: Transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings
• Number of Parameters: 3.09B
• Number of Parameters (Non-Embedding): 2.77B
• Number of Layers: 36
• Number of Attention Heads (GQA): 16 for Q and 2 for KV
• Context Length: Full 32,768 tokens and generation 8192 tokens

Training

The Qwen2.5 models are trained with a focus on enhancing knowledge in various domains, especially in coding and mathematics, and improving instruction-following capabilities. They are designed to handle long-context support up to 128K tokens and generate up to 8K tokens.

Guide: Running Locally

To run the Qwen2.5-3B-Instruct model locally, follow these steps:

1. Install Dependencies: Ensure you have the latest version of the Hugging Face Transformers library since versions below 4.37.0 may cause errors.

2. Load Model and Tokenizer:

   from transformers import AutoModelForCausalLM, AutoTokenizer
   
   model_name = "Qwen/Qwen2.5-3B-Instruct"
   model = AutoModelForCausalLM.from_pretrained(
       model_name,
       torch_dtype="auto",
       device_map="auto"
   )
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   

3. Generate Text:

   prompt = "Give me a short introduction to large language model."
   messages = [
       {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
       {"role": "user", "content": prompt}
   ]
   text = tokenizer.apply_chat_template(
       messages,
       tokenize=False,
       add_generation_prompt=True
   )
   model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
   
   generated_ids = model.generate(
       **model_inputs,
       max_new_tokens=512
   )
   response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
   

4. Use Cloud GPUs: For optimal performance, consider using cloud-based GPUs such as those provided by AWS, Google Cloud, or Azure.

License

The Qwen2.5-3B-Instruct model is licensed under the Qwen Research license. For more details, refer to the license link.