Euro L L M 1.7 B Instruct

Introduction

EuroLLM-1.7B-Instruct is a multilingual language model developed by the EuroLLM Project. It is designed to understand and generate text across European Union languages and other relevant languages. This model is the first instruction-tuned version of the EuroLLM series, emphasizing general instruction-following and machine translation.

Architecture

EuroLLM-1.7B-Instruct is a 1.7 billion parameter model using a dense Transformer architecture. Key features include:

• Grouped Query Attention (GQA): Uses 8 key-value heads to improve inference speed.
• Pre-layer Normalization: Enhances training stability using RMSNorm.
• Activation Function: SwiGLU is employed for better performance on downstream tasks.
• Rotary Positional Embeddings (RoPE): Used in all layers for improved performance and extended context length.

Training

The model was pre-trained on 4 trillion tokens using diverse data sources, such as web data and parallel data, and was further instruction-tuned on the EuroBlocks dataset. Training was conducted using 256 Nvidia H100 GPUs with a batch size of 3,072 sequences, employing the Adam optimizer and BF16 precision.

Guide: Running Locally

1. Install Transformers Library:

   pip install transformers
   

2. Load the Model and Tokenizer:

   from transformers import AutoModelForCausalLM, AutoTokenizer
   
   model_id = "utter-project/EuroLLM-1.7B-Instruct"
   tokenizer = AutoTokenizer.from_pretrained(model_id)
   model = AutoModelForCausalLM.from_pretrained(model_id)
   

3. Generate Text:

   text = '<|im_start|>system\n<|im_end|>\n<|im_start|>user\nTranslate the following English source text to Portuguese:\nEnglish: I am a language model for european languages. \nPortuguese: <|im_end|>\n<|im_start|>assistant\n'
   inputs = tokenizer(text, return_tensors="pt")
   outputs = model.generate(**inputs, max_new_tokens=20)
   print(tokenizer.decode(outputs[0], skip_special_tokens=True))
   

4. Suggested Cloud GPUs: Consider using cloud services like AWS, Google Cloud, or Azure for access to powerful GPUs such as Nvidia A100 or H100 for optimal performance.

License

EuroLLM-1.7B-Instruct is licensed under the Apache License 2.0, allowing for wide usage and modification.