opus mt en nl

Introduction

OPUS-MT-EN-NL is a machine translation model developed by the Language Technology Research Group at the University of Helsinki. It translates English text into Dutch using the OPUS dataset and the transformer-align model architecture. The model employs pre-processing techniques such as normalization and SentencePiece tokenization.

Architecture

The model utilizes a transformer architecture, specifically designed for text-to-text generation tasks. It aligns with the Marian NMT framework, which is known for efficient training and inference in neural machine translation tasks.

Training

The OPUS-MT-EN-NL model was trained on the OPUS dataset using a transformer-align architecture. Training involved normalization and tokenization through SentencePiece. Original model weights and test sets, including translation outputs and evaluation scores, are available for download, supporting reproducibility and model evaluation.

Guide: Running Locally

To run the OPUS-MT-EN-NL model locally, follow these steps:

1. Install Libraries: Ensure you have Python and install necessary libraries such as transformers and torch.

   pip install transformers torch
   

2. Download Model: Download the model weights from the provided link: opus-2019-12-04.zip.

3. Load the Model: Use the Hugging Face Transformers library to load the model.

   from transformers import MarianMTModel, MarianTokenizer
   
   model_name = 'Helsinki-NLP/opus-mt-en-nl'
   tokenizer = MarianTokenizer.from_pretrained(model_name)
   model = MarianMTModel.from_pretrained(model_name)
   

4. Translate Text: Input text in English and obtain translations in Dutch.

   src_texts = ["Hello, how are you?"]
   translated = model.generate(**tokenizer(src_texts, return_tensors="pt", padding=True))
   tgt_texts = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
   print(tgt_texts)
   

5. Consider Cloud GPUs: For faster inference, consider using cloud-based GPU services like AWS, Google Cloud, or Azure.

License

The OPUS-MT-EN-NL model is licensed under the Apache-2.0 License, allowing for use, modification, and distribution with fewer restrictions.