opus mt sv en

Introduction

OPUS-MT-SV-EN is a translation model developed by the Language Technology Research Group at the University of Helsinki. It is designed to translate text from Swedish (sv) to English (en), using the OPUS dataset and a transformer-based architecture.

Architecture

The model employs a transformer-align architecture, which is known for its efficiency in handling sequence-to-sequence tasks like translation. Pre-processing of the data includes normalization and SentencePiece tokenization to prepare the text for the model.

Training

The model was trained using the OPUS dataset, a large collection of multilingual parallel corpora. The training process involved fine-tuning the transformer architecture with the specific goal of improving translation accuracy from Swedish to English. The model weights are available for download, and the performance has been evaluated using a BLEU score of 64.5 and a chr-F score of 0.763 on the Tatoeba test set.

Guide: Running Locally

1. Set Up Environment: Ensure you have Python and PyTorch or TensorFlow installed. Install the Hugging Face Transformers library using pip install transformers.

2. Download Model Weights: Obtain the model weights from this link and extract them.

3. Load Model: Use the Transformers library to load the model and tokenizer:

   from transformers import MarianMTModel, MarianTokenizer
   
   model_name = 'Helsinki-NLP/opus-mt-sv-en'
   tokenizer = MarianTokenizer.from_pretrained(model_name)
   model = MarianMTModel.from_pretrained(model_name)
   

4. Run Translation: Input Swedish text and generate translations.

   text = "Din text här"
   translated = model.generate(**tokenizer(text, return_tensors="pt", padding=True))
   print([tokenizer.decode(t, skip_special_tokens=True) for t in translated])
   

5. Cloud GPUs: For faster processing, consider using cloud services like AWS, Google Cloud, or Azure, which provide GPU support.

License

The OPUS-MT-SV-EN model is licensed under the Apache 2.0 License, allowing for use, modification, and distribution under the terms of this license.