opus mt zh vi

Introduction

The OPUS-MT-ZH-VI model is a translation model designed for converting text from Chinese to Vietnamese. It is developed by the Language Technology Research Group at the University of Helsinki and is part of the Tatoeba Challenge project. This model utilizes the transformer architecture and has undergone specific preprocessing steps to enhance translation accuracy.

Architecture

The model employs a transformer architecture with alignment features. It is capable of handling multiple Chinese language scripts, including Simplified (Hans) and Traditional (Hant) Chinese, in both Latin and native glyph forms, and translates them into Vietnamese. The preprocessing involves normalization and applies SentencePiece tokenization with a 32,000 subword vocabulary.

Training

The model was trained using datasets from the OPUS collection, specifically focusing on pairs of Chinese and Vietnamese text. The training process involved normalizing the text and using the SentencePiece model to tokenize it. The pre-trained weights dated June 17, 2020, are available for download, and the training was evaluated using BLEU and chr-F metrics on a test set, achieving a BLEU score of 20.0 and a chr-F score of 0.385.

Guide: Running Locally

To run the OPUS-MT-ZH-VI model locally, follow these steps:

1. Set Up Environment: Ensure you have Python installed along with libraries like transformers and torch or tensorflow if using TensorFlow backend.
2. Download Model Weights: Obtain the model weights from the URL provided: opus-2020-06-17.zip.
3. Load the Model: Use the Hugging Face transformers library to load the model and tokenizer.
4. Perform Translation: Input Chinese text and use the model to perform translations into Vietnamese.

Cloud GPUs: It is recommended to use cloud-based GPUs, such as those offered by AWS, Google Cloud, or Azure, to handle the computational requirements efficiently.

License

The OPUS-MT-ZH-VI model is licensed under the Apache 2.0 License, which allows for free use, modification, and distribution of the software, provided that proper attribution is given to the original creators.