mengzi t5 base chinese correction

Introduction

The mengzi-t5-base-chinese-correction model is a Chinese spelling correction model based on the T5 architecture. It is designed to enhance the accuracy of text by correcting spelling errors. The model achieves near state-of-the-art performance on the SIGHAN2015 test dataset.

Architecture

The model is built on the T5 architecture, which is a text-to-text transformer model. The specific implementation uses PyTorch and is compatible with Hugging Face's Transformers library. The model has been fine-tuned on a Chinese spelling correction dataset without altering the original model structure.

Training

The training utilized the combined SIGHAN+Wang271K Chinese spelling correction dataset. This dataset includes 270,000 entries of text correction data. The model has been evaluated using the SIGHAN2015 test dataset, achieving a precision of 0.8321, recall of 0.6390, and an F1 score of 0.7229.

Guide: Running Locally

1. Install Dependencies:

   • Ensure Python is installed on your system.
   • Install the pycorrector package using pip:

     pip install -U pycorrector
     

2. Run the Model:

   • Use the following Python script to use the model for text correction:

     from pycorrector.t5.t5_corrector import T5Corrector
     nlp = T5Corrector("shibing624/mengzi-t5-base-chinese-correction").batch_t5_correct
     sentence = "今天新情很好"
     print(sentence, ' => ', nlp([sentence]))
     

   • The output will correct the sentence to "今天心情很好" with a correction from '新' to '心'.

3. Cloud GPU Suggestion:

   • For enhanced performance, consider using cloud-based GPU services such as AWS EC2, Google Cloud Platform, or Azure.

License

The mengzi-t5-base-chinese-correction model is licensed under the Apache-2.0 License, allowing wide use and modification, provided that proper credit is given.