m Mini L Mv2 L6 H384 distilled from X L M R Large

Introduction

The mMiniLMv2-L6-H384-distilled-from-XLMR-Large is a distilled version of the MiniLMv2 model, designed for efficient inference and deployment. It is based on the MiniLM model architecture developed by Microsoft, and is optimized for multilingual tasks, leveraging the capabilities of the XLM-RoBERTa model.

Architecture

MiniLMv2 is a smaller, faster variant of the XLM-RoBERTa (XLM-R) architecture, designed to provide competitive performance while reducing computational requirements. It uses a 6-layer transformer model with a hidden size of 384, making it suitable for applications where computational resources are limited.

Training

The model is distilled from the larger XLM-RoBERTa model, leveraging knowledge distillation techniques to retain performance while reducing model size. This process involves training the smaller model to mimic the outputs of the larger model, effectively transferring its knowledge.

Guide: Running Locally

To run this model locally, follow these steps:

1. Install Dependencies: Ensure you have Python and PyTorch installed. Use pip to install the transformers library from Hugging Face.

   pip install transformers
   

2. Load the Model: Use the Hugging Face Transformers library to load the model and tokenizer.

   from transformers import AutoModelForMaskedLM, AutoTokenizer
   
   tokenizer = AutoTokenizer.from_pretrained("nreimers/mMiniLMv2-L6-H384-distilled-from-XLMR-Large")
   model = AutoModelForMaskedLM.from_pretrained("nreimers/mMiniLMv2-L6-H384-distilled-from-XLMR-Large")
   

3. Inference: Perform tasks such as fill-mask predictions using the loaded model.

   inputs = tokenizer("Hello, my name is [MASK].", return_tensors="pt")
   outputs = model(**inputs)
   

4. Cloud GPUs: For improved performance, consider using cloud-based GPUs from providers like AWS, Google Cloud, or Azure.

License

The model and its code are available under the license specified by the original MiniLMv2 repository, which can be found on GitHub. Users should review the license for terms and conditions.