distill bert base spanish wwm cased finetuned spa squad2 es

Introduction

The model DISTILL-BERT-BASE-SPANISH-WWM-CASED-FINETUNED-SPA-SQUAD2-ES is a fine-tuned and distilled version of BETO, specifically designed for Spanish Question Answering (Q&A) tasks. It is trained on the SQuAD-es-v2.0 dataset using the distillation technique with the bert-base-multilingual-cased model as the teacher.

Architecture

The model is based on the BERT architecture, utilizing a distilled version to enhance performance by making it smaller, faster, and lighter compared to the original bert-base-spanish-wwm-cased-finetuned-spa-squad2-es. The distillation process involved using bert-base-multilingual-cased as the teacher model.

Training

Training was conducted on a Tesla P100 GPU with 25GB of RAM. The process involved five epochs with a learning rate of 3e-5. The dataset used for training was SQuAD-es-v2.0, and the model was trained with various hyperparameters to optimize performance for the Q&A task.

Guide: Running Locally

To run the model locally:

1. Install Dependencies: Ensure that you have the transformers library installed.
2. Load the Model:

   from transformers import pipeline
   
   nlp = pipeline(
       'question-answering', 
       model='mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es',
       tokenizer=(
           'mrm8488/distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es',  
           {"use_fast": False}
       )
   )
   

3. Run Inference: Use the pipeline to answer questions based on the provided context.

For enhanced performance, consider using cloud GPUs such as those offered by Google Colab or AWS.

License

The model is released under the Apache-2.0 license, allowing for wide usage and distribution with minimal restrictions.