B E T O espanhol Squad2

Introduction

The BETO-ESPANHOL-SQUAD2 model is a fine-tuned version of the BETO language model specifically designed for question answering (Q&A) tasks in Spanish. It utilizes the SQuAD-es-v2.0 dataset for training to enhance its performance in Q&A applications.

Architecture

The underlying architecture of the BETO-ESPANHOL-SQUAD2 is based on the 'dccuchile/bert-base-spanish-wwm-cased' model. BETO is a BERT model trained on a large Spanish corpus using the Whole Word Masking technique. It is comparable in size to a BERT-Base model and includes both uncased and cased versions. BETO has been tested against other models, including Multilingual BERT, on various Spanish benchmarks.

Training

The model was trained on a Tesla P100 GPU with 25GB of RAM. The training utilized the following command:

export SQUAD_DIR=path/to/nl_squad
python transformers/examples/question-answering/run_squad.py \
  --model_type bert \
  --model_name_or_path dccuchile/bert-base-spanish-wwm-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train_nl-v2.0.json \
  --predict_file $SQUAD_DIR/dev_nl-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --save_steps 5000 \
  --threads 4 \
  --version_2_with_negative

The model achieved an exact match score of 76.5050 and an F1 score of 86.0781.

Guide: Running Locally

1. Setup Environment: Ensure you have Python and necessary libraries installed, specifically PyTorch or TensorFlow, and Hugging Face Transformers.

2. Download Model: Obtain the BETO-ESPANHOL-SQUAD2 model from Hugging Face's model hub.

3. Prepare Data: Organize your data in a format compatible with SQuAD-es-v2.0.

4. Run Model: Execute the model using a script similar to the training script provided, adjusting paths and parameters as needed.

5. Hardware Recommendations: For efficient performance, utilizing cloud GPUs such as those provided by AWS, GCP, or Azure is recommended, especially for large-scale or production-level tasks.

License

The model and its associated resources are provided under a license specified by the BETO team. Ensure to review the terms at their GitHub repository before use.