bert tiny 5 finetuned squadv2 LLM Model

Introduction

BERT-TINY-5-FINETUNED-SQUADV2 is a compact version of the BERT model fine-tuned on SQuAD 2.0 for question answering tasks. Developed by Google Research and further refined by Manuel Romero, this model is tailored for environments with limited computational resources.

Architecture

BERT-TINY is one of 24 smaller BERT models, designed to operate efficiently with WordPiece masking. These models are compact, making them suitable for restricted computational settings, and are particularly beneficial when used with knowledge distillation techniques.

Training

The model was fine-tuned using a Tesla P100 GPU with 25GB RAM. The training leveraged the SQuAD 2.0 dataset, which includes both answerable and unanswerable questions to challenge models to discern when no answer is available. The fine-tuning script is accessible via the Hugging Face Transformers GitHub repository.

Results

Exact Match (EM): 57.12
F1 Score: 60.86
Model Size: 24.34 MB

Guide: Running Locally

Install the Transformers library:
```
pip install transformers
```

Import the pipeline and initialize:

from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="mrm8488/bert-tiny-5-finetuned-squadv2",
    tokenizer="mrm8488/bert-tiny-5-finetuned-squadv2"
)

Run a sample question:

qa_pipeline({
    'context': "Manuel Romero has been working hardly in the repository hugginface/transformers lately",
    'question': "Who has been working hard for hugginface/transformers lately?"
})

For enhanced performance, consider using cloud GPUs such as those offered by AWS, Google Cloud, or Azure.

License

The model is available under the Apache License 2.0, which allows for both commercial and non-commercial use, modification, and distribution.