robbert v2 dutch base

Introduction

RobBERT is a state-of-the-art Dutch language model based on the RoBERTa architecture. It is designed for various natural language processing tasks such as emotion detection, sentiment analysis, and named entity recognition, specifically tailored for the Dutch language.

Architecture

RobBERT employs the RoBERTa architecture, which is an optimized version of the original BERT model. It consists of 12 self-attention layers with 12 heads and contains 117 million trainable parameters. The model is pre-trained using the masked language model (MLM) task and does not utilize the next sentence prediction (NSP) task.

Training

RobBERT was pre-trained on the Dutch section of the OSCAR corpus, consisting of 39GB of data with 6.6 billion words. The training utilized the Fairseq library and was conducted on a computing cluster with up to 80 GPUs over two epochs. The training process involved the Adam optimizer with specific hyperparameters and employed a large batch size of 8192 sentences.

Guide: Running Locally

To run RobBERT locally, follow these steps:

1. Install Dependencies: Ensure you have Python and the Hugging Face Transformers library installed.
2. Load the Model:

   from transformers import RobertaTokenizer, RobertaForSequenceClassification
   tokenizer = RobertaTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
   model = RobertaForSequenceClassification.from_pretrained("pdelobelle/robbert-v2-dutch-base")
   

3. Fine-tune the Model: Use Hugging Face's notebooks or scripts to fine-tune RobBERT for your specific task.
4. Inference: Test the model using the masked language model head for zero-shot predictions.

For optimal performance, consider using cloud GPUs, such as those provided by AWS, Google Cloud, or Azure, to handle the computational demands effectively.

License

RobBERT is released under the MIT License, allowing for broad use and modification with attribution.