bert base greek uncased v1

Introduction

The BERT-BASE-GREEK-UNCASED-V1 model is a Greek version of the BERT pre-trained language model. It is developed by the Natural Language Processing Group at the Athens University of Economics and Business (AUEB). The model has been designed to handle tasks in the Greek language, such as named entity recognition and natural language inference.

Architecture

The model mirrors the English bert-base-uncased architecture, consisting of 12 layers, 768 hidden units, 12 attention heads, and a total of 110 million parameters. It is available for use with both PyTorch and TensorFlow 2.

Training

The model was trained using the Greek portions of several corpora: Wikipedia, European Parliament Proceedings, and OSCAR. Training followed the official BERT setup, with 1 million steps and batches of 256 sequences of length 512. A single Google Cloud TPU v3-8 was used, with support from TensorFlow Research Cloud and GCP research credits.

Guide: Running Locally

1. Install Dependencies: Install the required libraries using pip:

   pip install transformers
   pip install torch  # or tensorflow
   

2. Load the Model: Use the following Python code to load the model:

   from transformers import AutoTokenizer, AutoModel
   
   tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
   model = AutoModel.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
   

3. Pre-process Text: Text should be in lowercase with no Greek diacritics, though the default tokenizer now supports this natively.

4. Cloud GPUs: For large-scale tasks, consider using cloud services like Google Cloud or AWS for GPU support.

License

The model and its source code are published under licenses that allow for research and commercial use. Ensure you comply with any licensing requirements if you plan to use the model in a commercial setting.