Ancient Greek B E R T

Introduction

Ancient Greek BERT is a pioneering sub-word BERT model designed specifically for the Ancient Greek language. This model achieves state-of-the-art performance in tasks such as Part-of-Speech (PoS) Tagging and Morphological Analysis after fine-tuning. It is based on a standard 12-layer, 768-dimensional BERT-base architecture. The model, along with scripts for its usage and fine-tuning, is accessible via GitHub.

Architecture

The Ancient Greek BERT model is a pre-trained transformer model with 12 layers and a hidden size of 768. It is initialized from the AUEB NLP Group's Greek BERT and is further trained on various Ancient Greek datasets, including those from the First1KGreek Project and the Perseus Digital Library.

Training

The model is trained using monolingual data from multiple sources, including the First1KGreek Project and various Greek treebanks. Training was performed on four NVIDIA Tesla V100 16GB GPUs over 80 epochs, with a maximum sequence length of 512. The model achieves a perplexity of 4.8 on the test set and demonstrates high accuracy (>90%) on PoS Tagging and Morphological Analysis when fine-tuned.

Guide: Running Locally

1. Requirements
   Install necessary packages:

   pip install transformers
   pip install unicodedata
   pip install flair
   

2. Usage
   Load the model and tokenizer from the Hugging Face Model Hub:

   from transformers import AutoTokenizer, AutoModel
   tokenizer = AutoTokenizer.from_pretrained("pranaydeeps/Ancient-Greek-BERT")
   model = AutoModel.from_pretrained("pranaydeeps/Ancient-Greek-BERT")
   

3. Fine-Tuning
   For fine-tuning instructions, refer to the GitHub repository.

4. Cloud GPUs
   Consider using cloud-based GPUs such as AWS EC2, Google Cloud Platform, or Azure for efficient training and inference.

License

Please refer to the project's GitHub repository for license details and terms of use.