ModernBERT Domain Classifier

Introduction

The ModernBERT Domain Classifier is a fine-tuned version of the answerdotai/ModernBERT-base model. It is designed for text classification tasks and achieves an F1 score of 0.8914 on its evaluation set. The model utilizes the Transformers library and is compatible with end-to-end inference endpoints.

Architecture

The model is built upon the ModernBERT-base architecture, leveraging the capabilities of BERT for domain-specific text classification tasks. It is optimized for performance with adjustments made during the fine-tuning process.

Training

Training Procedure

The training involved fine-tuning on an unspecified dataset, using the following hyperparameters:

• Learning Rate: 5e-05
• Train Batch Size: 32
• Eval Batch Size: 16
• Seed: 42
• Optimizer: OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08
• LR Scheduler Type: Linear
• Number of Epochs: 5

Training Results

The model achieved the following results through the training process:

• Epoch 1: Validation Loss = 0.7298, F1 = 0.7744
• Epoch 2: Validation Loss = 0.4369, F1 = 0.8311
• Epoch 3: Validation Loss = 0.6091, F1 = 0.8399
• Epoch 4: Validation Loss = 0.4352, F1 = 0.8798
• Epoch 5: Validation Loss = 0.3927, F1 = 0.8914

Guide: Running Locally

To run the ModernBERT Domain Classifier locally, follow these steps:

1. Install Required Libraries:

   pip install transformers==4.48.0.dev0 torch==2.5.0 datasets==3.1.0 tokenizers==0.21.0
   

2. Load the Model: Use the Transformers library to load the model for inference.

3. Prepare the Data: Ensure your input data is pre-processed according to the model's requirements.

4. Run Inference: Use the model to make predictions on your data.

For enhanced performance, consider using cloud GPUs such as those provided by AWS, Google Cloud, or Azure.

License

The ModernBERT Domain Classifier is licensed under the Apache-2.0 License, allowing for both personal and commercial use, modification, and distribution.