fairlex scotus minilm

Introduction

The FAIRLEX-SCOTUS-MINILM model is part of a multilingual benchmark suite designed to evaluate fairness in legal text processing. This benchmark covers multiple jurisdictions and languages, assessing fairness across attributes like gender, age, nationality, language, and legal area. Despite various fine-tuning techniques, the study highlights ongoing disparities in model performance, underscoring the challenges in achieving fairness in legal natural language processing (NLP).

Architecture

The FAIRLEX-SCOTUS-MINILM model is based on a mini-sized BERT architecture. It consists of 6 Transformer blocks, 384 hidden units, and 12 attention heads. The model is warm-started from MiniLMv2 and utilizes a distilled version of RoBERTa for English datasets, with adaptations for other languages.

Training

The model underwent continued pre-training on specific legal corpora, including SCOTUS for the English language. This approach aims to enhance the model's understanding of legal texts while addressing biases across various fairness attributes.

Guide: Running Locally

To run the FAIRLEX-SCOTUS-MINILM model locally, follow these steps:

1. Installation: Ensure you have Python and the Hugging Face Transformers library installed:

   pip install transformers
   

2. Load the Model:

   from transformers import AutoTokenizer, AutoModel
   
   tokenizer = AutoTokenizer.from_pretrained("coastalcph/fairlex-scotus-minilm")
   model = AutoModel.from_pretrained("coastalcph/fairlex-scotus-minilm")
   

3. Inference: Use the model to process legal text inputs and evaluate fill-mask tasks.

Cloud GPUs: For intensive tasks, consider using cloud-based GPU services like AWS, Google Cloud, or Microsoft Azure to improve processing speed and efficiency.

License

The FAIRLEX-SCOTUS-MINILM model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (cc-by-nc-sa-4.0) license, which allows for sharing and adaptation with attribution for non-commercial purposes.