legal roberta base

Introduction

LEGAL-ROBERTA is a domain-specific language representation model fine-tuned on extensive legal corpora. It is designed to cater to legal language processing tasks by leveraging a large-scale dataset.

Architecture

LEGAL-ROBERTA is based on the ROBERTA architecture, a transformer model well-suited for NLP tasks. This model has been fine-tuned to specialize in legal text, making it adept at tasks such as fill-mask predictions within legal contexts.

Training

The model was initially based on the pretrained ROBERTA-BASE and further fine-tuned using legal-specific data. The training data originated from three main sources:

• Patent Litigations
• Caselaw Access Project (CAP)
• Google Patents Public Data

The training configuration included a learning rate of 5e-5 with decay, 3 epochs, and a total of 446,500 steps. Training was conducted on a dual GeForce GTX TITAN X setup.

Guide: Running Locally

1. Install the Transformers library:

   pip install transformers
   

2. Load the pretrained model:

   from transformers import AutoTokenizer, AutoModel
   
   tokenizer = AutoTokenizer.from_pretrained("saibo/legal-roberta-base")
   model = AutoModel.from_pretrained("saibo/legal-roberta-base")
   

3. Run Inference: Utilize the model for tasks like masked language modeling or other legal text processing tasks.

Cloud GPU Suggestion

For optimal performance, consider using cloud services that offer GPU resources, such as AWS EC2 with GPU instances, Google Cloud with Compute Engine, or Microsoft Azure with NV-series virtual machines.

License

LEGAL-ROBERTA is released under the Apache-2.0 license, allowing for broad use and modification under the terms of this license.