Financial Roberta

Introduction

Financial Roberta is a Masked Language Model trained using the Roberta architecture on the Financial Phrasebank corpus. It focuses on language tasks related to financial texts and is built using Hugging Face's transformers library.

Architecture

The model is based on the Roberta architecture with the following specifications:

• vocab_size: 56,000
• max_position_embeddings: 514
• num_attention_heads: 12
• num_hidden_layers: 6
• type_vocab_size: 1

Training

Financial Roberta was trained on the Financial Phrasebank corpus. The training process utilized the RobertaConfig from the transformers package and ran for 10 epochs with a GPU batch size of 64.

Guide: Running Locally

To use Financial Roberta locally, follow these steps:

1. Import Required Libraries:

   from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline
   

2. Load the Pre-trained Model:

   tokenizer = AutoTokenizer.from_pretrained("abhilash1910/financial_roberta")
   model = AutoModelWithLMHead.from_pretrained("abhilash1910/financial_roberta")
   

3. Set Up the Fill-Mask Pipeline:

   model_mask = pipeline('fill-mask', model='abhilash1910/financial_roberta')
   

4. Inference Example:

   result = model_mask("The company had a <mask> of 20% in 2020.")
   print(result)
   

For enhanced performance, consider using cloud GPUs such as those provided by AWS, Google Cloud, or Azure.

License

Refer to the Hugging Face site and repositories for specific license information.