mental roberta base

Introduction

MentalRoBERTa is a language model based on RoBERTa-Base, specifically pre-trained with mental health-related posts from Reddit. It aims to assist in the automatic detection of mental disorders in online content for non-clinical use, providing support for social workers in early prevention efforts. However, it does not replace professional psychiatric diagnosis.

Architecture

The model utilizes the RoBERTa-Base architecture, which includes 12 layers, 768 hidden units, and 12 attention heads. It is implemented using Hugging Face's Transformers library, which is widely used for building natural language processing models.

Training

The training process involved using four Nvidia Tesla V100 GPUs, with a batch size of 16 per GPU, and conducting evaluations every 1,000 steps over 624,000 iterations. This extensive training took approximately eight days to complete. The data used for training was carefully selected from publicly available, anonymous Reddit posts.

Guide: Running Locally

To run MentalRoBERTa locally, follow these steps:

1. Install Transformers Library:

   pip install transformers
   

2. Load the Model:

   from transformers import AutoTokenizer, AutoModel
   tokenizer = AutoTokenizer.from_pretrained("mental/mental-roberta-base")
   model = AutoModel.from_pretrained("mental/mental-roberta-base")
   

3. Authentication for Gated Models:
   Access to gated models requires authentication. Visit Hugging Face documentation for detailed instructions.

4. Consider Cloud GPUs:
   For efficient model training and inference, using cloud-based GPUs such as AWS, Google Cloud, or Azure is recommended.

License

MentalRoBERTa is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (cc-by-nc-4.0), allowing for non-commercial use with appropriate attribution.