BIOMED-ROBERTA-BASE

Introduction

BioMed-RoBERTa-base is a language model adapted from RoBERTa-base (Liu et al., 2019). It is specifically trained on 2.68 million scientific papers sourced from the Semantic Scholar corpus, amounting to 7.55 billion tokens and 47GB of data. Unlike many models, it uses full-text papers for training, enhancing its application in the biomedical domain.

Architecture

BioMed-RoBERTa-base builds upon the RoBERTa-base architecture. It leverages continued pretraining on a vast corpus of biomedical literature, enabling it to handle specialized tasks in this domain effectively. The adaptive pretraining procedure is detailed in the work by Gururangan et al., 2020.

Training

The model underwent continued pretraining with full-text scientific papers, using adaptive techniques to tailor the RoBERTa architecture for biomedical applications. This process was key to achieving its performance in specialized NLP tasks.

Evaluation

BioMed-RoBERTa-base demonstrates competitive performance compared to state-of-the-art models across various NLP tasks in the biomedical field. Performance metrics include:

• RCT-180K (Text Classification): 86.9 (0.2)
• ChemProt (Relation Extraction): 83.0 (0.7)
• JNLPBA (NER): 75.2 (0.1)
• BC5CDR (NER): 87.8 (0.1)
• NCBI-Disease (NER): 87.1 (0.8)

Guide: Running Locally

To run BioMed-RoBERTa-base locally:

1. Install the Hugging Face Transformers library.
2. Load the model using the Transformers API.
3. Fine-tune the model on your dataset as needed for specific tasks.

For enhanced performance, consider using cloud GPUs such as AWS EC2, Google Cloud Platform, or Azure.

License

Please refer to the specific Hugging Face repository for license details. Users are encouraged to cite the related paper when using this model:

@inproceedings{domains,
  author = {Suchin Gururangan and Ana Marasović and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
  title = {Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
  year = {2020},
  booktitle = {Proceedings of ACL},
}