Bio_ Discharge_ Summary_ B E R T

Introduction

The Bio+Discharge Summary BERT model is a variant of ClinicalBERT, specifically trained on discharge summaries from the MIMIC III database, using an initialization from BioBERT. This model is tailored for processing clinical notes, particularly focusing on the discharge summaries of ICU patients.

Architecture

The model is based on the BERT-Base architecture, featuring 12 layers, 768 hidden size, and 12 attention heads. It was initialized using BioBERT, which is built on top of the BERT-Base architecture and pre-trained on biomedical text.

Training

Pretraining Data

The model was trained using discharge summaries from the MIMIC III database, which includes approximately 880 million words from electronic health records.

Note Preprocessing

The notes were divided into sections using a rule-based approach, and further split into sentences using the SciSpacy tokenizer.

Pretraining Procedures

Training was conducted on a GeForce GTX TITAN X 12 GB GPU, using code from Google's BERT repository. The model began with parameters initialized from BioBERT.

Pretraining Hyperparameters

• Batch Size: 32
• Maximum Sequence Length: 128
• Learning Rate: 5 × 10⁻⁵
• Training Steps: 150,000
• Duplication Factor: 5
• Masked Language Model Probability: 0.15
• Max Predictions per Sequence: 20

Guide: Running Locally

To use the Bio+Discharge Summary BERT model locally, follow these steps:

1. Install Transformers Library:

   pip install transformers
   

2. Load the Model:

   from transformers import AutoTokenizer, AutoModel
   tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_Discharge_Summary_BERT")
   model = AutoModel.from_pretrained("emilyalsentzer/Bio_Discharge_Summary_BERT")
   

For optimal performance, using a cloud GPU service like AWS EC2, Google Cloud, or Azure with GPU support is recommended.

License

The Bio+Discharge Summary BERT model is available under the MIT License, allowing for free use, modification, and distribution.