Biomed V L P C X R B E R T specialized

Introduction

CXR-BERT-SPECIALIZED is a domain-specific language model designed for chest X-ray (CXR) analysis. It leverages an enhanced vocabulary, innovative pretraining techniques, and text augmentations to improve performance in radiology-specific tasks, such as natural language inference and masked language model token prediction.

Architecture

CXR-BERT-SPECIALIZED is built on a multi-modal contrastive learning framework, similar to CLIP. This model aligns text and image embeddings using the latent representation of the [CLS] token. The architecture involves continual pretraining of CXR-BERT-general, which was initially trained on biomedical literature and clinical notes from PubMed and MIMIC datasets.

Training

The model undergoes a multi-stage training process:

1. Pretraining CXR-BERT-general: Using masked language modeling on PubMed abstracts and MIMIC clinical notes.
2. Specializing for Chest X-ray: Further training from CXR-BERT-general to focus on the CXR domain.
3. Multi-modal Training: Joint training with a ResNet-50 image model using a contrastive learning framework. The image model is pre-trained with SimCLR on MIMIC-CXR images.

Guide: Running Locally

To run CXR-BERT-SPECIALIZED locally, follow these steps:

1. Install Dependencies: Ensure you have Python and PyTorch installed. Install the transformers library from Hugging Face.

   pip install torch transformers
   

2. Load the Model and Tokenizer:

   import torch
   from transformers import AutoModel, AutoTokenizer
   
   url = "microsoft/BiomedVLP-CXR-BERT-specialized"
   tokenizer = AutoTokenizer.from_pretrained(url, trust_remote_code=True)
   model = AutoModel.from_pretrained(url, trust_remote_code=True)
   

3. Run the Model: Use the following snippet to compute sentence embeddings and their cosine similarity:

   text_prompts = ["There is no pneumothorax or pleural effusion", "No pleural effusion or pneumothorax is seen", "The extent of the pleural effusion is constant."]
   tokenizer_output = tokenizer.batch_encode_plus(batch_text_or_text_pairs=text_prompts, add_special_tokens=True, padding='longest', return_tensors='pt')
   embeddings = model.get_projected_text_embeddings(input_ids=tokenizer_output.input_ids, attention_mask=tokenizer_output.attention_mask)
   sim = torch.mm(embeddings, embeddings.t())
   

4. Cloud GPU Suggestion: For efficient processing, consider using cloud services like AWS EC2 or Google Cloud Compute Engine with GPU support.

License

The CXR-BERT-SPECIALIZED model is released under the MIT License. This permits use, distribution, and modification, ensuring the model is accessible for research and development purposes.