ESM-2 Model Documentation

Introduction

ESM-2 is an advanced protein model developed for tasks involving protein sequences. It utilizes a masked language modeling objective, making it suitable for fine-tuning across various protein sequence tasks. More detailed information can be found in the accompanying research paper. The model is available in multiple configurations on the Hugging Face Hub, each varying in size and complexity.

Architecture

ESM-2 models have different checkpoints containing various numbers of layers and parameters. Larger models typically offer improved accuracy but require more computational resources. The available checkpoints are:

• esm2_t48_15B_UR50D: 48 layers, 15 billion parameters
• esm2_t36_3B_UR50D: 36 layers, 3 billion parameters
• esm2_t33_650M_UR50D: 33 layers, 650 million parameters
• esm2_t30_150M_UR50D: 30 layers, 150 million parameters
• esm2_t12_35M_UR50D: 12 layers, 35 million parameters
• esm2_t6_8M_UR50D: 6 layers, 8 million parameters

Training

The ESM-2 model is trained using a masked language modeling approach, which is effective for learning representations of protein sequences. The training datasets and more detailed methodologies are discussed in the research paper linked in the documentation. Users can fine-tune the ESM-2 models using provided demo notebooks available for both PyTorch and TensorFlow.

Guide: Running Locally

To run ESM-2 models locally, follow these steps:

1. Environment Setup: Ensure you have Python and necessary libraries installed. It's recommended to use a virtual environment.
2. Install Libraries: Install transformers and torch or tensorflow based on your preference.
3. Download Model: Use the Hugging Face transformers library to download the desired ESM-2 model checkpoint from the Hub.
4. Fine-tuning: Utilize the provided demo notebooks for guidance on fine-tuning the model on your specific tasks.

For computational efficiency, consider using cloud GPUs from services like AWS, Google Cloud, or Azure, especially for larger model variants.

License

The ESM-2 model is released under the MIT License, allowing for flexible usage and distribution.