all mpnet base v2

Introduction

The all-mpnet-base-v2 model is a part of the Sentence-Transformers framework, mapping sentences and paragraphs to a 768-dimensional dense vector space. This can be utilized for tasks like clustering and semantic search.

Architecture

The model is based on the microsoft/mpnet-base architecture and fine-tuned for sentence embeddings. It uses contrastive learning objectives to predict sentence pairs within a dataset. This approach enables the model to understand semantic similarities between different sentences.

Training

Pre-training

The model leverages the pre-trained microsoft/mpnet-base model. Details on the pre-training process are available on its model card.

Fine-tuning

Fine-tuning involves using contrastive objectives where cosine similarity is computed for sentence pairs in the batch. The model is trained with a cross-entropy loss comparing true pairs. The training utilized a TPU v3-8 for 100,000 steps with a batch size of 1024. Optimization was performed using the AdamW optimizer with a learning rate of 2e-5.

Training Data

The model was fine-tuned on over 1 billion sentence pairs from various datasets, including:

• Reddit comments
• S2ORC citation pairs
• WikiAnswers
• PAQ question-answer pairs
• Stack Exchange question pairs
• MS MARCO triplets
• Yahoo Answers

Guide: Running Locally

To run the all-mpnet-base-v2 model locally, follow these steps:

1. Install Sentence-Transformers:

   pip install -U sentence-transformers
   

2. Usage with Sentence-Transformers:

   from sentence_transformers import SentenceTransformer
   
   sentences = ["This is an example sentence", "Each sentence is converted"]
   model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')
   embeddings = model.encode(sentences)
   print(embeddings)
   

3. Usage with Hugging Face Transformers:

   from transformers import AutoTokenizer, AutoModel
   import torch
   import torch.nn.functional as F
   
   def mean_pooling(model_output, attention_mask):
       token_embeddings = model_output[0]
       input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
       return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
   
   sentences = ['This is an example sentence', 'Each sentence is converted']
   tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-mpnet-base-v2')
   model = AutoModel.from_pretrained('sentence-transformers/all-mpnet-base-v2')
   encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
   
   with torch.no_grad():
       model_output = model(**encoded_input)
   
   sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
   sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
   print("Sentence embeddings:")
   print(sentence_embeddings)
   

For better performance, consider using a cloud GPU service such as AWS EC2, Google Cloud Platform, or Azure.

License

The all-mpnet-base-v2 model is licensed under the Apache-2.0 License.