bert base nli mean tokens

Introduction

The bert-base-nli-mean-tokens model from Sentence-Transformers is designed to map sentences and paragraphs to a 768-dimensional dense vector space. It is suitable for tasks such as clustering and semantic search. However, this model is deprecated due to producing low-quality sentence embeddings, and users are advised to refer to recommended alternatives on SBERT.net.

Architecture

The model architecture comprises a SentenceTransformer with:

• A Transformer model (BertModel) with a maximum sequence length of 128.
• A Pooling layer configured to use mean pooling of token embeddings.

Training

The model, trained by Sentence-Transformers, was part of the development of Sentence-BERT, which uses Siamese BERT-Networks to generate sentence embeddings. Further details on the training methodology can be found in the publication "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks."

Guide: Running Locally

To run the model locally, follow these steps:

1. Install Sentence-Transformers Library
   Ensure you have the sentence-transformers library installed:

   pip install -U sentence-transformers
   

2. Using with Sentence-Transformers Library

   from sentence_transformers import SentenceTransformer
   sentences = ["This is an example sentence", "Each sentence is converted"]
   model = SentenceTransformer('sentence-transformers/bert-base-nli-mean-tokens')
   embeddings = model.encode(sentences)
   print(embeddings)
   

3. Using with Hugging Face Transformers
   If not using Sentence-Transformers, manage pooling manually:

   from transformers import AutoTokenizer, AutoModel
   import torch
   
   def mean_pooling(model_output, attention_mask):
       token_embeddings = model_output[0]
       input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
       return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
   
   sentences = ['This is an example sentence', 'Each sentence is converted']
   tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/bert-base-nli-mean-tokens')
   model = AutoModel.from_pretrained('sentence-transformers/bert-base-nli-mean-tokens')
   encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
   with torch.no_grad():
       model_output = model(**encoded_input)
   sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
   print("Sentence embeddings:")
   print(sentence_embeddings)
   

4. Consider Using Cloud GPUs
   For enhanced performance, consider using cloud GPUs from providers like AWS, Google Cloud, or Azure.

License

The model is licensed under the Apache 2.0 License, allowing for free use and distribution with proper acknowledgment of the original authors.