all roberta large v1

Introduction

The all-roberta-large-v1 model is part of the Sentence-Transformers library, designed to convert sentences and paragraphs into 1024-dimensional dense vector embeddings. These embeddings are suitable for tasks such as clustering, semantic search, and sentence similarity.

Architecture

The model is based on the roberta-large architecture, fine-tuned to enhance its ability to generate sentence embeddings. The sentence-transformers library facilitates the use of this model by providing an interface for easy integration into applications.

Training

Pre-training

The model uses the pre-trained roberta-large as its foundation. Detailed information about the pre-training procedure can be found in the corresponding model card.

Fine-tuning

Fine-tuning was conducted using a contrastive learning objective. The model was trained on a dataset comprising over 1 billion sentence pairs, leveraging a TPU v3-8 infrastructure. Training involved 400,000 steps with a batch size of 256, employing the AdamW optimizer with a learning rate of 2e-5. The sequence length was capped at 128 tokens.

Training Data

The model was fine-tuned using a diverse set of datasets sourced from platforms like Reddit, S2ORC, WikiAnswers, and others, totaling over 1.1 billion sentence pairs.

Guide: Running Locally

Basic Steps

1. Install Sentence-Transformers:

   pip install -U sentence-transformers
   

2. Using the Model:

   from sentence_transformers import SentenceTransformer
   
   sentences = ["This is an example sentence", "Each sentence is converted"]
   model = SentenceTransformer('sentence-transformers/all-roberta-large-v1')
   embeddings = model.encode(sentences)
   print(embeddings)
   

3. Alternative using Hugging Face Transformers:

   from transformers import AutoTokenizer, AutoModel
   import torch
   import torch.nn.functional as F
   
   def mean_pooling(model_output, attention_mask):
       token_embeddings = model_output[0]
       input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
       return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
   
   sentences = ['This is an example sentence', 'Each sentence is converted']
   tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-roberta-large-v1')
   model = AutoModel.from_pretrained('sentence-transformers/all-roberta-large-v1')
   
   encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
   
   with torch.no_grad():
       model_output = model(**encoded_input)
   
   sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
   sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
   
   print("Sentence embeddings:")
   print(sentence_embeddings)
   

Cloud GPUs

For performance optimization, consider using cloud-based GPUs from platforms like AWS, Google Cloud, or Azure, which offer various GPU options to accelerate model inference and fine-tuning tasks.

License

The all-roberta-large-v1 model is licensed under the Apache 2.0 License, allowing for both personal and commercial use with proper attribution.