granite embedding 278m multilingual

Introduction

The Granite-Embedding-278M-Multilingual model is a high-quality text embedding model developed by IBM's Granite Embedding Team. It features 278 million parameters and supports 12 languages, including English, German, Spanish, and Chinese. The model is designed primarily for text similarity, retrieval, and search applications. It uses a transformer-based architecture similar to XLM-RoBERTa and produces embeddings with a vector size of 768.

Architecture

Granite-Embedding-278M-Multilingual employs an encoder-only, XLM-RoBERTa-like architecture. It comprises 12 layers, each with 12 attention heads and an intermediate size of 3072. The model uses the GeLU activation function, supports a vocabulary size of 250,002, and can handle a maximum sequence length of 512 tokens.

Training

The model was trained using data from four main sources: open-source relevance-pair datasets, publicly available datasets, IBM-internal paired data, and IBM-generated synthetic data. The training did not use the MS-MARCO dataset due to its non-commercial license. Training was performed on IBM's Cognitive Compute Cluster using NVIDIA A100 GPUs.

Guide: Running Locally

1. Installation:

   • Install the necessary libraries:

     pip install sentence_transformers transformers torch
     

2. Using Sentence Transformers:

   • Load and use the model with the SentenceTransformer library:

     from sentence_transformers import SentenceTransformer, util
     model = SentenceTransformer("ibm-granite/granite-embedding-278m-multilingual")
     

3. Using Hugging Face Transformers:

   • Load the model with the Transformers library:

     from transformers import AutoModel, AutoTokenizer
     model = AutoModel.from_pretrained("ibm-granite/granite-embedding-278m-multilingual")
     tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-embedding-278m-multilingual")
     

4. GPU Recommendation:

   • For optimal performance, it is recommended to use cloud GPUs such as NVIDIA A100.

License

The Granite-Embedding-278M-Multilingual model is licensed under the Apache 2.0 License, which is a permissive license that allows for commercial use, modification, distribution, and private use.