spar wiki bm25 lexmodel query encoder

Introduction

The SPAR-Wiki-BM25-Lexmodel-Query-Encoder is a query encoder model from Meta AI, designed to imitate the behavior of the BM25 sparse retriever. It achieves this by leveraging a dense retrieval architecture to enhance lexical matching capabilities. The model is based on a BERT-base architecture and is trained on Wikipedia articles.

Architecture

The model incorporates a BERT-base dense retriever architecture, allowing it to simulate BM25's lexical retrieval capabilities. It is part of a broader framework that includes context encoders and can operate alongside a dense retriever to facilitate both lexical and semantic matching. This architecture is crucial for applications such as Open-Domain Question Answering.

Training

The SPAR-Wiki-BM25-Lexmodel-Query-Encoder was trained using Wikipedia articles, with a focus on mimicking the sparse retrieval performance of BM25. The training process involves using a BERT-base model as the backbone, ensuring compatibility with other dense retriever architectures for enhanced performance.

Guide: Running Locally

1. Setup Environment: Ensure you have Python and PyTorch installed. Use a virtual environment for better dependency management.

2. Install Transformers:

   pip install transformers
   

3. Load the Model:

   import torch
   from transformers import AutoTokenizer, AutoModel
   
   tokenizer = AutoTokenizer.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
   query_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-query-encoder')
   context_encoder = AutoModel.from_pretrained('facebook/spar-wiki-bm25-lexmodel-context-encoder')
   

4. Prepare Input: Tokenize your query and contexts.

   query_input = tokenizer("Where was Marie Curie born?", return_tensors='pt')
   ctx_input = tokenizer(["Context 1", "Context 2"], padding=True, truncation=True, return_tensors='pt')
   

5. Compute Embeddings:

   query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :]
   ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :]
   

6. Compute Similarity Scores: Use the dot product to calculate similarity.

   score1 = query_emb @ ctx_emb[0]
   score2 = query_emb @ ctx_emb[1]
   

Cloud GPUs: For improved performance, especially with large datasets, consider using cloud services such as AWS EC2, Google Cloud, or Azure to access powerful GPU resources.

License

The model and its code are part of the Meta AI research initiatives and are available under licenses that typically allow for research and non-commercial use. Refer to the specific repository for detailed license information.