List Con Ranker

Introduction

ListConRanker is a Listwise-encoded Contrastive text ReRanker that includes a ListTransformer module for listwise encoding. This model facilitates global contrastive information learning between passage features by clustering similar passages and distinguishing between similar and dissimilar passages. It incorporates ListAttention to maintain query features while learning global comparative information. The training employs Circle Loss, which addresses issues of low data efficiency and unsmooth gradient change.

Architecture

The key component of ListConRanker is the ListTransformer module, which encodes listwise information to enhance the clustering and differentiation of passage features. The model is designed to learn global comparative information effectively, supported by ListAttention, which preserves the integrity of query features during the process.

Training

The model undergoes a two-stage training process:

1. Stage One: Freeze the embedding model parameters and train the ListTransformer for 4 epochs with a batch size of 1024.
2. Stage Two: Unfreeze all parameters and train for an additional 2 epochs with a batch size of 256.

The training data comprises approximately 2.6 million queries, with datasets including cMedQA1.0, cMedQA2.0, MMarcoReranking, T2Reranking, among others.

Guide: Running Locally

1. Installation: Clone the repository and navigate to the project directory.

2. Dependencies: Install necessary dependencies using pip install -r requirements.txt.

3. Model Initialization: Import and initialize the ListConRanker model.

   from modules.listconranker import ListConRanker
   reranker = ListConRanker('./', use_fp16=True, list_transformer_layer=2)
   

4. Inference: Prepare your batch of queries and passages, then compute scores using:

   scores = reranker.compute_score(batch)
   

5. Iterative Inference: For iterative inference, which supports only a batch size of 1:

   scores = reranker.iterative_inference(batch[0])
   

6. Reproduce Results: Use provided scripts for evaluation:

   • Iterative Inference: python3 eval_listconranker_iterative_inference.py
   • Without Iterative Inference: python3 eval_listconranker.py

7. Hardware Suggestion: For optimal performance, utilize cloud GPUs such as those available on AWS, Google Cloud, or Azure.

License

This work is licensed under the MIT License. The model weights are available under the Creative Commons Attribution-NonCommercial 4.0 International License.