tapas base finetuned wtq

Introduction

TAPAS is a transformer model akin to BERT, designed for table-based question answering tasks. It is pre-trained on English data from Wikipedia and fine-tuned specifically for tasks like SQA, WikiSQL, and WTQ.

Architecture

The TAPAS model utilizes relative position embeddings and is designed to handle tabular data with associated text. It is pre-trained with two main objectives: Masked Language Modeling (MLM) and intermediate pre-training for numerical reasoning. The model architecture includes a cell selection head and an aggregation head for fine-tuning on specific tasks.

Training

Preprocessing

Data is lowercased and tokenized using WordPiece with a vocabulary size of 30,000. The input format consists of a question followed by a flattened table.

Fine-Tuning

Training is performed on 32 Cloud TPU v3 cores for 50,000 steps, with a sequence length of 512 and a batch size of 512. An Adam optimizer with a learning rate of 1.93581e-5 is used. The model is fine-tuned to select cells within the same column.

Guide: Running Locally

1. Install Dependencies:

   • Ensure you have Python and PyTorch installed.
   • Install the Transformers library: pip install transformers.

2. Download the Model:

   • Use the Hugging Face Model Hub to download tapas-base-finetuned-wtq.

3. Load the Model:

   • Use the Transformers library to load the model and tokenizer:

     from transformers import TapasTokenizer, TapasForQuestionAnswering
     
     tokenizer = TapasTokenizer.from_pretrained("google/tapas-base-finetuned-wtq")
     model = TapasForQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")
     

4. Run Inference:

   • Prepare your data and run inference through the model.

Cloud GPUs

For faster training and inference, consider using cloud GPU providers such as AWS, GCP, or Azure.

License

The TAPAS model is licensed under the Apache 2.0 License, allowing for both personal and commercial use.