twhin bert base

Introduction

TwHIN-BERT is a multilingual pre-trained language model designed for tweet representations, leveraging both textual and social information from Twitter. It processes over 7 billion tweets across 100+ languages and incorporates a social objective derived from the Twitter Heterogeneous Information Network.

Architecture

TwHIN-BERT introduces a socially-enriched pre-training approach that extends beyond the traditional text-based self-supervision methods like Masked Language Modeling (MLM). It incorporates social engagement data, enhancing its performance in tasks involving semantic understanding and social recommendations.

Training

Two versions of TwHIN-BERT have been released: TwHIN-BERT-base with 280 million parameters and TwHIN-BERT-large with 550 million parameters. These models are compatible with the Hugging Face Transformers library, allowing easy integration and use in various NLP tasks.

Guide: Running Locally

1. Installation
   Ensure you have the Hugging Face Transformers library installed:

   pip install transformers
   

2. Loading the Model
   Use the following Python code to load and use TwHIN-BERT:

   from transformers import AutoTokenizer, AutoModel
   
   tokenizer = AutoTokenizer.from_pretrained('Twitter/twhin-bert-base')
   model = AutoModel.from_pretrained('Twitter/twhin-bert-base')
   
   inputs = tokenizer("I'm using TwHIN-BERT! #TwHIN-BERT #NLP", return_tensors="pt")
   outputs = model(**inputs)
   

3. Running on Cloud GPUs
   For large-scale computations, consider using cloud platforms like AWS, Google Cloud, or Azure, which provide GPU instances suitable for model inference and fine-tuning.

License

TwHIN-BERT is released under the Apache-2.0 License, allowing for open use, modification, and distribution.