roberta small

Introduction

KLUE RoBERTa Small is a pretrained RoBERTa model for the Korean language, part of the KLUE benchmark, which evaluates Korean language understanding. It is implemented using the Transformers library from Hugging Face, utilizing PyTorch and Safetensors. The model is designed for fill-mask tasks in Korean.

Architecture

The KLUE RoBERTa Small model is based on the RoBERTa architecture, which is a robustly optimized BERT variant. It is tailored to Korean language tasks, making use of specific tokenizers to handle Korean text efficiently.

Training

The model was trained on a diverse dataset as part of the KLUE benchmark, which aims to improve Korean language understanding. Details on the model's training and evaluation can be found in the associated paper on arXiv: KLUE: Korean Language Understanding Evaluation.

Guide: Running Locally

To use KLUE RoBERTa Small, follow these basic steps:

1. Install the Transformers library:

   pip install transformers
   

2. Load the model and tokenizer:

   from transformers import AutoModel, AutoTokenizer
   
   model = AutoModel.from_pretrained("klue/roberta-small")
   tokenizer = AutoTokenizer.from_pretrained("klue/roberta-small")
   

   Note: Use BertTokenizer instead of RobertaTokenizer. By default, AutoTokenizer loads BertTokenizer.

3. Initialize and run your inference with the loaded model and tokenizer.

For optimal performance, consider using cloud GPUs from providers like AWS, GCP, or Azure.

License

Please refer to the model's GitHub repository for licensing details, usage guidelines, and restrictions associated with KLUE RoBERTa Small.