Java B E R T

Introduction

JavaBERT is a language model based on BERT, specifically pretrained on Java software code. Developed by the Software Engineering Group at Christian-Albrechts-University of Kiel (CAUKiel), the model is designed for fill-mask tasks in Java programming language contexts. It is licensed under Apache-2.0.

Architecture

JavaBERT utilizes a BERT-like architecture, focusing on the fill-mask task. The model uses a bert-base-cased tokenizer and was trained on a large corpus of Java files sourced from GitHub.

Training

Training Data

The model was trained using 2,998,345 Java files from open source projects on GitHub.

Training Procedure

JavaBERT was trained with a Masked Language Model (MLM) objective. Further details on preprocessing, speeds, sizes, and times are not provided.

Guide: Running Locally

To utilize JavaBERT locally, you can use the Hugging Face Transformers library. Here is a basic example:

from transformers import pipeline
pipe = pipeline('fill-mask', model='CAUKiel/JavaBERT')
output = pipe('public [MASK] isOdd(Integer num) {if (num % 2 == 0) {return "even";} else {return "odd";}}')

For optimal performance, especially with large datasets, consider using cloud GPUs such as those provided by AWS, Google Cloud, or Azure.

License

JavaBERT is released under the Apache-2.0 license, which permits use, distribution, and modification, provided that the license terms are followed.