cosmicroberta

Introduction

CosmiRoBERTa is a specialized version of the RoBERTa model, pre-trained for space science applications. It leverages a domain-specific corpus that includes data from the NASA Technical Reports Server (NTRS), SCOPUS abstracts, European Cooperation for Space Standardization (ECSS) requirements, among other sources. The corpus consists of approximately 75 million words. The model improves upon existing benchmarks in specific space-related tasks.

Architecture

CosmiRoBERTa builds upon the RoBERTa architecture, a robust transformer-based model known for its language understanding capabilities. The architecture is augmented with domain-specific knowledge from the space science sector, enhancing its performance in tasks such as Guidance, Navigation, and Control (GN&C), System Engineering, and others.

Training

The model has been trained using a comprehensive dataset of space science-related texts. It achieves superior performance on a subset of the CR task described in the SpaceTransformers paper, demonstrating improvements across various parameters and tasks compared to the base RoBERTa model. For instance, it shows enhanced performance in areas like System Engineering, Propulsion, and Thermal analysis.

Guide: Running Locally

1. Setup Environment: Ensure you have Python and PyTorch installed. Use a virtual environment for better dependency management.

2. Install Transformers Library: Run pip install transformers.

3. Load the Model: Use the Hugging Face Transformers library to load CosmiRoBERTa with the following code:

   from transformers import RobertaTokenizer, RobertaForMaskedLM
   
   tokenizer = RobertaTokenizer.from_pretrained('icelab/cosmicroberta')
   model = RobertaForMaskedLM.from_pretrained('icelab/cosmicroberta')
   

4. Inference: Use the model to perform tasks like mask-filling by tokenizing input text and obtaining predictions.

5. Hardware Recommendations: For optimal performance, especially with large datasets, consider using a cloud GPU service such as AWS EC2 with NVIDIA GPUs or Google Cloud's AI Platform.

License

CosmiRoBERTa is released under the MIT License, allowing for wide usage and adaptation. This permissive license enables users to freely use, modify, and distribute the model with minimal restrictions.