sentence compression

Introduction

The Sentence-Compression model is a fine-tuned version of distilbert-base-cased designed for token classification tasks. This model is optimized for compressing sentences while retaining their core meaning. It achieves a loss of 0.2973 and an accuracy of 0.8912 on the evaluation set.

Architecture

The model is based on the distilbert-base-cased architecture, leveraging the transformer-based approach for NLP tasks. The fine-tuning process enhances this pre-trained model to specialize in sentence compression tasks.

Training

Training Hyperparameters

• Learning Rate: 5e-05
• Training Batch Size: 16
• Evaluation Batch Size: 64
• Seed: 42
• Optimizer: Adam with betas (0.9, 0.999) and epsilon 1e-08
• Learning Rate Scheduler Type: Linear
• Warmup Steps: 500
• Number of Epochs: 3

Training Results

The model achieved the following metrics over three epochs:

• Epoch 1: Validation Loss: 0.2667, Accuracy: 0.8894, F1: 0.8283
• Epoch 2: Validation Loss: 0.2704, Accuracy: 0.8925, F1: 0.8372
• Epoch 3: Validation Loss: 0.2973, Accuracy: 0.8912, F1: 0.8367

Framework Versions

• Transformers: 4.12.5
• PyTorch: 1.10.0+cu113
• Datasets: 1.16.1
• Tokenizers: 0.10.3

Guide: Running Locally

To run the Sentence-Compression model locally, follow these steps:

1. Setup Environment: Ensure you have Python and Git installed. Create a virtual environment and activate it.
2. Install Dependencies: Use pip to install the requirements:

   pip install transformers==4.12.5 torch==1.10.0+cu113 datasets==1.16.1 tokenizers==0.10.3
   

3. Download Model: Clone the repository or download the model files from Hugging Face Hub.
4. Run the Model: Use the Hugging Face Transformers library to load and run the model.

   from transformers import AutoModelForTokenClassification, AutoTokenizer
   
   model = AutoModelForTokenClassification.from_pretrained("alexmaclean/sentence-compression")
   tokenizer = AutoTokenizer.from_pretrained("alexmaclean/sentence-compression")
   

5. Inference: Prepare your input data and run inference using the model.

Suggested Cloud GPUs

For optimal performance, consider using cloud GPUs such as NVIDIA Tesla T4 or V100 on platforms like AWS, Google Cloud, or Azure.

License

The Sentence-Compression model is licensed under the Apache 2.0 License, allowing for both personal and commercial use, distribution, and modification.