Prompt Task and Complexity Classifier

Introduction

The Prompt Task and Complexity Classifier model classifies English text prompts into task types and evaluates them along complexity dimensions. It distinguishes tasks across 11 categories and assesses complexity over 6 dimensions, ultimately producing an overall complexity score. This model is suitable for commercial use.

Architecture

The model is built on a DeBERTa backbone with multiple classification heads for task categorization and complexity assessment. It supports simultaneous predictions during inference with a default context length of 512 tokens, though DeBERTa can handle up to 12k tokens.

Training

The model was trained with 4024 English prompts, annotated according to task and complexity taxonomies. Task distribution includes categories like Open QA, Text Generation, and Summarization. Evaluation uses top-1 accuracy with n-fold cross-validation to ensure consistency, achieving high accuracy across complexity dimensions.

Guide: Running Locally

1. Prerequisites:

   • Python 3.10
   • Install necessary libraries: transformers, torch, and huggingface_hub.

2. Code Setup:

   • Use the provided PyTorch code snippet to load the model and tokenizer.
   • Prepare your prompts and encode them using the tokenizer.
   • Run the model to get task and complexity classifications.

3. Hardware Requirements:

   • NVIDIA GPU with Volta™ architecture or higher, compatible with CUDA 12+.

4. Cloud GPUs:

   • Consider cloud services like AWS EC2 with GPU support for efficient performance.

License

This model is released under the NVIDIA Open Model License Agreement.