Notic I A 7 B

Introduction

The NoticIA-7B model is designed to generate concise summaries of Spanish clickbait articles. This model aims to reduce the impact of low-quality journalism by providing readers with valuable summaries that reveal the core message of sensational headlines. Developed by Iker García-Ferrero and Begoña Altuna, it is a 7-billion parameter language model fine-tuned specifically for summarizing clickbait content in Spanish.

Architecture

NoticIA-7B is built on a decoder-only architecture, fine-tuned from the openchat-3.5-0106 model. It uses the Next Token Prediction (NTP) loss for training and employs the Low-Rank Adaptation (LoRA) technique to preserve prior knowledge. The model is optimized to generate high-quality summaries while minimizing the length, balancing brevity and informativeness.

Training

The model was trained using the NoticIA-it dataset, comprising 850 Spanish news articles with clickbait headlines and single-sentence summaries. The training process involved 3 epochs with a batch size of 64, utilizing a bfloat16 regime with LoRA and Deepspeed Zero3 optimizations. The training aimed to teach the model to effectively summarize clickbait headlines by extracting the key message from the article content.

Guide: Running Locally

To run the NoticIA-7B model locally, follow these steps:

1. Install Dependencies: Ensure you have torch, newspaper3k, transformers, and bitsandbytes installed.

   pip install torch newspaper3k transformers bitsandbytes
   

2. Load the Model and Tokenizer:

   from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
   
   tokenizer = AutoTokenizer.from_pretrained("somosnlp/NoticIA-7B")
   quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
   model = AutoModelForCausalLM.from_pretrained("somosnlp/NoticIA-7B", torch_dtype=torch.bfloat16, device_map="auto", quantization_config=quantization_config)
   

3. Test the Model: Use the model to generate summaries for clickbait articles by applying a chat-style prompt and processing the output.

4. Consider Cloud GPUs: Running models with 7 billion parameters locally can be resource-intensive. Consider using cloud services like AWS, Google Cloud, or Azure with GPU instances for better performance.

License

The NoticIA-7B model is released under the Apache 2.0 license, allowing for wide usage and modification under the terms of this open-source license.