Linkbricks Horizon A I Japanese Pro V3 70 B

Introduction

Linkbricks-Horizon-AI-Japanese-Pro-V3-70B is a language model developed by Saxo, a data scientist at Linkbricks, designed for text generation tasks. It utilizes a robust multi-lingual dataset to enhance its performance across various languages including Japanese, Korean, Chinese, and English. The model is built on the Saxo/Linkbricks-Horizon-AI-Japanese-Base-70B framework and optimized for handling complex logical and mathematical problems.

Architecture

The model leverages eight H100-80G GPUs for training, employing advanced techniques such as SFT (Supervised Fine-Tuning), DPO (Data Processing Optimization), and merging strategies. It operates with a 128k-context window, allowing it to handle extensive contextual information. The architecture supports function calls and uses Deepspeed Stage 3, rslora, and BAdam Layer Mode for enhanced performance.

Training

Training involves using a substantial corpus of 50 million Japanese news articles alongside multi-lingual datasets for cross-lingual learning. The model enhances capabilities in customer review analysis, social media post evaluation, coding, writing, and logical reasoning. The tokenizer is kept unchanged from the base model, ensuring consistency and efficiency in word processing.

Guide: Running Locally

1. Environment Setup: Install Python and necessary libraries, particularly the transformers library version 4.46.3.
2. Model Download: Access the model from the Hugging Face model hub.
3. Execution: Utilize a script to load the model and execute text generation tasks.
4. Hardware Suggestions: For optimal performance, it is recommended to use cloud GPUs such as NVIDIA's H100 series.

License

This model is released under the Apache 2.0 License, allowing for both personal and commercial use with attribution.