Huatuo G P T o1 8 B G G U F

Introduction

HuatuoGPT-o1-8B-GGUF is a text generation model optimized for medical applications, utilizing quantization techniques to reduce resource requirements. It is based on the HuatuoGPT-o1-8B model by FreedomIntelligence and supports English language processing for medical reasoning and problem verification.

Architecture

The model employs llama.cpp quantization (release b4381), using an imatrix option for efficient performance. Various quantized versions of the model are available, each optimized for different hardware and quality requirements.

Training

The model is trained using datasets focused on medical reasoning and verifiable problems, specifically:

• FreedomIntelligence/medical-o1-reasoning-SFT
• FreedomIntelligence/medical-o1-verifiable-problem

Guide: Running Locally

1. Install Hugging Face CLI:

   pip install -U "huggingface_hub[cli]"
   

2. Download the Model:
   Use the CLI to download your desired quantization file. For example:

   huggingface-cli download bartowski/HuatuoGPT-o1-8B-GGUF --include "HuatuoGPT-o1-8B-Q4_K_M.gguf" --local-dir ./
   

3. Choose the Right File:

   • Consider your hardware's RAM/VRAM capacity.
   • If using a GPU, select a file size 1-2GB smaller than your VRAM.
   • For maximum quality, combine system RAM and VRAM, and choose a file accordingly.

4. Run the Model:
   Load the model into your preferred environment, such as LM Studio, ensuring compatibility with your hardware setup.

5. Cloud GPU Option:
   Consider using cloud services like AWS or GCP for GPU resources if local hardware is insufficient.

License

The model is licensed under the Apache 2.0 License, allowing for both personal and commercial use with proper attribution.