codegemma 7b

Introduction

CodeGemma is a suite of lightweight, open-source models for code completion and generation, developed by Google. Built on the Gemma framework, these models excel in tasks like code completion, generation from natural language, and code chat. The collection includes a 7 billion parameter model for general purposes and a 7 billion parameter instruction-tuned variant for more interactive tasks.

Architecture

CodeGemma models are decoder-only models structured for both text-to-text and text-to-code applications. The architecture leverages a fill-in-the-middle (FIM) objective for tasks requiring code completion, utilizing specific tokens to denote prefixes, suffixes, and middle segments.

Training

The CodeGemma models are trained on 500 billion tokens sourced from public code repositories and synthetic datasets. Key training techniques include dependency graph-based packing and unit test-based lexical packing. The models employ rigorous safety measures, such as filtering sensitive content and ensuring alignment with ethical AI principles.

Guide: Running Locally

1. Installation: Install the transformers library to access CodeGemma models.
2. Setup: Use GemmaTokenizer and AutoModelForCausalLM from transformers to load models.
3. Model Inference: Create inputs using the tokenizer and generate outputs with the model's generate function.
4. Example: For code completion, prepare a prompt with <|fim_prefix|>, <|fim_suffix|>, and <|fim_middle|> tokens.

Cloud GPUs

For efficient processing, consider using cloud GPUs from providers like AWS, Google Cloud, or Azure.

License

CodeGemma models are released under the Gemma license. Users must review and agree to Google's usage terms before accessing the models. More details can be found on the Gemma License page.