Triage GitHub Issues with Local AI Models

What this is, in plain English

When a busy GitHub project gets a flood of issues and pull requests, someone has to sort each one into the right category (“this is about authentication”, “this is a UI bug”). This workflow automates that sorting using AI models that run on your own machine, free and private, no cloud bills. The example used open models (Gemma and Qwen) inside an “agent harness” (a program that drives an AI through a task, here Pi) to label issues for the OpenClaw project.

Be honest about the level: this is advanced and developer-only. Running models locally needs a capable computer (plenty of memory, ideally a GPU), you wire up an agent harness, and the source shares the approach rather than copy-paste config. This page explains what it does, what it is good for, and the realistic shape of building it.

What you can use it for

• Auto-label issues and PRs. Tag each new item by topic so nothing sits unsorted.
• Route work to the right people. Send a database issue to the database folks automatically.
• Tame a busy open-source repo. Keep a flood of contributions organized without manual triage.
• Keep it private and free. Sensitive repos get AI triage without sending anything to a cloud service.
• The reusable idea: classifying any stream of text (tickets, emails, messages) with a local model and a fixed label set.

Tools you need

• Gemma (free) and Qwen (free): open AI models you download and run locally (via a runner like Ollama or LM Studio).
• Pi (freemium): the agent harness that drives the model through the classification task.
• OpenClaw (free): the open-source repository used as the worked example.

How it actually works

You run a local model, point a harness at it, and have it label each new item. The realistic shape:

1. Run a local model. Use Ollama or LM Studio to download and serve an open model (a Gemma or Qwen variant). It exposes a local address (often http://localhost:11434).
2. Connect the agent harness. Point Pi at that local model address so it can send prompts and get answers.
3. Define your labels. Decide the fixed set of categories (for example local_models, agent_runtime, ui_tui). A clear, finite list is what makes classification reliable.
4. Prompt it to classify. For each new issue or pull request, feed the model the title, body, and a snippet of the code change (“diff”), and ask it to pick one or more labels from your set.
5. Get structured output. Have it return the answer as JSON (tidy, labelled data) so your automation can act on it. The author used a final_json step for this.
6. Optionally give read-only context. For trickier cases, let the agent run a few safe, read-only commands to inspect the code, carefully, since it is processing outside input.

Words you’ll see, explained

• Local model: an AI model that runs on your own computer instead of a cloud service.
• Agent harness: a program (Pi here) that drives an AI through a multi-step task.
• Classification / labels: sorting each item into one of a fixed set of categories.
• Structured output (JSON): answers in a tidy data format a program can read.
• Inference: running a model to get an answer (the part that needs the hardware).
• Diff: the snippet showing what a pull request changed.

Original source

Based on a Hugging Face blog post by Onur Solmaz and Ben Burtenshaw on using local, open-weight models (Gemma, Qwen) in an agent harness to triage GitHub issues and pull requests for the OpenClaw repository.

Notes & variations

• Do you even need local models? For low volume, a free cloud AI tier can classify issues with less setup. Go local for privacy, no per-item cost, or high volume, if your hardware can handle it.
• Common pitfall: giving an agent broad system (bash) access while it processes outside input is a security risk. Restrict it to the minimum read-only commands.
• Tip for better results: put a few correctly-labelled examples in the prompt. Showing the model what good classification looks like sharply improves its accuracy.