Build an Offline AI Assistant with Python and Ollama

What this is, in plain English

This is a developer’s account of building an AI assistant in Python that runs entirely on your own computer: no internet, no API keys, no cloud bills, no heavy libraries. The clever part is that it does not lean on a big AI model for everything. It first uses a small, hand-written “intent scorer” (simple word-matching that guesses what you are asking) and only calls a local AI model through Ollama when it needs more brainpower. That keeps it fast, private, and free.

Be honest about the level: this is an advanced coding project, not a copy-paste recipe. The original article shares the key ideas and code snippets, but not a complete, ready-to-run program, you assemble the surrounding code yourself. So this page explains what it is, what it is good for, and the realistic shape of building it, with the author’s core snippet to anchor the approach.

What you can use it for

• An offline study assistant. Answer questions from your own downloaded notes during exam prep, with no internet needed.
• Private document Q&A. Summarize internal policies or answer HR questions on a company machine, where sensitive data must never leave the building.
• An offline chatbot for low-connectivity areas. For example, quick crop-disease answers in a farming app that works without a signal.
• A cost-free prototype. Try out assistant ideas with zero per-message charges before committing to a paid API.
• Learning how assistants actually work. Building the intent-scoring and model-routing yourself demystifies what tools like the big chatbots do under the hood.

Tools you need

• Python (free): the language you build the assistant in. Version 3.8 or newer.
• Ollama (free): runs AI models locally, called only when the simple matching is not enough.

How it actually works

The first three steps are concrete setup; the rest is the design you build in Python.

1. Install Python. Download it from python.org and install. On Windows, tick “Add Python to PATH”. Check it worked:

   python --version

2. Install Ollama and pull a model. Get Ollama from its site, then download a model and try it:

   ollama run llama2

   It downloads the model, then opens a chat. Type bye to leave. (This is the local AI your assistant will call when needed.)

3. Start your Python file. Create offline_assistant.py in a folder; this will hold your code.

4. Build the intent scorer (the heart of it). Instead of a heavy AI for everything, the author scores how well your words match known patterns, exact matches, prefix matches, and even one-character typos. Their core function:

   def _score_intent(self, tokens: List[str], full_text: str, patterns: List[str]) -> float:
       score = 0.0
       for pattern in patterns:
           if " " in pattern:  # multi-word phrase: bonus by word count
               if pattern in full_text:
                   score += len(pattern.split()) * 1.5
           else:
               if pattern in tokens:
                   score += 1.0  # exact match
               elif any(t.startswith(pattern) or pattern.startswith(t) for t in tokens if len(t) >= 3 and len(pattern) >= 3):
                   score += 0.4  # prefix match ("perfor" ~ "performance")
               elif len(pattern) >= 5 and any(self._edit_distance(t, pattern) <= 1 for t in tokens if abs(len(t) - len(pattern)) <= 2 and len(t) >= 4):
                   score += 0.6  # typo tolerance (1-character difference)
       return score

   You supply your own patterns (lists of keywords per intent) and a simple _edit_distance function (a standard “Levenshtein distance”, widely available online).

5. Blend in a model when unsure (optional). The author adds a second opinion: if a trained classifier is very confident it overrides the keyword guess, and if both agree it strengthens the result. This “trust the confident source, agree to amplify” logic is the idea; you decide whether to add it.

6. Route hard questions to the local model (optional). When word-matching is not enough, send the question to Ollama using Python’s requests library (pip install requests) and use its reply. Now the assistant handles open-ended questions too, still fully offline.

Words you’ll see, explained

• Intent: what the user is actually asking for; the scorer guesses it from their words.
• Token: a single word the assistant breaks your sentence into.
• Edit distance (Levenshtein): how many single-letter changes turn one word into another; used here to forgive typos.
• Classifier: a small model that labels input into categories, used as the optional second opinion.
• Confidence: how sure a method is of its answer; the assistant trusts the more confident source.
• LLM: large language model, the bigger AI (run via Ollama) called only for harder questions.

Original source

Based on “How I Built an Offline AI Assistant in Python - No OpenAI, No LangChain, No Dependencies” by huckler on the DEV Community blog, sharing a privacy-first design that does most of its work without a heavy AI model.

Notes & variations

• Do you even need to build this? For casual use, running Ollama directly (or the GPT4All app) gives you an offline chatbot with no coding. Build this when you want a fast, custom assistant that only calls the model when it must.
• Common mistake: expecting the snippets to run as-is. They illustrate the approach; a working assistant needs the surrounding code (a main loop, input handling, your patterns, and _edit_distance).
• Tip for better results: start with a small, focused set of patterns and expand them as you test with real questions. A basic Levenshtein _edit_distance is easy to find and drop in.