Skip to content
OPQAI.
Sourced intermediate / 💻 Coding Free tools

Run a Local AI Model with Ollama for Privacy and Cost Control

Job to be done: Set up and run local LLMs for development, privacy, or cost control

🇳🇬 Ways to use this in Nigeria

Ideas to get you started, adapt to your situation.

  • 9-5 employee

    As an employee handling confidential company data, use a local LLM to summarize internal reports or draft sensitive communications, ensuring data privacy and compliance.

  • Student

    As a university student, run a local LLM on your laptop to summarize research papers or generate practice questions for exams, keeping your study data private and avoiding internet costs.

  • Entrepreneur

    As an entrepreneur developing an AI-powered product, use a local LLM for rapid prototyping and testing new features without incurring API costs or exposing proprietary data.

What you’ll get

You will have an AI model running directly on your computer, accessible through your terminal. This is useful for keeping your data private, avoiding API costs when experimenting, and developing AI applications without relying on external services.

Tools you need

  • Ollama (free): A tool that makes it easy to download and run large language models (LLMs) on your computer.
  • Terminal (free): The command line interface (like Command Prompt or PowerShell on Windows, or Terminal on macOS/Linux) where you will type commands to control Ollama.

Steps

  1. Install Ollama: Go to the Ollama website (https://ollama.com). Click the “Download” button for your operating system (macOS, Windows, or Linux) and follow the on-screen instructions to install the application.
    • You should see Ollama installed on your computer. On macOS, it will appear in your Applications folder. On Windows, it will run in the background, often with a small icon in your system tray.
  2. Open your Terminal: Open your computer’s terminal application.
    • On Windows: Search for “Command Prompt” or “PowerShell” in the Start menu and open it.
    • On macOS: Search for “Terminal” in Spotlight (Cmd + Space) and open it.
    • On Linux: Open your preferred terminal application.
    • You should see a window with text appear, showing a blinking cursor where you can type commands.
  3. Download an AI model: In your terminal, type the command to download an AI model. The author mentioned llama3.2, but llama3 is a widely available and recommended model for beginners. 
    ollama pull llama3
    • You will see progress updates as the model downloads. This can take several minutes depending on your internet speed and the model’s size. Once complete, you should see a message indicating success.
  4. Run the AI model: Once the model is downloaded, you can start interacting with it. 
    ollama run llama3
    • The terminal will change, and you will see a prompt like >>> or Send a message (/? for help). You can now type your questions or prompts directly into the terminal.
  5. Chat with your local AI: Type a question or command, then press Enter. 
    What is the capital of Nigeria?
    • The AI model will process your request and display its answer in the terminal. To exit the chat, type /bye and press Enter.

Original source

This workflow is inspired by “Running Local LLMs: A Practical Guide” by philk10, published on Hackernews. The article explores various ways to run AI models on your own computer, highlighting Ollama as a top choice for its ease of use.

Notes & variations

  • Free-tier alternatives: For quick chats without setup, freemium online services like ChatGPT (chatgpt.com), Claude (claude.ai), or Gemini (gemini.google.com) offer free tiers that are often more convenient.
  • Common mistake: Forgetting to pull the model before trying to run it. If you see an error like “model not found,” make sure you’ve completed the ollama pull step successfully.
  • Tip for better results: Experiment with different models. Ollama’s library includes many models like Mistral, Qwen, and DeepSeek. You can find a list on the Ollama website. Smaller models (e.g., 7B or 3B parameters) run faster on less powerful computers but might give less detailed answers.
  • Other tools: The author also mentions Llama.cpp for highly versatile, low-level control, and Llamafiles for single-executable models, which are more advanced options for developers.

Keep going

More Coding workflows