Why Run AI Locally?

What You Actually Need

Minimum Viable Setup

GPU Setup (Recommended)

What "VRAM" Actually Means

Installing Ollama

Your First Model

Choosing the Right Model for Your Hardware

If you have no GPU (CPU only)

If you have 6-8GB VRAM

If you have 10-12GB VRAM

If you have 16GB VRAM

The Ollama API (Actually Useful)

OpenAI-Compatible API Mode

Open WebUI: The Best UI for Ollama

Useful Local AI Tasks

Code Review and Debugging

Document Summarization

Writing First Drafts

Private Q&A

Scripting and Automation

Git Commit Messages

Model Chaining and Pipelines

The Thing Everyone Misses

Next Steps Running AI models locally has become surprisingly accessible. With Ollama, you can run capable language models on a laptop or desktop — no API keys, no subscriptions, no internet required. Here's a practical guide to getting set up, choosing the right model, and actually using local AI for something useful. Three main reasons people do this: Privacy. Your prompts never leave your machine. If you're processing code, client data, personal notes, or anything sensitive, local means you control where it goes. Cost. After hardware, inference is free. No per-token billing, no monthly subscriptions, no rate limits. Run it as much as you want. Ownership. The model doesn't change overnight, doesn't go down, doesn't require an internet connection. Works on a plane, in a basement, wherever. The tradeoff is hardware. You need a GPU with enough VRAM to fit the model, or you fall back to CPU inference (slow but usable for some tasks). VRAM is your bottleneck. A model loaded into VRAM runs fast (GPU inference). A model that overflows to RAM runs slow (partial CPU fallback). A model entirely on CPU is slower still but still works. Rule of thumb: a 7B model at Q4 quantization needs about 4-5GB VRAM. A 14B model needs 8-10GB. A 27B model needs 15-16GB. Ollama is available for Linux, macOS, and Windows. Installation is straightforward. macOS: Download from ollama.com — native app with menu bar integration. Windows: Native installer available at ollama.com. After install, Ollama runs as a background service and exposes a REST API at http://localhost:11434. After installing Ollama, pull and run your first model: The first pull downloads the model weights (~4-16GB depending on size). After that, launching is instant. Llama 3.2 3B — Fast enough for quick tasks. Good for summarizing, drafting, answering questions. Limited by small context. Phi-3 Mini — Microsoft's 3.8B model. Surprisingly capable for its size. Excellent for CPU inference. Llama 3.1 8B — Meta's flagship small model. Versatile, fast. Great starting point. Mistral 7B — Fast, efficient, strong for its size. Good instruction following. Qwen2.5 7B — Strong coding performance in a small package. Llama 3.1 8B Q8 — Higher quality than Q4, fits comfortably in 12GB. Qwen2.5 14B Q4 — Best quality/speed tradeoff in this range. Good at code and reasoning. Phi-4 14B — Microsoft's current flagship. Very capable for its size. Qwen2.5 32B Q3/Q4 — This is where it gets interesting. 32B class performance at 16GB. DeepSeek R1 14B — Reasoning-focused model. Slower but more careful reasoning. Great for complex tasks. Devstral 24B — Coding specialist. Excellent for code generation, review, debugging. Ollama exposes a REST API that you can call from any language: This is what makes Ollama powerful for automation. You can pipe it into scripts, build small apps, automate content generation — all running locally. Ollama also runs in OpenAI-compatible mode, which means any tool built for the OpenAI API works with Ollama: Tools like Continue (VS Code extension), Open WebUI, Obsidian AI, and OpenClaw all support connecting to a local Ollama instance this way. If you want a ChatGPT-style interface for your local models, Open WebUI is the best option. Then open http://localhost:3000. It connects to your local Ollama automatically. Features worth using: What people actually do with local AI: Paste a function. Ask what's wrong with it. No code ever leaves your machine. Feed a long PDF or article. Get a clean summary. Useful for research, reading, catching up. Brief → full draft in seconds. Edit down from there. Faster than staring at a blank page. Anything you'd normally Google but don't want tracked — medical questions, legal basics, financial concepts. Describe what you want a script to do. Get working Python or bash as a starting point. Paste your diff. Ask for a clean commit message. Small thing, constant annoyance solved. More advanced: you can chain Ollama calls to build small pipelines. Example: summarize a web page, then extract action items, then format as a structured report — three separate prompts, each feeding into the next. Libraries like LangChain, LlamaIndex, and OpenAI-SDK (pointed at Ollama's API) all support this. Local inference makes these workflows free to run as much as you want. People try local AI, get mediocre results, and blame the model. Usually the problem is the prompt. Local models are more sensitive to prompt quality than hosted models. They benefit from: The model is doing its job. Your job is giving it the right input. If this is interesting to you: The ecosystem moves fast — check ollama.com/library for new models as they drop. Running a homelab and want to go deeper? The Homelab Starter Guide covers self-hosting fundamentals including setting up Docker, securing your services, and building a proper local stack. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse