Running AI Models Locally with Ollama: A Complete Guide
Not every AI workload should go to a cloud API. If you are processing sensitive documents, working with proprietary code, handling regulated data, or simply want to avoid sending your prompts to a third-party server, running models locally is not a compromise — it is a requirement. Ollama has become the standard tool for making local model deployment accessible to individual developers and organizations alike.
Ollama is an open-source tool that packages, distributes, and runs large language models on local hardware. It handles model downloading, quantization, memory management, and inference serving behind a clean command-line interface and an OpenAI-compatible REST API. If you can install a command-line tool, you can run a local LLM. The barrier to entry that used to require CUDA configuration, model format conversion, and inference server setup has been reduced to a single command.
Installation
Ollama is available on macOS, Linux, and Windows. Installation is straightforward on all platforms:
On macOS, download the application from ollama.com or install via Homebrew with brew install ollama. On Linux, run the official install script: curl -fsSL https://ollama.com/install.sh | sh. On Windows, download the installer from ollama.com. All methods install both the Ollama runtime and the command-line interface.
Once installed, Ollama runs as a background service that serves models via a local API on port 11434. You can verify the installation with ollama --version and check that the service is running by visiting http://localhost:11434 in a browser, which should return a simple status response.
The Model Library
Ollama's model library includes dozens of open-source models optimized for local execution. The library covers a range of sizes and capabilities:
- Llama (Meta): The Llama model family from Meta is among the most popular open-source LLMs. Llama 3.2 is available in 1B, 3B, 11B, and 90B parameter sizes. The smaller variants run comfortably on laptops with 8GB of RAM. The larger variants need more capable hardware but offer performance approaching cloud models.
- Mistral: Mistral AI's models are known for strong performance relative to their size. Mistral 7B and Mixtral 8x7B (a mixture-of-experts model) are available through Ollama and perform well for code generation, analysis, and general reasoning tasks.
- Qwen (Alibaba Cloud): The Qwen model family offers strong multilingual performance. Qwen 2.5 is available in sizes from 0.5B to 72B parameters, making it one of the most flexible families for local deployment. The smaller Qwen models are particularly capable for their size.
- Gemma (Google): Google's Gemma models are available in 2B and 7B sizes, optimized for efficiency on consumer hardware. They perform well for text generation and summarization tasks.
- Code-specialized models: Models like CodeLlama, DeepSeek Coder, and StarCoder are optimized for code understanding and generation. These are particularly useful for development teams that want local code assistance without sending proprietary code to external services.
Downloading a model is a single command: ollama pull llama3.2 downloads and configures Llama 3.2 for local use. Models are stored locally and loaded into memory on demand. You can run multiple models on the same machine and switch between them per request.
The API: OpenAI-Compatible by Default
One of Ollama's most practical features is its OpenAI-compatible API. Ollama serves models through a REST API at http://localhost:11434 that includes an endpoint at /v1/chat/completions matching the OpenAI chat completions format. This means any tool, library, or application that works with the OpenAI API can work with Ollama by changing the base URL.
The compatibility is practical, not theoretical. The Python openai library works with Ollama out of the box by setting the base URL to the local Ollama endpoint. Frameworks that support custom OpenAI-compatible endpoints — which is most of them at this point — can use Ollama as a drop-in replacement for cloud API calls.
Using Ollama with the Platform
The platform's SmartModelRouter includes a built-in Ollama provider that makes local model usage a first-class capability, not an afterthought. Ollama is one of multiple providers the SmartModelRouter supports (spanning multiple model families across several provider integrations), and it integrates seamlessly with the platform's routing policies. You can configure the SmartModelRouter to send sensitive workloads to local Ollama models while routing tasks requiring frontier reasoning to cloud providers — all through policy configuration, not code changes.
Running a tool synthesis against a local Ollama model is a single command with the agenticode-cli:
oat synth "your intent description" --provider ollama --model llama3.2
This tells the orchestrator to use the local Ollama instance for model inference. The synthesized tool is generated using the local model, validated, sandboxed, and executed — all without any data leaving your machine. The entire workflow, from intent to execution, happens on your hardware.
For teams running Ollama on a shared server rather than individual machines, the SmartModelRouter supports remote Ollama instances:
oat synth "your intent" --provider ollama --model llama3.2 --base-url http://server:11434
This routes model inference to an Ollama instance running on a different machine on your network. The data travels between your machine and the Ollama server but never leaves your network. For organizations with dedicated GPU servers, this is a common pattern: run Ollama on the GPU machine and access it from development workstations.
For air-gapped deployments, Ollama combined with the platform's Kubernetes-native architecture provides a complete AI platform with zero internet connectivity. Deploy the platform stack and Ollama on an on-premises or air-gapped Kubernetes cluster, and you have the full platform — SmartModelRouter, Workflow Builder, Audit System, DLP Scanner, RBAC — running entirely within your security boundary.
Data Sovereignty: The Real Value Proposition
The technical capability of running models locally is interesting. The data sovereignty implications are transformative for certain use cases.
When you use a cloud AI API, your prompts and their context travel to the provider's infrastructure. Even with contractual guarantees about data handling, the data leaves your network. For many use cases, this is perfectly acceptable. For others, it is a dealbreaker.
- Regulated industries: Healthcare organizations processing patient data, financial institutions handling non-public financial information, and government agencies processing classified or sensitive information may have regulatory obligations that prohibit sending data to external AI services.
- Proprietary code: Development teams working on proprietary software may not want their source code sent to external services for code completion or analysis, regardless of the provider's data handling policies.
- Air-gapped environments: Some operational environments have no internet connectivity by design. Ollama enables AI capabilities in these environments because it has no cloud dependency.
- Cost sensitivity: For high-volume, lower-complexity workloads, running inference locally can be more cost-effective than paying per-token cloud API prices, particularly if the organization already has capable hardware.
Hardware Considerations
Local model performance depends on your hardware. Here is a practical guide:
CPU-only inference works for smaller models (1B-7B parameters) and is adequate for development and testing. Response times will be measured in seconds per response rather than milliseconds. An Apple Silicon Mac with 16GB of unified memory runs 7B models at reasonable interactive speeds.
GPU-accelerated inference is recommended for production workloads and larger models. NVIDIA GPUs with CUDA support provide the best performance. A single RTX 4090 with 24GB of VRAM can run 13B models at near-interactive speeds. For 70B+ models, you need multiple GPUs or enterprise-grade hardware.
Quantization is Ollama's key enabler for running large models on consumer hardware. Models are available in multiple quantization levels (Q4, Q5, Q8) that trade small amounts of output quality for significantly reduced memory requirements. A 7B model at Q4 quantization requires approximately 4GB of RAM. The same model at full precision would require 14GB.
When to Use Local vs. Cloud
Local models and cloud APIs are not competitors — they are complementary. The right choice depends on the specific requirements of each workload:
Use local models when data must not leave your network, when you need predictable costs regardless of volume, when you are operating in air-gapped or restricted environments, or when latency to cloud APIs is unacceptable.
Use cloud APIs when you need frontier model capabilities that exceed what local hardware can run, when burst capacity matters more than steady-state cost, or when the operational overhead of managing local infrastructure exceeds the cost of API calls.
The platform's SmartModelRouter supports both. You can configure routing policies that send sensitive workloads to local Ollama models while routing tasks that require frontier reasoning to cloud providers. The Intelligence Slider (0-100) lets you tune the cost-vs-quality tradeoff, and with automatic failover across all providers including Ollama instances, the system stays resilient. The agent code does not change. The routing policy determines where each request goes based on the classification rules you define. Local and cloud become deployment options, not architectural decisions.