[agenticwork]
← blog

architecture · april 22, 2025 · 10 min read

Building SmartModelRouter: How We Route Across Multiple LLM Providers With Minimal Overhead

One of the first components we built for the platform was SmartModelRouter. The reasoning was straightforward: if you are building AI agents for the enterprise, you cannot afford to be locked into a single LLM provider. Pricing changes. Models get deprecated. Outages happen. New capabilities emerge from competitors. And different tasks genuinely require different models.

We shipped SmartModelRouter in April 2025 as a core platform service. Here is how it works, why we built it this way, and what we learned.

The Three-Tier Architecture

Not every AI agent request requires a frontier model. A tool that extracts a date from an email does not need Opus-level reasoning. A task that plans a multi-step workflow across five systems does. The first insight behind SmartModelRouter is that most agentic workloads follow a predictable distribution:

  • 40-60% of requests are Economy Tier: text classification, data extraction, formatting, summarization. Models like Anthropic Haiku, OpenAI GPT-4o mini, or Google Gemini Flash handle these at $0.25-0.80 per million tokens.
  • 30-40% are Balanced Tier: multi-step reasoning, code generation, detailed analysis. Anthropic Sonnet, OpenAI GPT-4o, Google Gemini Pro. $3-15 per million tokens.
  • 5-15% are Premium Tier: frontier reasoning, complex planning, ambiguous decisions. Anthropic Opus with Extended Thinking, OpenAI reasoning models. $15+ per million tokens.

Without intelligent routing, enterprise teams default everything to the most capable model — paying premium prices for tasks that a $0.25/M model handles identically. SmartModelRouter analyzes task complexity before selecting a model, routing a significant share of typical agentic workloads to cheaper tiers, with the goal of reducing cost while preserving output quality.

The Intelligence Slider

We wanted a control that was intuitive enough for product managers but powerful enough for ML engineers. The Intelligence Slider is a 0-100 scale that controls inference depth and model selection:

  • 0-30: Economy models, minimal reasoning chains, fastest response times
  • 30-70: Balanced models, standard reasoning, good cost-quality tradeoff
  • 70-100: Premium models, extended thinking, maximum reasoning depth

The slider is not just a model picker. At higher settings, SmartModelRouter enables extended thinking mode on supporting models, increases context window allocation, and adjusts retry strategies for more thorough reasoning. At lower settings, it constrains output length, disables extended thinking, and prioritizes latency over depth.

The result: teams can tune cost-quality tradeoffs per workflow, per agent, or per task without touching agent code.

Automatic Failover Chains

Single-provider architectures have a single point of failure. When OpenAI has an outage, your agents stop working. When Anthropic rate-limits you during a usage spike, requests queue and timeout.

SmartModelRouter maintains failover chains for each model tier. If the primary model returns an error or exceeds latency thresholds, the request automatically routes to the next provider in the chain. The agent code never knows a failover happened. The audit trail records it.

Failover chains are configurable per deployment. An air-gapped environment might have Ollama as primary with no cloud fallback. A cloud-first deployment might chain Anthropic → OpenAI → Bedrock. A hybrid deployment might prefer local models for sensitive data with cloud failover for non-sensitive tasks.

Provider-Agnostic Agent Code

The most important design decision in SmartModelRouter: agent code never references a specific model or provider. Agents describe what they need (reasoning depth, output format, context requirements) and SmartModelRouter handles the mapping. This means:

  • Switching from OpenAI to Anthropic is a configuration change, not a code change
  • Adding a new provider (Bedrock, Ollama, a fine-tuned model) requires zero agent modifications
  • Cost optimization happens at the routing layer, not in application code
  • Model deprecations and upgrades are transparent to agents

We currently support Anthropic (Claude family), OpenAI (GPT-4, GPT-4o), Google (Gemini), AWS Bedrock, and self-hosted models via Ollama. Adding a new provider is implementing a single interface — the routing logic, failover chains, and cost optimization work automatically.

Minimal Routing Overhead

SmartModelRouter adds minimal overhead to any request. The routing decision is lightweight: analyze task metadata (complexity hint, required capabilities, security classification), check the Intelligence Slider setting, consult the failover chain health status, and select a provider. No LLM call is needed for the routing decision itself.

Model health is tracked with a sliding window. If a provider's p95 latency exceeds thresholds or error rate climbs above 5%, SmartModelRouter deprioritizes it in the chain until health recovers. This happens automatically — no operator intervention needed.

Integration with the Platform

SmartModelRouter is one layer of the platform stack. Every routing decision is recorded in the Audit System — which model was selected, why, what the alternatives were, and how much it cost. The DLP Scanner checks requests before they reach the model, ensuring sensitive data is redacted or routed to on-premise models only. Credential isolation ensures API keys for each provider are scoped and rotated independently.

For teams using OATS for tool synthesis, SmartModelRouter selects the model that generates the tool code. Economy models handle simple API calls. Premium models handle complex multi-step tool generation. The Intelligence Slider lets teams tune synthesis quality without changing their workflow.

In the Workflow Builder, each node in an agent pipeline can have its own routing policy. A data extraction step uses economy. An analysis step uses balanced. A decision step uses premium. Cost optimization becomes part of workflow design, not a separate concern.

What We Learned

Building SmartModelRouter taught us that most enterprise AI cost problems are routing problems. Teams are not overspending because models are expensive — they are overspending because every request goes to the same model regardless of complexity. The fix is not cheaper models. The fix is smarter routing.

The open question is how far automated routing can go. Today, SmartModelRouter uses task metadata and heuristics. Tomorrow, it could use historical performance data to learn which model performs best for specific task types within a specific organization. The architecture supports this — the routing layer is separate from the model layer, so the routing logic can evolve independently.

SmartModelRouter is available to all platform users. If you are running agentic workloads across multiple providers and want to stop managing model selection manually, join the waitlist.