Agents with Local LLMs

Most discussions about running AI agents with local models obsess over the wrong thing: model capability. The question isn’t whether your local LLM is smart enough. It’s whether your agent architecture can survive imperfection.

The Wrong Question

When developers evaluate local LLMs for agentic workflows, they benchmark reasoning ability, context windows, and tool-calling accuracy. They compare Llama against Mistral against Phi, searching for the model that approaches GPT-4’s reliability.

This framing assumes agents need near-perfect inference to function. It doesn’t hold up. The agents that work well locally aren’t built on better models—they’re built on better error boundaries.

Designing for Graceful Degradation

A cloud-hosted agent can often brute-force its way through ambiguity with raw capability. A local agent needs guardrails that catch failures before they cascade.

Consider a simple tool-calling agent using Ollama:

import ollama
import json

def parse_tool_call(response: str) -> dict | None:
    try:
        start = response.find('{')
        end = response.rfind('}') + 1
        if start == -1 or end == 0:
            return None
        return json.loads(response[start:end])
    except json.JSONDecodeError:
        return None

def agent_step(prompt: str, max_retries: int = 3) -> dict:
    for attempt in range(max_retries):
        response = ollama.chat(
            model='mistral',
            messages=[{'role': 'user', 'content': prompt}]
        )
        result = parse_tool_call(response['message']['content'])
        if result and 'action' in result:
            return result
    return {'action': 'fallback', 'reason': 'parse_failure'}

The retry logic and fallback path matter more than the model choice. Local models produce malformed JSON more often than cloud APIs. Your architecture either handles this or it doesn’t.

Structured Output as a Forcing Function

The most effective technique for local agent reliability is constraining output format at generation time. Outlines and similar libraries let you define schemas that the model must follow:

from outlines import models, generate
from pydantic import BaseModel

class ToolCall(BaseModel):
    action: str
    parameters: dict[str, str]
    reasoning: str

model = models.transformers("mistralai/Mistral-7B-v0.1")
generator = generate.json(model, ToolCall)

result = generator("Extract the search query from: find restaurants near me")

This eliminates an entire class of parsing failures. The model physically cannot produce invalid structure. You trade some flexibility for dramatically improved reliability—a reasonable exchange when running smaller models.

The Context Window Trap

Local models typically offer 4K to 32K context windows. Agent loops that accumulate conversation history hit these limits fast. The fix isn’t finding a model with more context; it’s aggressive summarization:

interface AgentMemory {
  recentMessages: Message[];
  summary: string;
  maxRecent: number;
}

function compressMemory(memory: AgentMemory, llm: LocalLLM): AgentMemory {
  if (memory.recentMessages.length <= memory.maxRecent) {
    return memory;
  }
  
  const oldest = memory.recentMessages.slice(0, -memory.maxRecent);
  const newSummary = llm.generate(
    `Summarize these interactions concisely:\n${formatMessages(oldest)}\n\nPrevious context: ${memory.summary}`
  );
  
  return {
    ...memory,
    recentMessages: memory.recentMessages.slice(-memory.maxRecent),
    summary: newSummary
  };
}

Tool Design Over Model Selection

Local agents perform dramatically better when tools are designed for ambiguity tolerance. Instead of requiring exact parameter formats, accept fuzzy inputs and normalize internally:

def search_tool(query: str, filters: str = "") -> list[dict]:
    # Normalize whatever the model produces
    normalized_filters = parse_flexible_filters(filters)
    
    # Validate and clamp to safe ranges
    if 'limit' in normalized_filters:
        normalized_filters['limit'] = min(int(normalized_filters['limit']), 50)
    
    return execute_search(query, normalized_filters)

The tool becomes the reliability layer. Your model can be sloppy; your tools cannot.

The Actual Tradeoff

Running agents locally isn’t about matching cloud performance at lower cost. It’s about building systems that function predictably within known constraints. The developers succeeding with local agents aren’t waiting for better models—they’re writing better error handling, tighter schemas, and more forgiving tool interfaces. The model is the least interesting part of the stack.