Local Function Calling and Structured Output: Making Small Models Reliable

When a frontier cloud model returns JSON, it almost always parses. When a 7B local model returns JSON, it sometimes adds a stray comma, drops a closing brace, or wraps the whole thing in apologetic prose. The model is not 'wrong'; it is undertrained for that exact format. The fix is constrained decoding — telling the inference engine to only allow tokens that keep the output valid.

Three levels of structured output

1. 1Hope: prompt the model nicely and parse the result. Fast to implement, brittle in production
2. 2JSON mode: ask Ollama to enforce JSON-shaped output. Better — but no schema constraint
3. 3Grammar / schema: feed a JSON schema or a GBNF grammar to the engine. Output is mathematically valid by construction

import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

schema = {
    "type": "object",
    "properties": {
        "intent": {"type": "string", "enum": ["book", "cancel", "reschedule"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
    },
    "required": ["intent", "confidence"],
}

resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": user_text}],
    response_format={"type": "json_schema", "json_schema": {"name": "intent", "schema": schema}},
)
result = json.loads(resp.choices[0].message.content)

Function calling on small models

Modern Ollama supports OpenAI-style tool calls — same shape as the cloud APIs. Many open-weight models (Llama 3.1+, Qwen 2.5+, Mistral) are tuned for tool use and work well. Older or off-brand models will hallucinate the wire format. Always test the specific model you plan to ship; do not assume that 'tool calling works' generalizes.

Apply this

1. 1Pick a real classification task in your work and write a JSON schema for the output
2. 2Implement it on Ollama with response_format JSON schema
3. 3Compare reliability against the same task with no schema constraint, on 50 inputs

The big idea: small local models become reliable when you constrain their output. The cloud's apparent reliability is partly that the engine already does this for you.