Probabilistic Systems: Why LLMs Do Not Act Like Code

An LLM does not return an answer. It returns a probability distribution over possible next tokens. Sampling from that distribution produces text. Even identical prompts will produce different text unless you control the randomness explicitly.

The sampling knobs

Why this breaks classical software instincts

• Unit tests that check exact output strings are brittle
• Caching by prompt hash works only at temperature 0 and even then not perfectly
• Error handling must account for plausible wrong answers, not just failures
• Rate limits and costs scale with token counts, which vary per call

# Robust evaluation of a probabilistic system
import statistics
import anthropic

client = anthropic.Anthropic()

def evaluate(prompt, n=10):
    outputs = []
    for _ in range(n):
        resp = client.messages.create(
            model="claude-opus-4-7",
            max_tokens=200,
            messages=[{"role": "user", "content": prompt}],
        )
        outputs.append(resp.content[0].text)
    # Measure pass@k, agreement, or similar
    return outputs

results = evaluate("Classify: 'great product'", n=10)
print(f"Unique outputs: {len(set(results))}")

Strategies for taming randomness

1. 1Constrain output format (JSON schema, regex, tool calls)
2. 2Use structured generation libraries like Outlines, Instructor, or native JSON mode
3. 3Ensemble multiple samples and majority-vote
4. 4Use a cheaper verifier model to grade an expensive generator
5. 5Cache known-good outputs and fall through to LLM only on cache miss

Testing probabilistic systems

• Behavioral tests: semantic properties (answer contains X, matches regex, passes a grader)
• Distributional tests: 95 percent of samples must score above threshold
• A/B tests in production with guardrails
• Canary prompts that track drift between model versions

“An LLM in production is a distribution, not a function. Design accordingly.”

The big idea: LLMs are stochastic systems. The tooling, testing, and architecture patterns that assume deterministic code need to be re-thought from the ground up when you put one in the loop.