Hermes For Structured JSON Output: Schemas That Work

Asking a model for JSON is easy. Asking it for a JSON object that always matches your schema is hard. Frontier API models offer schema-strict modes; open-weight models often need help. Hermes is responsive to good instructions, and when paired with grammar-constrained decoding (available in llama.cpp / Ollama), it can be very reliable.

Schemas that work in practice

1. 1Keep the schema flat where possible — fewer levels of nesting means fewer chances for the model to drop a brace.
2. 2Use enum lists for categorical fields — 'category' should be one of a fixed list, not free text.
3. 3Always include an 'id' field that echoes input — easier to map outputs back.
4. 4Add a 'confidence' field when you can — useful for routing low-confidence cases to a human.
5. 5Provide ONE example of the exact output you want, formatted as the schema, in the prompt.

Prompt skeleton:

SYSTEM: You will receive an input. Return ONLY a JSON object
matching this schema. Do not add commentary, do not wrap in code fences:

{
  "id": string,                   // echo input id
  "category": one of ["a","b","c"],
  "summary": string (max 30 words),
  "confidence": number 0.0-1.0
}

Example output for an input id="x1":
{"id":"x1","category":"b","summary":"...","confidence":0.78}

Now process the input below.

Grammar-constrained decoding

llama.cpp supports a grammar feature that physically prevents the model from emitting tokens that violate a JSON schema. When available, it is the strongest reliability tool in your kit — schema violations become impossible, not unlikely. Both Ollama and LM Studio expose access to this feature.

Applied exercise

1. 1Pick a real classification or extraction task you do.
2. 2Define a flat JSON schema for the output.
3. 3Prompt Hermes with the skeleton above and run on 25 inputs.
4. 4Compute schema-failure rate. If above 5%, try grammar-constrained decoding and recompute.

The big idea: structured output from open-weight models is solvable. Use grammar constraints when you can, validate always, and never trust the model to remember the schema mid-stream.