Valid JSON is not permission to act

Consider this hypothetical payload. It is valid:

{
  "action": "issue_refund",
  "orderId": "ord_1842",
  "amountMinor": 4999,
  "currency": "USD",
  "reason": "Customer requested a full refund"
}

It can satisfy a strict schema. Every required field is present. The action comes from an allowed enum. The amount is a positive integer. No unexpected properties slipped through.

It can still be wrong in every way that matters. The order may belong to another tenant. The original charge may have been smaller. The customer may have asked about a refund without approving one. The refund may already exist. The operator behind the request may not have authority to issue it.

Structured output makes the payload easier to parse. It does not make the claim true, the action permitted, or the write safe.

That distinction is the missing boundary in many connected agent workflows. Teams add JSON mode or a tool schema, see reliable objects arrive, and wire those objects directly into an API. The format problem is solved, so the result feels validated. In production, schema validation should be the first rejection point, not the last approval.

A schema answers a narrow question

JSON Schema is good at describing the shape of a document. It can require properties, restrict strings to enumerated values, apply numeric ranges, and reject undeclared fields with additionalProperties: false JSON Schema object reference. That is real protection. It prevents a large class of malformed or surprising payloads from reaching application code.

Model APIs can also constrain generation to that shape. OpenAI distinguishes JSON mode, which produces valid JSON, from Structured Outputs, which provides schema adherence on supported models and schemas. The same guide warns that structured outputs can still contain mistakes OpenAI Structured model outputs.

The schema cannot answer questions that live outside the document:

Question	Where the answer lives
Is this an integer in the allowed range?	Schema validator
Does this order exist in this tenant?	Application data
Does the amount match the refundable balance?	Business rules and current state
May this actor issue the refund?	Authorization policy
Has this logical refund already run?	Execution and deduplication state
Does the evidence support the reason?	Source review or a human decision

A schema can say that orderId must be a string. It cannot prove that the string identifies the right order. It can cap amountMinor at a configured maximum, but it cannot know whether the remaining refundable amount changed a moment ago. Even conditional schemas operate on the instance being validated; they do not replace a policy engine or a transaction against live state.

This is not an argument against structured output. It is an argument for giving it the job it can actually do.

Make the model propose; let the application dispose

The clean boundary is a proposal envelope. The model may choose an action and supply candidate arguments. Conventional code attaches trusted context, checks the proposal, and either rejects it or turns it into a command.

{
  "proposal": {
    "action": "issue_refund",
    "orderId": "ord_1842",
    "amountMinor": 4999,
    "currency": "USD",
    "reason": "Customer requested a full refund",
    "evidenceRefs": ["ticket_message_73"]
  },
  "trustedContext": {
    "actorId": "usr_921",
    "tenantId": "tenant_44",
    "workflowId": "support_refund_v3",
    "requestId": "req_b7f..."
  }
}

The model should not generate actorId, tenantId, approval state, credential scope, or policy version. Those values come from the authenticated session and the workflow runtime. If the model is allowed to declare its own authority, the application has confused a claim with a fact.

Run the assembled envelope through separate checks:

Shape. Validate the model-controlled proposal against a closed schema. Reject missing fields, unknown actions, unbounded strings, invalid currency codes, and additional properties. Do not quietly coerce a malformed value into something executable.

Meaning. Resolve identifiers inside the authenticated tenant. Load the order and ticket. Confirm the currency, refundable balance, current status, and evidence reference. This is also where stale proposals fail: if the order changed after the model formed its answer, revalidation should catch it.

Authority. Evaluate the authenticated actor, workflow, action, target, and amount against policy. The model can recommend issue_refund; it cannot grant the permission to do it. OWASP’s excessive-agency guidance calls for authorization in downstream systems and complete mediation of requests, rather than relying on an LLM to decide what is allowed OWASP LLM06:2025.

Execution safety. Require confirmation when the action’s impact warrants it, assign an idempotency key to the logical action, execute with the narrow credential for this run, and write an evidence record. A valid, authorized command can still duplicate a payment or message if retry behavior is unsafe; see assume your agent will run the workflow twice for that failure mode.

Keep the failures distinct. A schema error means the proposal is malformed. A business-rule error means the proposal no longer fits reality. An authorization error means the caller cannot take the action. A confirmation requirement means the action is possible but not yet approved. Collapsing all of those into “tool failed” makes both model recovery and incident review harder.

Tool schemas belong at both sides of the call

This boundary matters whether the model emits a response object or invokes a tool.

The current Model Context Protocol tools specification defines inputSchema for expected tool parameters and an optional outputSchema for structured results. It also says MCP tools are model-controlled, servers must validate tool inputs and implement access controls, and clients should request confirmation for sensitive operations MCP tools specification.

That division of responsibility is useful:

• The client can constrain and validate what the model asks to send.
• The tool server must validate again at the enforcement boundary.
• The downstream system must authorize against trusted identity and current policy.
• The client should validate structured results before using them in another model turn or application step.

Do not assume that an MCP tool annotation such as “read only” or “non-destructive” is a security control. The specification describes tool annotations as untrusted unless they come from trusted servers. Treat them as interface hints; enforce actual behavior with credentials, network access, server code, and policy.

Output validation deserves equal attention. A tool result may become the next model input, a rendered page, a file path, or an argument to another tool. OWASP’s improper-output guidance points to the consequences of passing model output directly into shells, SQL, file paths, or rendered content OWASP LLM05:2025. Validate and contextually encode at every boundary instead of trusting data because an earlier component called it structured.

Prompt injection makes the boundary non-optional

Suppose the refund proposal came from an agent reading a support ticket. The ticket is data from an external party, but it shares the model context with workflow instructions. The UK NCSC’s analysis is blunt: current LLMs do not provide a dependable security boundary between instructions and data inside a prompt Prompt injection is not SQL injection.

A strict schema can force an injected response into an allowed shape. That helps, but it can also produce a perfectly shaped attack:

{
  "action": "issue_refund",
  "orderId": "ord_attacker_controlled",
  "amountMinor": 4999,
  "currency": "USD",
  "reason": "Customer requested a full refund"
}

Nothing about prompt injection requires malformed JSON. The dangerous result is often syntactically excellent.

The enforcement layer limits what that result can do. Tenant-bound lookup rejects the foreign order. The live balance check rejects an excessive amount. Authorization rejects an actor without refund permission. Confirmation can stop a sensitive action even after the other checks pass. Narrow credentials limit the damage if an implementation bug remains.

This is why prompt wording is not the control plane. Prompts can improve behavior and reduce noise. Security decisions belong in code and policy outside the model.

Where the pattern earns its cost

Use this separation when model output can cause a durable or externally visible effect: payments, messages, account changes, deployments, record deletion, access grants, or writes into operational systems. It is especially useful when an agent reads untrusted content, works across tenants, or chains several tools.

It is usually too much machinery for a draft-only workflow. If a model produces a private summary that a person reads and discards, schema validation may be enough—or plain text may be better. A deterministic parser is not automatically valuable just because one is available.

There are also cases where an agent should not own the decision at all. A fixed rules engine is easier to test for stable, high-volume decisions. A conventional form may be better when the user already knows the required fields. A human approval queue is appropriate when the evidence is ambiguous and the cost of a wrong action is high. Separating proposal from execution makes those alternatives easier to swap in.

The costs are ordinary engineering costs: more application code, policy maintenance, state reads, error handling, tests, and latency. The benefit is not that the model becomes correct. The benefit is that correctness and authority stop depending on the model.

Review the boundary when the workflow changes

Revisit the proposal schema and enforcement path when you add an action, widen a credential, change a business rule, introduce a new tenant boundary, replace a model, update the tool protocol, or let the workflow read a new untrusted source. OpenAI recommends evals when designing structured-output schemas; use them to test representative valid proposals, malformed responses, stale state, cross-tenant identifiers, prompt-injected inputs, and refusals.

Also review what gets logged. Proposal payloads and rejection reasons may contain customer text, account identifiers, or operational details. Record enough to reconstruct the decision without copying more sensitive data than the audit needs. The broader pattern is covered in agent workflows need an evidence trail.

Structured output is an interface contract. Keep it. Make it strict. Then put a real decision boundary behind it.