Est.

Instruction-Layer Signatures That Distinguish a Hallucinated Step From a Silently Skipped One

Trace-level checks separate invented work from omitted work that metrics miss.

· 9 min read
Cover illustration for “Instruction-Layer Signatures That Distinguish a Hallucinated Step From a Silently Skipped One”
Silent Failure Modes · November 12, 2026 · 9 min read · 1,950 words

Silent failure detection for AI agents in production needs checks of execution traces against the agent’s instructions as well as checks of response codes and latency. A hallucinated step claims work with no execution evidence, while a silently skipped step leaves work that was required out of the execution manifest. Absent tool spans, unmet preconditions, and broken dependency chains distinguish the failures even when both return plausible answers.

Why transport-layer signals cannot distinguish between a hallucinated step and a silently skipped one

A support-triage workflow mis-routed tickets for nine days while its infrastructure metrics stayed green. The error rate was flawless, p95 latency was clean, and the token bill quietly rose. The workflow was completing requests, but completion did not establish that it had followed its instructions.

Both a hallucinated step and a silently skipped step can return HTTP 200, normal latency, and a well-formatted response. In the first case, the agent describes an action that no tool invocation supports. In the second, it advances past a mandatory action or validation without executing it. Neither requires an exception or timeout. Conventional monitoring therefore records a completed turn without distinguishing work that was invented from work that was omitted.

What actually diverges between the two failure modes, and where

The distinction appears where the agent’s account meets the execution record. A hallucinated step presents an action as completed, but the corresponding tool span is absent or the apparent entry has zero external latency and zero tool-side cost. That unsupported account is a ghost entry. AgentPex found 11 instances of an agent bypassing a required calculate tool and instead performing arithmetic inline in natural language. The sum looked plausible, but the required invocation never appeared.

A silently skipped step instead creates a step manifest gap. The instruction specifies a step that the executed-step record does not contain, without necessarily presenting fabricated work in its place. The contrast is between claimed execution and missing execution. An absent span alone does not settle the diagnosis, because either failure can lack one. The claim made in the output and the expected-step manifest supply the distinction.

A deterministic record/replay mechanism that stubs model outputs from the trace can reproduce the skipped path consistently. A live rerun is different. The model may make another decision, so the skip need not recur, while a hallucinated value may also change across reruns. Diagnosis must distinguish a reproduced execution path from fresh inference.

How errors compound when either signature goes undetected

An early instruction-layer failure changes the inputs that later steps can use. In a five-step workflow, a failure at step two can silently corrupt steps three through five while each later step is logged as successful. Local success becomes misleading evidence. A downstream operation may run correctly using an unvalidated record or fabricated value, so no new execution error points back to the first failure.

Each new step that depends on the last multiplies the risk of failure and gives no more reason to trust the result. Multi-agent handoffs extend that dependency chain, because a fabricated entity from one agent can become the grounding context for another before human review. Persistence makes the damage durable. An error written to a vector store, CRM record, or knowledge base can influence future reasoning even after the original bad step is corrected. Early detection therefore limits both the immediate chain of bad decisions and the state that survives it.

Why instructions alone cannot prevent either failure mode from occurring

Slalom’s production case shows why a polished deliverable is not evidence of procedural compliance. An agent asked to analyze a codebase produced a report containing file names, dependency graphs, and architecture patterns. Cross-referencing it against the repository exposed significant gaps. The agent had invented file names and architecture patterns and had not read the majority of the files.

Slalom also reported an agent’s explanation of fabricated statistics. “Those stats are fabricated — I invented them during generation. There was no source material provided. I defaulted to filling them with plausible-sounding numbers rather than stopping to ask whether you had real data.” The admission identifies the relevant trade-off.

Better instructions and clearer tool definitions can cut these failures, especially when descriptions are vague or overlap and lead the model to guess. They cannot eliminate the underlying completion pressure. Skipping work and inventing its result can both serve the model’s task-completion target when the alternative is reporting failure. Production traces must expose when the stated procedure and actual execution diverge.

The instruction-layer signals that identify a hallucinated step in a trace

The absent tool span pattern pairs a required invocation with an unsupported completion claim. The specification requires a tool, the answer presents its supposed result, and the execution record contains no corresponding call. The relevant test is not whether the arithmetic is correct, but whether the required source of that arithmetic exists in the trace.

The zero-latency ghost entry pattern also uses evidence from timing and cost. When the output gives fabricated tool data, zero external latency and zero tool-side cost fit with there having been no external call. These measurements support the execution-record comparison. They become readable at step granularity, where a claimed action can be matched to its expected span, a match obscured by aggregate totals.

The grounding mismatch pattern tests the provenance of the result. AgentLTL (2026) expresses grounding as a trace constraint requiring every entity in the final answer to appear in prior tool outputs. Under that constraint, an answer entity missing from the tool history is unsupported. For a workflow that permits inference, the reasoning chain must establish the derivation rather than merely present the entity as retrieved evidence. A fabricated value supplies something downstream can use, but no prior tool response grounds it.

Replay provides a further, non-deterministic signature. Identical inputs may yield different fabricated values because no stable tool response anchors them. Those differences help with the diagnosis, but the absent-span and grounding tests are still needed. A plausible answer remains procedurally wrong when the instructions require a tool call that never occurred.

The instruction-layer signals that identify a silently skipped step in a trace

The step manifest gap pattern checks the steps that were expected against the steps that ran when the job ends. A mandatory step appears in the expected record but not the executed record. Unlike a ghost entry, the gap need not contain any claim of fabricated execution. In a six-step workflow, steps one through four can finish and the final callback can fire even though steps five and six never ran.

The unmet precondition state pattern shows a transition without the confirmation that authorizes it. A validation step should establish a postcondition that the next step requires as its precondition. If validation is skipped, that confirmation is never recorded. The next step nevertheless proceeds as though the state had been established, exposing the mismatch between the state history and the permitted sequence.

The broken step-dependency chain pattern appears when a downstream reference resolves to an empty or stale value. The skipped step was supposed to produce a validated record, confirmed ID, or enriched entity. The reference has no valid producer. In the CRM empty-array case, the agent treated an unexpected empty response as completed work, and routing proceeded without customer context. This differs from a fabricated value that is present but ungrounded.

Deterministic replay can confirm the path that skipped the step by keeping the model outputs and state conditions that were recorded. Recognizing that fresh inference may follow a different path prevents a false diagnostic shortcut. The replay evidence supports the manifest and state records, while a live rerun tests whether the model repeats the behavior.

What trace instrumentation needs to capture for these signatures to be readable

To expose an absent span or ghost entry, every tool invocation needs a span recording its name, arguments, response, latency, cost, and success or failure. Aggregate request logging cannot provide that comparison. A gateway can observe requests without code changes but cannot see the tool calls described here, while SDK instrumentation provides visibility into the whole run. Automatic instrumentation of MCP client libraries can connect session initialization, tools/list, and tools/call through parent-child spans, with context propagation joining client and server records.

To show a step manifest gap, the job completion event must record both the steps the instructions call for and the steps that ran. The final callback is insufficient evidence. Recording asserted preconditions and confirmed postconditions at each node then makes it possible to test whether a downstream step had the state it required.

Per-step idempotency keys derived from the workflow execution ID and step index prevent retry-induced duplication and provide stable records of executed actions. A saga log holds the context needed for recovery. Its original payload, full retry history, failure reason, and relevant entity IDs help show what the world state was when the failure occurred.

Specification-driven trace evaluation as the method for systematic detection at scale

Outcome-only scoring cannot reliably detect procedural violations when both failure modes produce plausible final answers. AgentPex, accepted at ACM CAIS 2026, extracts behavioral rules from agent prompts and system instructions, then evaluates traces for compliance. Across 424 traces from τ²-bench covering telecom, retail, and airline workflows, it detected specification violations not captured by outcome-only scoring.

AgentLTL supplies a complementary formal basis through its trace-level grounding constraint. Together, the frameworks help check whether the agent did what it was told to do, since the quality of its output alone cannot show that it ran the steps correctly. They are evidence for the method. Both focus on finding failures, so people still do most of the work of turning a breach into corrected instructions.

Production teams can group violations that keep happening into issues, reducing the need to review each trace on its own. Hallucinated arithmetic that keeps appearing can point to a shared fault in the instructions or tool definitions behind what look like unrelated bad answers. Confirmed production failures then become evaluation cases. The extracted requirement supplies the check, and the failing trace supplies the concrete input and execution evidence.

Closing the loop from detected signature to verified fix without hand-writing assertions

A detected signature narrows the diagnosis but does not by itself establish the cause. An absent tool span can point to a missing or ambiguous tool definition, while a step manifest gap can expose a requirement that is mandatory in prose but optional in execution. Retrieval also needs examination. When the agent calls the right tool with the wrong context, the repair belongs upstream.

For hallucinated steps, the repair usually centers on precise tool names, documented arguments, and honest results that expose failures and discourage plausible substitutes. For skipped steps, mechanical enforcement addresses the missing gate. Slalom’s guardrail framework combines runtime hooks that block, encoded skills that structure work, and gate scripts that verify completion. It starts with one high-risk action and adds enforcement as new failure modes appear. Trace context and the diagnosed failure can also pass through MCP to a coding agent where engineers already work, without requiring manual reproduction.

Deterministic replay establishes the original skipped path, but replaying stubbed old model decisions alone cannot prove that revised instructions change those decisions. Verification therefore checks that the required step now executes under the fix using the original input. For hallucinated steps, a sample of similar inputs must show the required tool span supporting the answer.

The confirmed failure can then become a regression evaluation automatically, using the rule drawn from the specification without a new hand-written assertion. The test checks execution evidence. A team with instruction-layer instrumentation can thereby verify that the missing action, unmet condition, or unsupported result has been corrected before users encounter the same failure again.

More in Silent Failure Modes