What an LLM guardrail is
An LLM guardrail is code that sits at a boundary of a model call (before the prompt goes in, after the answer comes out, or around a tool the agent wants to use) and decides whether that step may go ahead. It can pass the step, block it, rewrite it, or hand it to a person for review.
The important word is outside. A system prompt is a request the model may or may not honour: it shares one token stream with the user's message, retrieved documents and tool output, and nothing in the architecture gives it the final say (see System Prompt Violations in Production Agents). A guardrail is enforced by your code, so when it says no, the step does not happen.
That is also its limit. A guardrail can only enforce a rule someone turned into a check before the run started. Everything an agent can get wrong that nobody wrote down passes straight through.
Where guardrails sit in an agent
A chat completion has two boundaries: the input and the output. An agent has many more, because every tool call, retrieval and handoff is a place where text from outside enters the context or an action leaves it. The common layers:
| Layer | Runs on | Typical checks | Blind spot |
|---|---|---|---|
| Input | The user's message, before the main model runs | Off-topic or disallowed requests, jailbreak and prompt-injection patterns, PII to mask | Instructions that arrive later, inside a tool result or retrieved document |
| Retrieval | Chunks pulled from a knowledge base before they enter the prompt | Untrusted sources, injected instructions, sensitive records | Whether the agent then uses the right chunk correctly |
| Tool / execution | Each tool call, before or after it runs | Argument schema, allowlisted actions, amount or scope limits, approval for irreversible actions | Arguments that are well formed but wrong, such as a plausible invented ID |
| Output | The final answer, before it reaches the user or the next system | Schema validity, toxicity, leaked secrets or PII, banned claims, grounding against sources | Steps the agent skipped, which leave nothing in the output to check |
| Dialog / flow | The conversation's path between turns | Keeping the agent on scripted flows for sensitive topics | Drift that stays inside allowed topics but breaks the operator's rules |
Frameworks name these layers differently. The OpenAI Agents SDK has input, output and tool guardrails; NVIDIA's NeMo Guardrails library has input, output, retrieval, dialog and execution rails. The shape is the same: a check at each place something enters or leaves the agent.
How guardrails decide
Every guardrail is one of three kinds of check, and most production setups mix them.
- Deterministic rules. Regular expressions, JSON schemas, allowlists, numeric limits, lookups against a system of record. Fast, cheap and predictable, and they only catch what the rule describes. A rule that a refund must be under a set amount says nothing about whether the order was eligible at all.
- Classifiers. A small model trained to label text: unsafe content, prompt injection, PII, topic. Meta's Llama Guard models, for example, classify prompts and responses against a safety taxonomy. They generalise beyond exact patterns but return a probability, so you choose a threshold and live with its false positives and misses.
- LLM-as-judge. A second model call asked whether the input or output breaks a stated rule. It is the most flexible kind and can read rules written in plain language, at the cost of latency, money, and a judge that can be wrong in the same confident way as the agent.
Timing matters as much as method. A blocking guardrail runs first, so a failed check means the main model or tool never starts. A guardrail that runs in parallel saves latency, but the work it was meant to stop may already have begun by the time it fires; the OpenAI Agents SDK documentation spells out this trade-off. For any tool call that moves money, deletes data or contacts a customer, run the check before the action, not alongside it.
Guardrail libraries and SDKs to know
These are the options teams most often start from. Features change quickly, so check each project's own documentation for what it supports today and how it is licensed.
- OpenAI Agents SDK guardrails. Input guardrails on the first user input, output guardrails on the agent's final output, and tool guardrails around function-tool calls. A guardrail that trips raises an exception and halts the run. Docs: openai.github.io/openai-agents-python/guardrails.
- NVIDIA NeMo Guardrails. An open-source library where rails are configured in YAML and Colang and applied at the input, output, retrieval, dialog and execution stages. Docs: docs.nvidia.com/nemo/guardrails.
- Guardrails AI. An open-source Python framework built around validators, each of which checks a value and says what to do when it fails, with community validators published on the Guardrails Hub. Docs: guardrailsai.com/docs.
- Llama Guard. Meta's safeguard models for classifying whether a prompt or response is unsafe, usable as the classifier inside any of the frameworks above. Model cards: the meta-llama organisation on Hugging Face.
- Your own code. Many of the most valuable guardrails are a few lines in the tool itself: re-read the record before writing to it, refuse an action outside a date window, require a human approval flag. Nothing about a guardrail needs a framework.
What guardrails don't catch
Guardrails are good at the loud failures: an abusive prompt, a leaked card number, a malformed tool call. The failures this site covers are the quiet ones, where every check passes because the output is well formed, safe and plausible, and still wrong.
- Policy applied to the wrong case. The refund is under the limit and in the right format, but the order was outside the return window. Agents also find other tool paths to the same outcome. See Detecting Policy-Window Violations in Agent Refund and Compensation Flows.
- Invented identifiers. An order ID that matches the schema and the regex, and does not exist, or belongs to someone else. See Fabricated Identifier Failures in Agentic Pipelines.
- Skipped steps. An output guardrail can only judge what was produced. If the agent never ran the verification step it was told to run, there is nothing on the boundary to inspect. See Instruction-Layer Signatures That Distinguish a Hallucinated Step From a Silently Skipped One.
- Drift over a long session. Rules stated once at the top of the context lose their hold as the session grows and gets compacted, and each individual turn still looks acceptable. See Instruction Drift in Long Agent Sessions.
- Unwritten rules. A guardrail encodes an explicit constraint. Many of the rules an agent breaks were implicit, so nobody wrote a check for them. See Implicit Versus Explicit Constraints in Agent Instructions.
- Failures between agents. Each agent's guardrails pass while the context handed from one to the next quietly loses what mattered. See Silent Failures in Multi-Agent Handoff Chains.
Guardrails and monitoring do different jobs
A guardrail asks one question about one step, in real time: may this go ahead? Monitoring asks a question about the whole run, afterwards: did the agent do what its instructions said? You need both, and neither replaces the other.
| Guardrails | Instruction-aware monitoring | |
|---|---|---|
| When | During the run, at each boundary | After the run, over the full trace |
| Scope | One input, output or tool call | The sequence of steps and the decisions between them |
| Rules | Known in advance and written as checks | The agent's own instructions, including rules nobody turned into a check |
| On failure | Blocks, rewrites or escalates the step | Groups recurring failures, alerts the team, feeds new checks and evals |
| Misses | Plausible wrong answers, skipped steps, drift | Nothing in real time: by design, it finds the failure after it happened |
The two form a loop. Monitoring that audits traces against the agent's instructions finds the failure patterns guardrails let through; the ones that can be expressed as a rule become new guardrails or regression evals, and the rest stay under watch. Monitoring Contracts That Encode Agent Intent, Not Just Agent Execution covers how to write those checks so they test intent rather than execution.
A checklist for adding guardrails to an agent
- List the rules in the system prompt. Every "never", "only" and "always" is a candidate guardrail. Add the implicit ones the team assumes but never wrote down.
- Sort them by cost of failure. Anything irreversible (payments, deletions, outbound messages, record changes) gets a blocking check on the tool call, not only on the final answer.
- Put each check at the layer where the evidence exists. Eligibility belongs in the tool, against the system of record; tone and leaked data belong on the output; injected instructions belong on input and retrieval.
- Prefer deterministic checks where a rule is exact. Use a classifier or a judge model only where the rule needs judgement, and record its threshold.
- Decide the action for each failure. Block, retry with feedback, rewrite, or route to a person. A guardrail that only logs is monitoring, and should be labelled that way.
- Log every trigger with the run it belongs to. A guardrail that fires is a signal about your instructions as much as about the input.
- Test each guardrail against real failures. Replay traces where the agent went wrong and confirm the check catches them, and replay good traces to measure false positives.
- Review what passed. Sample runs where every guardrail passed and read them against the instructions. That is where the failures no check describes will show up.
Questions
Are guardrails the same as a system prompt?
No. A system prompt tells the model what to do and the model may not comply. A guardrail is a check in your code that runs outside the model and can stop a step whatever the model decided.
Do guardrails stop prompt injection?
They reduce it. Input and retrieval guardrails catch known injection patterns, and tool guardrails limit what an injected instruction can make the agent do. None of them can recognise every instruction hidden in text the agent reads, so limit what tools can do on their own as well.
Are guardrails the same as evals?
No. Evals score an agent's behaviour on a set of cases, usually offline or on sampled traffic, to measure quality and catch regressions. Guardrails act on live runs, one step at a time. A failure found by monitoring often becomes both: a guardrail for the step and an eval case so it stays fixed.
What does an "LLM with no guardrails" mean?
Usually a model deployed without application-level checks around it: no input filtering, output validation or tool limits beyond what the model learned in training. Model providers train safety behaviour into their models, but that is not a substitute for checks on what your agent is allowed to do.
Do guardrails slow an agent down?
Deterministic checks add little. Classifier and judge-model guardrails add a model call each, which is why some frameworks let them run in parallel with the main model. Running in parallel trades safety for speed: the step may already be under way when the check fails.
Further reading
- System Prompt Violations in Production Agents
Agents can ignore their instructions while every dashboard reports success.
- Out-of-Scope Actions in Autonomous Agents
Agents can finish jobs flawlessly while crossing boundaries nobody checks.
- Calibrating Output Contracts to Catch Confident Incorrectness at the Instruction Layer
Instruction-layer contracts catch wrong answers that look right.
- Which Instruction-Layer Signals Separate Platforms Built for Agent Observability From Those Retrofitted for It
Purpose-built platforms store and evaluate instructions; retrofitted ones only log execution traces.