Est.

Guide · Instruction Adherence

LLM Guardrails for AI Agents: What They Block and What Gets Past Them

A guardrail is a check that runs outside the model and can stop a run. It enforces the rules you can write down in advance. It does not tell you whether the agent did the right thing.

By Instruction Surface Editors · Updated

What an LLM guardrail is

An LLM guardrail is code that sits at a boundary of a model call (before the prompt goes in, after the answer comes out, or around a tool the agent wants to use) and decides whether that step may go ahead. It can pass the step, block it, rewrite it, or hand it to a person for review.

The important word is outside. A system prompt is a request the model may or may not honour: it shares one token stream with the user's message, retrieved documents and tool output, and nothing in the architecture gives it the final say (see System Prompt Violations in Production Agents). A guardrail is enforced by your code, so when it says no, the step does not happen.

That is also its limit. A guardrail can only enforce a rule someone turned into a check before the run started. Everything an agent can get wrong that nobody wrote down passes straight through.

Where guardrails sit in an agent

A chat completion has two boundaries: the input and the output. An agent has many more, because every tool call, retrieval and handoff is a place where text from outside enters the context or an action leaves it. The common layers:

LayerRuns onTypical checksBlind spot
InputThe user's message, before the main model runsOff-topic or disallowed requests, jailbreak and prompt-injection patterns, PII to maskInstructions that arrive later, inside a tool result or retrieved document
RetrievalChunks pulled from a knowledge base before they enter the promptUntrusted sources, injected instructions, sensitive recordsWhether the agent then uses the right chunk correctly
Tool / executionEach tool call, before or after it runsArgument schema, allowlisted actions, amount or scope limits, approval for irreversible actionsArguments that are well formed but wrong, such as a plausible invented ID
OutputThe final answer, before it reaches the user or the next systemSchema validity, toxicity, leaked secrets or PII, banned claims, grounding against sourcesSteps the agent skipped, which leave nothing in the output to check
Dialog / flowThe conversation's path between turnsKeeping the agent on scripted flows for sensitive topicsDrift that stays inside allowed topics but breaks the operator's rules

Frameworks name these layers differently. The OpenAI Agents SDK has input, output and tool guardrails; NVIDIA's NeMo Guardrails library has input, output, retrieval, dialog and execution rails. The shape is the same: a check at each place something enters or leaves the agent.

How guardrails decide

Every guardrail is one of three kinds of check, and most production setups mix them.

  • Deterministic rules. Regular expressions, JSON schemas, allowlists, numeric limits, lookups against a system of record. Fast, cheap and predictable, and they only catch what the rule describes. A rule that a refund must be under a set amount says nothing about whether the order was eligible at all.
  • Classifiers. A small model trained to label text: unsafe content, prompt injection, PII, topic. Meta's Llama Guard models, for example, classify prompts and responses against a safety taxonomy. They generalise beyond exact patterns but return a probability, so you choose a threshold and live with its false positives and misses.
  • LLM-as-judge. A second model call asked whether the input or output breaks a stated rule. It is the most flexible kind and can read rules written in plain language, at the cost of latency, money, and a judge that can be wrong in the same confident way as the agent.

Timing matters as much as method. A blocking guardrail runs first, so a failed check means the main model or tool never starts. A guardrail that runs in parallel saves latency, but the work it was meant to stop may already have begun by the time it fires; the OpenAI Agents SDK documentation spells out this trade-off. For any tool call that moves money, deletes data or contacts a customer, run the check before the action, not alongside it.

Guardrail libraries and SDKs to know

These are the options teams most often start from. Features change quickly, so check each project's own documentation for what it supports today and how it is licensed.

  • OpenAI Agents SDK guardrails. Input guardrails on the first user input, output guardrails on the agent's final output, and tool guardrails around function-tool calls. A guardrail that trips raises an exception and halts the run. Docs: openai.github.io/openai-agents-python/guardrails.
  • NVIDIA NeMo Guardrails. An open-source library where rails are configured in YAML and Colang and applied at the input, output, retrieval, dialog and execution stages. Docs: docs.nvidia.com/nemo/guardrails.
  • Guardrails AI. An open-source Python framework built around validators, each of which checks a value and says what to do when it fails, with community validators published on the Guardrails Hub. Docs: guardrailsai.com/docs.
  • Llama Guard. Meta's safeguard models for classifying whether a prompt or response is unsafe, usable as the classifier inside any of the frameworks above. Model cards: the meta-llama organisation on Hugging Face.
  • Your own code. Many of the most valuable guardrails are a few lines in the tool itself: re-read the record before writing to it, refuse an action outside a date window, require a human approval flag. Nothing about a guardrail needs a framework.

What guardrails don't catch

Guardrails are good at the loud failures: an abusive prompt, a leaked card number, a malformed tool call. The failures this site covers are the quiet ones, where every check passes because the output is well formed, safe and plausible, and still wrong.

Guardrails and monitoring do different jobs

A guardrail asks one question about one step, in real time: may this go ahead? Monitoring asks a question about the whole run, afterwards: did the agent do what its instructions said? You need both, and neither replaces the other.

GuardrailsInstruction-aware monitoring
WhenDuring the run, at each boundaryAfter the run, over the full trace
ScopeOne input, output or tool callThe sequence of steps and the decisions between them
RulesKnown in advance and written as checksThe agent's own instructions, including rules nobody turned into a check
On failureBlocks, rewrites or escalates the stepGroups recurring failures, alerts the team, feeds new checks and evals
MissesPlausible wrong answers, skipped steps, driftNothing in real time: by design, it finds the failure after it happened

The two form a loop. Monitoring that audits traces against the agent's instructions finds the failure patterns guardrails let through; the ones that can be expressed as a rule become new guardrails or regression evals, and the rest stay under watch. Monitoring Contracts That Encode Agent Intent, Not Just Agent Execution covers how to write those checks so they test intent rather than execution.

A checklist for adding guardrails to an agent

  1. List the rules in the system prompt. Every "never", "only" and "always" is a candidate guardrail. Add the implicit ones the team assumes but never wrote down.
  2. Sort them by cost of failure. Anything irreversible (payments, deletions, outbound messages, record changes) gets a blocking check on the tool call, not only on the final answer.
  3. Put each check at the layer where the evidence exists. Eligibility belongs in the tool, against the system of record; tone and leaked data belong on the output; injected instructions belong on input and retrieval.
  4. Prefer deterministic checks where a rule is exact. Use a classifier or a judge model only where the rule needs judgement, and record its threshold.
  5. Decide the action for each failure. Block, retry with feedback, rewrite, or route to a person. A guardrail that only logs is monitoring, and should be labelled that way.
  6. Log every trigger with the run it belongs to. A guardrail that fires is a signal about your instructions as much as about the input.
  7. Test each guardrail against real failures. Replay traces where the agent went wrong and confirm the check catches them, and replay good traces to measure false positives.
  8. Review what passed. Sample runs where every guardrail passed and read them against the instructions. That is where the failures no check describes will show up.

Questions

Are guardrails the same as a system prompt?

No. A system prompt tells the model what to do and the model may not comply. A guardrail is a check in your code that runs outside the model and can stop a step whatever the model decided.

Do guardrails stop prompt injection?

They reduce it. Input and retrieval guardrails catch known injection patterns, and tool guardrails limit what an injected instruction can make the agent do. None of them can recognise every instruction hidden in text the agent reads, so limit what tools can do on their own as well.

Are guardrails the same as evals?

No. Evals score an agent's behaviour on a set of cases, usually offline or on sampled traffic, to measure quality and catch regressions. Guardrails act on live runs, one step at a time. A failure found by monitoring often becomes both: a guardrail for the step and an eval case so it stays fixed.

What does an "LLM with no guardrails" mean?

Usually a model deployed without application-level checks around it: no input filtering, output validation or tool limits beyond what the model learned in training. Model providers train safety behaviour into their models, but that is not a substitute for checks on what your agent is allowed to do.

Do guardrails slow an agent down?

Deterministic checks add little. Classifier and judge-model guardrails add a model call each, which is why some frameworks let them run in parallel with the main model. Running in parallel trades safety for speed: the step may already be under way when the check fails.

Further reading

Sources

  1. Guardrails - OpenAI Agents SDK
  2. Guardrail Types - NVIDIA NeMo Guardrails Library Developer Guide
  3. Validators - Guardrails AI
  4. guardrails-ai/guardrails on GitHub