Agent Safety, Guardrails & Governance

What this course covers and why it starts here

This course teaches you to find the risks in an AI agent and to contain them, from threat modeling to tool permissions. This first lesson answers one question: what makes an agent different from the software you already run?

Illustrative example, used throughout: a freight company has an assistant that answers staff questions about shipment status. It reads documents and writes answers, and a person decides what to do next. Now the team wants it to handle late shipments from start to finish. The new version can query carriers, rebook loads, update the warehouse system and email customers. A person does not check each step.

The model behind both versions is the same. What changes, and what this lesson is about, is what the system can do on its own.

What counts as an agent

An AI agent is software that uses a language model to pursue a goal. It picks its own actions, runs them through tools, and decides the next step from the result. Three simpler systems help draw the line.

• A chatbot answers one message at a time. A person reads the answer and acts on it.
• A fixed pipeline runs steps in an order a developer wrote. It may call a model at each step, but the code sets the order.
• An automation script runs set logic. The same input gives the same result.

A useful test is to ask who controls the loop. In a chatbot, the user does. In a pipeline, the developer does. In an agent, the model does: it chooses what to do next while it runs.

The line is not perfectly sharp. Some search systems rewrite their own queries and try again, and practitioners disagree on whether those are agents. For safety work, ask instead: can the system take actions that change things, without a person choosing each one? If yes, give it the care this course describes.

Persistent state: what the agent carries forward

Persistent state means the agent keeps memory and context from one step to the next. Ordinary software has state too, such as database rows. The difference is that an agent reasons over its state to choose actions, in ways no developer wrote down.

In the freight example, the agent sees one carrier return errors three times in a week. It stores a note that the carrier is unreliable. Next week it starts to avoid that carrier on its own. Maybe the errors came from a short outage, or the agent misread an error code. Either way, the stored belief keeps steering decisions until someone looks at it.

Two risks follow from this.

• A wrong belief compounds. Each action based on it can create evidence that seems to confirm it.
• A planted instruction can persist. A document the agent reads may hide text that tells it what to do. That text can stay in memory and shape later steps. This attack is called prompt injection, and chapter 4 covers it.

The safety question for state: can you see what the agent remembers, and can you reset it?

Tool access: the agent can act, not just answer

Tool access means the agent can call outside systems. It can query an API, write to a database, send email or run code. When a model only writes text and a person reads it, that person stands between its output and the world. A model with tools acts directly.

When your billing service calls a payments API, a developer wrote that call, and it was reviewed and tested. An agent's tool calls are different. The model decides while it runs which tool to call, with which inputs, in which order. Those inputs can come from user messages, retrieved documents or stored memory.

Take a customer database. A service with write access writes where its code says to. An agent with write access writes when its reasoning says to. The permission you grant is the real limit on what it can do. Its judgment is not a security boundary.

This is why least privilege matters for agents. Least privilege means giving a system only the permissions its task needs. If the agent needs to read shipment records and nothing more, give it a read-only key. Then a bad decision cannot change a record, because the key does not allow it.

Self-directed action sequencing: the agent plans its own path

A scheduled job or a deployment pipeline runs steps in an order a person designed. An agent builds its control flow as it goes.

Give the freight agent the goal "resolve this delayed shipment." It might check the carrier first, or the warehouse. It might rebook, wait or hand the case to a person. Each choice depends on what the last step returned.

The number of possible paths grows fast. Suppose an agent has 8 tools and may take up to 6 steps, and it can pick any tool at each step. Six-step runs alone give 8 × 8 × 8 × 8 × 8 × 8, or 262,144, possible sequences. Real limits cut that number down, but it stays far too large to test one by one.

This also changes what a failure looks like. A bug is code doing something its author did not intend. An agent can follow its goal faithfully and still choose a path you did not expect.

Each property is manageable alone. Together they give you a system that forms beliefs from memory, acts on them through real tools, and chains actions in an order no person reviewed.

The autonomy spectrum

Agents differ in how much they do without a person. The stages below are a teaching scale for this course, not an industry standard.

Stage	What the system does	Role of the person
0. Suggests	Proposes actions; a person carries them out	Does every action
1. Asks first	Takes actions, each one after approval	Approves each action
2. Asks for big ones	Takes routine actions alone; asks before high-impact ones	Approves the exceptions
3. Reports after	Runs long chains of actions; a person reviews results later	Reviews outcomes
4. Runs alone	Operates for long periods and handles its own exceptions	Reviews after the fact, if at all

As you move down the table, one run can chain more actions before a person sees any of them. A bad run keeps acting until someone steps in.

Watch for autonomy creep. A team builds a stage 2 system, then sets "routine" actions to approve themselves. Over months, the list of routine actions grows. On paper the system is stage 2. In practice it runs at stage 3.

Three questions place a system honestly:

1. Can it take any action without a person approving it?
2. Can it chain several actions, including high-impact ones, before a person reviews the outcome?
3. Can its stored state change how it behaves, without a person reviewing that state?

If all three answers are yes, the system is at stage 3 or above, whatever the design document says.

One caution: autonomy stage and risk are separate. A stage 1 agent with write access to a large patient database can carry more risk than a stage 3 agent that only reads public web pages. Rate both how much it does alone and what its tools can reach.

Why classic software safety practice falls short

Your current safety toolkit may include tests, code review, access control and monitoring. These practices rest on three assumptions that agents break.

• Same input, same output. Tests check that known inputs give known results. An agent can take different paths on the same input. Your tests cover the paths you thought of.
• A person wrote the control flow. Code review works because a developer wrote the logic and a reviewer can read it. An agent decides its control flow while it runs.
• Code limits behavior. Access control assumes a service does what its code says and no more. An agent uses its permissions in whatever way its reasoning finds useful for the goal.

The old practices are not wrong, just incomplete for agents. Keep them, and add controls on permissions and actions. Check every tool-call input in code before it runs. Test with hostile documents and failing tools, not only common cases.

A common misconception shows the gap. A team writes a rule into the system prompt, such as "do not email anyone outside the company," and counts it as a guardrail. But that rule is an instruction to the same model it is meant to limit. A prompt injection or an odd chain of reasoning can override it. A real guardrail sits outside the model. Here, the email tool itself would accept approved recipient domains and refuse the rest, whatever the model asks for.

Four controls to start with

Later chapters cover each in depth.

1. Least privilege (covers tool access). List every tool the agent can call and the permission behind it. Cut each to what the task needs, and remove tools it does not use.
2. Human approval for irreversible actions (covers action sequencing). An action is irreversible if you cannot cleanly undo it. Examples are a payment sent, an email delivered, a record deleted or a contract signed. Require a person to approve these before they run. Show the reviewer the exact action, such as the recipient, amount or record, taken from the tool call itself rather than the agent's summary.
3. Step limits (covers action sequencing). Cap how many actions one run may take. Where it fits, also cap spend and rate, such as a ceiling on rebookings per hour. Enforce the cap in the code that runs the agent, not in the prompt. When a run reaches it, the run stops and a person takes over. Give a person a way to stop a running agent at any time, known as a kill switch.
4. Logging (covers state and sequencing). Record each input, tool call, result and change to memory, with times. Logs let you rebuild what it did and which inputs led to it. Store them where the agent cannot edit or delete them, and protect them, since they hold customer data.

Applied to the freight agent, the plan might look like this. It gets read access to shipment and carrier data. It can draft rebookings, but a dispatcher approves each one before it goes to a carrier. Customer emails use approved templates, and the email tool sends only to the customer contact on file for that shipment. That recipient rule, enforced in the tool, replaces approval of each email. Each run stops after 10 tool calls. Every call is logged. Its stored notes about carriers are reviewed each week and cleared if wrong.

These controls do not depend on the agent judging well. They still hold when its judgment fails.

What to do this week

Pick one agent your team runs or plans to run, and work through this list.

• Place it on the autonomy spectrum, using the three questions above.
• List its tools and the permission each one uses. Mark any that are broader than the task needs.
• Mark every action it can take that you cannot undo. Note whether a person approves it first.
• Check whether a single run has a step limit.
• Pull one day of logs. See whether you can rebuild what the agent did and why.

The gaps you find are your starting list. The next lessons in chapter 1 sort agent failures into kinds of harm. They explain why safety work put off until later is harder to add. Then they set out the habit of starting from minimal access.