What is AI agent governance?
AI agent governance is the set of policies, controls, and audit mechanisms that determine what AI agents are allowed to do, when they require human approval, how their costs are managed, and how their actions are recorded for review and compliance.
It answers four questions:
- What can this agent access and modify?
- When does it need a human to approve before acting?
- How much can it spend (tokens, API calls, compute)?
- Who knows what it did, and when?
Governance is not about limiting what agents can do. It's about ensuring that what they do is intentional, safe, and accountable — so teams can give agents more autonomy over time with confidence, rather than pulling back after an incident.
Why governance matters now
Most teams that run AI agents start without governance. The prototype works; the agent is helpful; costs are low. Then one of these happens:
- An agent makes an unexpected API call that modifies production data
- Token costs appear on the billing dashboard three times higher than expected
- A security audit asks: "what did your AI agents do in production last quarter?"
- An agent loops unexpectedly and consumes $2,000 in a single session
These are not hypothetical. They are the actual incidents that prompt teams to build governance retroactively — under pressure, after damage. Building governance before the first production agent run is substantially cheaper.
The five pillars of AI agent governance
Access control (tool policy)
Define which tools and APIs each agent is allowed to call. Separate permissions into allow (no approval needed), review (requires human approval before execution), and block (never permitted). Apply the principle of least privilege: agents get access to exactly what they need for the current task, not permanent broad access.
Approval workflows
For actions that touch sensitive targets — production databases, billing APIs, credential stores, external communications — require a human to explicitly approve before execution. Approval gates should be fast (seconds, not hours) so they don't become the bottleneck, but they must be mandatory for defined action categories.
Token and cost budgets
Set per-session and per-agent spending limits. Implement circuit breakers that pause execution when a session exceeds expected token consumption, pending human review. Without budgets, a single misbehaving agent session can generate thousands of dollars in API costs before anyone notices.
Audit trail
Log every agent action: which tool was called, with what parameters, what the result was, whether it was approved or blocked, and who (human or system) triggered the original request. Logs must be immutable, queryable, and retained for compliance periods. The audit trail is what turns 'something went wrong' into 'here is exactly what happened and why'.
Separation of duties
High-risk actions should require separation — the entity that requests an action cannot also be the one that approves it. An agent that writes its own approval is not governed. Separation of duties applies to both human-agent and agent-agent workflows.
Governance vs the harness
Governance is the policy layer — the rules about what agents can do. The harness is the technical layer that enforces those rules. They are complementary:
| Layer | What it contains | Who owns it |
|---|---|---|
| Governance (policy) | Rules: what agents can do, approval thresholds, cost limits, retention requirements | Security, compliance, and engineering leadership |
| Harness (enforcement) | Technical controls: linters, policy engine, approval gates, budget tracking, audit logging | Platform engineering |
| Agent (execution) | Tool calls, reasoning, task completion | AI model + runtime framework |
Frequently asked questions
Does governance slow AI agents down?
Badly designed governance does. Well-designed governance adds milliseconds for policy evaluation and seconds for human approval on high-risk actions — with everything else flowing automatically. The goal is fast approval workflows for the actions that need them, and zero friction for actions that don't.
How is AI agent governance related to AI safety?
They overlap but are different in scope. AI safety research focuses on alignment and the long-term behaviour of advanced AI systems. AI agent governance is an engineering practice for controlling the behaviour of production AI agents today — access control, audit trails, cost management. Governance is tractable and implementable now; it doesn't require solving alignment.
Is AI agent governance required for SOC 2 compliance?
If AI agents take actions that affect in-scope production systems, then yes — those actions need to be auditable, access-controlled, and change-managed as part of your SOC 2 program. The specific controls depend on the trust services criteria you're targeting, but CC6 (logical access) and CC7 (system operations) are typically relevant.
Who is responsible for AI agent governance?
In most organisations: platform engineering implements the technical controls, security and compliance set the policies, and product/engineering leadership defines acceptable use. It requires cross-functional alignment — governance that engineering ignores because it's too slow is effectively no governance at all.
Related: the harness engineering checklist, cutting AI agent token costs, agents, context, and guardrails.