One model. Identical weights, identical training, run inside two different agent systems. In one it scored 78%. In the other, 42%. Same brain. Nearly double the result.
That is Anthropic's CORE benchmark, published earlier this year and presented at the AI Engineer Summit, and it should be required reading for anyone deploying AI inside a bank, a fintech, or anywhere an examiner will eventually knock. If you only read model headlines, the number makes no sense. The model didn't change. Everything around the model did. And "everything around the model" turns out to be the part that decides whether you have a product or a science experiment.
Anthropic has a clean way of naming the two halves, and I've adopted it wholesale. The model is the brain. Everything else is the harness: where the agent runs, what it's allowed to touch, what it remembers between sessions, how it manages its own context, how it checks its work, and what happens when it fails. The brain gets the magazine covers. The harness decides whether the brain is useful or dangerous.
At SavvyMoney we made a decision that surprises people when I describe it. We built our own harness, and our own brain layer, rather than buying a finished agent off the shelf. We call it AgentOS.
It started as a memory problem. Every engineer on the team was using an AI coding agent, and every instance started from zero: no awareness of our conventions, our past decisions, the errors we'd already debugged, or what anyone else was working on. Brilliant, and amnesiac. AgentOS began as the shared brain that fixes that, a persistent, searchable, team-wide knowledge layer that every engineer's agent connects to, so one person's hard-won lesson becomes everyone's context and a session can be handed off with its full history intact. That alone changed how fast the team moves.
But the moment you give a fleet of agents a shared brain and the ability to take action, you've built something that, in a regulated company, you are absolutely going to have to answer for. That's where my other hat comes on.
In a regulated company, the harness is the governance
Every AI governance framework asks for the same things. NIST AI RMF (opens in new tab), ISO 42001 (opens in new tab), the EU AI Act (opens in new tab), the newer state automated-decision rules: know what your system can do, control what it can act on, prove a human is in the loop where it matters, keep evidence that's current rather than annual. Every one of those requirements is a property of the harness, not the model.
You cannot buy a model that is "compliant." Compliance isn't a property a brain has. It's a property of the scaffolding you wrap around the brain: the permissions, the audit trail, the boundary between an output the system may act on and one a human must interpret first. The harness is not an optimization layer that makes the agent a little faster; it is the control, and it is the thing the examiner is actually auditing.
That turned build-versus-buy into a question I already knew the answer to. Would I outsource my control environment to a vendor whose roadmap I don't own and whose philosophy I can't see? In a company that has to prove its security to 1,500+ financial institutions and their regulators, no. You don't rent your controls.
Most of a real agent isn't the model
Anthropic's own engineers keep repeating that the model call is a small fraction of a production agent. The rest is plumbing, the unglamorous infrastructure nobody demos: session state that survives a crash, a permission pipeline that classifies every action before it runs, a token budget checked before the expensive call rather than after, a verification step that confirms the agent actually did what it claimed, an event log that records what the system did and not just what it said.
That list is also, almost word for word, a control catalog. So when I scoped the work, I told the team to ignore the model for the first month and build the plumbing. That's where both the reliability and the auditability live. The principles we organized around, in priority order:
- Scoped tool authority. Every agent gets the narrowest set of capabilities a task requires, gated by per-user access controls that default to deny on anything unknown, with an approval step for actions that mutate state or can't be undone. Least privilege isn't a nicety here; it's the difference between an agent that can read a repository and one that can change production.
- An audit trail as a first-class object. Every tool the system invoked, every permission granted or denied, every output it produced is structured, logged, and replayable. If you can't reconstruct what happened, you can't defend it, and you certainly can't put it in front of an auditor.
- The act-or-interpret boundary. I've argued before that this single design decision is the real AI control. Every output is labeled: something the system may act on, or something a human interprets first. The harness enforces the label. That's what turns "human oversight" from theater into a control with teeth.
- Durable state and verification. Work is modeled as explicit, recoverable steps rather than a chat transcript, so a crash mid-task doesn't double-fire a side effect. And the agent has to prove a result before it's trusted.
- Memory with provenance. The system remembers, but every memory carries where it came from, whether it was stated or inferred, and a visibility scope. Memory without provenance is accumulated hallucination, and an unmanaged retrieval layer is a fresh injection surface.
- One governed path for every model call. Every request to the brain runs through a single dispatcher, so the same audit log, the same content scrubbing, and the same refusal rules apply uniformly. No code path gets to route around them.
None of that is exotic. It's the same discipline that has always separated software that ships from software that demos: version control, least privilege, observability, rollback. The models are new. The engineering that makes them safe at scale is not.
Why we own the brain, too
The "brain" half of the decision is subtler. We did not build a foundation model; that would be malpractice for a company our size. We treat the model as what it is becoming, a commodity component. Benchmarks turn over every quarter, and the best model this month is rarely the best model next month. So the harness holds the model at arm's length, behind an interface, swappable. We don't build our identity on one vendor's weights.
What we do own is the intelligence that's actually ours: the institutional knowledge, the memory, the context that makes the system understand our environment instead of the internet's average opinion of it. That layer doesn't get rented back to us by whichever app wins this month. It stays in our control, portable across whatever brain we point it at. Own the harness, own the knowledge, rent the raw reasoning. That's the posture.
Why a combination of harnesses, not one
The question I get next is always the same: which harness? Wrong question. We don't bet AgentOS on a single agent framework any more than we bet it on a single model.
Agent products differ along a few axes that matter: where they run (local, cloud, or hybrid), who picks the model, what the interface contract is. Most of them die stuck in the middle. Adopt one framework wholesale and you inherit its single set of blind spots, and you concede your position on the things that are actually defensible: workflow depth, memory, and verification.
So we treat the agent ecosystem as a menu of patterns rather than a vendor decision. Our durable core stays untouched: a shared, searchable team-memory substrate, deep enterprise integrations, one governed model layer, compliance gating. Into that we graft the specific ideas that close real gaps. We borrow patterns; we refuse wholesale replacement. What each reference point contributes:
- OpenClaw. The local-first archetype: agents that run on your own hardware and actually take action in the real environment. That action-taking is exactly why local agents win adoption, and exactly why they punch holes through security boundaries. We want the upside — agents that do real work and improve themselves — without the ungoverned downside, so we reproduce the capability behind managed, gated, audited equivalents.
- Hermes. The most fully realized personal-agent harness in the open landscape, but a single-user shape. We're a shared team brain, so we don't replace anything; we graft on the few patterns that make agents stickier for an individual: automatic skill mining, a self-sharpening model of the user, frictionless in-conversation memory capture.
- OpenBrain. A portable second brain exposed over a standard tool protocol, so every tool you use reads and writes the same store. It validates our foundational thesis: own the substrate, expose it over an open protocol, keep one shared memory instead of many siloed per-vendor ones. The invariant we hold is strict. Human and agent read and write the same source of truth, with no sync layer, because a sync step means it's really a copy, and copies drift.
- The swarm pattern. From other open projects we pull isolation and inter-agent messaging, plus budget and coordination governance, into our own multi-agent coding harness: worker agents run in isolated worktrees, detect when they'd touch the same files, and operate under hard budget caps with pause-and-approve at the limit.
The composition shows up all the way down at dispatch. Hard, isolated tasks route to an autonomy-plus-correctness-gate shape; cross-tool coordination routes to an integrated, human-in-the-loop shape. We pick per task by the shape of the work, not by a leaderboard. That's the whole thesis, operationalized: don't bet on one.
Running on AWS Bedrock and AgentCore
The brain is interchangeable. The egress is not. Every model call in AgentOS goes through AWS Bedrock. We never call model vendors directly, and a thin internal facade exposes one client so application code never imports a vendor SDK.
The reason is governance, not preference. One cloud provider is one trust boundary and one egress path, which is dramatically easier to audit than several SaaS destinations under separate contracts, logging pipelines, and data-processing agreements. It inherits our existing cloud controls for free: network isolation, audit logging, encryption at rest, threat monitoring. And it keeps regulated data from fanning out to a dozen external APIs.
Inside that boundary we route across tiers — fast and cheap, a balanced default, deep reasoning, and a bulk tier — chosen per request by complexity, with prompt caching wired into the hot paths so we aren't paying repeatedly to resend large, stable tool and schema context every turn. Resilience stays inside the cloud too: we fail over across regions, not across vendors. Cross-vendor fallback would reintroduce the exact sprawl and data-boundary problems the single egress was built to eliminate.
For the capabilities that gave me the most pause, we lean on Bedrock AgentCore's managed runtimes instead of running them ourselves:
- Managed browser. Agent-driven web automation runs in a managed sandbox, not in our service. A standing privilege-escalation and CVE risk belongs on the provider's side of the boundary.
- Managed code interpreter. Executing model-generated code is the highest-risk thing an agent can do, so we delegate the sandbox lifecycle and keep that blast radius off our infrastructure. Conservative by default.
- Per-session runtime. Short, bounded, human-initiated coding tasks move onto managed per-session runtimes; long-running and fully autonomous jobs stay on our own container runtime, and routing fails closed to the self-hosted path. The managed path ships behind a kill-switch and a scoped canary until completion, cost accounting, and lifecycle are proven.
- Workload identity. Managed runtimes authenticate with short-lived, minted credentials instead of standing long-lived secrets: least privilege by construction, smaller blast radius.
Worth saying what we deliberately did not adopt. We don't use a cross-agent federation gateway for internal agent-to-agent delegation, and we don't outsource our core memory to a managed primitive. Coordination stays in-process. Memory is our own pipeline, because it's the differentiator I'm least willing to rent. Adopt the managed pieces that clearly reduce risk or toil; skip the ones that don't.
The five layers of guardrails
Letting agents act in a regulated environment is acceptable only because no single control is load-bearing. The danger the local-agent world proved is action-taking power plus broad data access. So we stack independent layers, and a gap in one is backstopped by another. Defense in depth, applied to agents.
- Content safety. A managed, continuously-updated filter screens text in and out for harmful content and prompt-injection attempts, with optional anonymization of sensitive personal data, returning a clear allow / block / anonymize verdict with reasons. It's one band in the stack, not the perimeter, and far better than hand-written "don't say X" instructions sprinkled across every prompt.
- Formal facts checking. Automated Reasoning evaluates the model's factual claims against a curated policy of verified organizational facts and returns an auditable verdict per claim, naming which premise was satisfied or contradicted, rather than an opaque confidence score. For regulated communications, compliance can read the policy and trace exactly why a claim was flagged, and improving coverage is a policy task rather than a code change.
- Output schema validation. Every tool result passes through one central middleware before it reaches the model: validate against the tool's declared contract, drop undeclared fields, truncate oversized payloads, log every modification. With a large catalog of tools written by many people, that stops schema drift and untyped-blob smuggling, and turns an advisory convention into a guarantee every new tool inherits for free.
- The lethal-trifecta gate. A runtime gate watches for the combination most exploitable via prompt injection: access to private data, exposure to untrusted content, and the ability to cause side effects, all in one run. Any two are fine. All three at once is exactly the thing you build a control to catch. Each tool declares which of the three it touches. The gate doesn't blanket-block, because some legitimate workflows need all three; it escalates to per-run authorization, and it's introspectable so a human can see why a sequence was flagged. That scales where per-tool review can't, and it catches the subtle path where untrusted content is stored, read back later, then acted on.
- Central dispatcher discipline. Because every model call goes through one dispatcher, three protections apply uniformly and can't be bypassed: a universal scrubber strips secrets and sensitive personal data, a regulated-data refusal instruction leads every call, and every invocation is fully audited. No individual code path can forget to scrub, skip the refusal posture, or miss logging. The control is structural, not dependent on each author remembering it.
This is a precondition, not an afterthought. Agents that browse, run code, and act are acceptable only because these five layers sit underneath them. The harness is the governance; the guardrails are how the governance holds.
How I framed it for the board
The hardest part was getting everyone to stop treating this as a tool purchase. A team that adopts an agent platform doesn't just buy a subscription. It starts building habits, processes, and verification steps around one vendor's idea of how work should happen. Six months in, that's not a tool you can swap; it's an architecture you're married to. I've watched this movie before. It was called the cloud, and the companies that treated it as a commodity spent the next decade paying migration costs.
So I framed the decision the way I'd frame any infrastructure commitment. Not "which agent is cheapest," but "which architecture matches how we have to operate, and what does it cost us to change our mind in eighteen months?" For a regulated business, that pointed straight at owning the layer where the controls live. We built incrementally, because you can't start with the finished system; it has to accumulate. Unglamorous plumbing first, then capability compounding on top of a control plane we trusted.
The payoff isn't only safety. It's provable maturity. When a partner or an examiner asks how our automated systems make decisions, we don't hand them a vendor's marketing page. We walk them through our own harness: the permissions, the boundary, the audit trail. The conversation that usually slows a deal down becomes the part where we look more serious than the alternative.
The bet
The industry is converging on a quiet consensus: the model becomes the commodity, and the harness becomes the product. In every regulated domain — finance, health, anything with an auditor — I'd put it more sharply. The harness becomes the governance. The brain is rented reasoning; the harness is the part you have to own, because it's the part you have to answer for.
That's why we built our own. If the decision were in front of me again tomorrow, I'd make the same call — only sooner.