Agents on Call, Part 1. The Scenario: Why an Ops Team Hires Agents

A mid-size B2B SaaS company running about 50 engineers across roughly 30 AWS accounts pages someone on-call around 40 times a week, and the page gets a real answer only after 25 to 35 minutes of manually pulling context: which account, which dashboard, which runbook, whether that runbook is even still accurate. That is before diagnosis starts, not instead of it. This series builds the fix: a small platform of AI agents that does that first pass automatically, reads everything it is allowed to read, changes nothing without a human's sign-off, and stops being useful the moment it stops being that disciplined.

The company, and what it is actually paying in toil

Call the company what it is: a mid-size B2B SaaS shop, about 50 engineers, roughly 30 AWS accounts spread across a handful of product lines, each with its own dev, staging, and production trio. Nobody set out to end up with 30 accounts. It happened the ordinary way, one team at a time asking for isolation, until AWS Organizations reflects the org chart more than the architecture.

The toil is not dramatic. It is compounding, and four numbers from three months of on-call data and one uncomfortable retro make the shape of it clear:

• About 40 pages a week land on whoever is on rotation, spread unevenly across the 30 accounts.
• 25 to 35 minutes of every page is spent assembling context before diagnosis even starts: the right account, the right dashboard, the right runbook, and whether that runbook still matches reality.
• The monthly FinOps review runs roughly three weeks behind actual spend, so a misconfigured autoscaling group or an idle GPU instance burns money for most of a month before it shows up on a report anyone reads.
• A recent runbook audit found that about one in three runbooks pointed at a resource, an ARN, or a console path that no longer existed.

None of those four is an emergency by itself. Together they are a standing tax on the team: somewhere around 30 to 40 engineer-hours a week spent on the mechanical part of operations, gathering context and re-verifying stale instructions, paid by people who were hired to build product, not to be human context-assemblers.

Why agents, and why now

The honest answer to "why now" is that the pieces finally exist as managed services instead of internal frameworks somebody has to babysit. Bedrock has offered a managed agent orchestration service, Agents for Amazon Bedrock, since it reached general availability in November 2023 with improved orchestration control and reasoning trace visibility. What changed is the tier above it. Amazon Bedrock AgentCore went generally available on 2025-10-13, across nine regions, packaging seven component services (Runtime, Memory, Identity, Gateway, Code Interpreter, Browser, Observability) that used to mean building a session store, a credential broker, and a tracing pipeline from scratch. In parallel, AWS open sourced the Strands Agents SDK in May 2025 under Apache 2.0 and shipped a production 1.0 release for Python on 2026-05-21, a week before this post's own timeline starts. The managed infrastructure and the open orchestration layer both cleared a production-readiness bar in the same few months. That window is when this platform gets built.

Build versus buy

The team looked at buying first, as it should have. An off-the-shelf AIOps or incident-copilot product would have shipped faster and needed no Terraform. It lost for three reasons specific to this company, not to agents in general.

• The mutation boundary is not negotiable. Whoever executes a fix has to assume a narrowly scoped IAM role in the affected account, gated by an internal approval step. A SaaS product asks you to either grant it broad cross-account access or bolt its own approval flow onto yours after the fact. Owning the platform means the approval gate is native from day one, not integrated later.
• Thirty accounts of tribal knowledge live in Slack threads and in engineers' heads, not in a format a vendor's ingestion pipeline understands. A Bedrock Knowledge Base over the existing runbook corpus, with the team controlling chunking and retrieval quality directly, is more tractable than exporting that knowledge to a third party and hoping its retrieval is good enough.
• The company already operates AWS at this scale. The Terraform, the CI, the on-call rotation, the account structure, all of it exists today. The marginal cost of building on AWS-native primitives is lower than the cost of a second vendor relationship, a second billing line, and a second security review.

The counter-argument is real: buying is faster, and faster has value when toil compounds weekly. The team's answer was to scope the build tightly (four agents, one approval pattern, on-demand inference to start, see the sizing math in a later part) rather than treat this as a platform team's forever-project. The companion code for the whole series lives at github.com/flightlesstux/agents-on-call; this post is where its docs/architecture.md starts.

The stack decision: three ways to run an agent on Bedrock

Once the team decided to build, the next decision was which layer to build on. Three real options existed, and the trade-off is not close to as simple as "use the newest thing":

The team picked AgentCore plus Strands. The managed pieces they did not want to own (session isolation, credential brokering, a cross-account gateway) came paired with an orchestration layer they could read and debug, instead of one they had to reverse-engineer from a console. AgentCore Runtime at general availability gives each agent session full isolation and an eight-hour execution window, with Agent2Agent protocol support if agents in this platform ever need to talk to agents outside it. The @aws/agentcore CLI, released April 22, 2026, added a create, dev, deploy, invoke, logs, and traces workflow that supports Strands directly (also LangGraph, LangChain, Google ADK, OpenAI Agents, or bring-your-own), which mattered for onboarding: nobody had to learn a brand-new deployment tool mid-project.

Classic Bedrock Agents stayed genuinely attractive on its own merits: two and a half years of production track record, less custom code to own, and a simpler mental model when an action group is all a use case needs. It lost here because four agents sharing a supervisor pattern across a hard multi-account IAM boundary needed more control over the loop than action groups expose. Self-built on the raw Converse API was rejected for the opposite reason: writing a session store, a credential broker, and a tracing pipeline from scratch, for something AWS already ships as AgentCore Runtime, Memory, Identity, and Observability, would have spent the first two months of the project reinventing infrastructure instead of writing agents.

Target architecture: hub-and-spoke across the org

The platform is multi-account, hub-and-spoke, and deliberately additive. If it goes down, the org degrades to exactly the status quo: humans get paged the way they always were. It never sits in the critical alerting path.

graph TD
  MGMT["management account
org root, SCPs, no workloads"]
  SEC["security account
org CloudTrail, GuardDuty,
Security Hub delegated admin"]
  OPS["ops-tooling account
the agent platform"]
  W1["workload account 1"]
  W2["workload account 2"]
  WN["workload account N
~30 total"]

  MGMT -. SCPs .-> SEC
  MGMT -. SCPs .-> OPS
  MGMT -. SCPs .-> W1
  MGMT -. SCPs .-> W2
  MGMT -. SCPs .-> WN

  W1 -- alarm events --> OPS
  W2 -- alarm events --> OPS
  WN -- alarm events --> OPS

  OPS -- "assume ops-readonly, read-only tools" --> W1
  OPS -- "assume ops-readonly, read-only tools" --> W2
  OPS -- "assume ops-readonly, read-only tools" --> WN
  OPS -. "assume ops-mutate, approved executor only" .-> W1

Each workload account exposes exactly two roles to ops-tooling: ops-readonly, assumable by every tool Lambda, built from AWS read-only managed policies with explicit denies layered on top; and ops-mutate, assumable by exactly one Lambda, the one that runs after a human approves a proposed action, scoped to an allowlist of specific SSM Automation documents. No agent, no other Lambda, no code path besides that single post-approval executor can ever assume ops-mutate. That separation, not a prompt instruction, is what makes read-only the default: it is enforced by IAM, not by asking a model nicely.

Inside ops-tooling

Zoom into the one account that the rest of this series lives in:

graph LR
  subgraph entry["Entry"]
    EB["EventBridge bus
cross-account alarms"]
    SLACK["Slack app
API Gateway + verifier Lambda"]
    SCHED["EventBridge Scheduler
daily cost sweep"]
  end
  subgraph agents["Agent layer"]
    SUP["supervisor"]
    TRI["incident-triage"]
    RUN["runbook"]
    COST["cost"]
    MEM["AgentCore Memory"]
    ID["AgentCore Identity"]
  end
  subgraph models["Model layer"]
    BR["Bedrock, cross-region
inference profiles"]
    GR["Bedrock Guardrails"]
    KB["Knowledge Base
runbook corpus"]
  end
  subgraph tools["Tool layer"]
    GW["AgentCore Gateway"]
    T1["cloudwatch-read"]
    T2["logs-read"]
    T3["cost-read"]
    T4["ssm-execute"]
  end
  subgraph gate["Approval gate"]
    SF["Step Functions
waitForTaskToken"]
    EXEC["post-approval executor"]
  end
  subgraph obs["Observability"]
    OTEL["OTEL traces to CloudWatch
GenAI observability"]
    LOG["invocation logs to S3"]
    BUD["Budgets + Cost
Anomaly Detection"]
  end

  EB --> SUP
  SLACK --> SUP
  SCHED --> COST
  SUP --> TRI
  SUP --> RUN
  SUP --> COST
  TRI --> MEM
  RUN --> MEM
  COST --> MEM
  SUP --> ID
  TRI --> BR
  RUN --> BR
  COST --> BR
  BR --> GR
  TRI --> KB
  RUN --> KB
  SUP --> GW
  GW --> T1
  GW --> T2
  GW --> T3
  GW --> T4
  T4 --> SF
  SF -- "context + Slack approval" --> SLACK
  SF -- approved --> EXEC
  BR --> OTEL
  GW --> LOG
  COST --> BUD

Four Strands agents run on AgentCore Runtime behind a supervisor: incident triage, runbook execution, and cost optimization. AgentCore Memory holds session and long-term state so the supervisor is not re-explaining context to a specialist on every hop, and AgentCore Identity brokers outbound credentials so no agent's code carries a long-lived secret. Every model call goes through cross-region inference profiles for reliability and burst headroom, with Bedrock Guardrails attached to every invocation and a Knowledge Base over the runbook corpus backing the triage and runbook agents. The tool layer is entirely AgentCore Gateway: least-privilege Lambdas behind MCP, three of them read-only and assuming ops-readonly in a spoke, one of them, ssm-execute, the only mutating tool, and it cannot reach a spoke account directly. It can only hand a proposal to Step Functions, which pauses on waitForTaskToken, posts the full context (diagnosis, blast radius, exact SSM document and parameters) to Slack, and resumes only when a human approves. Every hop from EventBridge to Gateway carries OTEL tracing into CloudWatch generative AI observability, Bedrock invocation logs land in S3, and Budgets plus Cost Anomaly Detection watch the platform's own spend with per-agent cost allocation tags.

Read-only is the prime directive

Every agent on this platform can look at anything relevant and change nothing, unless a human has already said yes. That is the one sentence the rest of this series has to keep being true to. It shows up as three separate, unglamorous IAM decisions rather than one clever mechanism: the tool Lambdas assume a read-only role built from AWS managed policies with explicit denies stacked on top; the only mutating tool cannot assume anything in a spoke account directly; and the sole path from "the agent proposed a fix" to "the fix ran" passes through a Step Functions wait state that only a human's Slack click can resume.

Read-only is not a prompt instruction telling a model to be careful. Models that are asked nicely are still occasionally not careful. Read-only here is an IAM boundary that a compromised prompt, a hallucinated tool call, or a plain bug cannot cross, because the credentials to cross it do not exist anywhere the agent can reach.

What done looks like

The finale of this series comes back to these numbers. They are the ones the toil inventory above set up, made measurable:

• Median time from page to an enriched Slack summary (logs, dashboards, likely cause) under 5 minutes, down from the current 25 to 35 minutes of manual gathering.
• Zero mutating actions execute without a Step Functions-recorded human approval, verified against CloudTrail weekly.
• The monthly FinOps lag drops from about three weeks to same-day, via the daily cost sweep surfacing anomalies before the next billing cycle closes.
• Runbook staleness (broken ARNs, dead console paths) drops from about one in three to under one in twenty, caught continuously instead of at an annual audit.
• A quarterly game day disables ops-tooling entirely and confirms paging and manual runbook execution still work exactly as before it existed, with no silent dependency created.
• Incident-triage diagnosis matches the eventual root cause often enough, measured against a numeric threshold the eval harness in a later part will define, that on-call stops re-verifying it by hand before acting on it.

Failure modes to watch from day one

Three things to watch before anyone touches Terraform, because scenario-setting is the wrong place to discover them in production. If a read tool cannot reach a spoke account, whether a role assumption is denied or a service is degraded, the agent has to say so and hand back to a human, not guess: a wrong diagnosis stated with confidence is worse than no diagnosis. If the approval gate itself stalls, both the wait state and the surrounding state machine need explicit timeouts, or a stuck proposal ages silently instead of failing loud. And if ops-tooling is down entirely, the platform has to degrade to exactly today's process, which only holds if nothing outside ops-tooling starts quietly depending on it existing.

Read this next

• Part 2, The Foundation: Terraform Before Tokens, where the account layout above becomes real HCL: IAM baseline, Bedrock model access, and the on-demand versus provisioned versus cross-region inference decision.
• AWS Multi-party Approval for Organizations on ercan.cloud, the same "gate the risky action behind a quorum of humans" pattern, from the raw AWS Organizations side rather than the agent side.

The two diagrams above, and the account and IAM layout they describe, also live as Markdown in the companion repository at github.com/flightlesstux/agents-on-call, alongside the Terraform and Python this series builds part by part. For the database and infrastructure side of running this kind of platform at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.