Agents on Call, Part 5. The Team: Supervisor and Three Specialists
A supervisor delegates to triage, runbook, and cost agents over the network, AgentCore Memory ties their findings together, and when one agent still wins.

Four agents now exist where Part 4 left one: a supervisor, a runbook agent, and a cost agent join incident-triage, coordinated not by Python function calls inside one process but by AgentCore Runtime's own InvokeAgentRuntime API, because all four still deploy as separate, isolated Runtime resources, the same isolation Part 3 chose AgentCore Runtime for in the first place. One more AWS resource ties them together: a single AgentCore Memory instance, shared across all four by actor ID, so a diagnosis triage writes at 3am is still legible to the runbook agent it hands off to seconds later, and to whichever agent looks at an incident shaped like this one again next month.
Part 1 picked AgentCore plus Strands and sketched four agents behind one supervisor as the target shape. Part 2 built the account boundary and the ops-readonly/ops-mutate split every agent since has reused. Part 3 shipped the first agent, two tools wired directly into its own process. Part 4 moved those tools, plus a second read tool and the platform's only mutating tool, behind AgentCore Gateway: four Lambdas one MCP client can discover instead of four things to wire by hand per agent. This part is where that investment starts paying for more than one caller: the runbook and cost agents both connect to the same Gateway and inherit all four tools without re-deriving cross-account credential handling from anything. The companion code lives at github.com/flightlesstux/agents-on-call, in agents/supervisor/, agents/runbook/, agents/cost_optimizer/, and this part's additions to terraform/10-agent-runtime/; every snippet below is excerpted from those files, not simplified for the post.
When one agent stops being enough
Part 3's triage agent had one system prompt, two tools, one job. That held fine through Part 4's four tools, because all four still served that one job: diagnose, never remediate. The pressure that actually forces a second agent is not tool count, it is job count. A runbook agent (turn a diagnosis into a proposed SSM Automation run) and a cost agent (read Cost Explorer once a day, unprompted, and propose rightsizing) are not variations on triage's job, they are different jobs, with different schedules and, if crammed into one system prompt, contradictory instructions living in the same context window. "You never take a mutating action, you have no tool that can" (triage's own rule, unchanged since Part 3) and "propose an ssm_execute call when a matched runbook chunk supports it" (the runbook agent's actual job) are not sentences one prompt states cleanly together; an agent asked to do both drifts one job's discipline toward the other's willingness to act, or loses the second job's nuance under the first job's caution. Splitting the work across processes, not just across prompt sections, is what keeps triage's own discipline legible two parts later: its system prompt has not changed a line since Part 3.
Context bloat is the quieter version of the same pressure. A triage agent also carrying Cost Explorer's quirks and the runbook corpus's retrieval behavior spends system-prompt tokens and tool schema weight on jobs it is not doing on any given invocation, most of the time. Four agents, each holding only what its own job needs, is a token-efficiency decision as much as an organizational one.
Agents as tools, the shape that fits four separate Runtimes
Strands Agents 1.0 shipped four new multi-agent primitives plus Agent-to-Agent protocol support in its production release, and "agents as tools" has more than one real shape in the SDK. Passing a child Agent instance directly into a parent's tools=[...] list, or wrapping one in a decorator that exposes it with a tool spec, both work well and both require the same thing: the sub-agent lives in the same Python process as its caller. That is not this platform's shape. Triage, runbook, and cost each deploy as their own aws_bedrockagentcore_agent_runtime resource, independently versioned, independently IAM-scoped, each in the session isolation Part 3 picked AgentCore Runtime for specifically. Collapsing three Runtimes into one process to use the in-process shortcut would undo that isolation for a syntactic convenience.
| Shape | How it works | Fits this platform? |
|---|---|---|
tools=[agent_instance] | Pass a Strands Agent object straight into a parent's own tools list | No: needs the sub-agent in the same process, collapsing three separate Runtimes' isolation into one |
.as_tool() | Wrap an in-process agent call in a decorator that exposes a tool spec | No, same reason: still one process, one blast radius if it crashes |
@tool calling InvokeAgentRuntime | A plain Strands tool function that calls another agent's already-deployed Runtime over the network | Yes: matches the four-separate-Runtimes shape Part 3's Terraform already committed to |
So the supervisor uses the third shape: @tool-decorated functions that call bedrock-agentcore's InvokeAgentRuntime data-plane operation against each specialist's runtime ARN. The trade-off is real and worth naming instead of hiding: an in-process call is a function call, this is a network hop with its own latency and its own way to fail partway. Paid deliberately, not by accident, for the isolation Part 3 already committed to:
def _invoke_specialist(agent_runtime_arn: str, actor_id: str, prompt: str) -> str:
"""Invoke one specialist's AgentCore Runtime and return its text result.
...
"""
response = _agentcore.invoke_agent_runtime(
agentRuntimeArn=agent_runtime_arn,
runtimeSessionId=_session_id,
contentType="application/json",
payload=json.dumps({"prompt": prompt, "session_id": _session_id}).encode("utf-8"),
)
if response["statusCode"] != 200:
raise RuntimeError(f"{actor_id} runtime invocation failed with status {response['statusCode']}")
body = json.loads(response["response"].read())
if "error" in body:
raise RuntimeError(f"{actor_id} returned an error: {body['error']}")
return body["result"]
@tool
def ask_triage_agent(alarm_context: str) -> str:
"""Ask the incident-triage specialist to diagnose a firing alarm.
...
"""
result = _invoke_specialist(TRIAGE_AGENT_RUNTIME_ARN, "incident-triage", alarm_context)
record_incident_event("supervisor", _session_id, {"delegated_to": "incident-triage", "result": result})
return resultTwo details worth pausing on. First, _invoke_specialist raises on a non-200 status or an error body instead of returning a string that reads like a normal finding: a supervisor that cannot tell "runbook found nothing" from "runbook's invocation failed" will synthesize false confidence into whatever it tells a human. Second, _session_id is minted once per supervisor invocation, not once per specialist call, and passed to every specialist unchanged: it is what lets AgentCore Runtime group all three specialists' traces under one incident, and what lets the shared memory below tell one incident's events apart from another's.
The runbook agent: Knowledge Base over SSM documents, proposes, never executes
The runbook agent's whole job is turning a diagnosis into a candidate remediation, sourced from a real document, not invented. It gets there with two tool sources: a new kb_retrieve tool over a Bedrock Knowledge Base built on the runbook corpus, and every tool Gateway already indexes, discovered at connect time rather than declared by hand:
@tool
def kb_retrieve(query: str, max_results: int = 5) -> str:
"""Search the runbook Knowledge Base for playbook chunks relevant to query.
...
"""
response = _bedrock_agent_runtime.retrieve(
knowledgeBaseId=RUNBOOK_KNOWLEDGE_BASE_ID,
retrievalQuery={"text": query},
retrievalConfiguration={"vectorSearchConfiguration": {"numberOfResults": max_results}},
)
...
return json.dumps({"query": query, "chunks": chunks})
_gateway = connect_gateway_tools()
agent = Agent(
model=model,
system_prompt=SYSTEM_PROMPT,
tools=[kb_retrieve, *_gateway.list_tools_sync()],
name="runbook",
...
)connect_gateway_tools() signs every request to Gateway's MCP endpoint with the runtime container's own execution-role credentials, SigV4, matching Part 4's choice of authorizer_type = "AWS_IAM" on the Gateway resource itself, and returns whatever tools that IAM role can reach: cloudwatch-read, logs-read, cost-read, and ssm-execute, the identical four Part 4 built, unlisted and unreferenced anywhere in this agent's own code. Whether the model actually calls ssm-execute is a system-prompt and IAM decision, not a code-time one. And ssm-execute has not changed since Part 4: it still requires diagnosis and blast_radius on every call, still only ever starts the Step Functions approval flow, never a spoke-account action directly. The runbook agent's system prompt makes that a hard rule rather than an implicit one: calling ssm_execute is reported to the supervisor as "sent to a human", never "done", regardless of how confident the model's own text sounds.
The cost agent: a scheduled sweep into the same gate
The cost agent is the one specialist nothing pages: EventBridge Scheduler invokes it once a day per spoke, unprompted by any alarm, the schedule itself defined entirely in Terraform rather than left as a claim in a docstring:
resource "aws_scheduler_schedule" "cost_agent_daily" {
name = "${var.platform_name}-cost-agent-daily"
description = "Daily trigger for the cost agent's Cost Explorer sweep across configured spoke accounts."
schedule_expression = "cron(0 6 * * ? *)"
flexible_time_window {
mode = "OFF"
}
target {
arn = "arn:aws:scheduler:::aws-sdk:bedrockagentcore:invokeAgentRuntime"
role_arn = aws_iam_role.cost_scheduler.arn
...
}
}It reads through the same Gateway connection as runbook (cost-read, plus cloudwatch-read and logs-read to corroborate a spend anomaly against real load before flagging it), and when a rightsizing action looks warranted, it proposes through the identical ssm-execute path runbook uses. One approval pipeline, two proposers: a human reviewing Slack should not need to know or care whether a pending proposal came from an incident diagnosis or an overnight sweep.
The one thing worth designing in deliberately is not re-litigating a denial. A daily sweep with no memory of yesterday will flag the same "idle" instance every morning until a human either approves it or gets tired of denying it, and neither outcome is good:
prior_episodes = retrieve_incident_context("cost", prompt)
if prior_episodes:
prompt = (
f"{prompt}\n\nNote: {len(prior_episodes)} related past cost proposal(s) "
"exist in memory; check whether any were denied before re-proposing."
)AgentCore Memory: one shared incident, four writers
AgentCore Memory bills short-term events at $0.25 per 1,000 CreateEvent calls and long-term retrieval at $0.50 per 1,000 RetrieveMemoryRecords calls, cheap enough that four agents each writing one event per invocation costs a rounding error next to a single model call's token price. The strategy doing the work here is EPISODIC, one of AWS's built-in types: it captures each session as a structured episode (context, reasoning, actions, outcomes) and runs its own reflection step across episodes to extract broader patterns, without this platform needing to prompt-engineer that extraction itself. Terraform support for the EPISODIC type on aws_bedrockagentcore_memory_strategy landed April 29, 2026, almost two months before this part's own date:
resource "aws_bedrockagentcore_memory" "incident" {
name = "${replace(var.platform_name, "-", "_")}_incident_memory"
description = "Shared short-term event stream and long-term episodic memory for one incident's supervisor, triage, runbook, and cost agents."
event_expiry_duration = var.memory_event_expiry_days
# memory_execution_role_arn deliberately omitted: per the fact table, it
# is required only when a memory uses a CUSTOM strategy with a
# model-processing override block (SEMANTIC_OVERRIDE,
# USER_PREFERENCE_OVERRIDE, and so on). EPISODIC below is a built-in
# type, not CUSTOM, so AWS runs its own reflection step without this
# platform needing to grant or manage a role for it.
tags = merge(var.tags, { Component = "memory" })
}
...
resource "aws_bedrockagentcore_memory_strategy" "episodic" {
name = "incident-episodes"
memory_id = aws_bedrockagentcore_memory.incident.id
type = "EPISODIC"
namespaces = ["/incidents/{actorId}"]
}Every agent writes into that one memory through the same two functions, which is what keeps four independently deployed Runtimes from drifting on how they read and write shared state:
def record_incident_event(actor_id: str, session_id: str, event: dict) -> str | None:
"""Write one short-term event to the shared incident memory.
...
"""
if not MEMORY_ID:
return None
response = _agentcore.create_event(
memoryId=MEMORY_ID,
actorId=actor_id,
sessionId=session_id,
eventTimestamp=time.time(),
payload=[
{
"conversational": {
"content": {"text": json.dumps(event, default=str)},
"role": "ASSISTANT",
}
}
],
)
return response["event"]["eventId"]Two things it deliberately does not do. It returns None instead of raising when the memory ID is unset, so an agent under local test without a provisioned memory resource degrades to "no shared memory" rather than failing a diagnosis over a missing environment variable. And actor_id is a string an agent passes in, not a value this function validates against anything, on purpose: adding a fifth agent later needs no schema change here, only a new actor writing into the same namespace shape.
Honest: when multi-agent is overkill
Four AgentCore Runtimes plus a memory resource plus cross-runtime networking is real infrastructure for what a smaller ops team could run as one agent with seven tools and a longer system prompt. AgentCore Runtime bills $0.0895 per vCPU-hour and $0.00945 per GB-hour with idle time free, so four small runtimes are not expensive in absolute terms, but they are four things to deploy, four IAM roles to audit, and one InvokeAgentRuntime hop's worth of latency and failure surface that a single process never has. The honest trigger for splitting is not "this platform has grown," it is the same one this part opened with: distinct jobs whose instructions contradict each other in one system prompt, or distinct schedules (an alarm-driven agent and an unprompted daily sweep) that do not share a natural single loop. An ops platform with one job, triage and nothing else, gets none of that pressure and should stay one agent with good tools: no cross-runtime call to reason about, no memory resource to keep from going stale, no for_each Terraform module coordinating three roles that could just as well be one. Splitting early, before a second real job exists, buys isolation nobody needed yet at the cost of a network hop nobody wanted yet either.
Failure modes to watch
Four worth naming before this runs against a real incident. Triage does not write to the shared memory yet: its Part 3 code predates this part's memory resource, and nothing in this part retrofits it, which means the reflection strategy's episodes are missing the one agent that runs first on almost every incident, a real gap, not an oversight to gloss over. A cross-runtime call failing partway is a different failure than an in-process one: _invoke_specialist raising mid-incident leaves whatever the supervisor already recorded to memory sitting there as a partial timeline, useful for a human picking up the incident by hand but easy to misread as complete if nobody checks. EPISODIC's reflection step runs asynchronously over closed sessions, so retrieve_incident_context returning an empty list means either "genuinely new" or "reflection has not run yet," and neither this code nor the agent's prompt can currently tell those apart. And the new agents' IAM policies have a real ordering dependency: the supervisor's role policy references the runbook and cost runtimes' own ARNs, so a Terraform apply interrupted between provisioning those runtimes and provisioning the supervisor's role leaves the next plan showing the supervisor's policy as not-yet-applied, not a lasting misconfiguration, but a reason to let an apply finish rather than trusting a partial one.
Read this next
- Part 4, Tools and the Gateway: MCP, Allowlists, Read-Only Default, for the four tools this part's runbook and cost agents connect to and reuse without re-deriving anything.
- Part 6, Guardrails: The Part Everyone Skips, where every one of these four agents' model invocations gets a content and prompt-injection layer none of them have yet.
- How to call multiple terraform modules in a single terragrunt file on ercan.cloud, the same "coordinate several independently deployed units without duplicating the wiring between them" problem this part's
for_each-based agent runtimes solve inside one module, from the multi-module side instead.
The full agents/supervisor/, agents/runbook/, agents/cost_optimizer/, and the memory and runtime Terraform behind them live in the companion repository at github.com/flightlesstux/agents-on-call. For the database and infrastructure side of running platforms like this one at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.
More from Ercan
Two more sites, same author, different ground.
Cloud, AWS, EKS, Terraform, platform engineering.
Field notes from production systems. EKS, IAM, Terraform at organization scale, observability, cost optimization.
Visit ercan.cloud →The hub. About, consulting, contact.
Personal hub for both writing tracks. Who I am, how the consulting works, how to reach me.
Visit ercanermis.com →