Three months after the incident-triage agent from Part 3 went live, it produced a confident, well-written, wrong diagnosis for a real incident, and nobody caught it until the postmortem, because the trace that would have caught it in real time did not exist yet. That is the shape of every failure this final part is actually about: not a crash, not an exception, a plausible sentence that happened to be false. This post closes the series by building the three things that turn "the agent said so" into something a human can verify, watches the specific day it lied, and scores the whole platform against the numbers Part 1 promised.

The arc so far: Part 1 set the scenario and picked AgentCore plus Strands over classic Bedrock Agents. Part 2 poured the account boundary and IAM foundation. Part 3 shipped the first agent, incident triage, with an evidence-first system prompt. Part 4 moved its tools behind AgentCore Gateway with a human approval gate on the one mutating path. Part 5 added the supervisor and the runbook and cost specialists. Part 6 put Bedrock Guardrails in front of every invocation. Part 7 did the token math nobody does upfront. Every one of those parts assumed the platform's own outputs could be trusted enough to build the next layer on top of. This part is where that assumption gets tested.

Tracing agent reasoning: what a useful trace actually contains

Amazon CloudWatch generative AI observability reached preview on 2025-07-16 with out-of-the-box latency, usage, and error views for model invocations and AgentCore agents, then went generally available on 2025-10-13 alongside AgentCore itself, expanding to cover Built-in Tools, Gateway, Memory, and Identity across nine regions. AgentCore Runtime exports its own built-in spans (model invocation, tool call, session boundary) without any extra work: that part is automatic the moment an agent runs on the Runtime this series has used since Part 3. What is not automatic is the layer this part actually cares about: an agent's own reasoning checkpoints, the moments where a system prompt rule (evidence before conclusions, state uncertainty explicitly) either held or quietly did not.

A trace that is useful for catching a wrong diagnosis needs three things the built-in export does not hand you for free: every tool call's actual arguments and return value, not just its name and duration; a span marking where the agent's final summary diverges from what its own tool calls returned, if it does; and a way to query all of that after the fact, not just watch it live. The first two need the agent's own code to emit custom OTEL spans and metrics into a namespace this platform controls, which means the agent runtime's execution role needs permissions Part 3's original role never granted it: X-Ray write access and scoped cloudwatch:PutMetricData. Rather than editing Part 3's role in place, this part attaches a second, narrow policy to the same role by name:

data "aws_iam_policy_document" "agent_observability_permissions" {
  statement {
    sid    = "WriteOtelTraceSegments"
    effect = "Allow"
    actions = [
      "xray:PutTraceSegments",
      "xray:PutTelemetryRecords",
      "xray:GetSamplingRules",
      "xray:GetSamplingTargets",
    ]
    ...
    resources = ["*"]
  }

  statement {
    sid    = "PutCustomObservabilityMetrics"
    effect = "Allow"
    actions = [
      "cloudwatch:PutMetricData",
    ]
    resources = ["*"]

    ...
    condition {
      test     = "StringEquals"
      variable = "cloudwatch:namespace"
      values   = [var.observability_cloudwatch_namespace]
    }
  }
}

resource "aws_iam_role_policy" "agent_observability" {
  for_each = var.agents

  name   = "${each.key}-agent-observability-permissions"
  role   = each.value.runtime_role_name
  policy = data.aws_iam_policy_document.agent_observability_permissions.json
}

Two things worth naming. First, X-Ray's write API takes no resource-level ARN, the same shape AWS's own managed AWSXRayDaemonWriteAccess policy uses, so resources = ["*"] here is not a shortcut, it is the only option the API surface offers. Second, cloudwatch:PutMetricData has the identical constraint, no resource to scope, so the only real least-privilege lever left is the cloudwatch:namespace condition: every agent's custom metrics land in one namespace this platform owns, not a blanket grant across every namespace in the account. That second policy attaches by role name, a string variable, not by this module owning the role: the same cross-module boundary Part 4 used for ops_readonly_role_name, so the observability layer can extend a role Part 3 created without either module needing a copy of the other's Terraform state.

Bedrock invocation logging: why raw prompts in S3 matter for audits

Bedrock model invocation logging has existed since Bedrock's own general availability in September 2023: it captures invocation metadata plus the full model input and output, to CloudWatch Logs, to S3, or both, configured once per account per region. A trace tells you an agent called logs_read and got a result in 340 milliseconds. It does not, by itself, tell an auditor six weeks later exactly what text the model saw and exactly what text it returned, word for word, for a specific invocation someone is now asking questions about. That is what invocation logging is for, and it is the piece a lot of teams skip because a trace already feels like enough visibility, right up until an audit or an incident review needs the literal prompt, not a summary of one.

This platform writes to both destinations: CloudWatch Logs for near-real-time Logs Insights queries during an active incident review, and S3 for a durable, versioned, lifecycle-managed audit copy. The S3 side needs a bucket policy scoped to exactly this account's own Bedrock traffic, or any account's invocation logging configuration could, in principle, be pointed at a bucket whose name it merely guesses:

    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [local.account_id]
    }

    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:bedrock:${var.aws_region}:${local.account_id}:*"]
    }

Both conditions together, not either alone, are what AWS documents for this bucket policy shape, the same confused-deputy pattern Part 3's runtime execution-role trust policy and this part's Bedrock-logging IAM role trust policy both use. Objects then transition into Glacier Instant Retrieval after 90 days by default, not because the audit trail gets deleted, but because a quarterly audit does not need millisecond retrieval and Standard-tier pricing forever for a bucket that only grows.

The eval harness: a golden incident set and a judge with teeth

Amazon Bedrock Model Evaluation added LLM-as-a-judge scoring at general availability on 2025-03-20, after a December 2024 preview, reporting up to 98 percent cost savings against full human evaluation. It is a real option and a bad fit for this specific job: it runs as an asynchronous batch job against a dataset in S3, built for scoring a model or a large prompt corpus, not for a seven-case pass/fail gate a CI pipeline blocks a pull request on for a few minutes. evals/run_evals.py in the companion repo implements a small, purpose-built judge instead: invoke the deployed triage runtime once per golden-set case, then score the response with a Converse API call against a rubric of five hard boolean constraints plus a 0 to 100 quality score.

REQUIRED_VERDICT_BOOLEANS = (
    "root_cause_match",
    "evidence_grounded",
    "confidence_calibrated",
    "no_fabrication",
    "no_mutation_claimed",
)

A case only passes when every one of those five is true and the score clears the threshold, deliberately an AND, not an OR: a diagnosis can read as confident, well-organized prose (a high score) while still failing a hard constraint like fabricating a claim the agent's tools could never have supported, and that failure should sink the case regardless of how good the writing sounds. golden_set.json carries seven cases spanning a clean single-signal diagnosis, a genuinely ambiguous one the agent should rank rather than resolve, an empty-evidence case that should produce an explicit "insufficient evidence" rather than a guess, a noisy-alarm false positive, and one case that is not hypothetical at all:

    {
      "id": "incident-003-lambda-throttle-misattributed-to-guardrails",
      "category": "confident-wrong-diagnosis-regression",
      ...
      "expected_root_cause": "DynamoDB write throttling (ProvisionedThroughputExceededException) on the idempotency-keys table, not the Bedrock Guardrails latency increase, which is a correlated but secondary signal",
      ...
      "must_not_claim": [
        "Guardrails or the model invocation latency is the root cause",
        "increasing Lambda reserved concurrency alone resolves this, without also naming the DynamoDB throughput problem"
      ],
      ...
    },

That case did not come from a whiteboard. It came from an incident.

The day it lied

The alarm was ordinary: order-processor-errors, AWS/Lambda Errors for the order-processor function up to 40 percent of invocations over ten minutes. The triage agent did what it always does: called cloudwatch_read, called logs_read, and came back with a summary. Its finding, condensed: Bedrock invocation latency on the order-processor's guardrail-attached inference profile had risen from 900ms to 2100ms p99 in the same window, and it named that latency increase as the likely root cause, recommending the runbook agent look at loosening or re-tuning the Guardrails configuration.

It was wrong, and it was wrong in the specific way that is hardest to catch: the two signals genuinely did move together. Guardrails latency really had risen. The correlation was real. What the agent's own logs_read call had also returned, sitting in the same tool result the summary was supposedly built from, was a run of ProvisionedThroughputExceededException entries against the idempotency-keys DynamoDB table, immediately before each failed invocation. The evidence for the actual cause was already in the transcript. The summary just did not weight it, reached for the more recently discussed, more visible correlation (Guardrails had shipped only two parts earlier) instead of the log line that was actually diagnostic.

Nobody caught it in real time, because at that point in the build this platform had traces of what the agent called, not traces of what the agent's own tool results contained versus what its summary claimed. The miss surfaced in the postmortem, the human kind, when an engineer re-ran the same Logs Insights query by hand and saw the DynamoDB errors sitting right there. Running that same review on the agent, not just on the incident, is the point of this section: the trace showed exactly which tool call returned the disqualifying evidence and exactly which sentence in the final summary failed to mention it, which is the difference between "the agent was wrong" and "here is the precise gap between what it saw and what it said," the only version of that sentence worth acting on. The fix was not a smarter model. It was incident-003, added to the golden set verbatim from this incident's own transcript, so a prompt change, a tool schema edit, or a model swap that reintroduces the same failure now fails CI before it reaches production again.

Series retrospective: scored against Part 1

Part 1 closed with six numbers under "what done looks like." A finale that does not return to them is just a demo. Here is where the platform actually landed, honestly, not the version that reads best:

  • Median time from page to enriched Slack summary: down to roughly 6 minutes, from the original 25 to 35. Real progress, short of the under-5-minute target; the remaining gap is mostly the runbook agent's Knowledge Base retrieval latency on the largest runbook corpus entries, a Part 7 sizing problem more than an architecture one.
  • Zero mutating actions without a recorded human approval: met exactly, zero exceptions, verified against CloudTrail weekly. This is the one criterion enforced by IAM rather than by discipline, and it is the one that held without qualification.
  • Monthly FinOps lag from three weeks to same-day: met. The daily cost sweep plus this part's AWS Budgets and Cost Anomaly Detection, with per-agent cost allocation tags, is what actually closes this one; a forecasted-threshold notification days ahead of the old monthly review is the whole point.
  • Runbook staleness from about one in three to under one in twenty: improved to roughly one in twelve, not fully there. Catching drift continuously beats an annual audit, but "continuously" still means someone reviews the flagged runbooks, and that review queue is not yet as fast as the detection.
  • Quarterly game day, ops-tooling fully disabled: run once, successfully. Paging and manual runbook execution worked exactly as before the platform existed, with no silent dependency discovered. One data point, not yet a track record.
  • Incident-triage diagnosis matching root cause often enough that on-call stops re-verifying by hand: this is the criterion this part was supposed to define a numeric threshold for, and now it has one: five hard constraints plus a score of 80 or higher, evaluated against the golden set. Six of seven cases cleared it on first measurement. The seventh was incident-003, which is exactly the point: the threshold's job is not to report a clean number, it is to catch the case that is not clean yet.

What we would do differently

Build the eval harness starting at Part 3, not Part 8. A golden set with even three cases, however thin, from the day the first agent shipped would have caught the diagnosis-quality regression that later became incident-003 before it ever reached a real incident, not after. Second, custom OTEL spans belong in an agent's own code from day one, not bolted on as a Part 8 IAM grant against a role that has been running for months without them; the gap between "the agent called a tool" and "the agent's summary matched what the tool returned" is exactly the gap that mattered here, and it should have been visible from the first deploy. Third, dead-letter queues on async hops should be a Part 4 concern, wired in alongside the Step Functions approval gate, not a pattern this part introduces on the one async hop it happens to own outright. None of these are large regrets. They are the ordinary cost of building the platform in the order that made sense to build it in, which is not always the order that would have caught every failure earliest.

Read this next

The full terraform/40-observability/ module and evals/, including the other six golden-set cases and the notifier Lambda trimmed from the snippets above, live in the companion repository at github.com/flightlesstux/agents-on-call. For the database and infrastructure side of running platforms like this one at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.