Agents on Call, Part 2. The Foundation: Terraform Before Tokens

Before any of the four agents from Part 1 can look at a log, price a workload, or propose a fix, this platform needs an account boundary and two IAM roles that make "read-only by default" a property enforced by AWS, not a promise made in a system prompt. That boundary, the model access request that has to happen days before anyone plans to demo anything, and the decision between on-demand, provisioned throughput, and cross-region inference profiles are what get built in this part, entirely in Terraform, before a single line of agent code exists.

Part 1 set the scene: a mid-size SaaS company with 30 AWS accounts and 40 pages a week, choosing to build a multi-agent ops platform on Bedrock AgentCore and Strands rather than buy one, with read-only enforced by IAM as the non-negotiable design constraint. This part is where that constraint stops being a diagram and becomes HCL. The companion code lives at github.com/flightlesstux/agents-on-call, in terraform/00-foundation/, and every snippet below is excerpted from that directory, not simplified for the post.

Why a dedicated ops-tooling account

The account layout from Part 1 puts the whole agent platform in one account, separate from the ~30 workload accounts it operates on:

A single dedicated account, instead of deploying agent infrastructure into each workload account, buys three things a shared account cannot. First, blast radius: a misconfigured Lambda or a leaked credential in ops-tooling cannot directly touch anything in a spoke, because reaching a spoke always requires an explicit sts:AssumeRole across an account boundary, not an implicit same-account permission. Second, one place to audit: CloudTrail in the security account sees every cross-account assume-role call from a single source account, instead of correlating agent activity scattered across 30 trails. Third, independent blast-radius for the platform itself: if ops-tooling has an incident, it degrades to the status quo everywhere else, exactly the additive-not-critical-path property Part 1 set as a requirement.

The two roles every spoke exposes

Every workload account exposes exactly two roles to ops-tooling, and nothing else. Both live in a reusable Terraform module, modules/spoke-roles/, meant to be applied once per spoke account: a provider alias per account works for a handful of spokes, and at roughly 30 accounts this runs from a per-account pipeline instead, for example one Terraform Cloud workspace per account or a Landing Zone Account Factory customization. The root module in this repo instantiates it once against a single provider so the example stays runnable with terraform validate.

ops-readonly: built from managed policies, capped by an explicit deny

The read tools (cloudwatch-read, logs-read, cost-read) all assume this one role. Its trust policy names the exact Lambda execution role ARNs allowed to assume it, plus a shared external ID as a second factor against the confused-deputy problem:

data "aws_iam_policy_document" "ops_readonly_trust" {
  statement {
    sid     = "AssumeFromOpsToolingReadTools"
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = var.ops_readonly_assumer_role_arns
    }

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [var.external_id]
    }
  }
}

resource "aws_iam_role" "ops_readonly" {
  name                 = "ops-readonly"
  assume_role_policy   = data.aws_iam_policy_document.ops_readonly_trust.json
  max_session_duration = 3600
}

Permissions come from two AWS managed policies: CloudWatchReadOnlyAccess covers both CloudWatch metrics and CloudWatch Logs reads in one attachment, which is why cloudwatch-read and logs-read can share a single IAM role instead of needing two, and AWSBillingReadOnlyAccess covers the Cost Explorer calls cost-read needs. On top of both, an explicit deny statement blocks a fixed list of mutating actions (iam:*, ssm:StartAutomationExecution, ec2:TerminateInstances, s3:PutObject*, and about a dozen more) regardless of what the attached managed policies grant. That last part is not decoration. AWS managed policies get new actions added over time as services add features, and an explicit deny is the one thing an IAM evaluation cannot override with an allow, no matter which policy, or which future version of a policy, produced it.

ops-mutate: one caller, one allowlist, no managed policy at all

The trust policy is narrower again: exactly one principal, the post-approval executor Lambda that only runs after a human clicks approve in Slack.

data "aws_iam_policy_document" "ops_mutate_trust" {
  statement {
    sid     = "AssumeFromPostApprovalExecutorOnly"
    effect  = "Allow"
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = [var.ops_mutate_assumer_role_arn]
    }

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [var.external_id]
    }
  }
}

The permission policy is the more interesting half. ssm:StartAutomationExecution supports resource-level restriction to specific SSM Automation document ARNs, so the allowlist of runbooks this role can run is enforced by IAM directly, not by application logic that could have a bug:

data "aws_iam_policy_document" "ops_mutate_permissions" {
  statement {
    sid       = "StartOnlyAllowlistedRunbooks"
    effect    = "Allow"
    actions   = ["ssm:StartAutomationExecution"]
    resources = var.allowed_ssm_automation_document_arns
  }

  statement {
    sid    = "InspectAndStopOwnExecutions"
    effect = "Allow"
    actions = [
      "ssm:GetAutomationExecution",
      "ssm:DescribeAutomationExecutions",
      "ssm:StopAutomationExecution",
    ]
    resources = ["*"]
  }
}

The second statement is on Resource = "*" for a specific reason: GetAutomationExecution and the other inspection calls operate on an execution ID that does not exist until StartAutomationExecution has already returned one, so there is nothing to scope them to in advance. The meaningful boundary on this role is which documents it can start, not which of its own already-running executions it can check on. And unlike ops-readonly, this role has zero AWS managed policy attachments. Every permission it holds is enumerated in that one policy document, so an AWS-side change to a managed policy can never silently widen what it can do.

Bedrock model access is a console-days problem

There is no Terraform resource for granting Bedrock foundation model access, and that gap is worth calling out before it costs a debugging session. Turning on a model for an account is a manual step, either the Bedrock console's "Model access" page or the equivalent one-off API call, that triggers an EULA acceptance workflow AWS processes asynchronously. It is not instant, and it is not idempotent infrastructure a plan and apply can express, so it cannot live in the same repo as the IAM roles above.

The practical effect: request access for every model this platform will call, in every region a cross-region inference profile can route to, before the first terraform apply, not after. A missing entitlement in one region of a multi-region profile fails only the invocations that get routed there, which makes the failure look intermittent, a flaky agent, a transient throttle, instead of what it actually is: one region out of several still waiting on a console click nobody made yet. Budget this as a day-one task, not a during-testing task, because the approval workflow itself is out of Terraform's, and this project's, control.

On-demand, provisioned, or cross-region: the decision rule is "on-demand until measured"

Three ways exist to call a Bedrock model, and picking between them before there is any traffic to measure is a guess dressed up as an architecture decision.

Cross-region inference profiles are the easy call: they cost nothing extra over on-demand and add burst capacity plus a failover path if one region throttles, so this platform uses them everywhere from day one. Provisioned Throughput is the harder call, and the rule this project uses is on-demand until measured: a Model Unit buys a fixed input and output tokens-per-minute ceiling that is billed hourly whether or not the platform sends a single request in a given hour, so committing to it before knowing the real request-rate distribution across four agents is committing to a number nobody can defend yet. Part 7 works through the actual token math and the point at which provisioned throughput's hourly rate beats on-demand's per-token rate; until that math exists, guessing a commitment level is just moving the risk from "the model is slow" to "the bill is wrong."

What Terraform actually creates here is not the cross-region profile itself, that part is AWS-managed and exists the moment a region supports it, but an application inference profile per agent that wraps the system-defined profile:

resource "aws_bedrock_inference_profile" "agent" {
  for_each = var.agents

  name        = "${var.platform_name}-${each.key}"
  description = "Application inference profile for the ${each.key} agent, tagged for per-agent cost allocation."

  model_source {
    copy_from = each.value.cross_region_profile_arn
  }

  tags = merge(var.tags, {
    Agent = each.key
  })
}

The Agent tag is the whole point of wrapping the system profile instead of calling it directly: without it, every agent's Bedrock spend lands in one undifferentiated line, and the cost-allocation-per-agent requirement from Part 1's architecture has nothing to key off. With it, the platform's own Budgets and Cost Anomaly Detection setup, which is meant to watch the agents watching everything else, can actually attribute spend to the agent that generated it.

The first terraform apply

Running terraform -chdir=terraform/00-foundation init then plan is a small plan by line count, two IAM roles, two inline policies, two managed policy attachments, and however many application inference profiles there are agents, but it is the one apply in this whole series that has to be right before anything downstream can be trusted. The committed root module instantiates spoke_roles once against the default provider, so this example applies into a single account to stay runnable with terraform validate and plan. A real deployment passes a spoke-scoped provider alias to that module instead; without it, the two roles land side by side in one account and the cross-account boundary this part is supposed to build does not exist. Every later part assumes ops-readonly and ops-mutate already exist with exactly this shape: get the trust policy's principal list wrong here, and a tool Lambda's AssumeRole call fails in Part 4 in a way that looks like a Gateway misconfiguration, not a Part 2 typo three weeks upstream.

One implementation detail that cost a few minutes the first time through: aws_iam_role.max_session_duration has a hard floor of 3600 seconds. The instinct for ops-mutate, given how narrowly scoped its blast radius already is, is to ask for the shortest possible session, but IAM does not allow a role-level ceiling under one hour. The actual credential lifetime the post-approval executor requests at runtime, via its own STS call, can and should still be much shorter than that ceiling; the role setting only bounds the maximum, it does not set the default.

State and backend hygiene

Nothing in this part touches a remote backend on purpose, because the point being made in this section is more important than the specific backend chosen: state has to live somewhere with locking and encryption before a second person, or a second CI run, ever applies against the same account, and it has to live outside anything an agent or a tool Lambda can reach. An S3 bucket with versioning and a DynamoDB lock table, or Terraform Cloud, both work; what matters is that ops-readonly and ops-mutate have no permission to read or write that state, and that the humans running terraform apply against this repo are a different, audited path from the platform the repo builds. Console drift is the other half of this discipline: production changes made by clicking around desync the state file from reality in ways that surface later as a confusing plan, not an immediate error, which is exactly the argument for treating IaC as the only path to production covered in more detail in IaC-First: Why We Never Touch the AWS Console in Production on ercan.cloud.

Failure modes to watch from this part specifically

Three things worth knowing before they show up as a confusing symptom two parts from now. A missing model entitlement in one region of a cross-region profile fails only requests routed to that region, so it presents as intermittent flakiness, not a clean error, until someone thinks to check the console access page instead of the code. A typo in ops_readonly_assumer_role_arns or ops_mutate_assumer_role_arn fails AssumeRole with an access-denied that gives no hint about which side, trust policy or caller, is wrong, so keep those Lambda role ARNs as Terraform outputs from wherever they're created and reference them, never retype an ARN by hand. And an empty allowed_ssm_automation_document_arns list is valid Terraform, the role applies cleanly, assumes cleanly, and then can never actually start anything; a plan-time validation block catches this specific case, but only because someone thought to add it, not because Terraform catches it on its own.

Read this next

• Part 1, The Scenario: Why an Ops Team Hires Agents, for the toil inventory, the stack decision, and the account layout this part builds in Terraform.
• Part 3, First Agent: Incident Triage in Strands, where the first Strands agent deploys onto AgentCore Runtime using the ops-readonly role built here.
• IaC-First: Why We Never Touch the AWS Console in Production on ercan.cloud, the state-drift and console-discipline argument this part's backend section leans on, from the general infrastructure side rather than the agent platform side.

The full terraform/00-foundation/ module, including the parts trimmed from the snippets above, lives in the companion repository at github.com/flightlesstux/agents-on-call. For the database and infrastructure side of running platforms like this one at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.