Amazon Bedrock runs batch inference at 50 percent of on-demand token pricing, and the only thing you give up is immediacy. You submit a file of requests, the job runs asynchronously when there is capacity, and you collect the results later. For any workload where no human is sitting there waiting on the response, paying full price for real-time inference is leaving half the money on the table for a speed nobody needed.

The mistake is defaulting every model call to the synchronous, real-time path because that is how the first prototype was written. Interactive chat has to be real-time. A nightly job that classifies yesterday's support tickets does not. Those are different latency requirements, and Bedrock prices them differently. The engineering question is simply which of your workloads actually need an answer now versus which need an answer eventually.

How batch works on Bedrock

Batch inference is a job, not a call. The flow is deliberately boring:

1. write requests as JSONL to S3   (one record per line)
2. create a batch inference job     (input S3 -> output S3)
3. job runs asynchronously          (minutes to hours)
4. read results from the output S3 prefix

Each input record carries a record ID and the same model input you would send in real time. The output is written back to S3, one result per input, keyed by that record ID so you can join results to inputs. There is no endpoint to keep warm, no concurrency to tune, no throttling to catch. You trade the request-response loop for a submit-and-collect loop, and you get the discounted rate for accepting that the job finishes on the service's schedule, not yours.

Where latency genuinely does not matter

The workloads that fit are the ones where the result feeds a process, not a person waiting on a screen:

  • Bulk classification and tagging. Categorize a backlog of documents, tickets, or products. The answer lands in a database, not in front of a user.
  • Enrichment pipelines. Summaries, extractions, or embeddings generated ahead of time and stored, so the real-time path just reads a precomputed value.
  • Offline evaluation. Scoring a model or prompt change across thousands of test cases. It is a report, and reports can wait an hour.
  • Periodic reporting. Anything that runs on a schedule and produces output a human reads later, not instantly.

The common thread: the deadline is measured in hours, and the volume is large enough that halving the token price is real money rather than rounding error.

The cost math

The trade is stark when you put it on paper. Take a job of one million records, each averaging a fixed token cost:

Real-time path:
  1,000,000 requests x on-demand token price
  + endpoint kept responsive
  + throttling handling under load

Batch path:
  1,000,000 records x (0.5 x on-demand token price)
  + S3 storage (negligible)
  + the willingness to wait

The batch column is half the token bill and less operational surface, because there is no live endpoint to protect from a burst. The break-even is not about volume, it is about the deadline. If the answer can wait, batch wins on both cost and simplicity. If it cannot, no discount makes an asynchronous job acceptable.

When batch is the wrong tool

Do not force it where the shape does not fit. Batch is wrong when a user is waiting on the output, when one request depends on the result of the previous one, or when the job is small enough that the discount is trivial and the async plumbing is not worth it. It is also wrong as a throttling workaround: if your real-time traffic is getting throttled, the fix is capacity planning and prompt caching, not shoving interactive requests through a job that returns hours later. Batch is for work that was always going to be deferred, not for hiding a latency problem.

The takeaway

Half of a large token bill is worth an architecture that most teams already have lying around: write to S3, submit a job, read the results later. The discount is not clever, it is just paying for deferred capacity instead of on-demand capacity. Audit your model calls and sort them by who is waiting. Everything with a human on the other end stays real-time. Everything else, the classification, enrichment, evaluation, and reporting that feeds a pipeline rather than a person, belongs in a batch job at 50 percent off. The savings are sitting in the workloads you never questioned.

Read this next

For the wider cost-optimization and pipeline playbook across AWS, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.