If you cannot put a number on whether a change made your agent better or worse, you are not engineering it, you are guessing with extra steps. The most common reason agent projects stall is not a weak model, it is the absence of an eval harness. Without one, every prompt tweak, tool change, and model swap is evaluated by someone eyeballing a handful of outputs and declaring it "seems better," which is how you ship a regression and find out from a user. The scoreboard has to exist before the agent, because you cannot improve what you cannot score.

The reframing: an eval set is your test suite for probabilistic code. You would not refactor a service with no tests and ship on the strength of it looking fine. An agent is harder to reason about than that service, not easier, so it needs the scoreboard more, not less.

Build the harness before the agent

An eval harness is unglamorous and small to start: a set of representative inputs, a definition of a good output for each, and a way to run the current system against all of them and get a score. Twenty to fifty real examples, drawn from actual or expected traffic, beats a thousand synthetic ones. The point is a fixed yardstick, so that "we improved the agent" becomes a claim you can check instead of a feeling.

Once it exists, the workflow inverts. Every change, new prompt, new tool, new model, runs against the harness first. A change that raises the score ships; a change that lowers it does not, no matter how clever it looked. On Bedrock, Model Evaluation gives you a managed path for this, with automatic metrics and human review, and it went generally available well before you would need it here. The tool matters less than the discipline: score, then decide.

LLM-as-a-judge, and where it lies to you

Scoring open-ended outputs by hand does not scale, so the standard move is to use a model as the judge: give a strong model the input, the agent's output, and a rubric, and ask it to grade. Bedrock Model Evaluation supports exactly this. It is genuinely useful and it is also a component that can be wrong, which teams forget the moment it starts emitting tidy numbers.

The judge has predictable failure modes, and you have to design around them:

  • Position and verbosity bias. Judges tend to prefer the first option shown and the longer, more confident-sounding answer, independent of correctness. Randomize order and control for length.
  • Self-preference. A judge from the same model family can rate outputs from its own family higher. Where it matters, use a judge from a different family than the model under test.
  • Rubric drift. A vague rubric ("is this answer good") produces vague, unstable scores. A specific rubric ("does the answer cite the policy section, and is the cited section correct") produces scores you can act on.
  • Confident agreement with a wrong answer. If the input contains a false premise, the judge may reward an output that goes along with it. Include adversarial cases where the correct behavior is to refuse or correct.

The judge is not ground truth. It is a fast, cheap approximation of ground truth that you calibrate against a small human-labeled set. If the judge and your humans disagree on that set, fix the rubric before you trust the judge on the rest.

Score the trajectory, not just the answer

For agents specifically, the final answer is only half of what you care about. An agent can produce a right answer by a wrong path: calling a tool it should not have, spending ten steps on a two-step task, or leaking data into a log along the way. Evaluate the trajectory as well as the output. Did it use the expected tools? Did it stay within its allowed actions? Did it finish in a sane number of steps? An agent that gets the right answer while calling a delete API it was never supposed to touch has failed the eval, whatever the final text says.

The takeaway

The order is not negotiable: eval harness first, agent second. The harness is a small, fixed set of real inputs with defined good outputs and a repeatable score, and it turns every change from a vibe check into a measured decision. Use an LLM-as-a-judge to scale the scoring, but treat it as a fallible component: control for its biases, give it a specific rubric, and calibrate it against human labels. Score the path, not only the result. You cannot ship what you cannot score, and the teams shipping reliable agents are the ones who built the scoreboard before they built the player.

Read this next

For the platform side of wiring evals into a delivery pipeline, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.