ercan.ai

Pick a Vector Database by What Breaks First

Wed, 29 Jul 2026 08:15:00 +0200

Almost every retrieval system should start with pgvector in the Postgres you already run, and the selection question is not "which vector database is fastest" but "which failure mode reaches me first". Published recall and QPS numbers converge across engines at the scales most teams actually operate. What does not converge is what happens when you add a tenant filter to every query, when the index needs a rebuild during business hours, or when the HNSW graph stops fitting in RAM. Those are the events that force a migration, and each engine fails at a different one.

One Door to the Models, Part 10: Observability, Governance, and the Numbers

Tue, 28 Jul 2026 09:00:00 +0200

The moment the gateway logs prompts and completions, it holds whatever a customer-service agent's user typed into a chat box, which may include an address, an account number, or a complaint about their health. Every other decision in this series traded cost against latency or control against convenience. This one trades debuggability against a data protection obligation, and it is the last one because it is the one that needs everything else already in place. Part 9 got the platform shipping. This part closes it: what to record, what to watch, what the regulation asks, and whether the numbers from Part 1 actually moved.

One Door to the Models, Part 9: Eval Gates and Model Retirement

Mon, 27 Jul 2026 09:00:00 +0200

In Azure Pipelines, an approval that is not completed within its timeout marks the stage skipped, not failed. A skipped stage is green in most dashboards and in most notification templates, so a model promotion that nobody looked at can complete a pipeline run that reports success while the change never went out, or worse, while a preceding stage already did half of it. Part 8 put applications on top of the gateway. This part gets the gateway itself into production repeatably, and turns the model retirement email from Part 1 into a database query.

One Door to the Models, Part 8: Orchestration on Top of the Gateway

Sun, 26 Jul 2026 09:00:00 +0200

The moment an application starts calling tools, the model's output stops being text to read and becomes an instruction to act, and every guarantee this series has built so far was about tokens. A framework can make that loop easy to write and, in the same motion, hide the throttling, reroute around the gateway's routing, and turn one metered request into eleven nobody planned for. Part 7 gave applications retrieval. This part is about what sits on top of it, and specifically about which jobs the framework should not be doing because the platform already does them.

One Door to the Models, Part 7: Azure AI Search or a Vector Database

Sat, 25 Jul 2026 09:00:00 +0200

Azure AI Search does not size by how many vectors you have. It sizes by tier: the vector index is bounded by the memory reserved for vector search on your SKU, and on the Serverless model by a hard 300 MB per index that fails the indexing job outright when exceeded. That single property, not a feature comparison, decides whether the company's retrieval layer belongs in Azure AI Search or in a dedicated vector database. Part 6 made repeated questions cheap. This part handles the far more common case, where the model needs facts it never saw, and the retail knowledge search from Part 1 finally gets built.

One Door to the Models, Part 6: Semantic Caching and Its Failure Modes

Fri, 24 Jul 2026 09:00:00 +0200

Microsoft's own policy documentation says a score threshold above 0.2 may lead to cache mismatch, which is a polite way of saying the gateway will answer a question the caller did not ask. Every other mechanism in this series has failed loudly: a 429, a 503, an expired batch job. A semantic cache fails by returning a fluent, well-formed, confident response that belongs to somebody else's prompt. Part 5 made spend attributable. This part makes it smaller, and treats the mechanism that does so as a correctness surface rather than a cost lever.

One Door to the Models, Part 5: Identity, Quota, and Chargeback

Thu, 23 Jul 2026 09:00:00 +0200

The chargeback design is decided by an Azure Monitor limit, not by an accounting preference: a custom metric allows 10 dimension keys, API Management already spends 5 of them on defaults, and the active time series a policy produces is the product of the unique values of every dimension you add. Three dimensions with ten values each is a thousand time series, against a cap of 50,000 active time series per region per subscription in a twelve hour window. Put a user ID or a request ID on that metric and the platform's own telemetry becomes the outage. Part 4 finished the three request paths. This part answers the question the whole series exists for: which team spent what, and how the gateway stops one of them from spending everyone else's quota.

One Door to the Models, Part 4: Async Work Off the Request Path

Wed, 22 Jul 2026 09:00:00 +0200

Azure's batch inference path does not look like an API call at all. You upload a JSONL file, create a job, and collect an output file up to 24 hours later, which means there is no per-request transaction for a gateway policy to sit in front of. Every guarantee Part 3 built, token limits, per-tenant metering, routing, circuit breaking, applies to requests through API Management. Batch has none of them by construction. This part moves the work that should never have been on the request path in the first place, and deals with the fact that doing so opens a second door into the models.

One Door to the Models, Part 3: The Provider Abstraction and Streaming

Tue, 21 Jul 2026 09:00:00 +0200

The gateway's guarantees are not properties of the gateway. They are properties of the transport the caller picked, and they get weaker with every step toward real-time. A plain request-response call is metered from the usage block the model actually returned. The same call with stream: true has its prompt tokens estimated whatever the policy says, and its completion tokens estimated too. A WebSocket connection cannot be load balanced across backends at all, because once established it is pinned one-to-one between client and backend. Part 2 built the infrastructure. This part puts an API on it, and the honest version of that API is one that tells each caller which guarantees it just gave up.

Enterprise MCP Auth: Agents Finally Get Service Accounts

Tue, 21 Jul 2026 00:30:00 +0200

MCP now has a specified way for an agent to authenticate without a human in the loop. SEP-1046 adds the OAuth client_credentials grant to the Model Context Protocol as an official extension, with private_key_jwt (RFC 7523 JWT client assertions) as the recommended client authentication method and plain client secrets allowed for compatibility. It was accepted in October 2025, and over the past months it has landed in the TypeScript, Python, and now Ruby SDKs. If you run agents in an enterprise, this is the piece of the puzzle you were waiting for, whether you knew its SEP number or not.

One Door to the Models, Part 2: Terraform, Bicep, or ARM

Mon, 20 Jul 2026 09:00:00 +0200

The most consequential line in this platform's Terraform is one nobody writes: version_upgrade_option on an Azure model deployment defaults to OnceNewDefaultVersionAvailable, so the deployment moves itself to a newer model version whenever Azure decides a new default exists. A gateway whose entire justification is owning the model lifecycle cannot leave that on a default. This part builds the infrastructure layer under Part 1's architecture, picks between Terraform, Bicep, and ARM on grounds that are actually load-bearing, and turns off the two defaults that quietly take the lifecycle back.

One Door to the Models, Part 1: The Case for a Central LLM Gateway

Sun, 19 Jul 2026 09:00:00 +0200

A large corporate IT organization has five generative AI applications in production, eleven model deployments spread over three Azure subscriptions, a monthly bill somewhere around 38,000 euros, and no reliable way to answer which of the five spent it. Nothing is broken. Every app works, every team shipped, every security review passed at the time. What does not exist is a single place where a token gets counted, a caller gets identified, or a model gets retired, and that absence is what this ten-part series builds: one governed GenAI gateway on Azure that every application goes through, and nothing goes around.

Agents on Call, Part 8. Production: Observability, Evals, and the Day It Lies

Thu, 16 Jul 2026 09:00:00 +0200

Three months after the incident-triage agent from Part 3 went live, it produced a confident, well-written, wrong diagnosis for a real incident, and nobody caught it until the postmortem, because the trace that would have caught it in real time did not exist yet. That is the shape of every failure this final part is actually about: not a crash, not an exception, a plausible sentence that happened to be false. This post closes the series by building the three things that turn "the agent said so" into something a human can verify, watches the specific day it lied, and scores the whole platform against the numbers Part 1 promised.

Kiro After the Hype: What AI IDEs Actually Changed

Wed, 15 Jul 2026 17:00:00 +0200

Eight months after Kiro reached general availability, the durable idea turns out to be the spec, and the least durable part turns out to be the IDE. Kiro shipped as an agentic IDE whose pitch was that you work at the specification level and let the agent implement. The part that stuck is the artifact: a written statement of intent that a human reviews before code exists. The part that quietly lost is the assumption that the editor is where this happens. Kiro shipped a CLI at GA, added headless mode, and the terminal is where the interesting usage went.

Why Your AI Pilot Died in Procurement

Mon, 13 Jul 2026 19:00:00 +0200

The pilot worked. That was never the question. The demo landed, the accuracy was defensible, the users liked it, and then it spent five months in a queue and quietly stopped being mentioned. Nobody killed it. It expired. If you have watched this happen twice, you have probably concluded that the organization is broken, and I want to argue for a less satisfying explanation: the pilot was scoped to answer a question nobody was blocking on.

Per-App Bedrock Cost Tracking with Inference Profiles

Fri, 10 Jul 2026 14:00:00 +0200

An application inference profile is a tagged wrapper around a model ARN, and it is the only thing standing between you and a Bedrock bill that says one number for the whole company. You create a profile that points at a model, attach cost allocation tags, and invoke the profile ARN instead of the model ID. The tags ride along to Cost Explorer and the Cost and Usage Report. That is the entire mechanism, and it takes an afternoon.

Agents on Call, Part 7. Sizing: Token Math Nobody Does Upfront

Thu, 09 Jul 2026 09:00:00 +0200

A single deep incident investigation on this platform, eight rounds of tool calls before the triage agent has enough evidence to propose a diagnosis, costs 32,950 tokens, and 31,600 of them are input, because Bedrock's Converse API resends the whole growing transcript on every round. Multiply that by real alarm volume and the platform's entire monthly on-demand model bill comes out to about $14, one of the smaller line items in a total monthly cost near $21. Buying Provisioned Throughput on day one, the thing Part 1 explicitly deferred, would have cost $15,768 a month for a workload running at roughly a four-hundredth of the utilization it needs to break even. None of that is obvious from a diagram. It only shows up once someone does the arithmetic, which is what this part does, end to end, with every number traceable to either a Bedrock price sheet or a formula shown in full.

EU AI Act, August 2: The Deadline That Didn't Move

Wed, 08 Jul 2026 11:00:00 +0200

The Digital Omnibus on AI moved the deadlines that made headlines and left alone the one that will actually affect your build. High-risk obligations slipped by more than a year. Article 50 transparency did not slip. It applies from 2 August 2026, which is roughly three weeks out, and it is the provision that catches ordinary teams doing ordinary things: generating text, images, or audio with a model and putting the output in front of a person.

SageMaker vs Bedrock: An Org Decision, Not a Technical One

Mon, 06 Jul 2026 09:00:00 +0200

SageMaker versus Bedrock is not a technical comparison. It is a question about whether your organization employs people whose job is to own models. If you have a team that trains, evaluates, and takes pager duty for a model artifact, SageMaker describes work they are already doing. If you do not have that team, and most companies shipping AI features do not, Bedrock is not a compromise. It is a correct reading of your org chart.

Prompt Injection via Your Own Docs: The RAG Attack Surface

Sat, 04 Jul 2026 10:00:00 +0200

The moment you put retrieval in front of a model, every document in your knowledge base becomes executable input. Not executable in the shell sense. Executable in the sense that a paragraph sitting in a Confluence page can change what your agent does, because retrieval will fetch it, paste it into the context window, and the model will read it with the same attention it gives your system prompt. Teams threat-model the user's message box carefully and then ingest 40,000 pages from six systems without asking who can write to them.

M365 Security 101: AI Pilot and Business Impact Reports

Fri, 03 Jul 2026 13:30:00 +0200

Security teams do not have a detection problem. They have a remediation problem and a communication problem, and those are the two places where AI in security actually pays for itself. A modern scanner will happily hand you 800 findings against a compliance benchmark. The findings are not the hard part. The hard part is that most of them never get fixed, and the ones that do get fixed are invisible to the people who approve the budget. This post is a 101 on the two AI features I think address that directly: automated remediation with an approval gate, and AI-generated reporting for non-technical readers. I will use Aether365, a Microsoft 365 security platform, as the running example. Full disclosure up front: Aether365 is my product. Judge the arguments on their merits.

Agents on Call, Part 6. Guardrails: The Part Everyone Skips

Thu, 02 Jul 2026 09:00:00 +0200

A Bedrock Guardrail attached to every agent invocation catches three things IAM cannot see at all: a prompt-injected instruction hidden inside a log line, a customer's PII arriving in a tool result before the model summarizes it, and an agent's own text steering a human toward skipping the approval gate. None of that is an access-control problem, so none of it shows up in an IAM policy, no matter how carefully Part 2 through Part 4 scoped one. This part builds that guardrail in Terraform, in terraform/30-guardrails/, and spends more time on what it does not catch than on the demo-friendly parts, because the gap between the two is exactly where an incident goes sideways.

Trust the Model, Audit the Binary

Thu, 02 Jul 2026 03:30:00 +0200

The client binary of a coding agent is the most privileged piece of software on your machine, and almost nobody audits it. It reads your repo, runs your shell, holds your credentials, and auto-updates on a schedule you do not control. We have collectively decided to treat that binary as boring plumbing while we argue about model alignment. Last week Claude Code demonstrated why that is exactly backwards: for roughly three months it shipped hidden logic that classified requests routed through China-linked proxies and encoded the result steganographically into its own system prompt. Nobody noticed until a developer decompiled the binary.

AWS Monthly (June '26): Agents Get a Feedback Loop

Tue, 30 Jun 2026 18:00:00 +0200

June 2026 was the month AWS stopped shipping agent capabilities and started shipping the loop that improves them. The Summit in New York on 17 June was the centre of it: AgentCore gained optimization capabilities that read production traces and tell you what your agents are getting wrong, Web Search went generally available, the managed harness reached GA a day later, and AWS Continuum arrived as an AI-native security service that earns permission to act rather than assuming it. Elsewhere in the month, Anthropic launched and then lost Claude Fable 5 to an export directive, which taught everyone building on a frontier model something uncomfortable about supply chains.

Agents on Call, Part 5. The Team: Supervisor and Three Specialists

Thu, 25 Jun 2026 09:00:00 +0200

Four agents now exist where Part 4 left one: a supervisor, a runbook agent, and a cost agent join incident-triage, coordinated not by Python function calls inside one process but by AgentCore Runtime's own InvokeAgentRuntime API, because all four still deploy as separate, isolated Runtime resources, the same isolation Part 3 chose AgentCore Runtime for in the first place. One more AWS resource ties them together: a single AgentCore Memory instance, shared across all four by actor ID, so a diagnosis triage writes at 3am is still legible to the runbook agent it hands off to seconds later, and to whichever agent looks at an incident shaped like this one again next month.

AWS Built a Sandbox for AI-Generated Code: Lambda MicroVMs

Tue, 23 Jun 2026 02:00:00 +0200

AWS Lambda MicroVMs are a new serverless primitive built, in AWS's own framing, to run code generated by users or AI in isolated, stateful environments. That second word is the one to notice. The launch is not really about functions. It is about giving an AI agent a safe place to execute the code it just wrote, and the fact that AWS led with "users or AI" tells you who this is for.

Your Multi-Agent System's Real Limit Is Tokens Per Minute

Sun, 21 Jun 2026 16:00:00 +0200

Amazon Bedrock now surfaces its Mantle endpoint's tokens-per-minute quotas in the standard AWS Service Quotas console. You can read the per-model input-tokens-per-minute and output-tokens-per-minute limits directly, and request increases through the same workflow you already use for everything else in AWS. That sounds like a small console change. For anyone running multi-agent systems in production, it is the difference between planning your capacity and discovering it as a wall of 429s.

Estonia Is Giving AI Agents an ID. That Is the Easy Part

Sat, 20 Jun 2026 16:00:00 +0200

Estonia intends to issue official digital identities, "AI ID codes", to AI agents. The office of Prime Minister Kristen Michal says the goal is to let AI "act on behalf of people, companies or organisations within clearly defined limits and in a manner that is both verifiable and auditable." If it ships, Estonia becomes the first country to give AI agents a state-recognised identity. That is a genuinely useful move, and it is also the easy part. The hard part is everything the ID is supposed to carry.

Le Chaton Fat: The Fattest AI Model That Never Existed

Thu, 18 Jun 2026 11:00:00 +0200

Let me save you a search: there is no model called Le Chaton Fat. No weights, no API, no benchmark. It is not a leak, it is not a roadmap, it is not a Mistral product. It is a joke, and over the last few days it became the funniest one the AI community has told itself in a while.

Because a few people genuinely went looking for the download link, I think the bit deserves a proper write-up. So here is what is going on with the fattest model that never existed.

Agents on Call, Part 4. Tools and the Gateway: MCP, Allowlists, Read-Only Default

Thu, 18 Jun 2026 09:00:00 +0200

Four tools now sit behind one AgentCore Gateway instead of being wired one Python function at a time into each agent: cloudwatch-read, logs-read, and cost-read assume a read-only role in a spoke account and can never mutate anything, and ssm-execute, the platform's only mutating tool, cannot reach a spoke directly either, it can only start a Step Functions execution that pauses for a human's Slack approval. That pause is not a UI nicety. It is the one place in the whole platform where an AWS credential capable of changing something in a spoke account gets minted, and it only gets minted after a person clicks approve.

AI Coding Agents Need Staging Environments Too

Fri, 12 Jun 2026 09:00:00 +0200

Every environment discipline your team built for humans applies to coding agents, and most teams quietly dropped all of it the moment the agent got good. The same organisation that would not let a new hire near production on day one will hand an agent a long-lived credential and a task description, then act surprised when something gets dropped that should not have.

The thesis is boring on purpose: an agent is an unusually fast contributor with no institutional judgment, and the environment ladder exists precisely for contributors like that. Dev, staging, production, with a gate between each. It was never about the human. It was always about the blast radius.

Agents on Call, Part 3. First Agent: Incident Triage in Strands

Thu, 11 Jun 2026 09:00:00 +0200

The first working agent in this series is about 260 lines of Python: a Strands Agent, two read-only tools that assume a cross-account IAM role before calling boto3, and a system prompt whose entire job is to stop the model from sounding confident about something it has not actually checked. No AgentCore Gateway yet, no supervisor, no multi-agent handoff: those come in later parts. This part is about getting one agent doing one job correctly, deployed to AgentCore Runtime, before adding anything that makes debugging it harder.

Claude Code in CI: Letting an Agent Fix the Build

Mon, 08 Jun 2026 14:00:00 +0200

Putting a coding agent in CI is a permissions design problem wearing a productivity costume. The mechanics are an afternoon's work: headless mode has shipped since Claude Code 2.0 in September 2025, and anthropics/claude-code-action@v1 wraps the whole thing in a GitHub Action. You can have an agent commenting on pull requests before lunch. Whether you should have it pushing commits is a different question, and it is the only one that matters.

Agents on Call, Part 2. The Foundation: Terraform Before Tokens

Thu, 04 Jun 2026 09:00:00 +0200

Before any of the four agents from Part 1 can look at a log, price a workload, or propose a fix, this platform needs an account boundary and two IAM roles that make "read-only by default" a property enforced by AWS, not a promise made in a system prompt. That boundary, the model access request that has to happen days before anyone plans to demo anything, and the decision between on-demand, provisioned throughput, and cross-region inference profiles are what get built in this part, entirely in Terraform, before a single line of agent code exists.

The Context Window Is Not Your Friend

Wed, 03 Jun 2026 10:00:00 +0200

A large context window is a capacity limit, not a retrieval strategy. The fact that a model accepts several hundred thousand tokens does not mean it reads them with even attention, and it certainly does not mean you should pay to send them. Every long-context benchmark that has bothered to measure position and length says the same thing: accuracy falls as input grows, and evidence buried in the middle of a prompt gets used least.

AWS Monthly (May '26): Agents Get a Wallet

Sun, 31 May 2026 19:00:00 +0200

May 2026 was the month AWS handed agents a wallet and hardened the toolchain that builds them. The headline was Amazon Bedrock AgentCore Payments in preview, the first managed way for an agent to autonomously pay for the APIs, content, and services it uses. Around it, AWS shipped the Agent Toolkit for AWS and took a managed Model Context Protocol server to general availability, both aimed at making AI coding agents build on AWS with fewer errors and tighter controls. Read together, the month says the agent story is moving past "can it reason" toward "can it transact, and can we trust the tools that build it."

Agents on Call, Part 1. The Scenario: Why an Ops Team Hires Agents

Thu, 28 May 2026 09:00:00 +0200

A mid-size B2B SaaS company running about 50 engineers across roughly 30 AWS accounts pages someone on-call around 40 times a week, and the page gets a real answer only after 25 to 35 minutes of manually pulling context: which account, which dashboard, which runbook, whether that runbook is even still accurate. That is before diagnosis starts, not instead of it. This series builds the fix: a small platform of AI agents that does that first pass automatically, reads everything it is allowed to read, changes nothing without a human's sign-off, and stops being useful the moment it stops being that disciplined.

Logging Prompts Without Logging PII

Wed, 27 May 2026 14:00:00 +0200

You cannot debug an LLM application without logging prompts, and you cannot keep those prompts if they are full of names, emails, and account numbers. Users type personally identifiable information straight into the box, so the log that helps you understand a bad response is also a growing store of regulated data sitting in CloudWatch or S3 with the wrong retention and the wrong access controls. The fix is not to stop logging. It is to redact before storage and to put a retention policy on what remains, so the debugging value survives and the liability does not.

Now

Mon, 25 May 2026 10:00:00 +0200

(Last updated: May 2026)

Writing

Publishing field notes here on AI, LLMs, and applied ML. Also maintain ercan.cloud for cloud and platform engineering, and short-form AI news at news.ercan.ai. Building awsmonthly.cloud, a monthly AWS news digest (not launched yet).

Consulting

Taking on a small number of AI and applied-ML consulting engagements. Bedrock architecture, LLM cost optimization, agentic pipeline design, interim AI platform lead. If what you are shipping overlaps with my writing, reach out on LinkedIn.

Consulting & Advisory

Sun, 24 May 2026 10:00:00 +0200

I take on a small number of consulting engagements each year. I like this work. It keeps me inside real production constraints, which is where the useful patterns come from. Every engagement feeds the writing on this site, and the writing feeds the engagements.

Services

Bedrock and LLM platform architecture. Cost-aware foundation for production LLM workloads. Model routing, inference profiling, guardrail design, multi-region failover. You ship features, not infrastructure.
LLM cost optimization. Most teams overpay 60-90% on Bedrock Knowledge Base, OpenSearch Serverless, and provisioned throughput because the pricing model is non-obvious. I find the waste, restructure the stack, and hand you the cost dashboard.
Agentic pipeline design. Moving from chat to agents. Tool-use architecture, agent loop design, human-in-the-loop checkpoints, tracing and observability for non-deterministic workflows.
S3 Vectors and RAG architecture. Native S3 vector search, chunking strategy, embedding model selection, retrieval evaluation. Skip the “just add a vector DB” reflex.

How I work

Project-based. You have a specific feature, pipeline, or migration. I build it alongside your team and transfer ownership with documentation.

About

Sat, 23 May 2026 10:00:00 +0200

I’m Ercan Ermis. Senior cloud platform engineer based in the Netherlands. I write here about AI, LLMs, agents, and the engineering work it takes to ship them.

How I got here

The first computer in my life was an Amstrad with two 5.25-inch floppy drives, bought by my father in 1986 for his business. The real switch flipped in 1998, fourth grade, when my teacher installed Linux on one of the Windows 95 machines in our school computer lab and said “this is Linux, it is free software.” Then Pac-Man appeared on that black screen and I was done. Why does this thing work, how does it work, what else can I make it do. More than thirty years later I am still asking those questions and still at the keyboard. Hours in front of a computer are still where I feel most comfortable. The AI work on this site is a continuation of the same curiosity, just with a new set of primitives.

When Haiku Beats Opus: Model Right-Sizing on Bedrock

Fri, 22 May 2026 11:00:00 +0200

Most production LLM traffic is not hard. It is classification, extraction, short rewrites, and routing, the kind of work a small fast model does correctly and cheaply. Yet the common pattern is to wire every call to the biggest available model, because it "just works," and then wonder why the bill and the latency are both high. On Amazon Bedrock, Claude Haiku 4.5 and Claude Opus 4.5 are both available, and the engineering win is not picking one, it is routing by task class so the cheap model handles the majority and the expensive one is an escalation path, not a default.

Agentic RAG Is Mostly Latency You Don't Need

Mon, 18 May 2026 16:00:00 +0200

Agentic RAG replaces a single retrieval with a loop: the model retrieves, reads, decides it needs more, retrieves again, and repeats until it is satisfied. Each hop is a full model round trip plus a search, and the hops are sequential because each one depends on the last. For most questions, this buys a marginally better answer at several times the latency, when one well-constructed query would have returned the same context in a single pass. Multi-hop retrieval is a real tool for a narrow class of questions, and a default that quietly triples your response time everywhere else.

Evals Before Agents: You Can't Ship What You Can't Score

Thu, 14 May 2026 13:00:00 +0200

If you cannot put a number on whether a change made your agent better or worse, you are not engineering it, you are guessing with extra steps. The most common reason agent projects stall is not a weak model, it is the absence of an eval harness. Without one, every prompt tweak, tool change, and model swap is evaluated by someone eyeballing a handful of outputs and declaring it "seems better," which is how you ship a regression and find out from a user. The scoreboard has to exist before the agent, because you cannot improve what you cannot score.

Semantic Caching: Two Different Questions, One Answer

Mon, 11 May 2026 09:00:00 +0200

A semantic cache answers "how do I reset my password" with the response it already generated for "I forgot my login," because the two mean the same thing. Instead of matching the exact string of a request, it embeds the request and looks for a stored request whose embedding is close enough, then returns that cached answer without calling the model. When it works, you skip a model call entirely: no tokens, no latency, no cost. When it misjudges "close enough," it serves a confident answer to a question the user did not ask.

Step Functions Is the Most Underrated Agent Orchestrator

Thu, 07 May 2026 15:00:00 +0200

Most of what people call an agent is a fixed sequence of steps with one or two model calls in the middle. The control flow is knowable in advance, but teams hand it to the model anyway, then spend weeks making a non-deterministic loop behave. When the shape of the work is known, the orchestrator should be deterministic, and AWS Step Functions is a better fit for that job than a hand-rolled agent loop. It already solves retries, timeouts, error handling, parallelism, and human approval, the exact things an agent framework asks you to reimplement.

LLM Gateways: Why Every Platform Team Builds One Eventually

Mon, 04 May 2026 10:00:00 +0200

The first LLM feature ships as a direct call from one service to Bedrock. The tenth ships the same way, from ten services, with ten sets of credentials, ten retry policies, and no single place to see what anyone is spending. That is the moment a platform team discovers it needs an LLM gateway, usually a quarter after it would have been cheap to build one. A gateway is the shared front door every model call passes through, and it exists to own the four things individual teams keep reinventing badly: authentication, quota, routing, and audit.

AWS Monthly (Apr '26): OpenAI Lands on Bedrock

Thu, 30 Apr 2026 18:00:00 +0200

April 2026 was the month the model menu on Amazon Bedrock changed shape. The headline was not a new AWS capability but a new tenant: OpenAI's frontier models, its Codex coding agent, and OpenAI-powered Managed Agents arrived on Bedrock in limited preview. Alongside it, AgentCore spent the month lowering the effort it takes to get from an idea to a running agent. Read together, the two stories say the same thing. Bedrock is positioning itself as the neutral place enterprises run whichever frontier model they want, and AWS is racing to make the surrounding developer workflow fast enough that the choice of model is the only decision left.

Cross-Region Inference: Cheap Resilience or Residency Trap?

Sun, 26 Apr 2026 13:00:00 +0200

Amazon Bedrock cross-region inference gives you higher effective throughput and fewer regional throttling errors at no routing surcharge, which is close to free resilience. The trap is that a global inference profile can send your prompt to whichever region has capacity, and if that prompt carries regulated data, "wherever there is capacity" is not an answer your compliance team will accept. The feature is genuinely useful. Whether it is a win or a violation depends entirely on which kind of inference profile you pick, and that choice is easy to make without reading what it means.

Your LLM Bill Is an Observability Problem

Tue, 21 Apr 2026 15:00:00 +0200

When your Amazon Bedrock bill jumps and nobody can say which feature caused it, you do not have a pricing problem. You have an observability problem. The invoice tells you the account spent more on tokens. It does not tell you which agent, which tenant, or which code path did the spending, and without that attribution every cost conversation is a guess. You cannot optimise what you cannot measure, and most teams are measuring the total and nothing underneath it.

Batch Inference on Bedrock: Half Price If You Can Wait

Fri, 17 Apr 2026 11:00:00 +0200

Amazon Bedrock runs batch inference at 50 percent of on-demand token pricing, and the only thing you give up is immediacy. You submit a file of requests, the job runs asynchronously when there is capacity, and you collect the results later. For any workload where no human is sitting there waiting on the response, paying full price for real-time inference is leaving half the money on the table for a speed nobody needed.

Multi-Tenant LLM Apps: Isolating Customers on a Shared Model

Mon, 13 Apr 2026 16:00:00 +0200

When many customers share one Bedrock model, the model gives you no isolation and was never going to. It is a stateless function: same input, same behavior, no memory of who called it. Every boundary between tenants has to be built in the layers around the model, not expected from it. Teams get this backwards, assume the managed service handles separation, and ship an app where one tenant's data, cost, and load bleed into another's. The shared model is fine. The shared everything-else is the problem.

Agent Memory Is a Database Problem, Not a Prompt Problem

Thu, 09 Apr 2026 09:00:00 +0200

An agent that remembers by pasting its entire history into the next prompt does not have memory. It has a growing bill, a hard token ceiling, and a latency curve that gets worse with every turn. Real memory is a storage decision: what you persist, where you put it, and how you fetch only the relevant slice back at inference time. That is a database problem, and treating it as a prompt problem is how agents that demo well fall over in week two.

Structured Output Beats Clever Parsing

Mon, 06 Apr 2026 14:00:00 +0200

If your application still pulls JSON out of model prose with a regex and a retry loop, you are solving a problem Amazon Bedrock now solves at the decoding layer. Structured outputs, generally available on Bedrock since February 2026, constrain the model to a JSON Schema while it generates tokens, so the response conforms to your shape by construction rather than by hope. The regex was never the fix. It was the symptom of asking a model to "please return JSON" and then cleaning up when it did not.

Prompt Caching on Bedrock: The 90% Discount Most Teams Ignore

Thu, 02 Apr 2026 10:00:00 +0200

Amazon Bedrock prompt caching reads a cached prefix at roughly a 90 percent discount, but a cache write costs more than a normal input token, so a cache that never gets a hit makes your bill worse, not better. The feature has been generally available since April 2025, and the one-hour cache duration that shipped in January 2026 makes it useful for whole sessions and batch jobs. Most teams still leave it off, or turn it on in the wrong place and quietly pay a premium. The discount is real. Whether you capture it depends entirely on where you put the cache breakpoint.

AWS Monthly (Mar '26): Governance Comes for the Agents

Tue, 31 Mar 2026 20:00:00 +0200

March 2026 was the month AWS stopped shipping agent capabilities and started shipping agent controls. The headline launches were not new models or flashier demos. They were the boring, load-bearing pieces you need before an agent is allowed near production: an authorization layer, a quality-evaluation layer, and a healthcare stack that had to be governed to exist at all. The pattern across the month is the same one every technology hits when it grows up. The interesting work moves from "can it do the thing" to "can you prove what it did and stop it doing the wrong one."

Streaming Responses Are a UX Decision, Not a Performance One

Tue, 24 Mar 2026 15:00:00 +0200

Streaming does not make your model faster. It makes the wait feel shorter. The total time to generate a response is almost identical whether you stream it or not. What streaming changes is when the user sees the first token, and that single number, time to first token, drives the entire perception of speed. Treat streaming as a UX decision, because that is what it is, and you will make better calls about when to use it and when it actively hurts.

Bedrock Agents vs Rolling Your Own Loop

Wed, 18 Mar 2026 10:00:00 +0200

A managed agent framework trades control for speed, and the trade is worth it right up until the day it is not. Amazon Bedrock Agents will run the reason-act loop for you: it plans, decides which tool to call, invokes it, feeds the result back to the model, and repeats until the task is done. That is real work you do not have to write. The question is not whether it saves you time. It is whether the parts it hides are parts you can afford to stop seeing.

IAM for LLM Apps: Least Privilege When the Caller Is a Model

Sat, 14 Mar 2026 17:00:00 +0200

An agent is a caller you cannot fully predict, which is exactly the caller that should hold the least privilege. The instinct with a new agent is to hand it a broad role so it "just works" while you iterate. That instinct is how you end up with a language model holding credentials that can read every bucket and delete every table, driven by text an attacker can influence. Least privilege was always the rule. A non-deterministic caller makes it non-negotiable.

Someone Registered antrophic.com and Points It Straight to OpenAI

Wed, 11 Mar 2026 12:34:48 +0300

A typo-squatted domain, a suspicious redirect, and a little too much to be called a coincidence.

Here's the real domain: anthropic.com. The AI safety company behind Claude, founded by former OpenAI researchers, doing genuinely important work on making AI systems safe and interpretable.

Now try mistyping it. Drop the second "h". You get antrophic.com, a domain that exists, is registered, and according to multiple reports, redirects directly to openai.com.

Let that sink in for a second.

Stop Fine-Tuning. You Need RAG, a Cache, and Better Prompts

Mon, 09 Mar 2026 11:00:00 +0200

For most teams reaching for fine-tuning on Amazon Bedrock, the correct answer is retrieval, a prompt cache, and better prompts, in that order. Fine-tuning is the tool you consider after those three are exhausted, not before. The reason is not ideology. It is the bill. A custom fine-tuned model on Bedrock has to be served through Provisioned Throughput, and that pricing model changes the economics of your whole application.

Fine-tuning feels like the serious move. You have proprietary data, you want the model to "know" it, so you train. But most of what people fine-tune for is not knowledge the weights need to absorb. It is context the model needs at inference time, formatting it can be told to follow, and instructions that were never written clearly. All three have cheaper solutions.

Knowledge Base Chunking Is Where Your RAG Quality Dies

Thu, 05 Mar 2026 14:00:00 +0200

When a RAG system gives a wrong or half-right answer, the model is usually not the reason. The chunking is. If the passage that holds the answer never makes it into the retrieved context, no model can answer from it, and no amount of prompt tuning changes that. Chunking decides what can be retrieved at all, which makes it the first thing to inspect and the last thing most teams look at.

Bedrock Guardrails Won't Save You From Prompt Injection

Tue, 03 Mar 2026 09:00:00 +0200

Amazon Bedrock Guardrails is a content filter, not a security boundary. It classifies text against topic, toxicity, and PII policies and blocks what crosses a threshold. That is genuinely useful for keeping a customer support bot from talking about a competitor or leaking a phone number. It is not what stops a prompt injection from turning your agent into a confused deputy, because prompt injection is an authorization problem and Guardrails does not authorize anything.

Cutting Amazon Bedrock Knowledge Base Costs by ~90%: Migrating from OpenSearch Serverless to Aurora Serverless v2 with pgvector

Sat, 21 Feb 2026 21:30:00 +0300

TL;DR

If you are running an Amazon Bedrock Knowledge Base backed by OpenSearch Serverless (AOSS), you are paying a ~$700/month floor before you ingest a single document. For most small and mid-sized RAG workloads, swapping AOSS for Aurora PostgreSQL Serverless v2 with the pgvector extension drops that floor to under $50/month a ~90% cost reduction; while remaining a fully supported, first-class vector store for Bedrock Knowledge Bases.

This post walks through the why, the math, the trade-offs, and the migration path, using a real production infrastructure as the reference implementation.

AWS Monthly (Dec '25): The Kiro Era Begins

Wed, 31 Dec 2025 20:23:45 +0300

We ended the year with the General Availability of Kiro (Frontier Agents). Kiro is not just a chatbot; it’s a Virtual Software Development Team.

Technically, Kiro is an Autonomous Coding Agent. You can assign it a ticket from Jira, and it will:

Pull context from your Git repos.
Draft the implementation in a secure sandbox.
Run the unit and integration tests.
Submit the Pull Request for your review.

It understands the context of your entire codebase, not just a single file. We also saw the launch of Database Savings Plans, which finally aggregate your spend across RDS, Aurora, and DynamoDB into one flexible commitment.

AWS re:Invent 2025: The "Agentic" Era

Sun, 14 Dec 2025 20:25:19 +0300

If 2024 was about talking to LLMs, re:Invent 2025 was about letting them actually do the work. Here is the comprehensive breakdown of the most significant announcements.

1. The Amazon Nova 2 Model Family

AWS didn't just update their models; they built a specialized fleet for different agentic roles:

Nova 2 Lite: Optimized for speed and cost. Equal or better than Gemini Flash 2.5 on 14/18 benchmarks.
Nova 2 Pro: The "Reasoning" heavy-lifter. Best for complex multi-step tasks and long-range planning.
Nova 2 Sonic: A speech-to-speech model for low-latency conversational AI.
Nova 2 Omni: The true multimodal star. It processes text, images, video, and speech simultaneously with a 1M token context window.
Nova Act: Generally Available and purpose-built for UI automation (browser-based tasks) with >90% reliability.

2. Custom Silicon: Graviton5 & Trainium3

The hardware story was about decoupling performance from cost:

AWS Monthly (Oct '25): Industrializing AI Training

Fri, 31 Oct 2025 20:18:03 +0300

October was the "Quiet before the re:Invent storm," but it brought the unveiling of Project Rainier.

Project Rainier is a massive AI compute cluster featuring over 500,000 Trainium2 chips. Technically, this is one of the largest dedicated AI training environments on the planet. For us, it means the "Industrialization of AI." The price-to-performance ratio of Trainium2 via the Neuron SDK is now significantly better than comparable GPU instances for transformer workloads.

AWS Monthly (June '25): S3 Becomes Your Vector DB

Mon, 30 Jun 2025 20:08:00 +0300

June saw a tectonic shift in the AI data stack with the preview of Amazon S3 Vector Search.

For the last two years, we have been told we need a specialized vector database (Pinecone, Milvus, etc.) for Retrieval-Augmented Generation (RAG). AWS simplified the equation: "Just store your vectors in S3." Technically, this adds a native vector indexing layer to S3 buckets. You can store embeddings as metadata on your objects and perform KNN (K-Nearest Neighbor) searches directly via API.

AWS Monthly (Feb '25): Automated Code Evolution

Fri, 28 Feb 2025 19:49:00 +0300

February was the month Amazon Q Developer stopped being a "helper" and started acting like a "senior engineer." The headline was the launch of Q-driven Refactoring.

This isn't just basic code completion. Technically, Q now uses a combination of Symbolic AI and LLMs to perform full repository dependency mapping. It can autonomously handle complex migrations, such as refactoring a legacy Java 8 microservice to Java 21. It identifies deprecated libraries, suggests modern replacements, and rewrites boilerplate to align with modern design patterns, such as Hexagonal Architecture.