Skip to content

Blog

Deep dives into building production AI systems on AWS. No fluff, just real code, real gotchas, real architecture decisions.

Series · 7 parts

Production AI, Honestly

1

Field Notes: Three things I learned diagnosing a production Bedrock workload

Three findings from a real customer engagement on AWS Bedrock: what a load test was actually doing, why p95 latency was 45 seconds, and the prompt-caching default that costs every team money. Plus the three CloudWatch metrics that catch all three.

8 min read
2

Field Notes: Turning prompt caching on for a production Bedrock workload

Strands' BedrockModel ships with prompt caching off. Two kwargs turn it on, one per-model gotcha catches you, and a 10-turn driver measures 99.9% and 99.8% hit ratios against an 8,156-token production system prefix. The usage block proves it in seconds.

9 min read
3

Every dashboard was green while the agent burned six figures a year

The most expensive AI agent failures don't throw an error, they hide. One ran at a six-figure-a-year rate for days while every dashboard stayed green, because the signals that catch it are the ones nobody watches. The two instruments you are missing.

6 min read
4

Field Notes: The AgentCore Memory write that returns success and reads back empty

AgentCore long-term memory has a read-after-write gotcha the docs skip: a direct BatchCreateMemoryRecords write returns 201 and stays unsearchable for 15 to 30 seconds. Measured, with the two-tier model that explains it.

8 min read
5

The LLM is not a security boundary

Designing a production agent over sensitive data: no control makes the flow hole-free. You rank the layers, assume each one leaks, and stack them so no single hole reaches the data. Here is the code that does it.

13 min read
6

Your LLM security diagram defends the wrong layer

The LLM security diagram you have seen a dozen times is a threat map. Read as a defence it makes you patch every box at the layer the attacker controls. The fix is one deterministic boundary the diagram leaves out, in code the model never touches.

9 min read
7

Field Notes: A Gateway policy can't see the user if your agent calls with its own token

A re:Post question asked how to stop an AgentCore agent acting beyond the user who asked. I built a demo to find out. A Cedar policy does nothing if the agent calls with its own token. Two gotchas, and the real limit: a stale token claim.

7 min read
Series · 4 parts

Evals for Production AI

Series · 3 parts

Building an AWS DevOps Agent that Knows Your Org

Series · 6 parts

Ultimate Guide to Building AI Agents on AWS with Bedrock AgentCore

1

Part 1: Why I Chose Amazon Bedrock AgentCore (And What Lambda Gets Wrong for AI Agents)

Before writing a single line of agent code, I spent a week figuring out where to run it. Here's the architecture decision that changed everything, and the Lambda limitations that forced my hand.

9 min read
2

Part 2: CDK Infrastructure for Amazon Bedrock AgentCore (And Every Gotcha You'll Hit)

A complete CDK v2 TypeScript stack for Bedrock AgentCore, with inline comments for every deployment trap: naming constraints, ECR bootstrap, missing L1 constructs, VPC endpoint conflicts, and more.

14 min read
3

Part 3: Building the AI Agent with Strands Agents SDK, Prompt Caching, and AgentCore Memory

How to build the Python agent that runs inside AgentCore: Strands SDK setup, prompt caching that cuts costs by 90%, dual-model strategy, tool definitions, and AgentCore Memory integration.

11 min read
4

Part 4: Running Your AgentCore Agent Locally with Docker (The Right Way)

How to build and run your AgentCore container locally with real AWS credentials, the correct linux/amd64 platform flag, the .env.local pattern, and how to test with curl.

7 min read
5

Part 5: CI/CD for Bedrock AgentCore with GitHub Actions and AWS OIDC (No Stored Credentials)

How to build a complete CI/CD pipeline for AgentCore using GitHub Actions OIDC: no stored AWS keys, dual-tag ECR strategy, automated Runtime updates, and multi-environment promotion.

10 min read
6

Part 6: Cost & Performance for Bedrock AgentCore: Prompt Caching, Model Selection, and CloudWatch Alarms

Real cost breakdown of running an AgentCore agent: prompt caching savings, when to use Nova Pro vs Claude Sonnet, PriceClass_100, idle timeouts, and how to set alarms before your bill surprises you.

9 min read

All Posts

Field Notes: Where a real production hit ratio sits against the prompt caching break-even point

The prompt caching break-even formula exists already, against illustrative numbers. This post checks it against a real measured production hit ratio, 99.8%, and flags where the pricing it rests on may not carry over to Bedrock's own rate.

Field Notes: 25 seconds to detect and hand off. Nothing executes until a human says yes.

Two agents, on two different frameworks, detect a real AWS incident and hand it off over A2A. One proposes a fix. Nothing executes until a human approves, enforced by Cedar.

LangGraph builds the agent. AgentCore runs it.

LangGraph and AgentCore are different layers, not rivals. Who owns each design primitive, and why your runtime decides how hard checkpointing works.

The ML you need to operate LLMs, not train them

You do not need to train a transformer to run one well. What tokens, context budget and sampling knobs actually mean, verified against the current Bedrock Converse API, including a temperature/top_p gotcha that silently breaks on newer Claude models.

The harness is one integer column

The fix for a poison-pill row re-billing a model every sixty seconds forever was not a rewrite. It was one INT column, capped at five, incremented atomically in the database. The fix itself shipped with a gap, and that gap is the actual lesson.

Introducing the AI Architect Roadmap

Eight rungs mapping what shipping production Agentic AI on AWS actually takes, marked honestly as covered, strong, flagship or gap rather than filled in to look finished.

Not a Python tutorial: the patterns that bite in production agents

The senior-lens Python patterns that actually bite in a production agent loop: why boto3 blocking the event loop is the first one, and four more that follow from it.

You probably don't need to fine-tune

A post framed fine-tuning as a 2026 interview trap. Here is the AWS version: the ladder before you touch weights, the four job types and three serving paths Bedrock forks 'fine-tune' into, and what each one costs, priced today.

What a Year 10 study system taught me about production AI failure modes

A personal Bedrock-adjacent build that went through three iterations and an architecture pivot. Five lessons that map directly to production AWS AI work.