#genai

I deliberately broke my summariser's prompt, then failed to detect it two different ways: with an LLM judge over a golden dataset, and with a judge-free deterministic assertion. Removing the judge changed nothing. Here is the calculation that would have told me before I started.

I built an LLM-as-judge eval on my own blog and got a suspiciously perfect 16/16. Here's the three-round test I ran before trusting that number: single-variable corruption, and a self-consistency check the research says most teams skip.

The LLM security diagram you have seen a dozen times is a threat map. Read as a defence it makes you patch every box at the layer the attacker controls. The fix is one deterministic boundary the diagram leaves out, in code the model never touches.

Designing a production agent over sensitive data: no control makes the flow hole-free. You rank the layers, assume each one leaks, and stack them so no single hole reaches the data. Here is the code that does it.