-
My Uptime Monitor Was Set to Every 5 Minutes. In July It Ran 15 Times a Day.
My GitHub Actions health check was configured for 288 runs a day. In July it ran 15. It had been degrading every month since January and nothing told me, because a scheduled run that never happens looks exactly like a scheduled run that found nothing wrong. — read more
-
Nobody Alerts on Silence: Wiring Sentry Into an LLM Pipeline
My LLM router silently fell back to a CPU container and burned an hour at 390% CPU with zero errors. Here is how I made silence audible — a route-reason span attribute, a throttle that turns 106 events into 1, and why I skipped Sentry’s OpenTelemetry bridge. Then, on day one in production, it caught… — read more
-
My Backup Leaked 156 GB, Filled the Disk, and Broke Itself. The Fix Was One Letter.
The nightly backup verify started flapping with “postgres did not become ready”. df -h said 61% used. The real disk — the one Docker lives on — was at 100%: 57 orphaned pgdata volumes, one leaked per run, for 34 days. The fix: docker rm -f → docker rm -fv. — read more
-
Stuck at “Preparing… 0/0” Forever. My State Machine Had No Exit.
Some books stayed at “Preparing… 0/0” forever. The status Indexing had no way out: not Failed, so no retry; not Ready, so no answers. The fix: 202 early, an atomic claim, and a sweep that gives every state an exit. — read more
-
Nothing Crashed. Nothing Errored. My CPU Sat at 390% for an Hour.
A user PDF stuck at “Preparing… 0/0”, a CPU-only container at 390%, and zero traffic to OpenAI. The bug: a silent fallback after a reliability fix. The fix: one config block. — read more
-
Your RAG Eval Isn’t Flaky. Your Retrieval Is Non-Deterministic.
Same query.Same documents.Same model.And the RAG eval can still hand back a different Recall@8. Not because the model is flaky. Because of an ORDER BY clause. I didn’t find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model… — read more
-
AI Wrote a Thread-Safe Counter. The CPU Made It 5x Slower.
Four threads, no locks, no shared state, no bugs — and still 5x slower than it should be. The code was correct. The memory layout was not. A 30-line C# demo of false sharing, and why AI-generated code makes this knowledge more valuable, not less. — read more
-
An AI Feature Has No “Tests Pass” Moment. So I Write the Eval First.
A reader on chapter 3 must never get spoilers from chapter 30. That one rule taught me why, for AI features, the eval is the spec — and why I write it before the feature. — read more
-
The AI Engineer Interview Is a Backend Interview: 18 Real Questions, Answered for .NET Developers
Open any “AI engineer interview prep” guide and you’ll find the same thing: prompting tricks, model trivia, and Python code. Here’s what the guides miss. When you look at what companies actually ask in 2026 — design a RAG system, debug retrieval failures, keep latency under 800ms, build LLM-as-judge evals — these are not prompt… — read more
-
From a Number to a Gate: Evals in CI and Production
Part 5, the finale, of a series on building production AI on .NET. We’ve built the pieces — what evals are, error analysis, golden datasets, and a trustworthy judge. Now we make them earn their keep. By now you can produce a defensible quality score for an AI feature. But a score you only look… — read more