Vasyl’s Dev Notes

Open any “AI engineer interview prep” guide and you’ll find the same thing: prompting tricks, model trivia, and Python code.

Here’s what the guides miss. When you look at what companies actually ask in 2026 — design a RAG system, debug retrieval failures, keep latency under 800ms, build LLM-as-judge evals — these are not prompt questions. They are backend engineering questions with an unreliable dependency in the middle.

That’s good news if you come from backend. The model is the smallest part of the system. Everything around it — contracts, retries, observability, deployment — is the engineering you already know.

One more thing: almost every answer online is written in Python. I’m a .NET engineer. So every answer below uses the .NET stack — ASP.NET Core, Polly, OpenTelemetry, Semantic Kernel. The patterns transfer to any language, but if you interview as a C# developer, these are the tools to name.

Let’s go through the questions, grouped the way interviews actually flow: design it, keep it alive, prove it works, ship it.

Part 1: “Design it” — RAG and system design

1. Design a RAG system for a customer support chatbot

This is the most common opening question in AI interviews right now. The expected flow:

documents → chunking → embeddings → vector store → retrieval → re-ranking → LLM → answer with citations

Don’t just draw the boxes. Interviewers listen for trade-offs:

• Chunking: small chunks retrieve precisely but lose context; large chunks keep context but dilute relevance. Start around 300–800 tokens with overlap, then tune against an eval set — not by feel.
• Retrieval: pure vector search misses exact terms (error codes, product names). Hybrid search (vector + keyword, merged with RRF) is the default answer in 2026, not a bonus point.
• Re-ranking: improves quality, adds latency and cost. Use it when the top-k results are noisy, skip it when retrieval is already clean.

.NET stack to name: Azure AI Search (hybrid search built in), Qdrant or pgvector as alternatives, Microsoft.Extensions.AI for embeddings, Semantic Kernel for orchestration.

2. Your RAG system gives wrong answers. How do you debug it?

The trap is jumping to “improve the prompt.” The senior answer is: split the pipeline and measure each stage.

1. Retrieval first. For a set of test questions, check: are the right documents even in the top-k results? Measure recall@k. If the right chunk isn’t retrieved, no prompt will save you.
2. Generation second. If retrieval is fine, check whether the model ignores the context or contradicts it (groundedness).
3. Common root causes: bad chunking that splits answers across chunk boundaries, stale index after document updates, embedding model mismatch between indexing and querying.

The one-liner that lands well: “Most RAG bugs are retrieval bugs, not model bugs.”

3. Vector database vs. adding pgvector to Postgres — how do you choose?

Boring answer, strong signal: start with pgvector if you already run Postgres and have under a few million vectors. One less system to operate, transactional consistency with your data, good enough performance. Move to a dedicated store (Qdrant, Azure AI Search) when you need scale, advanced filtering, or built-in hybrid search. Interviewers reward “I don’t add infrastructure until I have to” far more than name-dropping databases.

4. How would you handle sensitive data in a RAG pipeline?

Filter at retrieval time, not generation time. Apply the user’s permissions as a filter in the vector query (every serious vector store supports metadata filters), so the model never sees documents the user can’t access. Add PII redaction before indexing, and never rely on the prompt (“don’t reveal secrets”) as a security boundary — the prompt is not an ACL.

Part 2: “Keep it alive” — reliability and cost

5. An LLM provider starts returning 429s and timeouts. What does your code do?

This is where .NET shines in an interview. The answer is Polly:

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .HandleResult(r => r.StatusCode == HttpStatusCode.TooManyRequests),
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions())
    .AddTimeout(TimeSpan.FromSeconds(30))
    .Build();

Retry with exponential backoff and jitter for transient failures. Circuit breaker so a dead provider doesn’t cascade through the system. Then the part most candidates forget: a fallback — a cheaper model, a cached answer, or an honest “try again later.” A 429 storm should degrade quality, not availability.

6. What’s different about retrying LLM calls vs. retrying a normal HTTP call?

Two things. First, retries cost real money — every retry burns tokens, so you cap attempts and monitor retry spend. Second, idempotency: retrying “generate text” is safe, retrying an agent’s tool call that sends an email is not. Non-idempotent tool calls need idempotency keys or confirm-before-execute, exactly like payment APIs.

7. How do you control LLM costs in production?

Four levers, in order of impact:

1. Model routing — send simple queries to a cheap model, hard ones to an expensive model. Often 80% of traffic doesn’t need the frontier model.
2. Caching — exact-match first (free wins), then semantic caching for paraphrased repeats.
3. Token budgets — cap context size per request; trim conversation history instead of sending it all.
4. Cost as a metric — cost-per-request on the same dashboard as latency and error rate. You can’t control what you don’t measure.

8. How do you keep latency acceptable when every request hits an LLM?

Reframe it: you usually can’t make the model much faster, so you change what the user experiences. Stream tokens so time-to-first-token is the felt latency (in ASP.NET Core: IAsyncEnumerable over Server-Sent Events). Run retrieval and other I/O concurrently with Task.WhenAll. Cache aggressively. And move non-interactive work (summaries, enrichment) to background queues so it never blocks a user.

Part 3: “Prove it works” — evaluation and observability

9. How do you test a system whose output is non-deterministic?

The question that separates AI engineers from API callers. Three layers:

• Offline evals: a golden dataset of inputs and expected properties. Score with deterministic checks where possible (did it cite a source? is the JSON valid? does it contain the required fact?) and LLM-as-judge where not.
• CI gate: evals run in the pipeline; a prompt or model change that drops scores below threshold fails the build — same as a failing unit test.
• Online: user feedback signals plus a sampled review of production traces.

.NET stack to name: Microsoft.Extensions.AI.Evaluation, or eval scripts wired into GitHub Actions / Azure DevOps.

10. What is LLM-as-judge, and what are its failure modes?

Using a strong model to score another model’s output against a rubric. It scales where human review doesn’t — but say the failure modes out loud: position bias (prefers the first answer shown), self-preference (rates its own model family higher), score drift when the judge model updates. Mitigations: pin the judge model version, randomize answer order, calibrate the judge against a small human-labeled set before trusting it.

11. What do you monitor in an LLM application?

Everything you’d monitor in any service — latency, error rate, throughput — plus the LLM-specific layer: token usage and cost per request, eval scores on sampled traffic, retrieval quality, and refusal/fallback rates. The implementation answer for .NET: OpenTelemetry is built into the platform, and Microsoft.Extensions.AI emits LLM spans following semantic conventions — model name, token counts, latency per call. Export to Application Insights or Langfuse, and log prompt/response pairs (with PII redaction) so you can replay any failure.

12. A user reports the assistant “said something wrong” — walk me through your process

This tests whether observability is real or theoretical. The answer: find the trace by request ID; see the full chain — input, retrieved chunks, final prompt, model response, tool calls. Classify the failure: retrieval miss, model hallucination despite good context, or stale source data. Then close the loop: add the case to the eval set so the fix is protected by a regression test. If you can’t replay the request, you can’t debug the system — that’s the point to make.

Part 4: “Ship it” — agents, deployment, CI/CD

13. How do you stop an agent from looping forever or burning your budget?

Treat the agent loop like any untrusted loop: hard limits. Max iterations, max tokens, max wall-clock time per run, and a budget cap per user/session. Log every step (thought → tool call → result) as structured spans so a runaway run is visible in real time, not in the monthly invoice.

14. Where does agent state live?

Not in process memory. The agent loop must survive restarts and scale-outs, so state goes to Redis or a database, keyed by conversation/run ID. For long multi-step workflows, name Durable Functions (or Temporal): each step is checkpointed, so a crash resumes at the last completed step instead of restarting — and you get retries and human-approval steps for free.

15. How is an LLM API endpoint different from a regular REST endpoint?

Long-running and streaming. Concretely in ASP.NET Core: return IAsyncEnumerable<string> over SSE instead of a single JSON body; honor CancellationToken everywhere so a user who closes the tab doesn’t keep burning tokens; set timeouts that match real model latency, not the 100ms default mindset; and rate-limit by token cost, not just request count.

16. What does CI/CD look like for an AI application?

Standard pipeline plus two additions. One: prompts are code — versioned in git, reviewed in PRs, never edited live in production. Two: an eval stage — every change to prompts, models, or retrieval runs the eval suite, and quality regression fails the deploy. Everything else is the pipeline you already have: build, test, containerize, deploy.

17. How do you deploy a new model or prompt version safely?

Canary or blue-green, with one twist: the canary metric includes eval scores and cost, not just errors and latency. A new model version can return 200 OK on every request while quietly giving worse answers — HTTP metrics won’t catch that. Route a few percent of traffic, compare scores, then promote. Feature flags for model/prompt selection give you instant rollback without a redeploy.

18. Why should we hire a .NET developer for an AI role when the ecosystem is Python?

The honest framing: the model layer is language-agnostic — it’s an HTTP call. The hard parts of production AI are reliability, observability, and deployment, and .NET is excellent at exactly those: built-in OpenTelemetry, Polly, strong typing for tool contracts, first-class async. Python wins for training and research; for serving and orchestration, the platform with the strongest engineering tooling wins. And with Microsoft.Extensions.AI and Semantic Kernel, the AI layer itself is no longer a gap.

The cheat sheet

The takeaway

If you strip the AI vocabulary out of these 18 questions, you’re left with: design a pipeline, handle failures, test the untestable, watch it in production, ship it safely. That’s a backend interview.

The model is an unreliable dependency — a third-party API that sometimes lies. Everything around it is classic engineering. If you’ve built reliable systems in .NET (or anywhere), you’re closer to AI engineering than the job titles suggest. You don’t need to become a researcher. You need to apply the discipline you already have to a new kind of dependency.

What questions came up in your AI interviews? I’m collecting them — drop yours in the comments.