Hiring · August 2, 2026 · 13 min read

AI engineer interview questions: how to evaluate answers

AI engineer interview questions for hiring managers: what strong and weak answers to RAG, agent design and reliability sound like, plus a scoring approach.

← Part of The five pillars of hiring: what assessments measure

This guide is written for the hiring manager or engineering lead running the loop for an AI engineer — the person you are dropping into production LLM features who has to keep them honest, fast and affordable. It is not written for the candidate, though a candidate is welcome to read it and learn exactly what you are looking for. That admission matters, because it names the problem these AI engineer interview questions have to survive: in 2026, candidates prep with an assistant, and any question you can write down cleanly can be answered cleanly by someone who has never shipped a retrieval pipeline in their life. A memorisable question list is a leaked exam. So this guide gives you more than a list. For each question it tells you what a strong answer covers, what a weak one sounds like, where junior and senior answers diverge, a scoring approach that keeps your interviewers honest, and — for the skills that decide the hire — the point where you stop asking and start watching them work. Use it in the loop, after a short work sample and before the offer, not as a screen on its own.

How do you score interview answers consistently?

Decide what a good answer sounds like before you meet the candidate, not while they are talking. That single habit is the difference between a structured interview and a rapport contest. The rest follows from it: ask every candidate the same questions in the same order, and grade each answer against written anchors rather than gut feel.

A workable rubric is a one-to-five scale with behavioural anchors written out in advance. For a given question, a 5 might read "names the failure mode unprompted, gives a concrete example from their own work, and describes how they measured the fix"; a 3, "gives a correct textbook answer with no lived example"; a 1, "recites a definition or steers away from the question". Write those anchors for every question you plan to ask. It takes an afternoon and it is the highest-leverage hour in the whole process, because it converts a vague impression into a comparable score. This is generic structured-interview science, not anything proprietary — the structured interviews guide covers the practice. Each interviewer scores independently and only compares notes afterwards, so the loudest voice in the debrief does not quietly become the decision.

Retrieval-augmented systems

Retrieval is where most LLM product features live and where most of them quietly fail. These questions probe whether the candidate understands retrieval as an engineering system with failure modes, or as a recipe they memorised. The tell is specificity: a strong answer names where retrieval breaks; a weak one recites the stages in order and stops.

• Walk me through what happens between a user's question and the model's answer in a retrieval-augmented feature — strong answers describe chunking, indexing, ranking and the prompt assembly as choices with trade-offs; red flag: reciting the stages as a fixed pipeline with no decisions inside it.
• Your retrieval feature returns confident, fluent, wrong answers. How do you diagnose it? — strong answers separate a retrieval failure from a generation failure before touching the prompt; red flag: jumping straight to prompt tweaks without checking what was retrieved.
• How do you decide chunk size and overlap for a corpus? — strong answers tie it to the shape of the documents and the queries and admit they would measure it; red flag: quoting a single number as universally correct.
• What do you do when retrieval returns nothing useful for a query? — strong answers have a designed fallback and never let the model invent context; red flag: assuming retrieval always returns something relevant.
• How do you evaluate whether your retrieval is actually good, separate from the model's answer? — strong answers name retrieval-specific metrics and a labelled set; red flag: judging retrieval only by whether the final answer looked right.
• When would you not use retrieval at all, and just put the context in the prompt? — strong answers reason about corpus size, freshness and cost; red flag: treating retrieval as mandatory for every LLM feature.

The retrieval-failure question is your flagship here, so spend real time on it. A junior answer treats the whole feature as one black box: the answer was wrong, so they rewrite the prompt, add a line telling the model not to make things up, and hope. A senior answer refuses to touch the prompt until they know which half broke. They pull the retrieved chunks for the failing query and read them. If the right document was never retrieved, the prompt was never the problem — the bug is upstream, and no amount of prompt scolding will fix a document that never reached the context window. If the right document was retrieved and the model still answered wrong, now it is a generation problem. The candidate who recites the retrieval stages flawlessly, then cannot say why their retriever returned the wrong document, has memorised the map and never walked the ground.

Agent and tool design

Agents are the most over-reached-for pattern in the field right now, which makes restraint the signal. The strongest AI engineers talk you out of agents as often as into them. These questions are less about whether the candidate can wire a tool call and more about whether they know when the whole architecture is a mistake. If your team is building genuinely agentic products, pair this with how to hire an AI agent engineer.

• When would you not use an agent, and reach for a fixed workflow instead? — strong answers default to the simplest thing that works and reserve agents for genuinely open-ended tasks; red flag: treating agents as the obvious modern choice for everything.
• A tool your agent calls returns garbage, but the model believes it and carries on. How do you design around that? — strong answers validate tool output and never assume a tool succeeded; red flag: trusting tool responses because the model seemed confident.
• How do you stop an agent looping on a malformed tool call at 2am? — strong answers describe loop limits, timeouts and observability; red flag: no failure containment and no way to see what the agent did.
• How do you decide which actions an agent is allowed to take autonomously versus needing a human in the loop? — strong answers reason about blast radius and reversibility; red flag: full autonomy with no thought to what an irreversible mistake costs.
• How do you test an agent when its path through the tools changes every run? — strong answers evaluate behaviour and outcomes, not a single fixed transcript; red flag: assuming a happy-path demo means it works.
• Describe an agent design you scrapped or simplified. What made you pull back? — strong answers show earned scepticism from real pain; red flag: never having questioned an agentic approach.

"When would you not use an agent" is the flagship of this group, and it is a trap for the fashionable candidate. A junior answer, especially one that has read the current wave of agent tutorials, reaches for an agent by default and describes elaborate multi-step orchestration for a task that is really three sequential API calls. They mistake sophistication for judgement. A senior answer starts from the other end. They ask what the task actually needs, note that agents add latency, cost, non-determinism and a whole category of failure modes, and reserve them for problems where the path genuinely cannot be known in advance. They will happily tell you the last thing they shipped could have been an agent but was better as a plain workflow. The engineer who talks you out of an agent has usually cleaned up after one.

Evaluation and reliability

This is the defining competency of the role, so weight it accordingly. An AI engineer's entire value is knowing whether a feature works — and being able to prove it — before it reaches a user. The single most useful thing you can ask is how they knew a past feature worked; the answer sorts the genuine from the rebadged faster than anything else in the loop.

• How did you know your last LLM feature actually worked before you shipped it? — strong answers describe an eval set, offline and online checks, and a metric they trusted; red flag: "it looked right when I tried it a few times".
• What is the difference between offline and online evaluation, and when do you need both? — strong answers use each for what it is good at; red flag: conflating a demo with an evaluation.
• How do you build an eval set for a feature that has never had one? — strong answers start from real failing examples and grow it deliberately; red flag: no method beyond eyeballing outputs.
• The underlying model gets silently updated by the provider and your feature quietly degrades. How would you catch that? — strong answers have a regression suite that runs continuously; red flag: assuming a feature that worked last week still works today.
• How do you guard against prompt injection in a feature that takes untrusted user input? — strong answers treat model input as hostile and design boundaries; red flag: no awareness that user text can hijack the model.
• How do you measure hallucination in a way you can act on? — strong answers ground the claim in retrievable sources and count it; red flag: treating hallucination as unmeasurable and therefore ignored.

"How did you know it worked?" deserves its own probe because it is the fastest impostor detector in the loop. Ask it, then stay silent and let them fill the space. The rebadged candidate — the one whose AI engineering is really an afternoon following an API tutorial — talks about the prompt they were proud of and the demo that impressed a stakeholder. A genuine AI engineer talks about the eval set: how they assembled it from real failing cases, what regressions it caught, the uncomfortable number that told them the feature was worse than everyone assumed. The junior version knows evaluation matters but has only run it offline, once, before launch. The senior version has an evaluation that never stops running, because they have been burned by a provider updating a model underneath a feature that was working the day before. Same question, three different answers, and you can hear the seniority in which one they give.

Production judgement: cost, latency and fallback

The last group separates people who have shipped LLM features to real users from people who have built impressive prototypes. Production judgement shows up in tokens, milliseconds and the plan for the day something breaks. These are unglamorous questions, and that is exactly why they discriminate — nobody rehearses them, so the answers are usually honest.

• What does your feature cost per thousand calls, and how would you find out? — strong answers reason in tokens and know where the cost hides; red flag: never having looked at the cost of a feature they built.
• How do you decide when a cheaper, smaller model is good enough for part of a feature? — strong answers route by task difficulty and measure the quality trade-off; red flag: defaulting to the biggest model everywhere.
• The model provider has an outage in the middle of your peak. What happens to your feature? — strong answers have a fallback and degrade gracefully; red flag: a single hard dependency with no plan B.
• How do you bring the latency of a slow prompt chain under control? — strong answers name caching, parallelism and trimming the chain; red flag: accepting slow as inherent to LLMs.
• How do you version and roll back a prompt the way you would code? — strong answers treat prompts as versioned artefacts with a rollback path; red flag: prompts edited live in production with no history.
• What do you log so you can debug an LLM feature after it misbehaves for a real user? — strong answers capture inputs, retrieved context and model output; red flag: no logging beyond a success or failure flag.

The cost question is quietly one of the most revealing in the loop, because so few candidates have ever been forced to hold a budget. Ask what a feature costs per thousand calls and a junior candidate often blinks — they built the thing, shipped it, and never once looked at the bill, because someone else owned that line. Not disqualifying in a junior, but it tells you where they are. A senior candidate answers in the currency of the job: this feature is expensive because it stuffs the whole document into every prompt, so I would cache the retrieval, route easy queries to a cheaper model, and trim the chain first. They have felt the cost line hurt and it changed how they build. You are not testing arithmetic. You are testing whether the number is real to them.

When should you stop asking and start testing?

The moment a candidate can answer every question above without hesitation is the moment you should trust the questions least. Interviews sample claims; work samples sample the work. A confident description of how you would diagnose a broken retrieval feature is a claim. Watching someone pull the retrieved chunks, read them, and only then decide whether to touch the prompt is evidence — and in 2026, with assistants able to produce the confident description on demand, the gap has never mattered more.

So run the work sample before the interview, not after. Give the candidate a job-shaped task — a retrieval feature that returns fluent, wrong answers, an agent that loops on a bad tool call, a prompt chain quietly blowing the latency budget — inside an environment where a model is genuinely available rather than banned, because banning AI from an AI engineer's assessment tests a job nobody does. A work sample test like this predicts on-the-job behaviour far better than any question can, and it tells you which probes are worth spending scarce interview time on. If the sample already shows sound evaluation discipline, go deeper instead on the production judgement or agent restraint it left ambiguous. Let the report choose your questions.

This is the point where an assessment platform earns its place in the loop, ours included. Our AI Sandbox runs exactly this kind of job-shaped session for AI engineering roles, and the report tells you where to point the interview afterwards — at a capability level, without dictating your decision. The wider hiring case for the role sits in how to hire an AI engineer; the discipline of reading how someone works with a model, rather than whether they can, is how to assess AI fluency, scored against the 4D framework of Delegation, Description, Discernment and Diligence — the last two carry the most weight for an AI engineer, because catching the model being wrong is the entire job. You can build a homegrown version with a broken repository and a stopwatch; if you would rather see ours, book a demo. Either way the principle holds: ask the questions to open the conversation, then stop asking and start watching.

The best AI engineers are not the ones with the smoothest answers. They are the ones who, handed a feature that looks fine, refuse to trust it until they have measured it — and who will tell you the uncomfortable number the demo hid. Interview for that instinct, then verify it in a work sample, or you will keep mistaking a well-prepped answer for the thing that ships.