Hiring · August 2, 2026 · 12 min read

Prompt engineer interview questions: how to evaluate answers

Prompt engineer interview questions for the hiring manager: 24 questions grouped by competency, what strong and weak answers sound like, and when to test instead.

← Part of The five pillars of hiring: what assessments measure

This guide is for the hiring manager sitting across from a prompt engineer candidate, trying to work out whether the fluent answers describe a real discipline or a memorised script. That distinction has never been harder to draw, because the candidate almost certainly rehearsed with an assistant. Any question with a tidy, lookup-able answer — define few-shot prompting, list three ways to reduce hallucination — is now a leaked exam: they read the same threads you do, feed them to a model, and arrive with a polished recitation that says nothing about whether they can do the work. So this is not a list of questions to fire off and tick. It is a guide to evaluating the answers: what a strong reply sounds like versus a weak one, how to score consistently, and — the part every listicle skips — when to stop asking and start watching the person work. The role has consolidated from a phrasing gimmick into a systems discipline built on measurement, and clever phrasing is now the clearest impostor signal.

How do you score interview answers consistently?

Before the questions, the scoring — because unstructured interviews reward whoever is most like the interviewer, not whoever can do the job. Decide the questions once, ask every candidate the same ones in the same order, and write down what a weak, adequate and strong answer contains before anyone sits down. That is behaviourally anchored scoring, and it is the single change that most improves how accurately an interview predicts performance.

Anchoring works because it forces you to commit to a standard while you are still calm, rather than talking yourself into a likeable candidate at four in the afternoon. A practical rubric runs one to five, with behavioural anchors: for a debugging question, a one 'reaches for random wording changes, no hypothesis, no measurement'; a five 'groups the failures by type, states a theory, changes one variable, and checks it held across the whole set'. You are scoring not how articulate someone is but whether the behaviour they describe matches the job. The mechanics of running this well — anchor wording, panel calibration, reducing halo effects — sit in our structured interviews guide, and they are generic interview science, not anything proprietary to us.

Questions on evaluation and measurement habits

This is the heart of the role, so spend the most anchoring effort here. A prompt engineer who cannot measure is a copywriter with a larger vocabulary. Every question below asks the same thing from a different angle: do you turn 'the output feels off' into a number, or into another round of tinkering?

• How do you know when a prompt has actually improved? — a strong answer reaches for a case set and a metric; the red flag is 'it reads better', because a feeling is not a measurement.
• Walk me through how you built an evaluation set for a real prompt. — strong candidates collect real failures, label expected outputs, and run changes against the whole set; the red flag is treating three hand-picked examples as coverage.
• Tell me about a time you thought a prompt was fixed and it was not. — you want a specific regression they missed and caught; the red flag is a candidate who cannot recall being wrong, which means they never measured closely enough.
• How do you decide a change is worth shipping when the metric rises on some cases and falls on others? — strong answers weigh which failures matter most; the red flag is optimising a headline number while breaking edge cases.
• How large does an evaluation set need to be before you trust it? — listen for judgement about coverage, not a magic number; the red flag is 'a couple of examples is fine' or a rigid rule applied blind to the problem.
• When two prompts score the same on your set, how do you choose? — strong candidates talk about robustness and behaviour on unseen cases; the red flag is choosing whichever is cleverer for its own sake.

The flagship question here is the first, and the gap between junior and senior answers is stark. A junior prompt engineer describes eyeballing a few outputs and deciding they look better — not wrong so much as unscaled; it works until the prompt matters. A senior candidate answers structurally: they name the case set, tell you it holds a representative spread of production failure types, and read the diff rather than the absolute score, because a change that fixes ten cases and silently breaks two is a net loss dressed as a win. The junior answer runs out of road after 'it looks better'.

The second flagship is the regression story. Ask any experienced prompt engineer about a time a fix did not hold and you will hear a real one, told with the rueful texture of someone who has been burned: a model update quietly turned clean output subtly worse, nobody noticed until a downstream number moved, and they traced it back and put a guard in place so the next one surfaces automatically. A candidate who cannot produce this story has never shipped prompts that mattered, or never measured them closely enough — either way disqualifying for a role built on keeping output honest over time.

Questions on retrieval and context design

Modern prompt engineering rarely stops at the prompt. Most consequential work feeds the model the right context — retrieved documents, tool outputs, structured state — and a strong candidate knows a great instruction over bad context still fails. This group probes whether they see the whole system or just the text box.

• How do you decide what context to give the model versus what to leave out? — strong answers treat context as a budget spent on what changes the answer; the red flag is stuffing everything in and burying the signal.
• Tell me about a time the model was wrong and the fix was in the retrieval, not the prompt. — you want someone who locates a failure in the right layer; the red flag is believing every problem yields to better wording.
• How do you structure retrieved information so the model actually uses it? — strong candidates order, label and format context deliberately; the red flag is pasting raw chunks in and hoping.
• What do you do when the retriever returns something plausible but wrong? — listen for whether they detect and handle bad context; the red flag is no awareness that retrieval can quietly poison a good prompt.
• How do you keep a long context window from degrading the answer? — strong answers know more context is not free and dilutes attention; the red flag is treating a large window as licence to stop curating.
• How would you debug an answer correct in isolation but wrong for the user's situation? — you want someone who checks whether the model got the right context; the red flag is stopping at 'the output looks fine'.

The flagship here is the retrieval-versus-prompt diagnosis, and it separates the systems thinker from the wordsmith cleanly. A junior answer treats a wrong output as a prompt to be re-worded, and spends an afternoon adjusting phrasing on a problem no phrasing can fix. A senior answer asks what the model was actually given: was the right document retrieved at all? If it was and the answer is still wrong, that is an instruction problem; if not, no prompt will save it. That instinct — locating the failure in the correct layer before touching anything — distinguishes the role from the certificate, and echoes the point our how to hire a prompt engineer guide makes about candidates who insist a prompt can fix everything.

Questions on model-behaviour debugging

When a prompt misbehaves, the fix depends entirely on why. A candidate who cannot tell a hallucination from format drift from an over-eager refusal will apply the wrong remedy and make things worse. This group tests whether they read failure modes accurately rather than react to them.

• The model keeps returning JSON wrapped in a chatty apology. How do you diagnose and fix it? — strong candidates name it as format drift and reason about instructions and examples; the red flag is treating a formatting failure as a factual one.
• How do you tell a hallucination apart from a retrieval miss apart from a refusal? — you want someone who distinguishes the three, because the fix for each differs; the red flag is lumping all wrong answers into 'the model is bad'.
• A prompt works nine times out of ten and fails the tenth unpredictably. How do you approach it? — strong answers hunt for the pattern rather than declare the model random; the red flag is accepting non-determinism as licence to stop investigating.
• How do you make a prompt survive a model version change? — listen for building against behaviour rather than over-fitting to one model's quirks; the red flag is a prompt so tuned to today's model it is brittle by design.
• When the model refuses a legitimate request, how do you resolve it without weakening safety? — strong candidates reframe precisely rather than bully the model or disable guardrails; the red flag is treating every refusal as an obstacle to hack.
• How do you reproduce an intermittent failure reliably enough to fix it? — you want a systematic approach to isolating variables; the red flag is 'I just try again until it happens', with no method to repeat it.

The flagship in this group is the intermittent-failure question, because it exposes temperament as much as skill. A weaker candidate hears 'fails one time in ten unpredictably' and shrugs — the model is non-deterministic, what can you do. A strong candidate hears a pattern waiting to be found: they collect the failing cases, look for what the one-in-ten share, and treat variance as a signal rather than an excuse. Ask what they would do if they could not reproduce it, and the good answer is not 'give up' but instrumenting the system to capture the next occurrence with enough context to diagnose it.

Questions on working practices: versioning, review and collaboration

Prompts that matter are code, and they need the same discipline: version control, review, documentation, and a way to work with the people whose product depends on them. This group tells you whether a candidate treats prompt engineering as a solo craft or an engineering discipline practised with a team.

• How do you version and roll back prompts in production? — strong candidates treat prompts as versioned artefacts with a rollback path; the red flag is editing live prompts in place with no history and no way back.
• How do you review someone else's prompt change before it ships? — you want a real review focused on the evaluation results, not just readability; the red flag is 'I read it and it looked fine', with no test set in the loop.
• How do you document a prompt so the next person understands why it is shaped the way it is? — strong answers capture the failure modes the prompt defends against; the red flag is undocumented cleverness only the author can maintain.
• Tell me about a time you had to explain a prompt trade-off to a non-technical stakeholder. — listen for plain language and honesty about limits; the red flag is hiding behind jargon or over-promising.
• How do you keep prompts consistent across a team so the product does not develop split personalities? — you want shared conventions and a source of truth; the red flag is every engineer keeping their own private style.
• How do you decide when a prompt has grown too complex and needs to be split or redesigned? — strong candidates recognise unmaintainable sprawl and refactor; the red flag is bolting yet another instruction onto a prompt already collapsing under its weight.

The flagship here is the stakeholder-explanation question, and it matters more than its soft framing suggests. Prompt engineers sit next to operations and brand teams who feel every regression directly, and the ability to say 'this is a retrieval limitation, not something I can prompt away' — without condescension or over-promising — keeps those relationships intact. A junior candidate retreats into jargon or promises a fix that does not exist; a senior one explains the trade-off in the stakeholder's own terms and manages the expectation before it breaks. This is the measurement-plus-communication blend our prompt engineering by role hub describes: the technical judgement is necessary, but the honesty about limits is what makes it usable by a team.

When should you stop asking and start testing?

Here is the uncomfortable truth about every question above: an interview samples claims, not work. A candidate can describe a beautiful measurement loop they have never run, and a rehearsed answer is now indistinguishable from a lived one until you probe for texture — and even then, texture can be invented. The questions are worth asking, not worth trusting on their own. At some point the honest move is to stop asking someone to describe the job and watch them do it.

The most predictive exercise for this role is almost embarrassingly simple: hand the candidate a mediocre prompt and a set of cases it fails on, then watch them diagnose and iterate with a model genuinely available. Do they read the failures before touching the prompt, group them by type, change one variable, and check the fix across the whole set — or reword on instinct and eyeball a single case? Run it before the interview loop, not after, and let what you see decide which questions above deserve your limited time — spending the conversation on the retrieval-diagnosis and stakeholder-communication areas a sample covers less directly.

This is the bridge we build our whole approach around: a realistic, role-relevant task with AI tools at hand, observed as candidate evaluation rather than a quiz. It is what a prompt engineering skills assessment does — put the person in something close to a Tuesday morning and watch how they direct the model, catch it being confidently wrong, and verify before they ship. Reading those signals is its own skill; how to assess AI fluency covers what strong and weak collaboration looks like, alongside the sibling guide on AI engineer interview questions for the adjacent role. Weigh our framing as you like — the logic holds: the loop is what ships next quarter, and the only honest way to see it is to watch it. To see the assessment side, book a demo.

The best prompt engineer interviews do not reward the smoothest answers. They reward the candidate whose story survives a hard follow-up and whose work sample shows the same loop they described. Ask the questions, score them against anchors, then stop asking and watch — because clever phrasing is the impostor signal, and only the work tells the truth.