Skills assessment
AI Engineer skills assessment
AI engineer is the most rebadged title in tech right now: an afternoon following an API tutorial gets written up as AI engineering, and the skill that actually matters — knowing how you know a feature works — never shows up on a CV. The role builds LLM-backed product features on models someone else trained: retrieval pipelines, evaluation harnesses, prompt and tool orchestration, all inside a latency and cost budget. A rebadged candidate can describe prompts they wrote; the genuine one can tell you how they proved the feature worked.
That is why prompt trivia and a polished résumé are the wrong screen. The only reliable signal is job-shaped work: put a realistic LLM feature that is broken in an interesting way in front of the candidate, give them a model to work with, and watch how they direct it, verify its output, and correct it when it is confidently wrong. H-Evaluate generates that task per job description, in an environment where a model is genuinely available — per-job generation, so no candidate has rehearsed it — and scores the AI-fluency behaviour that defines the role.
What to assess
The competencies that predict performance in this role, mapped to the five hiring pillars.
Broken-feature work sample
Working a realistic broken feature with a model at hand — a retrieval endpoint returning fluent wrong answers, an agent looping on a bad tool call — and being watched for how they direct, verify, and correct the model rather than trusting it.
Retrieval & orchestration craft
Real pipelines rather than a single embedding call — chunking, ranking, and handling the case where retrieval returns nothing useful — plus enough software craft to run the feature in production with logging and fallbacks.
Evaluation discipline
The defining skill: building the eval set before trusting the fix, distinguishing offline from online evaluation, and treating 'it looked right when I tried it' as a confession, not evidence.
Failure-mode & cost judgement
Naming hallucination, prompt injection, and drift unprompted, designing for the model being wrong, and reasoning in tokens and milliseconds — knowing when a cheaper model is good enough and what a feature costs at scale.
Ownership & collaboration
How they handled a feature that shipped and then broke in production — the verification story and what they changed — rather than a tidy narrative about a prompt they were proud of.
AI fluency
The working relationship between the person and the model, scored on the 4D framework with Discernment and Diligence weighted most — catching the model being confidently wrong is the whole point of the role.
How to structure the assessment
- 1Build the task around a feature broken in an interesting way — a retrieval endpoint returning confidently wrong answers, a prompt chain quietly blowing the latency budget — not a whiteboard puzzle.
- 2Run it in an environment where a model is genuinely available rather than banned, because that is the only way to see the AI-fluency behaviour that defines the role.
- 3Watch for whether they reach for an eval before trusting their own fix, name a failure mode you did not mention, and notice the cost implication of the model they picked.
- 4Weight Discernment and Diligence above the other competencies — catching the model being wrong matters more than prompt fluency.
- 5Keep it short and score against the same rubric: one job-shaped work sample plus one structured interview beats six rounds of vibes.
Signals that predict success
- +Reaches for an eval set before trusting their own change
- +Names a failure mode — hallucination, prompt injection, drift — unprompted
- +Reasons about cost and latency without being asked, and knows when a cheaper model is enough
- +Can tell you exactly how they knew their last feature worked
Red flags to watch for
- –Describes prompts they wrote but cannot say how they knew the feature worked
- –Trusts 'it looked right when I tried it' as evidence rather than building an eval
- –Designs as if the model is always right, with no plan for it being wrong
- –Ships a prototype with a nice demo but cannot run it in production
Assessment vs. interview
You cannot interview evaluation discipline into visibility with questions alone, because the rebadged candidate has read the same blog posts you have. A work sample with a model at hand shows it directly — you watch whether they reach for an eval before trusting a fix, and whether they catch the model being confidently wrong. Use the assessment to gather that evidence, then use the structured interview to walk their work sample with them: why that fix, how they verified it, what they would check before shipping. The verification story matters more than the fix.
Skills assessment
Configure this assessment by role and seniority
Watch the emphasis shift in real time as you change the role and level — no signup.
Related reading
How to hire an AI engineer: skills to test in 2026
How to hire an AI engineer in 2026: what the role really does, the evaluation discipline that sets it apart from a machine learning engineer, and how to test it.
How to assess AI fluency: a practical guide by role
How to assess AI fluency in hiring: what to measure, how to score each part, what strong vs weak looks like, and the tasks that reveal real judgement.
How to hire a machine learning engineer: ship models
A skills-first guide on how to hire a machine learning engineer: what to assess, the work sample that predicts production success, and questions that work.
Frequently asked questions
How do you assess an AI engineer?
Give them a job-shaped task with a model at hand and watch them work — direct it, verify its output, correct it. A retrieval feature that returns confidently wrong answers is a good prompt. You are looking for someone who reaches for an eval before they trust their own fix, who names failure modes unprompted, and who reasons about cost and latency without being asked. Polished prompt trivia is not the signal.
What skills should an AI engineer test cover?
Evaluation discipline first: the ability to prove a feature works before it ships, and to write the eval set that proves it. Then failure-mode thinking around hallucination, prompt injection, and drift, plus latency and cost engineering, retrieval and orchestration craft, and enough software craft to run it in production. Model-training depth is optional — that is the machine learning engineer.
What is the difference between an AI engineer and a machine learning engineer?
A machine learning engineer trains, tunes, and serves models — the research-flavoured end. An AI engineer builds product features on top of models someone else trained: retrieval pipelines, prompt and tool orchestration, evaluation harnesses, and the latency and cost budget around them. The AI engineer treats the model as a component to engineer around. Most teams shipping LLM features want the second person, not the first.
Is an AI engineer a real job or a rebadged software engineer?
It is a real role with a distinct core skill: evaluation discipline. The impostor version is real too — plenty of CVs rebadge API-tutorial experience as AI engineering. The tell is whether the person can tell you how they knew a feature worked. A rebadged candidate describes prompts they wrote; a genuine AI engineer describes the eval set, the failure modes they hunted, and the cost budget they held.