Hiring · July 29, 2026 · 9 min read

How to hire an AI engineer: skills to test in 2026

How to hire an AI engineer in 2026: what the role really does, the evaluation discipline that sets it apart from a machine learning engineer, and how to test it.

← Part of The five pillars of hiring: what assessments measure

This guide is for the hiring managers and engineering leaders opening a role that did not exist three years ago. An AI engineer builds LLM-backed product features — retrieval pipelines, evaluation harnesses, prompt and tool orchestration, all inside a latency and cost budget — and the job exists because foundation models turned "train the model" into "engineer around the model". You need one when several model-dependent features are already in production and the work of keeping them honest never stops; you do not need one when a single chatbot feature could sit comfortably inside your existing team. The role usually reports into engineering, sitting close to the product surface it powers rather than off in a research group. Here is the uncomfortable part: this is the most rebadged title in tech right now, and the skill that actually matters — knowing how you know the feature works — is the one that never shows up on a CV.

What does an AI engineer actually do?

An AI engineer ships product features built on models they did not train. A typical week: tighten a retrieval pipeline that keeps surfacing stale documents, write an eval set that catches the regression before users do, trim a prompt chain that quietly doubled the latency budget, and chase down why the agent loops on a malformed tool call at 2am. The model is a component; the engineering is everything around it.

That is the honest version of the job, and it looks nothing like the demo. Notice how little of it is "write a clever prompt". The prompt is the easy hour. The hard days are spent on the things that make an LLM feature safe to ship to real users:

• Build and maintain the retrieval pipeline — chunking, indexing, ranking — so the model answers from the right context instead of confidently inventing one.
• Orchestrate prompts and tools: chaining calls, wiring the model to functions, handling the case where a tool returns garbage and the model believes it anyway.
• Own the evaluation harness — the eval sets, the regression suite, the offline and online checks that answer the only question that matters: does this feature actually work?
• Hold a latency and cost budget: cache what you can, route cheap requests to cheap models, and know to the cent what a feature costs per thousand calls.
• Hunt failure modes before customers find them — hallucination, prompt injection, silent drift as the underlying model changes beneath you.
• Run it all in production with real logging, real fallbacks, and a plan for the day the provider deprecates the model you built on.

Do you actually need one?

You need an AI engineer once you have real LLM surface area — several model-backed features live, an evaluation burden that never sleeps, and a cost line that has started to hurt. Below that, a strong software engineer with genuine AI fluency covers it, and opening the role too early just means paying a premium for someone who will spend most of their time on ordinary backend work.

So be honest about your scale before you post the job. If you have one chatbot feature and a wrapper around a single API, the person you want is a good software engineer who can work fluently with a model — read how to hire a software engineer and assess for AI fluency on top. If your problem is training or fine-tuning models rather than building on them, you want the research-flavoured end of the spectrum instead — that is a different hire entirely, covered in how to hire a machine learning engineer. The AI engineer earns their keep in the messy middle: production LLM features at enough volume that evaluation, cost and failure-mode work becomes a full-time discipline rather than a side quest.

What skills separate a strong AI engineer from a rebadged one?

Every title minted by the AI boom attracts rebadged CVs, and this one attracts the most: an afternoon following an API tutorial gets written up as "AI engineering". The rebadged candidate can describe prompts they wrote. The real one can tell you how they knew the feature worked — which is the entire job. Three skills draw the line, and none of them photograph well on a résumé.

• Evaluation discipline — the defining skill. They build the eval set before they trust the fix, distinguish offline from online evaluation, and treat "it looked right when I tried it" as a confession, not evidence.
• Failure-mode thinking — they name hallucination, prompt injection and drift unprompted, and design for the model being wrong rather than assuming it is right. They know a feature that worked last week can break when the provider silently updates the model.
• Cost and latency engineering — they reason in tokens and milliseconds, know when a cheaper model is good enough, and can tell you what a feature costs at scale without reaching for a calculator.
• Retrieval and orchestration craft — real pipelines, not a single embedding call: they think about chunking, ranking, and what happens when retrieval returns nothing useful.
• Enough software engineering to ship — logging, fallbacks, versioning the prompt like code. An AI engineer who cannot run their feature in production is a prototyper with a nice demo.

The fastest way to catch a rebadged AI engineer is to ask how they knew their last feature worked. The genuine one talks about the eval set and the failure modes they hunted. The impostor talks about the prompt they were proud of. Same question, completely different answer.

How do you test those skills?

You cannot interview evaluation discipline into visibility with questions alone, because the rebadged candidate has read the same blog posts you have. The only reliable signal is job-shaped work: put a realistic LLM feature in front of them, give them a model to work with, and watch how they direct it, verify its output, and correct it when it is confidently wrong. That is the AI-fluency lens — you are assessing the working relationship between the person and the model, not their memory of prompt patterns.

Build the task around a feature that is broken in an interesting way: a retrieval endpoint returning fluent, wrong answers; an agent that loops on a bad tool call; a prompt chain quietly blowing the latency budget. Then observe. Do they reach for an eval before they trust their own change? Do they name a failure mode you did not mention? Do they notice the cost implication of the model they just picked? A work sample test like this predicts on-the-job behaviour far better than a whiteboard, and running it inside an AI Sandbox — where a model is genuinely available rather than banned — is the only way to see the AI-fluency behaviour that defines the role. For scoring it consistently, how to assess AI fluency and the 4D framework — Delegation, Description, Discernment, Diligence — give you a rubric; Discernment and Diligence carry the most weight for an AI engineer, because catching the model being wrong is the whole point. Yes, this is the sort of exercise our platform runs; a homegrown version with a broken repo and a stopwatch will teach you most of the same things.

What does the interview loop look like?

Keep it short — strong AI engineers have several offers and a slow loop loses them. Aim for one job-shaped work sample plus one structured interview, not six rounds of vibes. A structured interview means the same questions, same order, and the same scorecard for every candidate, which is what makes the comparison fair and the decision defensible.

• Screen on skills, not pedigree — a short, role-relevant task everyone takes on the same terms, replacing the résumé sort. This is AI-native hiring, and it widens the pool to self-taught engineers who learned this craft because there was no degree for it.
• The work sample — the broken-feature session in the sandbox, scored on evaluation discipline, failure-mode thinking, and cost and latency awareness.
• A technical interviewer walks the candidate's work sample with them: why that fix, how did you verify it, what would you check before shipping? The verification story matters more than the fix.
• A structured behavioural conversation on ownership and collaboration — how they handled a feature that shipped and then broke in production.
• A scorecard per capability, filled in independently before anyone compares notes, so the loudest voice in the debrief does not become the decision.

What about compensation and seniority?

We do not publish salary figures — the market for this role reprices every quarter and any number here would be wrong by the time you read it. Qualitatively, AI engineers command a premium over general software engineers right now because the supply of people with genuine evaluation discipline is thin and the demand is not. Do not let that premium tempt you into over-levelling: a mid-level engineer with real failure-mode instincts is worth more to an LLM feature team than a senior title with prompt-tutorial depth. Benchmark against your own market for software engineering, add for demonstrated AI-native skill you have actually observed in the work sample, and anchor the offer on evidence rather than on the buzzword in the title.

First 90 days: what good looks like

A strong AI engineer spends the first month reducing your uncertainty, not adding features. By week four they should have built or hardened an eval set for a feature that had none, and told you something uncomfortable and true about how well it actually performs. That is the tell: a good hire makes your model-backed features measurable before they make them fancier.

• Weeks 1–4 — maps the existing LLM features, finds the ones running without real evaluation, and stands up an eval set that surfaces a regression nobody had noticed.
• Weeks 4–8 — ships a failure-mode fix that matters: a prompt-injection hole closed, a retrieval pipeline that stops hallucinating on empty context, a fallback for the day the model misbehaves.
• Weeks 8–12 — brings a cost or latency line under control, and leaves behind a way to keep it there: a budget, a cheaper routing path, a dashboard the team will actually watch.
• Throughout — makes the team's relationship with the model more honest. The features are not necessarily flashier by day 90; they are measurable, cheaper to run, and less likely to embarrass you in production.