LLM Evaluation Engineer hiring kit

Every step to hire the engineer who proves a language-model feature works — and says plainly where the numbers stop meaning anything.

About this kit — the 5Ws

An LLM evaluation engineer builds the evidence behind every claim that a language-model feature works. That means assembling test sets that look like real use, choosing who or what marks each answer — a person, a fixed rule, or another model acting as a grader — checking those graders agree with each other and with people, and tracking whether a new model or prompt quietly made things worse. The hardest part of the job is saying what a score does not prove.

Calibrated for:: Mid-level to senior individual contributors who have already measured something real in production — a feature, a model or a ranking system — and reported results that others acted on.
Timeline:: Set an approval-to-offer target at intake, allowing for panel availability, candidate adjustments, references and any lawful checks.

Who: Engineering and data leaders, and the recruiters working with them, who need someone dedicated to measuring language-model quality rather than building the features themselves.
What: This kit defines what good evaluation work looks like, tests it with a work sample where an automated grader and human reviewers disagree, and gives you structured interview questions, a rubric and a way to review your own hiring process afterwards.
When: Use it once release decisions on model or prompt changes are waiting on someone to say whether quality held — from the first business case through to reviewing how long the search took.
Where: Technology teams shipping features built on large language models.
Why: Plenty of candidates can produce a score. Far fewer can tell you whether the score deserves trust. The work sample and questions here put each candidate in front of a grader that disagrees with people, so you compare how they found the cause and what they refused to over-claim.

The competency model · 6

Use these competencies to compare job-related evidence. Review any apparent gap in a must-have before deciding.

Evaluation set designMust-have · 20
Builds test sets from real usage and known failure types, writes labelling guidance two reviewers can apply the same way, and knows which cases are missing.
Grader design and calibrationMust-have · 20
Picks between human review, fixed rules and a model acting as grader for each check, and tests a model grader against human labels before letting it decide anything.
Measurement and statistical honestyMust-have · 20
Measures how often graders agree, looks for systematic slants such as preferring longer answers, and states how much a small test set can and cannot support.
Regression tracking across model and prompt changesMust-have · 15
Keeps results comparable between versions — same cases, same grader, recorded settings — so a drop caused by a new model or prompt shows up before release, not after.
Evaluation tooling and reproducibilityCan learn · 10
Writes evaluation code a colleague can rerun and get the same result: versioned test sets, pinned grader prompts, logged outputs and a cost per run someone has checked.
Honest reporting and stakeholder communicationMust-have · 15
Writes results a product owner can act on, separates what the numbers show from what they suggest, and names the cases a release decision should not rest on.

What's required and what's optional

Your hiring journey for LLM Evaluation Engineer. Select a stage to jump to it.

Required for every hire
Optional — use when it adds evidence
AI boost — a big drafting win
Colour = phase

Phase 1
Plan
1. 01Plan the roleRequired
2. 02Agree successRequiredAI boost
Phase 2
Attract
1. 03Write the jobRequiredAI boost
2. 04Find candidatesOptionalAI boost
Phase 3
Select
Phase 4
Decide
Phase 5
Onboard & learn
1. 12Help them settle inRequired
2. 13Measure time to hireRequiredAI boost

AI drafts, people decide: Use AI to draft, rewrite and summarise your own material — never to rank, score, screen out or decide on candidates, and never with a candidate's personal data.

Plan

Attract

Select

Decide

Onboard & learn

Solid = required · dashed = optional · arrow keys move between steps

Stage 1 of 13Required

0 of 3 done

Plan the role

Workforce planning and requisition approval

Who runs it: Hiring manager + finance / HR business partner
Done when: An approved requisition with a written business case, budget and a named decision-maker.

Key tool

Answer these before the role is approved

1Which release decisions in the next year will depend on knowing whether a model or prompt change made answers better or worse?
2When the last model or prompt change went out, how did the team decide it was safe?
3Could an engineer or data scientist already here take on evaluation as their main job within two quarters?

AI as an advantage4D framework

Delegate: Hand AI the first draft of the business case and the build / borrow / buy comparison. The decision, the budget and the sign-off stay with people.

Show the prompt and the checks

Describe: You are helping a hiring manager write a business case for a LLM Evaluation Engineer role. Context: I will paste the problem the hire solves, the work it involves, the capacity we already have and the time we have. Produce: (1) a one-sentence problem statement in business terms; (2) a build / borrow / buy comparison with the strongest argument for each; (3) the questions a finance partner would ask; and (4) a short recommendation with the evidence it rests on. Use British English and plain language. Do not invent figures — leave [brackets] where I need to supply numbers. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check every number and claim against your own notes and strike anything the draft assumed. Make sure 'buy' is argued from ongoing work, not from a job title.
Diligence: The hiring manager owns the final case and the budget request. No candidate information belongs here; AI drafts the case, people decide whether to open the role.

Don't miss: Hiring an evaluation engineer to produce a quality score for a launch that has already been decided — the role only pays off when results can still change a decision.

Framework

Build · Borrow · Buy

Before opening a role, decide whether to build the capability in your current team, borrow it (contractor, agency, partner), or buy it (a permanent hire).

Write the problem the hire solves in one sentence, in business terms — not the job title.
Test Build: could an existing person grow into this within the time you have?
Test Borrow: is the need time-boxed, or does it end once a system is shipped?
Choose Buy only when the work is ongoing, core to the product and nobody inside can absorb it.
Record the decision, the budget and the one person who signs off the final hire.

Why it works: Writing the build, borrow or buy decision down exposes a vague or temporary need before it becomes a permanent requisition — and gives every later stage an anchor to check against.

• Name who acts on the results: usually the engineering or product lead who signs off releases of model-backed features.

Is this a dedicated role?

Decide whether the work is sustained and substantial enough for a dedicated owner by writing down the workload, the risk, the capacity you already have and the outcomes you expect.

Answer these before the role is approved

1.Which release decisions in the next year will depend on knowing whether a model or prompt change made answers better or worse?
Listen for: Named features and upcoming changes — a model upgrade, a new retrieval source, a prompt rewrite.
Red flag: 'We want better AI quality' with no decision that would use the results.
2.When the last model or prompt change went out, how did the team decide it was safe?
Listen for: An honest description — often a few people reading outputs. That gap is the case for this hire.
3.Could an engineer or data scientist already here take on evaluation as their main job within two quarters?
Listen for: A named person with time and interest, or a clear reason the work keeps losing to feature delivery.
4.Is the measurement need tied to one launch, or will new model versions and prompts keep arriving?
Listen for: Repeated changes across several features point to a permanent role; one launch may suit a contractor.
5.What has it cost when a change went out and quality dropped without anyone noticing?
Listen for: A concrete consequence: complaints, a rollback, support load, or a decision reversed later.

Build, borrow or buy?

Build: give a data scientist or quality engineer protected time and a mentor who has run model graders against human labels.
Borrow: bring in a contractor to set up the first test sets and grader checks, with a written method the team keeps using after they leave.
Buy: hire a permanent evaluation engineer when several features ship model or prompt changes and each needs comparable evidence.

Watch out for

Mixing this role up with an AI engineer. If you need features built and also tested, hire an AI engineer; if several teams ship changes and nobody trusts the measurements, hire an evaluation specialist.

Stage 2 of 13Required

0 of 2 done

Agree success

Intake meeting and competency model

Who runs it: Recruiter + hiring manager
Done when: A signed-off competency model: 5–7 competencies, each marked must-have or trainable, with weights.

Key tool

Ask the hiring manager

1Which of the systems or areas this person will own worries you most, and why?
2When something there went wrong recently, how did you find out?
3In 12 months, what will be shipped, fixed or running for you to call this hire a success? Agree two or three measurable outcomes.
4Who will they work with most closely?
5What could a strong candidate learn in their first three months? Those capabilities are 'trainable' — never use them to filter applications.

AI as an advantageAI boost4D framework

Delegate: Hand AI your raw intake notes to turn into a draft competency model. Choosing the must-haves and the weights stays with the hiring manager and recruiter.

Show the prompt and the checks

Describe: You are helping a recruiter and a hiring manager build a competency model for a LLM Evaluation Engineer role. Context: I will paste the notes from our intake meeting, including the 12-month outcomes and what the hiring manager worries about most. Produce a table of 5–7 observable competencies with, for each: a one-line definition; whether it is a must-have or trainable, with the reason; what evidence of it looks like in real work; and a suggested weight so the weights total 100. Use British English and inclusive wording, avoid years-of-experience or named employers as evidence, and keep every competency job-related. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that each competency is observable and job-related, that nothing describes a type of person rather than the work, and that trainable items haven't been marked must-have.
Diligence: The hiring manager signs off the final model and weights before anyone meets a candidate. No candidate data goes into the prompt; AI drafts, people decide.

Don't miss: Making a named evaluation framework or tracing product a must-have — the tools are young and change often; the measurement thinking carries over.

Framework

Competency model (must-have vs trainable)

A short list of the observable capabilities the job needs, split into what the hire must bring on day one and what you will teach.

Start from the 12-month outcomes, then ask which capabilities produce them.
Keep 5–7 competencies. With more, interviewers struggle to score each one separately.
Mark each must-have or trainable. Only must-haves can count against a candidate.
Give each a weight so they sum to 100, agreed before anyone meets a candidate.
Write what evidence of each looks like, so interviewers know what they are listening for.

Why it works: Agreeing the criteria before meeting candidates stops the bar moving from one candidate to the next, and makes it harder for 'I'll know it when I see it' to decide the hire.

Core intake questions (ask for every role)

Book 45 minutes with the hiring manager before anything is advertised, and write the answers down.
Which of the systems or areas this person will own worries you most, and why?
When something there went wrong recently, how did you find out?
In 12 months, what will be shipped, fixed or running for you to call this hire a success? Agree two or three measurable outcomes.
Who will they work with most closely?
What could a strong candidate learn in their first three months? Those capabilities are 'trainable' — never use them to filter applications.
Who is on the interview panel, and is their time held for the next four to six weeks?

Questions specific to this role

1.Who marks model outputs today — engineers, support staff, contracted reviewers, a model grader — and how often do they disagree?
Listen for: Whether a labelling process exists at all, and whether anyone has looked at disagreement. It decides how much of the first quarter goes on labelling guidance.
2.Can a failed evaluation hold a release, or are results advisory?
Listen for: Whether the role has teeth. Advisory-only results call for stronger persuasion skills; gating results call for stronger statistical care.
3.Do you already use a model to grade outputs, and has anyone compared its verdicts with people's?
Listen for: Whether there is an unchecked automated grader already in use — a risk this hire may inherit.

What success looks like in 12 months

Each model-backed feature that ships changes has a versioned test set and a grader whose agreement with people has been measured.
Model and prompt changes are compared on the same cases and grader before release, and results are written up in one standard format.
At least one release was changed, delayed or approved because of an evaluation result the team trusted.
Product owners can say, in their own words, what each feature's evaluation does not cover.

Watch out for

Expecting this person to also own model training. Building a model and judging one are different jobs, and combining them weakens the independence of the judging.

Stage 3 of 13Required

0 of 3 done

Write the job

Job description and pay band

Who runs it: Hiring manager (content) + recruiter (publishing)
Done when: A published job description that states outcomes, the must-haves, how AI tools are used and the pay information the law requires.

Key tool

LLM Evaluation Engineer

We're hiring an LLM evaluation engineer to build the evidence behind our language-model features in [product]. You'll design test sets, choose and check graders, track results across model and prompt changes, and report what the numbers do and don't show — working with [engineering/product] and the people who decide releases.

What you'll do

•Build and maintain test sets drawn from real usage and known failure types, with labelling guidance reviewers can apply consistently.
•Choose between human review, fixed rules and model graders for each check, and measure how well each grader agrees with people.
•Look for grader bias — preferring longer answers, a confident tone, or the first option shown — and correct for it.
•Run comparable evaluations on every model and prompt change, and flag drops before release.
•Keep evaluation code, test sets and grader prompts versioned so any result can be rerun.
•Write results for product and engineering leads, including what the evaluation cannot tell them.

What you need

•You have built an evaluation for a production system — model, feature or ranking — that someone used to make a decision.
•You have checked an automated grader or metric against human judgement and changed it because of what you found.
•You can explain agreement measures and the limits of a small sample in plain terms.
•You write maintainable [Python] for data and evaluation work, with tests where it matters.

Nice to have

•Experience writing labelling guidance and running reviewer calibration sessions.
•Experience evaluating retrieval systems, search ranking or recommendation.
•Experience with experiment design or online testing alongside offline evaluation.
•Awareness of how AI-specific rules, such as the EU AI Act, may ask for documented testing.

How we use AI tools

•Where company policy permits, you use approved AI tools at work — and since model graders are part of this job, you're expected to test their verdicts against people before relying on them.
•You can say when a fixed rule or a human reviewer is a better grader than a model, and why.
•In the work sample you can use whichever AI tools you normally would. Tell us what you used, and how you tested what it produced.

Pay

Salary range: [currency, minimum–maximum], plus [equity/bonus/benefits]. The range reflects [location/working pattern]. We place every offer within it using the same pre-set, job-related, gender-neutral factors — level, how much release authority evaluation results carry, depth of experience checking automated graders against human judgement, number of features and teams the evaluation work covers — never previous salary or negotiating style.

AI as an advantageAI boost4D framework

Delegate: Hand AI the first draft of the job description from your signed-off competency model. The must-haves, the pay range and the final wording stay with people.

Show the prompt and the checks

Describe: You are helping write an outcome-based job description for a LLM Evaluation Engineer role. Context: I will paste the competency model (must-have and trainable, with evidence), the 12-month outcomes, how the team uses AI tools, the location and working pattern, and the pay range. Produce: a summary paragraph that leads with the problem and what the hire will own; 'What you'll do' as outcomes, not tasks; 'What you need' containing only the must-haves; 'Nice to have' for the trainable items; 'How we use AI tools'; and a pay section that states the range. Use British English and inclusive, gender-neutral wording, avoid hype, and replace any 'X years' with the capability it stands for. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that 'What you need' contains only the must-haves, that no degree, employer or years requirement crept in, and that the pay range and location are exactly what was approved.
Diligence: The hiring manager owns the content and the recruiter owns publishing, including the pay information the law requires for each location. AI drafts the advert; people decide what it says.

Don't miss: Asking for a research doctorate by default. Some evaluation work is research-heavy, but most product evaluation needs careful measurement and plain writing more than publications.

Stage 4 of 13Optional

0 of 3 done

Find candidates

Sourcing strategy

Who runs it: Recruiter / sourcer
Done when: A shortlist-ready pipeline from at least three channels, with outreach tracked.

Use when:: Inbound applications won't give you enough qualified candidates.
Skip when:: Your shortlist already comes from inbound applications or referrals.

Key tool

Search query

("LLM" OR "language model" OR "generative AI" OR "search relevance") AND ("evaluation" OR "evals" OR "annotation" OR "inter-annotator agreement" OR "LLM-as-a-judge") AND ("test set" OR "benchmark" OR "metrics" OR "regression") AND (Python) NOT ("intern" OR "student")

First outreach message

AI as an advantageAI boost4D framework

Delegate: Hand AI the search queries and the first draft of the outreach template. Choosing channels, personalising to a candidate's public work and sending stay with the recruiter.

Show the prompt and the checks

Describe: You are helping a recruiter source candidates for a LLM Evaluation Engineer role. Context: I will paste the must-have competencies, the kind of work we want evidence of, the channels we are using and the pay range. Produce: (1) one search query per channel, using OR for alternatives, AND for things that must all appear and NOT to exclude, with a one-line note on what each query looks for; and (2) a first-outreach template of under 120 words that opens with a [placeholder for something the candidate built], states the problem we are hiring to solve and the pay range, and ends with a low-pressure question. Use British English and inclusive wording. Do not refer to any specific person. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the queries search for evidence of the work rather than job titles alone, and that the template claims nothing about the role or the pay you can't back up.
Diligence: The recruiter personalises every message to something the candidate actually built, and sends it. No names or profile data go into the prompt; AI drafts, people decide whom to contact.

Don't miss: Limiting the search to 'evaluation engineer' titles. Strong candidates often sit in search relevance, data science, quality engineering or annotation operations.

Framework

Channel mix + evidence-first outreach

Spread sourcing across channels where the work is visible, and open outreach with the problem, not the perks.

Pick channels where candidates show their work, not only where they list job titles.
Write one search query per channel. OR joins alternatives, AND joins things that must all appear, NOT excludes. Keep the version that works best.
Personalise the first line to something the candidate actually built or wrote.
Lead with the problem and the pay range. Save the perks for later.
Track reply rates by channel each week and drop the channel that isn't working.

Why it works: Searching for evidence of the work can surface candidates whose current title differs from the vacancy, or whose CV undersells what they have done.

Sourcing basics

Adapt the search query for each platform — the same words behave differently on each site.
Candidates with adjacent titles may have relevant evidence. Search for the work and outcomes the role needs, not the current title.
Channel activity is a lead, not evidence. Review linked work and examples against the written competencies.

Where to look

Published evaluation write-ups and open test sets

A published method or test set shows how the person chose cases and handled disagreement — reviewable evidence, not a claim.

Talks and posts critiquing benchmarks or grader reliability

People who question what a score measures in public are showing the exact habit this role depends on.

Search relevance, data science and quality engineering teams

Engineers who have judged search results or run labelling programmes bring transferable measurement experience under different titles.

Referrals from your engineers and annotation partners

Colleagues who reviewed outputs alongside the candidate can point to a specific test set or report you can read.

LinkedIn and similar networks, searched by measurement work rather than title

Profiles that describe agreement checks, test-set building or release metrics are a better lead than the words 'evaluation engineer'.

Boolean search string

("LLM" OR "language model" OR "generative AI" OR "search relevance") AND ("evaluation" OR "evals" OR "annotation" OR "inter-annotator agreement" OR "LLM-as-a-judge") AND ("test set" OR "benchmark" OR "metrics" OR "regression") AND (Python) NOT ("intern" OR "student")

First outreach message

Hi [name] — your [post/talk/repo] on [specific measurement] stood out, especially [one detail]. At [company] we ship changes to [n] language-model features, and our automated grading doesn't yet match what our reviewers think. We're hiring someone to own that evidence, paying [range]. Would a 20-minute call about the problem be worth your time?

Stage 5 of 13Required

0 of 3 done

Check applications

Application screening

Who runs it: Recruiter
Done when: Every application reviewed against the same written knockouts within five working days.

Key tool

Reject only if one of these is missing

•Can meet the role's work-authorisation requirements by the start date, with sponsorship where the employer offers it. Don't ask for documents at this stage.
•Describes at least one evaluation they built that was used to make a real product, model or release decision.
•Has written production or analysis code in Python, or an equivalent language, beyond notebooks for coursework.

AI as an advantage4D framework

Delegate: Hand AI the job of turning your written knockouts into a consistent checklist with reason codes. Reviewing each application stays with the recruiter.

Show the prompt and the checks

Describe: You are helping a recruiter prepare an application-screening checklist for a LLM Evaluation Engineer role. Context: I will paste the must-have competencies and our written knockout criteria. Produce: (1) each knockout as a yes/no question answerable from an application alone; (2) a short list of reason codes for a 'no'; and (3) a note of any criterion that looks like a proxy — a degree, employer brand or employment gap — rather than a job-related requirement, with a suggested rewrite. Use British English. Do not evaluate any applicant and do not ask for applicant details. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that each question is binary and job-related, and that every flagged proxy has been removed or justified in writing.
Diligence: A person reviews every application against the checklist and records the reason. Never paste CVs or applicant details into an AI tool, and never let a tool screen anyone out — in New York City and the EU that can bring automated-decision rules into play.

Don't miss: Counting leaderboard positions or public benchmark scores as evidence — reporting a score is different from checking whether it measured the right thing.

Framework

Knockout criteria

A small set of yes/no requirements, written in advance, applied to every applicant the same way.

Derive knockouts only from must-have competencies and genuine legal requirements.
Keep them binary and answerable from the application itself.
Apply them in the same order to every application and log the reason for each rejection.
Do not use proxies such as university, employer brand or employment gaps as knockouts.
Anything ambiguous gets a second review or moves forward to the skills stage.

Why it works: Early screening rejects people before richer evidence exists, so its criteria need the most care. Written criteria make decisions reviewable — but each one must also be job-related, applied consistently and checked for unjustified adverse impact.

Screening rules

Do not use degrees, employment gaps or certifications as filters unless they are demonstrably job-related and justified for this role.
Judge the claims in a CV, not how polished it is — a well-written CV may or may not have had AI help, and either way it is the evidence that matters.

Knockout criteria

Can meet the role's work-authorisation requirements by the start date, with sponsorship where the employer offers it. Don't ask for documents at this stage.
Describes at least one evaluation they built that was used to make a real product, model or release decision.
Has written production or analysis code in Python, or an equivalent language, beyond notebooks for coursework.

Green flags

Mentions checking a metric or grader against human labels, and what changed as a result.
Describes a result they reported that was disappointing or inconclusive.
Links to a test set, method write-up or report you can read.

Red flags

Lists benchmark names and scores with no account of how cases were chosen or marked.
Describes model grading as if the grader's verdict were ground truth.
Cannot point to any decision their evaluation work changed.

Watch out for

Marking down candidates from search, recommendation or annotation backgrounds because their past systems were not language models — the measurement problems are closely related.

Compliance checkpointNew York CityEU

For a New York City employment decision, work out whether a tool is an 'automated employment decision tool' (AEDT) under Local Law 144 — one that uses covered techniques to substantially assist or replace a discretionary decision. Before using a covered tool, make sure an independent bias audit was completed no more than a year earlier, publish the required summary, and give covered candidates or employees the required notice at least ten business days before use. In the EU, the AI Act (Annex III) lists AI used to recruit, filter applications or evaluate candidates as high-risk, subject to the Article 6(3) exception; the related obligations apply from 2 December 2027 under the amended timetable. A manual screen does not exempt an AI tool used elsewhere in the process.

Sources: NYC DCWP — Automated employment decision tools · EU AI Act — consolidated Regulation (EU) 2024/1689 · checked 2026-09-25

Stage 6 of 13Optional

0 of 3 done

Recruiter call

Recruiter screen

Who runs it: Recruiter
Done when: A 20–30 minute call that confirms logistics, motivation and one or two must-have signals.

Use when:: You are hiring at volume, or pay and logistics need confirming early.
Skip when:: You have only a handful of applicants — fold the pay and logistics questions into application screening instead.

Key tool

Ask every candidate, in this order

1Tell me about an evaluation you built. What decision was it for, and who used the result?
2How did you decide whether each output in that evaluation was right or wrong?
3Tell me about a result you reported that wasn't the answer people hoped for.
4Our range for this role is [range]. Does that work for you?

AI as an advantage4D framework

Delegate: Hand AI the first draft of the screening-call script and its evidence anchors. Running the call and taking the notes stay with the recruiter.

Show the prompt and the checks

Describe: You are helping a recruiter write a 20–30 minute structured screening call for a LLM Evaluation Engineer role. Context: I will paste the must-have competencies, the logistics we need to confirm (location, working pattern, notice period) and the pay range. Produce: an opening that explains the process, the timeline and the pay range; 4–6 questions in a fixed order, each with one line on what a strong answer contains; and a closing that asks whether the published range works. Use British English and inclusive wording. Do not include questions about salary history, age, family, health or any other protected characteristic. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that every question is job-related and lawful, that the pay question asks whether the range works rather than for history, and that the anchors describe evidence, not impressions.
Diligence: The recruiter asks the same questions of everyone and writes the notes. No candidate details go into the prompt; AI drafts the script, people run the call and decide.

Don't miss: Expecting the recruiter to judge statistical depth. Their part is motivation, logistics and whether the candidate can describe one evaluation they built in plain terms.

Framework

Structured screening call

The same short set of questions for every candidate, with notes taken against what a strong answer contains.

Open by explaining the process, timeline and pay range. No surprises later.
Confirm logistics: location, working pattern, notice period and whether the range works for them.
Ask the same 4–6 questions in the same order, and note evidence, not impressions.
Leave time for the candidate's questions. Don't score their questions unless they test a written, job-related competency.
Decide within 48 hours and tell the candidate either way.

Why it works: Using the same questions and written evidence anchors makes answers easier to compare, and a short, predictable call respects the candidate's time.

Screening call basics

Set the decision timetable before sourcing and tell candidates when to expect each update.
Ask every candidate what draws them to this particular role, and listen for interest in the work described rather than the title.
Confirm notice period and whether they are in other processes, so you can plan the timetable.
End every call by asking whether the published pay range works — never ask for salary history.

Recruiter screen questions

1.Tell me about an evaluation you built. What decision was it for, and who used the result?
Listen for: A specific system, a specific decision and the candidate's own part in it.
Red flag: Describes a dashboard nobody acted on, or can't name who used the results.
2.How did you decide whether each output in that evaluation was right or wrong?
Listen for: A stated method — human reviewers with guidance, a fixed rule, or a model grader checked against people.
Red flag: 'The model graded it' with no mention of checking the grader.
3.Tell me about a result you reported that wasn't the answer people hoped for.
Listen for: Willingness to report an inconclusive or negative result, and how they explained it.
Red flag: Every story ends in a clear win.
4.Our range for this role is [range]. Does that work for you?
Listen for: A clear answer on whether the published range works. Don't ask for salary history — this process sets pay from the published band and pre-set, job-related factors.

Compliance checkpointUS (state and city)EU

Don't ask for salary history. In the EU, national laws implementing Article 5(2) of Directive (EU) 2023/970 must prohibit the question. US rules vary, but jurisdictions including California and New York prohibit employers from seeking salary history. Ask whether the published range works instead.

Sources: EU Pay Transparency Directive 2023/970, Article 5 · California Labor Commissioner — salary history · New York State DOL — salary history ban · checked 2026-09-25

Stage 7 of 13Required

0 of 3 done

Work sample

Skills assessment / work sample

Who runs it: Hiring manager (design) + recruiter (logistics)
Done when: Every shortlisted candidate completes the same job-shaped task, marked against criteria written in advance.

Key tool

Find out why the automated grader disagrees with people

A fictional energy supplier runs a support assistant that answers billing and tariff questions from a library of policy pages. You receive 40 labelled cases: the customer question, the passages the assistant retrieved, its answer, verdicts from two human reviewers, and a verdict from a model-based grader, plus the grader's prompt. The grader passes answers the reviewers fail, and the team is about to use it to approve a release. Treat the 40 cases as a small exploratory set and report counts, not rates. Find out why the grader and the reviewers disagree, change the evaluation — the grader prompt, its inputs, the labels or the guidance — but not the assistant, and write what you would still not trust.

Planned for up to three hours (adjustments available on request). At the agreed stopping time, write a short note on what you would examine next, and what you deliberately left out.

AI as an advantage4D framework

Delegate: Hand AI the first draft of the work-sample brief and its marking sheet. Choosing the task, setting the timebox and marking stay with the hiring manager.

Show the prompt and the checks

Describe: You are helping a hiring manager design a work sample for a LLM Evaluation Engineer role. Context: I will paste the competency model, a real problem the team solved recently with anything confidential removed, the timebox, and our AI-tool policy for candidates. Produce: a candidate-facing brief (what to do, what to hand in, the time cap, how to request an adjustment, which AI tools are allowed and how to report their use); and a marking sheet with 4–6 criteria tied to named competencies, each with what a 1, a 3 and a 5 look like. Use British English and plain, inclusive wording. Do not design anything that needs more than the stated time. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the task fits the timebox, that every criterion maps to a competency, and that the brief gives all candidates the same starting materials.
Diligence: The hiring manager owns the task and the criteria, and people mark every submission. Do not paste candidate submissions into an AI tool for scoring; AI drafts the brief, people mark and decide.

H-EvaluateH-Evaluate can generate a job-specific assessment from your job description, so you don't have to build the task by hand.How H-Evaluate helps →

Don't miss: Letting candidates 'fix' the disagreement by changing the assistant — the brief is about repairing the measurement, and answers that tune the assistant miss the point of the task.

Framework

Work-sample test

A small, realistic slice of the job, marked against criteria written in advance. Give candidates equivalent materials and arrange reasonable adjustments where required.

Take a real problem the team solved recently and shrink it to fit the timebox.
Write the marking criteria first, tied to the competency model.
Timebox honestly — for take-home work, a few hours at most — and say so in the brief.
State the AI-tool policy in writing. Where the job permits these tools, use equivalent conditions in the assessment and evaluate how candidates verify their output.
Mark without CV details where you can, and have two people mark borderline submissions.

Why it works: A well-designed work sample gives direct evidence of job-relevant skill rather than a proxy for it, and gives candidates a realistic preview of the work.

Assessment rules for every candidate

State the planned time cap and how to request an adjustment. At the agreed stopping time, ask candidates to note what they would examine next and what they left out — never reward unpaid overtime.
Give every candidate the same starting materials, and use a second marker for borderline submissions.
Say in writing which AI tools are allowed and ask candidates to tell you how they used them.
Invite every candidate to ask for adjustments, and answer each request promptly.
Pay for anything the task costs the candidate to run, such as model or cloud usage.

The work sample

Find out why the automated grader disagrees with people

Timebox: Planned for up to three hours (adjustments available on request). At the agreed stopping time, write a short note on what you would examine next, and what you deliberately left out.

What the candidate hands in

Your diagnosis: the main reasons the grader and the reviewers disagree, each backed by specific case numbers.
Your revised evaluation — grader prompt, inputs, corrected labels or reviewer guidance — with a note on why each change was needed.
A proposed comparison: how you would measure grader–reviewer agreement for your revised evaluation, and what result would make you change your mind.
A short note for the release owner: what the revised evaluation can support, and what you would still not trust.

Marking criteria (write these first)

Notices that the two human reviewers also disagree on some cases, and treats those cases separately rather than as grader errors.
Spots that the grader never sees the retrieved passages, so it cannot tell a supported answer from a fluent made-up one.
Finds the slant toward longer, confident answers and tests the fix on the affected cases.
Checks whether any reference labels are out of date with the policy pages instead of trusting them as given.
Keeps the assistant untouched and does not claim more from 40 cases than they can support.
Scored by the hiring manager under honest reporting: the closing note is specific enough that the release owner could act on it without a follow-up meeting.

Adjustments: Candidates can ask for more time, or to talk through their diagnosis live with an interviewer instead of writing it all up. The task is designed to be done offline with no model access; a candidate who wants to rerun the grader can ask for a spending-limited key, but it isn't expected.

Watch out for

Rewarding a high agreement number on its own. A candidate who reaches modest agreement and explains which cases remain unreliable has done the harder part.

Compliance checkpointUSGreat BritainEU

In the US, covered employers must provide a reasonable accommodation to a qualified applicant with a disability unless it would cause undue hardship (ADA). In Great Britain, the Equality Act 2010 requires reasonable adjustments in recruitment. In the EU, the duty comes from national equality law. Don't ask for more medical information than the law allows.

Sources: EEOC — prohibited practices · Equality Act 2010 · checked 2026-09-25

Stage 8 of 13Required

0 of 3 done

Structured interviews

Structured interview loop

Who runs it: Interview panel (named in advance)
Done when: Every competency has a lead interviewer and at least one second source of evidence, with the same questions for every candidate.

Key tool

Interview questions by pillar

Free in this kit: Domain questions for every round. The other three pillars and the AI-fluency check are measured by H-Evaluate.

Free in this kit

Senior engineer or data scientist who scored the work sample submissions · Grader design and calibration · Measurement and statistical honesty

1.Take the case where your revised grader still disagrees with both reviewers. What would you need to see to decide whether the grader or the reviewers are wrong?
Grader design and calibration
Good answerGoes back to the policy page and retrieved passages, considers whether the guidance itself is ambiguous, and proposes a third reviewer or an adjudication step rather than assuming people are right by default.
Weak answerDeclares the humans correct without checking, or changes the grader prompt until that single case flips.
2.Your revised grader agrees with the reviewers on more cases than before. How do you know you haven't just fitted the grader prompt to these 40 cases?
Measurement and statistical honesty
Good answerHolds back fresh cases or asks for a new labelled batch to confirm the gain, and treats the 40 as the set used for tuning rather than proof.
Weak answerTreats the improved count on the same 40 cases as confirmation the grader is now reliable.
3.The two human reviewers disagreed on several cases. How did you handle those in your agreement numbers, and why?
Measurement and statistical honesty
Good answerReports reviewer–reviewer agreement alongside the grader's agreement as context, separates and explains the disputed cases, and says how they change what the grader's score means.
Weak answerPicks one reviewer's verdicts as truth, or averages the two without saying so.

Optional follow-ups (2)

4.You need a grader for a new check: whether an answer quotes the correct tariff price. Would you use a model, a fixed rule or a person, and why?
Grader design and calibration
Good answerPrefers a fixed rule comparing the quoted price with the source data, and keeps a model or person only for parts a rule cannot check, such as whether the price was explained clearly.
Weak answerDefaults to a model grader for everything because it is already set up.
5.How would you check whether a model grader favours answers presented first when it compares two responses side by side?
Grader design and calibration
Good answerRuns each pair in both orders, counts how often the verdict flips, and swaps or randomises order in the real evaluation if it does.
Weak answerHasn't considered order effects, or assumes a stronger model would not have them.

AI as an advantageAI boost4D framework

Delegate: Hand AI the drafting of probe questions and strong / weak anchors for one competency. Asking, listening and scoring stay with the interviewer.

Show the prompt and the checks

Describe: You are helping an interviewer prepare a structured technical interview for a LLM Evaluation Engineer role. Context: I will paste one competency, its evidence description and the core question we ask every candidate. Produce: four probes in this fixed order — what would you do, why that choice, what else did you consider, how would you know it worked — tailored to the question; a one-line description of a strong answer and of a weak answer; and two follow-ups to use only if time allows. Use British English and inclusive wording. Keep everything about the work, nothing about the person. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the probes test the named competency rather than trivia, that the anchors describe observable evidence, and that no question touches a protected characteristic.
Diligence: Interviewers ask the same core questions of every candidate and score alone against the rubric. Interview notes and candidate names never go into an AI tool; AI drafts the questions, people interview and decide.

H-EvaluateH-Evaluate can add evidence on Cognitive, Situational judgement, Behavioural and AI fluency for your panel to review alongside these interviews.How H-Evaluate helps →

Don't miss: Accepting 'we used a model as judge' as an answer without asking how that judge was checked against people.

Framework

Structured technical interview

Every candidate gets the same job-specific questions, each interviewer leads on specific competencies, and every answer is probed with the same four follow-ups: what would you do, why, what else did you consider, and how would you know it worked.

Give each competency a lead interviewer, plus a second source of evidence — the work sample, another round or an independent marker. Avoid asking the same thing twice, but never let one person's view be the only evidence for a critical competency.
Ask the three core questions for your round, in order, for every candidate. Use an optional follow-up only if there is time.
Probe each answer: why that choice, what else they considered, and how they would know it worked — until you hear the candidate's own technical decision.
Take notes on evidence during the interview. Don't rate until it's over.
Fill in your interviewer sheet and submit it before you talk to any other interviewer.

Why it works: Structured interviews make answers comparable: every candidate faces the same job-related questions and is scored against the same anchors.

Interview rules

Use the same core questions in the same order for every candidate.
In the work-sample review, understand the candidate's decisions against the rubric — don't turn it into a debate about your preferred solution.
Avoid trivia disconnected from a real job decision. Use a realistic debugging, design or evaluation problem instead.

Watch out for

Drifting into model architecture or training questions. This role is about judging outputs, and those questions favour a different job.

Compliance checkpointUSUKEU

Don't ask about protected characteristics — such as age, disability, pregnancy or family plans, religion, race or nationality — unless applicable law permits the question for a specific, necessary, job-related purpose, and HR or counsel has confirmed that basis beforehand. Ask only whether the candidate can meet the role's work-authorisation requirements. If a candidate volunteers protected information, don't probe; record only what is lawfully needed, keep it separate and restrict access.

Sources: EEOC — prohibited practices · Equality Act 2010 · checked 2026-09-25

Where to find more interview questions

Use web question banks for inspiration only: ask every candidate the same job-related questions, check each one against this kit's competencies, and never ask about protected characteristics. Don't reuse this kit's work sample from a public bank — candidates can find questions that circulate online, so a widely posted question tests preparation more than skill.

Show 7 sites

US OPM — Structured interviews(opens in a new tab)
US federal guidance on building structured interviews, with a guide, training material and sample questions with rating scales.
FreeFor interviewers
Acas — Interviewing job applicants(opens in a new tab)
UK public guidance on preparing fair interviews, asking applicants the same questions and avoiding discriminatory ones.
FreeFor interviewers
Workable — Interview questions library(opens in a new tab)
Sample interview questions for employers, listed by job role, including machine learning, data and engineering roles.
FreeFor interviewers
Indeed for Employers — Interview questions(opens in a new tab)
Sample interview questions for employers by job title and skill, with notes on what to look for in answers.
FreeFor interviewers
Tech Interview Handbook(opens in a new tab)
Open-source software-engineering interview preparation: coding, system design and behavioural questions.
FreeFor candidates and interviewers
Chip Huyen — Machine Learning Interviews Book(opens in a new tab)
Free online book with knowledge questions on maths, computer science and machine learning, grouped by difficulty.
FreeFor candidates and interviewers
Blind — Interview questions channel(opens in a new tab)
Anonymous posts by tech workers about questions they report being asked. Unverified; some content needs sign-in.
Free, some content needs sign-inCandidate reports, unverified

Links checked 2026-10-05. External sites; we don't control their content.

Stage 9 of 13Required

0 of 2 done

Compare evidence

Scoring, calibration and debrief

Who runs it: Hiring manager (chairs) + panel
Done when: A documented hire / no-hire decision traceable to scored evidence against the competency model.

Key tool

Interactive scorecard

Score one candidate against the rubric. Everything stays in your browser. Download it as a spreadsheet for your records.

Evaluation set design Must-have

Weight

What do 1, 3 and 5 look like?

1: Writes test cases from imagination with no link to real use or known failures.
3: Draws cases from real usage once prompted and covers the obvious failure types.
5: Builds sets from logs and failure history by default, writes clear labelling guidance, and can name what the set still misses.

Grader design and calibration Must-have

Weight

What do 1, 3 and 5 look like?

1: Treats a model grader's verdict as the truth.
3: Compares a grader with human labels when asked and fixes the most visible problems.
5: Chooses the grader type per check, measures agreement with people before trusting it, and tests for slants such as length or order preference unprompted.

Measurement and statistical honesty Must-have

Weight

What do 1, 3 and 5 look like?

1: Reports single numbers with no sense of sample size or noise.
3: Mentions sample size and agreement; occasionally over-reads a small difference.
5: Reports reviewer agreement alongside grader agreement and explains the disputed cases, separates tuning cases from confirming ones, and states plainly what a small set cannot support.

Regression tracking across model and prompt changes Must-have

Weight

What do 1, 3 and 5 look like?

1: Compares versions on different cases or graders without noticing.
3: Keeps cases and grader fixed between versions; reviews overall scores more than individual cases.
5: Compares versions case by case, separates real drops from run-to-run variation, and catches regressions before release.

Evaluation tooling and reproducibility Can learn

Weight

What do 1, 3 and 5 look like?

1: Results live in a notebook nobody else can rerun.
3: Versions code and test sets; grader settings are partly recorded.
5: Anyone on the team can rerun a past evaluation and get the same result, with costs per run tracked.

Honest reporting and stakeholder communication Must-have

Weight

What do 1, 3 and 5 look like?

1: Reports only favourable numbers, or buries the finding in method.
3: Reports clearly when asked, including limits, but doesn't always lead with the decision.
5: Leads with what the evidence supports, states what it doesn't, and reports inconclusive results without spin.

Weighted score– / 5

AI as an advantage4D framework

Delegate: Hand AI the drafting of a blank debrief worksheet and agenda from the competency rubric alone. People add the evidence locally; the scores and the decision stay with the panel.

Show the prompt and the checks

Describe: You are helping a hiring manager prepare a debrief for a LLM Evaluation Engineer role. Context: I will paste the competency rubric only — the competencies, their weights and their 1, 3 and 5 anchors. Produce: (1) a blank debrief worksheet with one row per competency and empty columns for each interviewer's independent score, the evidence they cite, the spread of scores and an 'evidence gap?' box; (2) a 45-minute agenda: confirm every scorecard is in and reveal all scores at once, discuss the widest spreads using evidence only, review any must-have scored 1, then look at the weighted total and decide (hire, no hire or one targeted follow-up), and record the reasons; and (3) neutral questions the chair can ask to draw out evidence rather than adjectives. Leave every evidence and score cell empty. Do not suggest a score, a rating or a recommendation. Use British English. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the worksheet follows your rubric exactly — same competencies, weights and anchors — and that the agenda keeps independent scores hidden until the reveal.
Diligence: Interviewers fill in the evidence and scores locally, alone, before the debrief, and the panel decides together. Candidate notes and personal data never go into an AI tool; AI never scores, ranks or compares candidates.

H-EvaluateH-Evaluate gives you a report on each candidate's assessment evidence to bring into this debrief.How H-Evaluate helps →

Don't miss: Letting a strong design round cover for a candidate who over-claimed from 40 cases in the work sample.

Stage 10 of 13Optional

0 of 2 done

Check references

References and background checks

Who runs it: Recruiter (references) + HR (checks)
Done when: At least two references that test specific competencies, plus only the checks that are lawful and relevant to the role.

Use when:: It is lawful where you hire, and specific evidence from a referee would change the decision.
Skip when:: Local law or company policy rules it out.

Key tool

Ask each referee

1Can you recall an evaluation result from [candidate] that changed what your team shipped or when?
2When [candidate]'s results were inconvenient, how did they present them?
3How did they decide whether to trust an automated metric or grader?

AI as an advantage4D framework

Delegate: Hand AI the drafting of reference questions aimed at the competency the panel was least sure about. Choosing referees, making the calls and interpreting the answers stay with the recruiter.

Show the prompt and the checks

Describe: You are helping a recruiter write competency-based reference questions for a LLM Evaluation Engineer role. Context: I will paste the competency model and the one or two competencies where our evidence is thinnest — no candidate details. Produce: 3–4 questions per competency that ask for specific examples ('tell me about a time…') rather than ratings; an opening question about how the referee worked with the person and for how long; and a closing question about what their next manager could do to help them do their best work. Use British English and inclusive wording. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the questions ask for examples rather than opinions, that they stay on job-related competencies, and that none asks about health, family or other protected matters.
Diligence: References are taken with the candidate's consent and interpreted by people. Referee answers and candidate details never go into an AI tool; AI drafts the questions, people decide.

Don't miss: Asking only whether the candidate was rigorous. Almost anyone will say yes; ask for a result they reported that changed a plan.

Framework

Competency-based referencing

Treat references as a last structured conversation about the candidate's work, aimed at the competencies where your evidence is thinnest.

With the candidate's consent, ask for two people who saw their relevant work directly — ideally a former manager and a peer.
Accept suitable alternatives if a manager or peer isn't available, and never penalise a candidate for a former employer's no-reference policy.
Aim your questions at the one or two competencies the debrief was least sure about.
Ask for examples, not ratings. 'Tell me about a time they…' works here too.
Set the timing of each background check by the law and policy that apply.

Why it works: Generic reference questions tend to produce little job-related evidence. Asking for specific examples tied to the role's competencies turns referencing into something you can use.

Core reference questions and checks

Open with: 'How did you work together, and for how long?' — it tells you how much weight the rest can carry.
Ask: 'Tell me about a time something they owned went wrong. What did they do?'
Close every reference with: 'What should their next manager do to help them do their best work?'
Verify only claims that were material to the decision — don't verify a degree the role never required.
Run criminal-record checks only where the role justifies them and local law allows; many roles don't need one.
Schedule formal work-authorisation checks at the legally required time: before employment starts in the UK; for US Form I-9, Section 1 by the first day of paid work and Section 2 within three business days.

Reference questions specific to this role

1.Can you recall an evaluation result from [candidate] that changed what your team shipped or when?
Listen for: A specific decision that moved because of their work, not general praise for rigour.
2.When [candidate]'s results were inconvenient, how did they present them?
Listen for: Evidence they reported unwelcome results plainly and held their position when the evidence supported it.
3.How did they decide whether to trust an automated metric or grader?
Listen for: A second source confirming they checked graders against people rather than accepting them.

Compliance checkpointUS

If a consumer reporting agency provides a background report, the Fair Credit Reporting Act requires a clear, stand-alone written disclosure and the candidate's written authorisation before you order it. Before rejecting someone because of the report, give them a copy and the FCRA summary of rights and a reasonable opportunity to review and dispute it before any adverse action; if you still reject, send a final adverse-action notice. State and local 'fair chance' laws may delay criminal-history questions or checks until after a conditional offer.

Sources: FTC — Background checks: what employers need to know · FTC — Using consumer reports: what employers need to know · checked 2026-09-25

Stage 11 of 13Required

0 of 2 done

Agree the offer

Offer, negotiation and close

Who runs it: Hiring manager + recruiter
Done when: An accepted written offer within the approved band, with a start date.

Key tool

Build the offer around

•Weight: their evaluation results feed real release decisions, not a report filed after launch.
•Resources: a labelling budget and grader model allowance they decide how to spend.
•Independence: a reporting line that lets them deliver unwelcome results without pressure from the team being measured.
•Visibility: time and support to publish evaluation methods externally where the company allows.

AI as an advantage4D framework

Delegate: Hand AI the first draft of the offer message built around what the candidate said matters to them. The figure, the approval and the conversation stay with the hiring manager.

Show the prompt and the checks

Describe: You are helping a hiring manager draft an offer message for a LLM Evaluation Engineer role. Context: I will paste three general priority categories (for example: learning, flexibility, scope) — not the candidate's words, notes or personal details; the parts of the offer; and the start-date options — without the salary figure. Produce: a warm, plain-English message that links each part of the offer to one of their priorities, says what happens next and when the written offer follows, and invites questions. Use British English and inclusive wording, and leave [brackets] for figures and names. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that nothing promised is outside policy or the approved band, that every claim about the role is true, and that the message reads as personal rather than templated.
Diligence: The hiring manager places the offer in the band using the same pre-set factors for everyone and makes the verbal offer personally. Figures and personal details are filled in by people, never pasted into AI.

Don't miss: Offering the role without saying whether evaluation results carry weight in release decisions — strong candidates ask, and a vague answer reads as a reporting job.

Framework

Close plan (motivators → offer → objections)

Build the offer around what the candidate told you they care about, anticipate their objections, and make sure the verbal and written offers match.

List the candidate's top three motivators from your notes at every stage.
Make the verbal offer personally, and link each part of it to a motivator.
Ask what else they are weighing and when they have to decide.
Place the offer in the band using the same pre-set, job-related factors for everyone, and record why.
Send the written offer promptly after the verbal one.

Why it works: A close plan ties the offer to the candidate's stated priorities and makes approvals explicit before the final interview.

Offer checklist and negotiation

Stay inside the approved band. If the candidate is worth more, re-level them with evidence rather than stretching the band.
Trade in things that cost less than salary where policy allows: start date, learning budget, working pattern.
Before the written offer, ask: 'What would make this an easy yes?'
Agree approval and the pay band before the final interview, so the offer isn't held up.
The hiring manager makes the verbal offer; the written offer follows within a working day.
Record which pre-set factors placed the offer where it is in the band.
Tell unsuccessful finalists personally, without avoidable delay, with brief feedback.

What will close this candidate

Weight: their evaluation results feed real release decisions, not a report filed after launch.
Resources: a labelling budget and grader model allowance they decide how to spend.
Independence: a reporting line that lets them deliver unwelcome results without pressure from the team being measured.
Visibility: time and support to publish evaluation methods externally where the company allows.

Compliance checkpointEUUS

Use objective, gender-neutral, pre-agreed factors to place candidates within the band, apply them consistently and record the decision as an internal pay-equity control. Check the national law implementing Directive (EU) 2023/970 and any local record-keeping rules; this recommendation is good practice, not a universal statutory documentation rule.

Sources: EU Pay Transparency Directive 2023/970 · checked 2026-09-25

Stage 12 of 13Required

0 of 2 done

Help them settle in

Pre-boarding and 30-60-90 onboarding

Who runs it: Hiring manager + buddy + HR
Done when: A new hire who has worked through a written 30-60-90 plan tied to the 12-month outcomes, with a check-in at each milestone.

Key tool

Before day one
Access to logged model outputs and existing test sets, cleared with whoever owns that data, before the first day.
Days 1–30: learn
Meets the product, engineering and support people who act on model quality, one at a time.
Days 31–60: contribute
Builds or rebuilds a versioned test set for the feature with the weakest evidence.
Days 61–90: own
Owns release evidence for one feature, with a standard results write-up used by its release owner.

AI as an advantage4D framework

Delegate: Hand AI the first draft of the 30-60-90 plan from the 12-month outcomes. Agreeing the plan with the new hire stays with the manager.

Show the prompt and the checks

Describe: You are helping a hiring manager draft a 30-60-90 day onboarding plan for a new LLM Evaluation Engineer. Context: I will paste the 12-month outcomes from the intake meeting, the systems and people the hire will work with, and anything they will need before day one. Produce: a pre-boarding checklist (equipment, access, a named buddy, a first-week schedule); then three sections — days 1–30 learn, days 31–60 contribute, days 61–90 own — each with 3–4 concrete, checkable expectations and one milestone check-in question. Use British English and plain wording. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that each expectation is realistic for a new starter, ties back to a 12-month outcome, and offers the hire support rather than a test.
Diligence: The manager agrees the plan with the new hire in week one and owns the check-ins. The new hire's personal details stay out of the prompt.

Don't miss: Asking the new hire to build a company-wide evaluation platform first, before they've checked whether any existing test set or grader can be trusted.

Framework

30-60-90 day plan

Three phases — learn, contribute, own — with written expectations for each, agreed with the new hire in week one.

Before day one: equipment, access, a named buddy and a first-week schedule.
Days 1–30, learn: systems, people and the current state of the work.
Days 31–60, contribute: a first scoped piece of work shipped with support.
Days 61–90, own: an area of responsibility with clear outcomes.
Hold a check-in at each milestone against the written plan.

Why it works: Hiring doesn't end at the signature. A written onboarding plan makes expectations and support visible to the new hire and their manager.

Before day one
- Access to logged model outputs and existing test sets, cleared with whoever owns that data, before the first day.
- A grader model key with a spending limit and access to the labelling tool.
- A named buddy who has shipped a model-backed feature here.
- A reading list: the features in scope, past incidents where quality dropped, and any existing evaluation reports.
Days 1–30: learn
- Meets the product, engineering and support people who act on model quality, one at a time.
- Lists every model-backed feature with whatever evidence of quality exists today, however informal.
- Checks one existing grader or metric against a small batch of human labels and shares what they find.
Days 31–60: contribute
- Builds or rebuilds a versioned test set for the feature with the weakest evidence.
- Writes labelling guidance and runs one calibration session with reviewers to measure their agreement.
- Sets up a repeatable comparison that runs on every model or prompt change for that feature.
Days 61–90: own
- Owns release evidence for one feature, with a standard results write-up used by its release owner.
- Reports at least one regression or inconclusive result that the team acted on.
- Presents the evaluation approach and its limits to the wider team in plain language.

Compliance checkpointUSUK

US: the employee completes Form I-9 Section 1 no later than their first day of paid work, and the employer completes Section 2 within three business days after that day (on the first day if the job lasts less than three business days). UK: complete the prescribed right-to-work check before employment starts, and apply it the same way to everyone.

Sources: USCIS — Form I-9 instructions · GOV.UK — Check a job applicant's right to work · checked 2026-09-25

Stage 13 of 13Required

0 of 3 done

Measure time to hire

Measure the process: mean time to hire

Who runs it: Recruiter / talent lead + hiring manager
Done when: A stage-by-stage timing record for this hire, the slowest step named, and one change agreed before the next requisition opens.

Key tool

Record when the hire accepts

•Time to hire: days from the successful candidate entering the pipeline (application or first contact) to the accepted offer. Average it across hires for mean time to hire.
•Time to fill, recorded separately: days from requisition approval to the accepted offer.
•Days per stage, from the stage's first action to its last — and whether the time was spent waiting or working.
•Offer-acceptance rate: offers accepted divided by offers made.
•Where candidates dropped out, and the reason you were given.
•Days spent preparing the 40 labelled cases, reviewer verdicts and grader output before the first work sample could go out.

Check in at 90 days whether the hire is meeting the outcomes in the job description — that tells you whether the process picked well.

Read: mean time to hire Read: time to hire vs time to fill

AI as an advantageAI boost4D framework

Delegate: Hand AI the job of building a blank timing worksheet and the instructions for reading it. Entering dates, working out the timings and deciding what to change stay with the recruiter and the hiring manager.

Show the prompt and the checks

Describe: You are helping a recruiter and a hiring manager set up a timing review for a LLM Evaluation Engineer hire. Context: I will list the stage names we used, in order (for example requisition approved, sourcing, screening, recruiter call, work sample, interviews, debrief, references, offer accepted). Produce: (1) a blank worksheet with one row per stage and columns for start date, end date, days in stage, days spent waiting (scheduling, approvals, no reply) and number of candidates who entered and left; (2) step-by-step instructions for us to work out, from our own records, time to hire (the successful candidate's entry to the pipeline, by application or first contact, to the accepted offer; average it across hires for mean time to hire) and, separately, time to fill (requisition approval to the accepted offer); (3) five questions we should ask about the slowest stage, separating waiting from working; (4) a short template for recording one change to make before the next hire. Use British English. Do not include benchmarks or example figures. Ask me questions if anything is missing. Rule: do not ask for, and I will not paste, candidate names, interview notes, CVs, submissions or any other personal data. If any appears, stop and tell me to remove it.
Discern: Check that the worksheet covers every stage you actually ran, that the two definitions are not mixed up, and that the questions point at the process rather than at any person.
Diligence: People fill in the dates and do the arithmetic from their own records, choose the one thing to fix and write it down. No dates, names or details about any candidate go into the prompt.

Don't miss: Assuming a slow search means a thin market before checking how long the work-sample data took to prepare and approve.

Framework

Mean time to hire, by stage

Time to hire is the number of days from the successful candidate entering your pipeline (their application or your first contact) to their accepted offer; mean time to hire is that figure averaged across hires. Time to fill — requisition approval to acceptance — is a separate measure. Record both per stage, so you can see where the time went rather than only the total.

When the hire accepts, write down the date each stage started and ended, and the day the successful candidate entered the pipeline.
Work out the days per stage, the time to hire and the time to fill, and mark the slowest stage.
Note the offer-acceptance rate and the stage where most candidates dropped out.
Separate waiting time (scheduling, approvals, silence) from working time, and decide which one to shorten.
Fix one thing before the next hire, and write down what you changed.

Why it works: A single total hides where the delay sits. Per-stage timing points at the one step to fix, and the acceptance rate and drop-out points show whether candidates kept pace with the process.

Record for every hire

Time to hire: days from the successful candidate entering the pipeline (application or first contact) to the accepted offer. Average it across hires for mean time to hire.
Time to fill, recorded separately: days from requisition approval to the accepted offer.
Days per stage, from the stage's first action to its last — and whether the time was spent waiting or working.
Offer-acceptance rate: offers accepted divided by offers made.
Where candidates dropped out, and the reason you were given.
Check in at 90 days whether the hire is meeting the outcomes in the job description — that tells you whether the process picked well.

What to log for this role

Days spent preparing the 40 labelled cases, reviewer verdicts and grader output before the first work sample could go out.
How many candidates withdrew after receiving the grader-disagreement brief, and any reason they volunteered.
Whether the labelling budget and grader allowance promised at offer matched what was actually set up by the first day.

Where this role's hiring slows down

Watch for the work-sample data needing fresh approval for each search because it was built from real logs instead of fictional cases.
Watch for the work-sample review depending on the one person who knows the seeded disagreements, with no written answer key for a backup marker.
Watch for the system design round overlapping the work-sample review on grader questions instead of testing set design and regression tracking.

Compliance checkpointUKEU

Under the EU and UK GDPR, tell candidates how long you will keep their data, or the criteria used to decide. Give that notice when you collect data from them directly, or within the Article 14 time limit when it comes from another source (such as a referral or a sourcing tool). Delete or anonymise data when it is no longer needed, keeping only what a documented lawful purpose still requires.

Sources: GDPR — Regulation (EU) 2016/679, Articles 5, 13 and 14 · ICO — UK GDPR guidance · checked 2026-09-25

General information, not legal advice. Rules depend on the employer, the role and where the work is done — confirm what applies with HR or counsel.

Frequently asked questions

What does an LLM evaluation engineer do?

They build the evidence that a language-model feature works: test sets drawn from real use, graders that mark each answer — people, fixed rules or a model — checks that those graders agree with human judgement, and comparisons across model and prompt changes. They also report what the results do not show.

Is an LLM evaluation engineer just an AI engineer who writes tests?

Not quite. An AI engineer builds model-backed features and usually evaluates their own work. An evaluation engineer specialises in the measurement itself — test-set design, grader calibration, agreement and regression tracking — often across several teams' features. If evaluation keeps losing to feature delivery, a dedicated role helps.

How do you interview an LLM evaluation engineer?

Give them a measurement problem rather than a modelling one. This kit's work sample hands over a model grader that disagrees with human reviewers and asks the candidate to fix the evaluation, not the assistant. Follow it with structured questions on test-set design, regression tracking and reporting, scored on a shared rubric.

Can we use a model to grade our model's outputs?

Often, for parts of the job — but only after checking the grader's verdicts against people on a labelled sample. Model graders can favour longer or more confident answers, or the option shown first. A good evaluation engineer measures those slants and uses fixed rules or people where they mark more reliably.

Do we need a dedicated LLM evaluation engineer?

Write down which model-backed features ship changes, how each change is judged safe today, and who does that judging. If the answer is a few people reading outputs, and changes are frequent across several features, the measurement work is sustained enough for a dedicated owner.

Go deeper

Related hiring kits

Guides

Skills tests

See your first AI-generated assessment in 5 minutes

No credit card. No question library to learn. Paste a job description and watch H-Evaluate build.

Request early access