Assessing AI ability in engineers requires looking past tool familiarity and into working habits: how they direct models, review output, and recover when the model is confidently wrong. A developer who has used AI in production will show this in how they describe tradeoffs, not just in which tools they name.
Why the usual interview fails
Asking 'do you use Copilot or Cursor?' tells you almost nothing. Tool use is near-universal now. The question is how an engineer uses these tools, what they check, and where they decide to stop relying on them.
A standard coding test, done alone and timed, also gives limited signal. Engineers who rely on AI assistance in their actual work will naturally reach for it during a test. Banning it measures the wrong thing. Allowing it without a structured debrief gives you no way to distinguish the engineer from the output.
What to look at instead
How they describe a recent piece of AI-assisted work. Ask them to walk through something they built with AI help in the last month. You are listening for: what prompt or context they gave the model, what came back that was wrong or incomplete, what they changed, and how they decided it was ready to ship. Vague answers, or answers that treat AI as a black box they trusted uncritically, are informative.
How they review AI-generated code. A useful exercise is to give them a short piece of code they did not write, some of which has subtle errors. Watch whether they read it or skim it. Watch whether they catch issues that are plausible-looking but wrong. This overlaps with normal code review skill but is sharpened by the fact that AI output tends to fail in particular ways: confident hallucination of APIs, logic that is structurally sound but behaviourally wrong, security assumptions baked in silently.
How they talk about context. Engineers who use AI well tend to have a clear view of what to put in a prompt, how much history to include, and when a conversation has drifted far enough that starting fresh produces better results. If they have never thought about context engineering, that is a signal. If they have strong opinions about it, that is also a signal, and worth exploring.
How they handle an incomplete spec. Give them a brief that is intentionally missing details. Do they ask clarifying questions, make stated assumptions, and flag what they would want confirmed before generating anything significant? Or do they fill gaps silently and deliver confidently? AI tools amplify both good and bad habits around ambiguity.
Where they say the model was wrong. Ask directly: 'tell me about a time the model gave you something that looked right and wasn't.' If they cannot recall a specific instance, they are either not using AI meaningfully in production or not reviewing the output. Both matter.
Signals that do not mean much
Knowing which tools exist is not evidence of skill. Being able to describe what a RAG pipeline or an MCP server is does not tell you whether someone can build or evaluate one. Certificates and course completions in this area have limited predictive value; the field has moved faster than most curricula.
Structuring the debrief
If you allow AI use during a technical exercise, build in a 20-minute debrief immediately after. Ask them to explain every part of what was produced. Ask what the model got wrong and how they caught it. Ask what they would do differently with more time. This approach is more useful than any take-home that arrives polished and without a conversation.
For more on what has changed structurally in this process, see what replaced the coding test.
What we test for
At Miyagami, we assess every engineer twice. The first assessment runs without AI tools and covers fundamentals including debugging, prioritisation and communication. The second gives candidates their normal AI tooling and watches what they actually do with it: whether they verify AI output before trusting it, how they handle tool orchestration, and whether they maintain ownership over code they did not type. Judgement by risk and leverage are scored across both. Full details of what we look for and why are at how we vet.
Short answers
Should I ban AI tools during a technical interview?
Banning AI tools measures a way of working that differs from most production environments. Allowing them with a structured debrief gives more useful signal: you can see what the engineer accepted, what they caught, and how they explain the result.
What is the single most useful interview question for AI ability?
Ask the engineer to describe a recent piece of work they built with AI assistance and where the model was wrong. The quality and specificity of the answer tells you more than any tool-familiarity question.
Does knowing about AI tools like RAG or agents signal strong AI ability?
Conceptual knowledge is a weak signal on its own. The more useful evidence is whether the engineer can describe how they used a tool in production, what went wrong, and how they caught it.
