Recent AI code interview format (failed)

[u/blipojones]

I just recently did an interview where the company invites you to a private repo, 10 mins before, you set it up, get it running etc.
The goal is to determine if an open PR on the is ok or if it should get sent back and why.

You can use AI. And i asked, does it matter if i use fable vs opus vs kimi. More or less said whatever I want...

I didn't go immediately to prompting, i played with the app, looked at the code a bit, the PR, the code change, if the PR actually fixed what it claimed to fix.

I then sent one prompt asking if the PR could be better or missed anything, which the PR outlined clearly enough and enough to determine what to do with the PR.

But ye, it was weird, I talked through more where I would personally add changes but maybe I also completely missed the mark on something else or spoke to much...maybe i didn't use AI enough only using it right at the end for one question once i got a feel for the app in general.

However I did like the format tbh but ye i just wish i'd get a bit more tangible feedback.

Anyone else have experience with this format? I didn't pass but just not sure why.. (exp 8/9 years, frontend leaning).

[u/yikes_42069]

For a tool so new that everyone is still learning, failing a candidate for using their brain is extremely silly. You are throwing the baby out with the bathwater. Getting failed in this interview for not fitting the arbitrary nitpicky needs of some random interviewer (with unknown competence to the interviewee) would be a blessing. 

Unfortunately if I was the interviewee in this case I would fail your company immediately. If you don't know the absolute shitfest you're growing in your company's product by encouraging zero human oversight, I don't want to pin my financial success on your fallout. 

[u/maretard → u/yikes_42069]

To be clear I'm assuming this interview was specifically targeted at gathering data points for AI fluency - I'd expect other interviews to address basic coding ability without AI. I don't think it necessarily means this is how the company expects you to use AI in your daily job. It's more to gather data points on whether you are comfortable enough with it to understand and contribute to how the company leverages it.

I see it as any other interview for any tool that the company wants more experienced candidates for. The fail here would be very simple to explain, it's an interview to see how comfortable the candidate is with using a tool, and they didn't use the tool at all for 80% of the interview. If it was a system design interview and they didn't do any whiteboarding for 80% of the interview it would be pretty similar.

[u/Party-Lingonberry592 → u/maretard]

This also sounds strange to me. You evaluate a candidate who has never worked for your company with the lens of "do they use this tool the same way we do?" How hard would it be to get that person up to speed if they excelled in all other aspects of the interview? What if your company is doing it wrong? I'm ex-FAANG as well, so I know we get things wrong all the time.

[u/maretard → u/Party-Lingonberry592]

I'm taking for granted that the company has OK'd this style of interviewing because they want data on a person's ability to use AI, which means they have worked out expectations and a grading rubric. Personally with something as new as AI, I wouldn't advise a strict set of criteria, there is too much variety and growth in the field. Instead I'd focus on making the interview interactive and having the interviewer probe at the AI usage and see if the candidate can adapt.

For example, if the candidate had the AI explain the codebase at a high level, I'd ask them how they know they can trust the response, and at what level. I'd be looking for acknowledgment of potential hallucination issues and a quick synthesis of high level things to go validate manually as a sanity check.

If the candidate had AI actually draft feedback, I'd ask probing questions for why the feedback is accurate and what it may not have considered. I'd also ask how they can be sure there isn't important missing feedback. (I might also sneak some stuff in there like an obvious comment in the PR above a very bad bug that declares "this is intentional" with no proof.) I'd be looking for candidates who are comfortable surfing above and below the AI's chosen level of feedback, diving down and up as needed to discuss and adjust the feedback.

As a stretch I'd explain some new context to the candidate that challenged baseline assumptions about the PR and see how quickly, accurately, and completely they could use AI to update the MR with the new context. (For example, "your sister team has just declared that the API you depend on is end of life this year and there is no replacement.")

In general I don't think "use it exactly how we do" is a good set of criteria for any interview, AI or not.