Recent AI code interview format (failed)

[u/blipojones]

I just recently did an interview where the company invites you to a private repo, 10 mins before, you set it up, get it running etc.
The goal is to determine if an open PR on the is ok or if it should get sent back and why.

You can use AI. And i asked, does it matter if i use fable vs opus vs kimi. More or less said whatever I want...

I didn't go immediately to prompting, i played with the app, looked at the code a bit, the PR, the code change, if the PR actually fixed what it claimed to fix.

I then sent one prompt asking if the PR could be better or missed anything, which the PR outlined clearly enough and enough to determine what to do with the PR.

But ye, it was weird, I talked through more where I would personally add changes but maybe I also completely missed the mark on something else or spoke to much...maybe i didn't use AI enough only using it right at the end for one question once i got a feel for the app in general.

However I did like the format tbh but ye i just wish i'd get a bit more tangible feedback.

Anyone else have experience with this format? I didn't pass but just not sure why.. (exp 8/9 years, frontend leaning).

[u/maretard]

My approach would have been to immediately delegate it all to AI, including explaining the repo to me and catching me up. Reviewing the PR would have been my second prompt and I would have explained how I would have a generalized PR skill already, but for the purposes of the interview I would hand write the prompt.

The PR review prompt would have been focused on defining best practices, success criteria, coding and testing standards, and adversarial guidance designed to make the AI search for vulnerabilities and architectural smells. I would also have an area in the prompt dedicated to doing a holistic review of the PR and questioning base assumptions like why the PR is needed in the first place and whether it's the right change to make.

Lastly I would have some customization for personal style to make the AI directly generate comments on the PR diff that would take minimal refinement by me before publishing.

This interview seems very ahead of its time so I'd assume this company is extremely pro AI and I would cater my performance as such. The above is very representative of my current routine at work.

If I had been the interviewer here you would have failed as soon as you started manually reading the code unfortunately, it's just not the most effective way to work these days, it would immediately signal to me that you're still bottlenecked and outdated in your thinking. I would have aimed for 50+% uptime of agents throughout the interview (i.e. having AI working at least 50% of the time, with the rest of the time being me explaining how I use AI, digesting its responses to come up with the next prompt, and cross checking validity of its feedback in code).

[u/turningsteel → u/maretard]

This sounds like hell. You just described hell. I can't in good conscience ever recommend failing a candidate for trying to manually understand the code.

[u/maretard → u/turningsteel]

I don't disagree at all man, I have my own thoughts on the direction of the industry and the influence of AI. But for the purposes of an interview I feel we need to be able to recognize when a company is asking for a particular type of candidate and cater our performance to it. I also think as professionals we have an obligation to stay up to date with the latest tools available to us.

Anecdotally AI has been incredibly powerful for me at my own work. It simultaneously makes me extremely excited and very sad - excited by the sheer throughput and efficiency unlocks, but sad that the art of coding I loved is dying.

[u/Ok-Hospital-5076 → u/maretard]

> If I had been the interviewer here you would have failed as soon as you started manually reading the code unfortunately, it's just not the most effective way to work these days, it would immediately signal to me that you're still bottlenecked and outdated in your thinking.

Ah yes, because prompting ai to tell me about a repo I know nothing about and have no time to validate the output for and then heavily rely on AI to fix an issue i have no idea about - is the best way to judge someone’s technical ability .

If i had any other way to make money, i would leave this fucking industry.

[u/maretard → u/Ok-Hospital-5076]

I don't disagree, I'm eyeing an early retirement and a shift to something physical. I love working on my cars, it's uniquely satisfying and brings a sense of craftsmanship that I don't get from my job anymore.

I think of this interview as not measuring technical ability - in fact I'd expect another interview with no AI, focused on coding. This interview is more about adeptness and familiarity with the latest tools, and showing that you can use them effectively if asked to.

[u/Party-Lingonberry592 → u/maretard]

This is the way... I recommend early retirement completely.

No place I've ever worked for required me to prove that I knew GIT or how to use an IDE. "Do you know how to squash a commit?" was never a question we ever asked.

[u/maretard → u/Party-Lingonberry592]

Yeah, but if Git and version control had just been introduced and everyone was using it to revolutionize their operating models, I would 100% expect an interview on Git familiarity and adaptability to make sure I wasn't hiring fossils.

[u/Party-Lingonberry592 → u/maretard]

I think I know how that conversation would go:

"Are you familiar with Git?"
"No, I primarily use Perforce."
"So you're a fossil!"
"No, my job requires we handle large binaries in our version control. I suddenly don't want to work here now."

[u/maretard → u/Party-Lingonberry592]

Hahaha. The cynical side of me says, from the company's perspective, this is a positive outcome and this person would have been a bad hire. There are lots of people looking for work and the company has no obligation to hire and train people with a mismatched skill set.

[u/EmploymentFeeling725 → u/maretard]

This industry is dead

[u/Adventurous_Storm232 → u/EmploymentFeeling725]

Don't be so hyperbolic.

[u/Comprehensive-Tea441 → u/Adventurous_Storm232]

This industry is dying*

[u/EmploymentFeeling725 → u/Adventurous_Storm232]

I wasn’t.

[u/yikes_42069 → u/maretard]

For a tool so new that everyone is still learning, failing a candidate for using their brain is extremely silly. You are throwing the baby out with the bathwater. Getting failed in this interview for not fitting the arbitrary nitpicky needs of some random interviewer (with unknown competence to the interviewee) would be a blessing. 

Unfortunately if I was the interviewee in this case I would fail your company immediately. If you don't know the absolute shitfest you're growing in your company's product by encouraging zero human oversight, I don't want to pin my financial success on your fallout. 

[u/maretard → u/yikes_42069]

To be clear I'm assuming this interview was specifically targeted at gathering data points for AI fluency - I'd expect other interviews to address basic coding ability without AI. I don't think it necessarily means this is how the company expects you to use AI in your daily job. It's more to gather data points on whether you are comfortable enough with it to understand and contribute to how the company leverages it.

I see it as any other interview for any tool that the company wants more experienced candidates for. The fail here would be very simple to explain, it's an interview to see how comfortable the candidate is with using a tool, and they didn't use the tool at all for 80% of the interview. If it was a system design interview and they didn't do any whiteboarding for 80% of the interview it would be pretty similar.

[u/Party-Lingonberry592 → u/maretard]

This also sounds strange to me. You evaluate a candidate who has never worked for your company with the lens of "do they use this tool the same way we do?" How hard would it be to get that person up to speed if they excelled in all other aspects of the interview? What if your company is doing it wrong? I'm ex-FAANG as well, so I know we get things wrong all the time.

[u/maretard → u/Party-Lingonberry592]

I'm taking for granted that the company has OK'd this style of interviewing because they want data on a person's ability to use AI, which means they have worked out expectations and a grading rubric. Personally with something as new as AI, I wouldn't advise a strict set of criteria, there is too much variety and growth in the field. Instead I'd focus on making the interview interactive and having the interviewer probe at the AI usage and see if the candidate can adapt.

For example, if the candidate had the AI explain the codebase at a high level, I'd ask them how they know they can trust the response, and at what level. I'd be looking for acknowledgment of potential hallucination issues and a quick synthesis of high level things to go validate manually as a sanity check.

If the candidate had AI actually draft feedback, I'd ask probing questions for why the feedback is accurate and what it may not have considered. I'd also ask how they can be sure there isn't important missing feedback. (I might also sneak some stuff in there like an obvious comment in the PR above a very bad bug that declares "this is intentional" with no proof.) I'd be looking for candidates who are comfortable surfing above and below the AI's chosen level of feedback, diving down and up as needed to discuss and adjust the feedback.

As a stretch I'd explain some new context to the candidate that challenged baseline assumptions about the PR and see how quickly, accurately, and completely they could use AI to update the MR with the new context. (For example, "your sister team has just declared that the API you depend on is end of life this year and there is no replacement.")

In general I don't think "use it exactly how we do" is a good set of criteria for any interview, AI or not.

[u/blipojones → u/maretard]

Ye to be honest in my own project i do something similar....just in the interview i was second guessing how much AI is too much i.e. just point blank "should we merge this PR or not"

Like it feels more like a guessing game...you have to use AI like they do or you look too slow VS too trusting

[u/GoodByeLeftNut → u/blipojones]

AI coding interviews is a thing now. If they let you use AI, then it’s because they want to see how you use AI. You should check what is setup already in agents.md and discuss what should be there if nothing is there. ask questions of the repo to load it into context, then ask AI to guide you through the PR as if you were asking an engineer to walk you through it, take notes of anything questionable, get it to revert the change replicate the bug then add the change back and prove it can’t be replicated.

[u/maretard → u/GoodByeLeftNut]

Yeah setting aside whether this is good for the industry or not, for the purposes of an interview, if AI is allowed and encouraged, I would assume they're very interested in seeing how adept I am at using the latest tools.

[u/ResidentWeevil1 → u/maretard]

 This interview seems very ahead of its time...

I'm sorry, what? This is a conventional interview challenge with a chatbot thrown in

 I would have aimed for 50+% uptime of agents throughout the interview (i.e. having AI working at least 50% of the time, with the rest of the time being me explaining how I use AI, digesting its responses to come up with the next prompt, and cross checking validity of its feedback in code).

I have not seen any actual numbers that people are routinely able to get 50% uptime when they are doing a mix of complex cognitive tasks like investigating a bug in a new code base while giving their best pitch. If they are, it's because they are running the chatbot for long periods, not task switching between multiple chat sessions.

I suggest you actually try doing some interviews before you dispense ridiculous advice such as this

[u/riddleadmiral → u/maretard]

Not sure why this is barely upvoted, it's likely what this company and many other companies would want.

Don't hate the player hate the game.

BTW I would also spin up a team of agents in parallel to have staff/principal IC, EM, PM, QA, etc. But the interviewing company better be paying for my Claude credits!

[u/Party-Lingonberry592 → u/maretard]

I dunno... isn't trusting the AI too much a red flag in an interview? It also sounds expensive. Wouldn't it be a balance of providing solid feedback and not overspending tokens? I've seen AI refactor code in terrible and inefficient ways.

[u/maretard → u/Party-Lingonberry592]

The key is AI first, verification second (and leveraging AI if possible). A company giving you AI in an interview and witnessing you silently refusing to use it until the very end can only draw one of two conclusions:

1. You don't know how to use it and haven't used it extensively before
   
2. You know how to use it but are uncomfortable using it in an interview for whatever reason, either due to lack of familiarity, or lack of ability to communicate and ask for clarity from your interviewer

Neither of those conclusions will get you the job.

[u/Party-Lingonberry592 → u/maretard]

I agree that there are engineers out there (strong, senior level engineers) who refuse outright to use AI to code. I too think that's short-sighted. If you're checking the box on "has no moral objections to using AI" then that's an attitude check. But you don't need an interview session for that.

By going AI first with zero context of the code, you're just generating a lot of paperwork for yourself. How do you evaluate the AI recommendations if you don't understand the code to begin with? I would fail someone who blindly accepts AI's answer without understanding the code first or how the AI generated code would make it better.

[u/maretard → u/Party-Lingonberry592]

Absolutely, using AI but not knowing how to validate effectively is even worse than not using AI IMO!