Towards better and less biased Software Engineer Interviews

Engineering interviews can take on weird and painful forms with toxic side effects.

Generally engineers are put through the gauntlet as technical interviews are a hybrid of performing live on stage and answering potpourri technical questions. It’s stressful and difficult for the interviewee to prepare and perform for these, its also difficult for the interviewee to know how to give a good interview.

I suspect the state of tech interviews isn’t poor because of any malice or willful desire to punish all those involved. I think we’re all just continuously learning how to do it better and possibly optimizing for the wrong metrics. So, one way we can improve this situation is to review some of the common bad practices in engineer interviews and think of alternatives. Here’s my list.

1. The ‘guess what I’m thinking’ interview.

Any candidate that knows <this term> and how to use <this method> is clearly a good fit. I’ll just get them to say <it> demonstrating they know how to use it.

Why this is a bad interview:

• The candidate cannot read your mind. Simply guessing what you’re thinking isn’t testing their domain knowledge, its testing their ability to guess what you’re thinking.
• They might have a valid alternative method or solution to the specific one you’re thinking of.
• Can they just google the topic and read wikipedia for 5 minutes to understand <term>? If so, you’re likely not getting a very good signal from this interview.

Alternative:

Prepare a specific problem that is in the domain of the field you’d like to talk about. This doesn’t necessarily mean white board coding, you could ask “How would you go about solving this situation, with these constraints?”

This will require the candidate to demonstrate an understanding of the field instead of <specific google-able term>.

For example lets say you get it in your head that “Amortization is so important… how do I get them to demonstrate they understand this concept”? Stop and turn things around. What is a specific problem that requires amortization? Better yet what you’re really trying to do is speed up a code base, so one possibility is to give the candidate some sub-optimal code and ask them to optimize it.

2. The ‘solve this problem we’re actively working on but haven’t quite figured out’ interview.

What better way to see if a candidate will be a good fit for your team, than to have them work on the very problem you’re working on now?

Why this is a bad interview:

Your depth of understanding on this problem is so wildly different than the candidate’s, it is very difficult to get a good signal from this type of interview. You’ll understand the ins and outs of every finer point, its difficult for candidates to just intuit these.

At worst you’ll come off as having the candidate solve the problem for you. This is a big red flag for a lot of candidates!

Alternative:

Build a problem that is similar, but solvable. Turn it into something sufficiently generic and simple that anyone with the actually skills or background you care about could solve it just by reading it. Again, preparing a succinct problem definition with a real solution is necessary here. If done properly many candidates will love that you’re giving them a problem that is relevant to the work they might be doing, but again be careful not to fall into the trap of expecting of expecting specialized knowledge of the problem you’re giving.

3. The ‘lets just chat about tech stuff’ interview.

I’m going to use my superior 6th sense of predicting when a candidate knows their stuff.

Why this is a bad interview:

If it looks like a genius and quacks like a genius it must be a genius, right? This interview style is so vague it is difficult to objectively delineate a “good tech conversation” vs a “bad one”. If you’re a firm believer in your ability to “sniff out” a good candidate, I’d like you to go through the exercise of writing out exactly what qualities of the conversation determine the candidate’s ability to carry out the work you want them to do. Try to be data driven here and look for hard evidence.

Finally try not to mislead yourself. Know that we’re all susceptible to first impressions, self exposure, confirmation bias, and other biases that cause us to select the wrong candidate for the wrong reasons. By leaning on your “technical 6th sense” you are likely only hiring charismatic candidates, or worse, candidates that have specific physical or behavioral characteristics.

Alternative:

Prepare a collection of specific questions with specific correct answers and positive signals. Do not let the conversation meander and hold candidates accountable for a response.

A good set of litmus tests for whether these are good questions is:

• ask yourself if you could give them to several candidates or just one candidate with a very specific background
• be able to answer “why is this question a good indicator of a candidates ability to do the job?”
• after you’ve given a specific question to a couple of candidates, do some of them get it wrong and others get it right

4. Expecting Senior Engineers to problem solve like a sharp recent grad.

I’m still working towards grey hair status, but in the mean time I’ve been fortunate to work with some seriously talented Experienced Engineers. Sometimes, these are people with 15+ years experience, who enjoy the art of problem solving and have a wealth of knowledge that make them invaluable to your team and organization. Not only do they produce work product, they know to resolve interpersonal conflicts when comments in PRs get heated, they build APIs that prevent technical debt and choose 3rd party dependencies that will cause you to deliver faster and more reliably.

These same candidates likely won’t your standard tech interview.

Why this is a bad interview:

See, it turns out that when you’re saving companies months on their release timeline by fixing data pipelines, correcting flaky tests, championing awesome 3rd party tooling and generally encouraging a top notch engineering culture… you get worse at the kind of problem solving we often test in engineer interviews. These are inherently different mental muscles. And that’s OK! Just because this person can’t invent a data structure to generate space filling curves in log(n) time, doesn’t mean we can’t get some signal on their value.

Alternative:

So the problem is most companies build their interview process to examine the ‘above average engineer’, and this is totally natural. For example, if I need to interview 50 engineers to fill 5 engineer slots, I simply do not have the resources to produce 50 custom interviews tailored to each candidate. So what we’re going to do is this: have two interview pipelines.

If during your recruiting process you suspect that an incoming candidate has Experienced status and that is something you’re interested in, I suggest building and preparing a separate interview for these candidates. Set expectations with the panel and figure out what are the strengths of this candidate that you need to test for. This won’t be the same for every organization but here are some suggestions:
1. Lower your expectations on the problem solving portion of the interview. The candidate should understand the ins and outs of software engineering (they’re still going to be producing code) but they might take longer or require a few hints to actually put the solution together.

2. You will likely want to focus on architecture and system design. They should be excellent at gathering requirements and avoiding common pitfalls around premature optimization and taking on risky technical debt.

3. Dive deep into their experience and find out what they’ve done at prior jobs that might be useful to you.

Conclusion: A lack of preparation can lead to bias

As interviewers we all make the above mistakes; we don’t come prepared and end up falling back on whatever is easiest. Engineering managers often forgo the pre-interview sync, not setting expectations properly ahead of time. The organization as a whole chooses not to train interview panelists in the interest of meeting hiring goals faster.

All of the above decrease the odds that you’re going to get a good read on a candidate and increasing the odds that you’ll fall back on personal biases.

This is because when we arrive to interview without knowing what our expectations are, we go with our gut feeling or intuition. This is a big source of bias. We go with the candidate that looks or talks like us, instead of the candidate that objectively did the best in the interview.

You’re going to see a huge difference in the teams you build, when you start holding candidates to a measurable bar instead of expecting a fuzzy “good feeling”.

Candidates will notice it too and are more likely to enjoy a process that is structured, well practiced and for those are that high value because of their experience — more tailored to them. Demonstrate to them that you’re a good coworker and a sharp interviewer, that your company is full of experts and therefor destined for success.