I was reminded recently how much I dislike live coding exercises to determine whether a software engineer should advance in an interview process. And the reason I dislike them is because they give absolutely no signal as to whether a candidate is right for the job.
The Urban Dictionary entry for “LeetCode” is spot on:
There is a mysterious legend that says you don’t actually have to know a thing about software development to get a job at top tech companies, as long as you know how to traverse a BST on a whiteboard.
Yep. In the same way that being a Jeopardy winner does not qualify one to do brain surgery, being able to quickly regurgitate LeetCode trivia does not indicate that a software developer has the practical skills necessary to solve problems for your organization.
The signal you’re getting
When you set a timer and ask candidates to solve the clever problem you copied off a website, you are getting some insight into their abilities.
- The candidate’s ability to interview. Some people are better than others at just getting through an interview process. This is not necessarily a bad thing. Some candidates who are adept at interviewing may also be adept at building applications. But the latter is not the signal you’ll get through this type of screen.
- The candidate’s grace under pressure. This type of tech screen will tell you whether the candidate is able to recall information while being actively judged under an unreasonable deadline. If that scenario reflects day to day life at your company, hiring should not be your focus. You’ve first got a culture problem to repair.
Some candidates are just nervous interviewers whose recall in such high pressure situations is not indicative of their typical work (guilty!). It would be a shame to lose amazing candidates because of an overly prescriptive screening format.
The signal you should be getting
Obviously it’s important to determine whether a candidate has the technical expertise needed in the job for which they’re interviewing. Here is what you actually want to learn from the tech screen:
- The candidate’s ability to solve relevant problems. I want to know if the candidate can solve practical problems unique to their area of expertise. For instance, a front end candidate is less likely to need to immediately recall a good way to determine which three page path is most common and more likely to need to know how to create a styled React component that can fetch data and adapt to different window sizes.
- The candidate’s ability to talk through a problem and to ask questions. A good candidate will ask clarifying questions. They’ll be able to explain their assumptions and the direction they chose.
- The candidate’s ability to evaluate buy vs build. In real life development jobs, we often have to determine if it makes more sense to use a third party solution or to build in-house. A good candidate will sometimes select a third party snippet, library, or framework. Your test should factor in this possibility.
- The candidate’s ability to learn. Sometimes a candidate just doesn’t know something off the top of their head, just like everyone else at your company. A good tech screen isn’t binary. If a candidate learns something in the process and can talk through it, they’ve demonstrated an extremely important skill.
Fuck it, we’ll do it async
To get the right information from a tech screen, I’ve had much better luck with giving candidates a take home project. For front end candidates specifically, here’s what that looks like.
- Use a prepared GitHub repo template. Build some kind of a fake web app (don’t make it anything like your real product, you’re not looking for free work here) that can be installed quickly, probably in React (I’ve used Next.js). Unless you’re looking for your first engineer, you probably don’t need to know about the candidate’s experience at procedural things like setting up the app in the first place—though that could be a useful question in a subsequent interview. Again, we want to get to signals about day-to-day development work as quickly as possible. So give candidates something they can clone, install, and start working with.
- Create a list of tasks. The web app you give candidates might have some bugs. It might need a new component built or an existing component styled. It might even be missing a key dependency. Maybe it needs some data fetched and massaged. You want something like 10 tasks that emulate real life work and progress in difficulty.
- Let the candidate know how they can ask questions. Provide a document (the repo README, probably) with all instructions and tasks. Provide an email address for the candidate to send any questions they have before they start or while they’re working.
- Don’t require all the tasks to be complete. Rather, instruct the candidate to stop working after 2 or 3 hours regardless of completion—and make it clear that the number of tasks they completed doesn’t necessarily determine whether they continue in the hiring process or not. The number of tasks completed—and how well they’re completed—is a useful metric in determining a level for the candidate (mid, senior, etc.).
- Have the candidate answer some follow up questions in writing. In the instructions document, ask the candidate to write out what they had trouble with and how they solved it. Ask them what they would improve about the app or what they would’ve built differently. Here’s where you’re learning how they talk through problems, their approach to architecture, and their ability to collaborate.
- Follow up with a short call to go over their work on the project and their answers to the written questions. Ask the candidate further questions about the decisions they made, and give them a chance to explain. Even gather feedback about the project itself and tweak it as necessary for the next candidate.
We don’t care what the candidate googled while they worked on the project. If they found a third party library that completed one of the tasks for them, wonderful. If they only completed a third of the tasks but did an excellent job and explained their decisions well, maybe there’s a mid level position for them.
This is how we work in real life, and the candidate’s code and explanations will give us plenty of insight into their abilities. Be picky! But make sure you’re not unnecessarily filtering out the best candidates through antiquated screening methods.