Estimated reading time: 7 minutes
Combine the depth of open and closed questions for enhanced interviews
When it comes to interviewing tech candidates, there are two main camps: those who like open questions and those who like closed questions, as well as many flavors in between.
Having conducted over 1000 interviews in 4 years, I’ve had the chance to experiment with different types of questions and to review the success rate and quality of hired candidates.
One thing that struck me was that open questions easily uncovered qualities like motivation and learning capability, whereas other qualities like technical skills or specific work habits were easier to spot with closed questions.
By melding open and closed questions together, I came up with “scalar interviews.”
But before we begin, let’s look at the two camps.
What are open questions?
Open questions are more common, requiring the interviewee to elaborate on their answers. This helps the candidate offer more detail while demonstrating their communication skills.
One of the greater advantages of this approach is that it encourages the candidate to open up more. While the candidate is explaining their solution, you also get to observe how they think about a problem in real-time. As open questions are more difficult to prepare for in advance, you can see how much a candidate draws from their opinion and experience when answering.
While this method has many perks, there are also things to look out for. If, as an interviewer, you’re not skilled at managing talkative candidates who provide lengthy or off-topic responses, you may struggle to extract the specific information you need.
Further, answers are open to interpretation by the interviewer. Therefore, two respective interviewers questioning the same candidate may have different perspectives on the answers supplied.
I also observed that open questions can sometimes make the candidate (consciously or unconsciously) provide what they think is an expected answer, affecting the response’s quality.
Lastly, answers to open questions are often longer, meaning fewer questions get asked in a given time frame than closed questions.
Some examples of open questions are:
- What do you see as your strengths?
- What type of working environment do you excel in?
- Describe a complex project that you were assigned to. What approach did you take to complete it? What was the outcome?
- Give me an example of a high-pressure situation you faced in the last three months. How did you handle it?
What are closed questions?
Closed questions are a little less popular than the other camp; however, they can also provide valuable information to the interviewer.
Closed questions have a lot of advantages, allowing you to measure how deep a candidate’s knowledge is in a specific domain. For instance, you can outright ask someone if they’ve used SQL for data analysis or, depending on their answer, if they’ve used window functions in SQL.
Candidate scoring is easier as all closed questions have a “yes” or “no” answer. This allows us to convert answers into scorecards for a more straightforward candidate comparison. Given the binary yes/no answers, there is no room for the opinions/feelings of the interviewer to sway candidate scoring.
Finally, since the answers to these questions are more straightforward, an interviewer can ask about 5–7 times more questions.
On the other hand, closed questions don’t help the interviewer deep dive into the “why.” We might be able to see the candidate’s specific SQL knowledge, but to understand how they gained it, we need to follow up with an open-ended question. The limiting aspect of the answer “no” to SQL questions can also lead you to miss relevant or adjacent experiences. Not to mention the fact that interviewers miss out on evaluating someone’s communication abilities in the process.
Lastly, questions like these can influence the interviewee’s answers. For example, if you ask someone whether they’ve ever used technology X, they’re more likely to respond with a “yes” than a “no,” even if their experience is limited.
Some examples of closed questions are:
- Do you have experience working remotely?
- List your top two priorities for improving the technology infrastructure.
- Where can someone go to learn more about what you do?
- Which products have you used out of the GCP toolset?
More like this
How scalar interviews bring the best of both worlds
With scalar interviews, the goal is to merge the advantages of open and closed interviews into one approach. They allow us to pose open questions while ensuring that the candidate stays on a specific topic.
The idea is very similar to that of a personality test, where each question is tailored to measure a certain quality.
The first step is to decide which qualities matter for you and the position you’re hiring for.
For my recent hire at an early-stage start-up, I prioritized independence and adaptability to thrive in the dynamic environment of start-ups without sacrificing quality.
When building scalar questions around these qualities, think about the spectrum of the characteristics. For example, the scale for independence would be someone extremely autonomous versus someone who requires a lot of direction. Once the scope is set, we then ask the candidates to mark themselves on it and explain their choice.
With scalar questions, it is important to define the spectrum clearly. The interviewer has to explain the spectrum and the two opposing sides of it, alongside a couple of examples of the pros and cons of both. This allows the interviewee to see that there is no right or wrong answer, and they’re being asked for their personal preference (rather than something they think the interviewer wants to hear).
Examples of scalar interview questions
Some examples of scalar questions can be found below. The goal is to provide the candidate with as much context in the lead up to your question.
- “I see engineers on a spectrum. On one end are “pleasers” who say yes to everything. This is generally desired as it helps keep pushing things forward and provides the engineer with learning opportunities. However, they might sometimes say yes to things they don’t understand, which leads to overcommitment and missed deadlines. On the other hand, there are overly cautious engineers who tend to say no and push back a lot. However, when they commit, they are extremely accurate in their estimations and almost always deliver on time. Where are you on this spectrum and why?”
- “There is often a spectrum when it comes to engineers and testing. Some are skeptical, believing that creating and maintaining a good test suite is difficult without a consistent return on investment. Others are all too happy to test, believing that it’s essential for code safety and refactoring. Where are you on this spectrum and why?”
- “Companies have a life cycle. In large, well-structured enterprise engineers can focus on their small domain. In contrast, start-ups/scale-ups with minimal structure require you to take on much more than your job description. The upside, however, is that you mean a lot to that organization. One great idea can change the future of the company. Where on this spectrum do you feel the happiest and why?”
Scalar questions don’t always have to have two points. I’ve experimented with three, which have also worked. I’d advise against going over this number as having any more dimensions seems to confuse the interviewee.
An example (devised by Eugene Braginets) is the following:
- I believe that there are three types of engineers: tech-driven engineers (those who enjoy working on architecture, performance, and algorithms), user-driven engineers (those who enjoy working on UI, UX, clear API design, clean code, clear error messages), and finally social engineers (those who enjoy to coach junior devs, onboard newcomers, give tech talks, write blog posts). I assume that you’re not one or the other but rather a mixture of the three. If you were a pie chart, how large would the slices be and why?
This question helps the interviewer understand what the interviewee values most and why, while keeping the focus on engineering responsibilities. The interviewee can then explain their reasoning and hopefully back it up with some examples.
Once the interview is done, it’s straightforward to assess whether the candidate meets your ideal profile.
Final thoughts
Finding the right candidate for a tech company has become increasingly challenging. The gap between top and average candidates is narrowing as more people access resources to develop in-demand skills. Using scalar questions can uncover insights into candidates that go well beyond their résumés or their technical expertise. Done thoughtfully, your interview process becomes more than a hiring tool – it becomes a meaningful exchange that reflects your team’s culture, values, and vision. With the right approach, you’ll be able to attract team members who can contribute to driving business success.
