I wrote the following memo 5 years ago (November 2017) immediately after Mashgin raised its Series A. It summarized my thoughts and learnings on hiring at the time. I also added a few updated comments as I read over it 5 years later (all of my 2022 comments are [bracketed] and italicized). Hopefully others find it useful!
Interviewing is actually not very helpful. Or at the very least it is extremely difficult to judge how someone will perform with just interviews, especially when unstructured. (See here, or all the data Google collected.)
Why?
People interviewing usually have relatively little experience, and thus have a poor "base rate" to judge the candidate against.
Answers to questions generally have very little correlation with actual performance.
It's difficult to extract enough information (even in long interview processes) to make a proper call. Imagine going on a 3-hour date, thinking it over for a few days, then asking the person to get married.
So what are other ways to know if someone will be good?
You're friends with them or have worked with them already.
Pointed reference checks from trusted people.
"Trial" side project or task requiring interaction with team. Getting as close as possible to a real working environment.
Recruiter who is both (1) very familiar with your needs/culture, and (2) specialized in hiring for that role.
But short of these things interviews are still necessary. Regardless of the specific process, it is important to have a set plan and follow it for every candidate.
Some advice:
Prepare: don't go into an interview cold. Know what you want to get out of them and have a clear plan for how to evaluate them.
Let them do the talking. You should only guide them and push them. Ask follow ups: Why? What did you do about it? How come?
Brain teasers don't work, and aren't indicative of anything.
The most effective questions are situational rather than just having them recall the past. "Instead of asking candidates to describe how they handled a unique situation in a previous job or organization, it’s more fruitful to describe consistent situations that candidates could face in this job or organization, and ask them what they would do — or how they would reason."
Encourage them to ask questions -- about your questions, you, or the company.
Be transparent and open about your entire hiring process.
Get away from your desk or room: Take them out, take a tour of offices, etc.
The main things you're trying to get are:
Excitement test. Would hiring this person make you more excited to work at the company?
What can they do now, and how quickly could they be productive?
How is this person going to be performing in 1 year from now?
How long does it take for them to learn something new?
What's their growth mindset and can they continually get better?
Will they work well with the team?
How long are they willing to keep pushing on a good project until giving up?
How hard is it for them to change their mind or adjust course?
Do they do the right thing even when they don't have to?
Aside from specific skills, what traits are the best indicators of these?
Integrity: not just honesty, but integrity with themselves, their ideas, and "doing the right thing" when necessary. They seek out truth and embrace failure.
Social intelligence: works well with others and is empathetic/caring.
Intelligence
Raw intelligence
Creativity in problem solving
Adaptability
[One of the best ways I found to test for this is to ask about something they really enjoyed working on. Then grill them with questions about it, diving as deep as possible into the details.]
Drive: is self motivated and can push themselves to get things done, even if it's not enjoyable work (grit). More internally motivated than externally.
[Curiosity: This could be part of intelligence or drive, but it needs to be tested for somehow. I liked to ask questions like "What things are interested in outside of work?" or even better "Pick a topic that's not part of your day job (hobby, book, subject) and take a few minutes to explain it."]
Warren Buffett: "In looking for someone to hire, you look for three qualities: integrity, intelligence and energy. Without the first the other two will kill you."
There's also a problem with hiring the "best" — they are either extremely expensive or have unlimited options so will want to work elsewhere. This is like the Moneyball problem in baseball: the best teams will have the best reputation and most resources to get the best players. But these players aren't necessarily the only best — they are just the ones who look really good based on the most obvious metrics.
So what do you do? Look for talent in places with low competition, that require more work, or who are too "different":
Growth potential -- people who are young or with little experience in the area, but are smart, driven, and internally motivated. You want people at the start of their "performance curve" -- in the 80th percentile that can move up to the 97th over time. [I think I'd change these numbers now. Finding someone in 80th percentile is too low. If inexperienced, you still want them in 90th with ability to move to 99th.]
Interest -- people with unusually strong interest in your product or mission.
Small fish in a big pond -- picked-over, under-utilized talent in large companies who can thrive on a smaller team. [You have to be careful here. Many people, although talented, can work under the bureaucracy of a big company for years and it drains them of the ability to get things done fast.]
Different -- too outside the traditional track to be easily seen or picked up by others.
What about at the team level? What's the right mix of people?
Diversity of thought and backgrounds is very important. You want people with good traits (good character, drive, etc.) and driven toward the same goal(s) but with a wide variety of experiences/backgrounds, and hence ways to think about problems. You don't want to hire a bunch of clones — that may work short term for some problems but will break when things change. See here for facts about workplace diversity in general. [Addendum: this is less important at the very beginning (seed) stage of a startup. With only a handful of people you may want similar types to get along better.]
Good materials:
http://firstround.com/review/The-anatomy-of-the-perfect-technical-interview-from-a-former-Amazon-VP/
https://www.nytimes.com/guides/business/how-to-hire-the-right-person
Stripe is a good case study of hiring processes:
On-site interviews for Engineering: What to expect (PDF) [Link is now broken -- if anyone still has access to this let me know.]
How Stripe built one of Silicon Valley's best engineering teams [Greg Brockman (now president at OpenAI) talks about how Stripe built out their engineering department.]