Years ago, real estate success story and $1.4 billion company Trulia was in its infancy and on the hunt for a VP of Engineering that would take the site to the next level. It wasn’t going to be easy. The initial team was a close-knit group of hardcore developers led by a pair of seasoned founders, and they weren’t going to let just anyone lead the technology organization.

Less than a mile away, another company was just getting started: a nascent Riviera Partners, founded by tech recruiters Michael Morell and Ali Behnam. Individually, they had a roster of wins, but their company’s engineering-focused model was still new and untested. Trulia bet on them, and after looking at half a dozen candidates, they landed on Daniele Farnedi.

“He had all the right DNA of a great VP of Eng, but he hadn’t been the No. 1 engineering leader anywhere before,” says Morell. “At Trulia, he went from 10 folks in the company to now. He’s still the head of technology six years later, and after going public. It really came down to finding someone who was smart, but also very curious about solving problems, with a high like-ability factor. He had success leading teams in the past, at Trulia he just gave it his own flavor.”

Since this search, Riviera has helped build out the technical leadership at Twitter, Uber, Hulu and Zappos, among others. With 70 employees ensconced in the same waterside office building as Dropbox (another client), the company has developed something of a secret sauce to successful hiring.

Over the past few decades, Morell and Behnam have developed a finely tuned sense of what can make or break a VP of Engineering hire, and they shared their knowledge in this exclusive First Round Review interview.

The Biggest Mistake You Can Make

When recruiting for a VP of Engineering, the biggest mistake a startup can make is thinking they know exactly what they need. It’s an easy trap to fall into. You think you know your team, the challenge at hand, and the kind of person it would take to lead everyone to glory. But according to Behnam, “there’s nothing worse than hiring in your own image.”

Morell and Behnam say they see this all the time. Many young, first-time CEOs say they want to find someone who – like themselves – is willing to work 18 to 20 hours a day and weekends. They want someone who codes and can be hands-on. And they want someone who has a strong network to hire from and can manage and grow team from 5 engineers to 500. But most of the time, this person doesn’t exist.

“If you can get a guy who can take you all the way from early stage through an exit, then you’re very, very lucky. If you want someone who can scale a team, that person will probably need a bit more experience which comes with age, and those people usually have families and can’t work those hours,” says Morell. “And the coding, that goes away once you get past eight or ten people. Founders see coding as a proxy for somebody being technical, but that’s not what you want your VP doing day-to-day.”

Another mistake is to hire based solely on general examples of success, and the pedigree you think a VP of Engineering needs to have. “Facebook is an obvious success story for most people, so they tell us, ‘Get us someone from Facebook,’” Morell says. “But when you really drill down, they can’t really tell you why, other than they think that’s what they are supposed to want, or that they think Facebook’s success is replicable just through one hire.”

Want to learn how to interview? Read: The anatomy of the perfect technical interview from a former Amazon VP

Instead, Morell advises designing the role you want to fill by outlining the specific pain points and problems you’re trying to solve – and then determine your requirements and measure them rigorously (i.e. determine how important they are to you). “If you don’t measure them, your requirements are sure to change over time, especially as you interview people and like one thing more in one guy, and something else in another. The job description you had starts to shift to accommodate people.” But, if you know your number one need is hiring a team of engineers, you better have conviction this person can do that – and be world-class at it. During the interview process, you want to see a very specific pattern of that skill in the candidate’s background.

Built to Scale

Naturally, the recruiting process will vary depending on the size of your company. As Behnam puts it, there are “different generals for different battles and different wars.” Riviera helped Twitter find its VP of Engineering when it was trying to solve the ‘Fail Whale’ problem, but since then, two others have held the post. They started working with Warby Parker and One Kings Lane on the ground floor, but were brought in by Zappos much later.

A successful VP of Engineering hire should get you a minimum of 18 to 24 months.

Some VP Engineering candidates relish the chance to build something out of nothing. Others are fascinated by the people engineering required to make multiple management tiers churn out great work. Only so often do you come across someone who can take an engineering team from start to exit, Morell says.

“A successful VP of Engineering hire should get you a minimum of 18 to 24 months,” he says. “You should feel very comfortable if you think they can get you that far with hopes that they can go even further. But there are different skills for different phases of a company’s lifecycle. There are those who like breaking paths through the jungle, those who are really good on dirt roads, and those who are best on paved roads.” It’s up to the founding team to recognize which phase they are in to find the right person to get them to the next stage.

“At the earlier level, there’s a lot more what I think of as hand-to-hand combat, and the person has to be willing to roll up their sleeves, fix some bugs,” says Behnam. “Then when you get to the later stage, like Twitter is at now, you need someone who can be a conductor. Maybe they don’t know how to play every instrument in the orchestra, but they know how it should sound, and when it’s out of key or out of tune. The secret there is their ability to work through other people, to pass down communication, to make sure the vision gets translated.”

For very young companies, the question may be when to hire for a VP role in the first place. “Let’s say one of your founders is a great hands-on technologist — you need to look out for that moment when they start spending most of their time doing things they’re less experienced in,” Morell says. “You might be at six or eight people, and now he’s spending most of his time in meetings, managing schedules or on recruiting. That’s when you need to bring someone in to lead the team.”

The real challenge (and need) is being able to anticipate and prepare for this moment, he says. “Even in the best case scenario, you’re looking at three or more months before you get a director or VP settled, so you can’t wait till you’re having problems to start the search,” Morell says. “I’ve seen it a lot where people think they can go a lot further with what they’ve got, and the need for a VP or VP-like role arrives before the actual person. That can be a bad situation.”

When it comes to finding that rare candidate who can scale with a company from this very early stage, Riviera’s founders say the biggest variable is whether someone can work through a layer of management between themselves and the army of engineers below. If they can’t, that doesn’t mean they won’t be right for a smaller startup, but they may eventually need to step aside. A person either scales faster, slower or at the same rate of an organization’s growth – and this rate changes over time.

“Those that can’t manage through a set of managers in a company of let’s say 75 or 200, top out at a certain number. They’re great at getting you to a certain point but then its time for the company to look for a different set of skills,” Morell says.

Finding the right VP for the right stage is also about modulating expectations. “Every startup wants to change the world. Very few do,” Behnam says. “You have to have a sense of self, and where your startup is ranked in the world. Very few companies can get whomever they want. You have to look yourself in the mirror and say, ‘You know what, today I’m not ten out of ten. I’m seven out of ten in the world of the Valley, and I’m going to get a seven out of ten for my VP of Engineering, and that’s the person who will get me to that next step.’”

While Riviera has seen ‘seven’ companies land VPs who could be considered eights or nines, Behnam and Morell say this only happens when a startup has a firm grasp on reality. It might be a battle to get there, but they need to realize it themselves.

In addition, it’s hard to embrace an unpredictable choice when you make a VP of Engineering a committee hire, Morell says. “As a young CEO, you may be trying to make sure that no one gets upset, but what ends up happening is you dilute the result. You can’t give everyone veto power or you end up with the average person everyone liked. You lose sight of what actually makes for a success.” This is almost always true when inexperienced members of the team are involved in hiring a leadership position.

Smart Isn’t Enough

It may go without saying that a good VP Eng needs to have a high IQ, but the chiefs at Riviera are focused on a rarer type of intelligence. “People need to leave the VP’s office and feel like they’re smarter than when they walked in,” says Behnam. “They have to be looked at as someone who isn’t just pulling levers in the ivory tower – they have to bring something more to the table. They need to set up robust recruiting programs to get the greatest people, build brand awareness, they have to be out there interacting with people, the tech blogs, letting their people make the world aware about the cool stuff they’re doing.”

Call it EQ – the drive to help the company in more ways than engineering building blocks. Successful VPs need to be able to transcend the tech requirements alone, and they need to make their team members feel like key participants at the same time. “At the end of the day, a VP is typically measured by how much productivity they are able to get out of their team,” Behnam adds. Doing this creatively, in a way that breeds serious loyalty is what separates the good from the great.

“This is why you have to look for people that others follow around,” Morell shares. “A good VP has a following, and good guys who move with them — they travel in packs. These are the guys who have an approach to make developers productive.”

This is one of the clearest ways to determine top candidates. Interviews, while important, can be much less revealing. As Morell points out, “If someone is smart, they can give you the right answer to any question you ask, but there’s no way to know if they actually did what they said they did without referencing. There’s no replacement for thorough reference checking.”

And when Morell says this, he’s not talking about provided references. “No one should have a perfect reference check,” he says. “And I’ve never called a provided reference that didn’t say the candidate was great.” To cut to the chase, Morell doesn’t just advise contacting other people the candidate has worked with and generally asking around, he says the best tactic is to listen for extremes. “You listen to that person who can’t stop compulsively repeating how great the person is, or that other person who says the candidate was absolutely the worst person he ever worked for. Ultimately, your goal should be consistency,” Morell says. You want to ask enough people to get a constant portrait of the person.

Aside from references – and maybe an entourage – the next best thing to look at is track record. Past experiences, products, and projects can be indicators of future success if read the right way. In terms of what to look for, Morell says he combs backgrounds for people who have:

• Worked at great companies in leadership roles, and have seen what success looks like.

• Been consistently promoted up the chain of responsibility, ideally at multiple companies so they have more than one perspective on technical leadership.

• Built and delivered products that have scaled in the marketplace.

• Made mistakes — even very public ones — and can speak openly and humbly about what happened and how the problem was solved.

• Spent time working across an entire organization, and built relationships to understand the ins and outs of the whole business.

• Spoken externally on a regular basis to help build the brand of a company or product.

• Recognized and helped recruit amazing people, learned how to effectively manage non-performers, and developed their team members into successful leaders in their own right.

VPs of engineering have the responsibility to create a team-based culture that motivates an extreme, almost fanatical work ethic — especially in early days. To find these candidates, Morell and Behnam often times look for talented engineers at the director level, maybe even a tier lower, who have created an inspiring sense of purpose in a team-based environment.

“A VP of Engineering needs to be able to create a vision and a purpose for why people are doing what they do, so they’re more connected,” says Behnam. “It’s imperative that they are able to communicate that down to the next one, two, three levels so that people know why they are doing the things they do day-to-day, and why they should remain excited about their work.”

The best VPs, the magical ones, are able to help their engineers solve problems in a way that makes them feel like it was their idea, that they’ve solved the problem themselves.

It also helps if the VP can speak fluidly about, and even pitch in on the day-to-day work. There’s no need to code on a regular basis, but being familiar with the technology at all levels of a company can be clutch. As Morell says, “Engineers respect other engineers.” The best person for the job is someone who can help “unstick” the engineers on their team when they’ve hit a dead end, he explains. Or, “They’ve got to be able to get in the mud and roll around,” as Behnam puts it.

“The best VPs, the magical ones, are able to help their engineers solve problems in a way that makes them feel like it was their idea, that they’ve solved the problem themselves,” Morell adds. “They say, ‘You may want to consider doing this, or you may want to think about that,’ so that they ultimately get to the right answer, and then it’s their answer. No one likes to be told what to do.”

None of this is to say that it’s a mistake to hire someone who hasn’t been a VP before. Plenty can be learned about someone who has had success at the director level, Morell says. In fact, hiring a first-time VP has its merits. “The guys who are first timers have no choice but to do everything in their power to achieve that win,” he says. “Those are the ones who will do whatever it takes. It’s their one shot to create a legacy as a great VP.” On top of that, seasoned executives can cost more money and equity, and — if they’re jumping from a large company to a small one — may not be used to life in the trenches. “So many companies want someone who has ‘been there and done that,’ but just becomes someone is untested at the VP level doesn’t mean they can’t successfully step into those shoes.”

There are also strong arguments for considering candidates from other sectors. Riviera has seen it work out again and again. More recently, Riviera helped Uber track down its perfect CTO, Thuan Pham, by putting this lesson to work.

“At first we thought we needed a consumer guy, but we ended up with a guy from VMWare — definitely a case of not getting hung up on domain,” says Behnam. “In the end, it came down to the complexity and the scale of the problems the company needed to solve. What they’re doing at VMWare is hugely complicated when you pull back the curtain. The products they build are heavy on computer science. And even though Uber is a classic consumer company, it’s got some big challenges to solve. We thought we needed a classic consumer guy but that wasn’t the case in the end.”

Search Like It’s Your Job

Finding the right VP of Engineering isn’t about investing a certain length of time, the Riviera team says. While most searches should close in under 90 days, the timeline varies. But there are certain tactics that consistently yield better results.

“Candidates come on and off the market, so you have a finite pool and a finite window — a shelf life,” says Morell. “Candidates are like orchids. They bloom for a certain amount of time and then go dormant when they join another company, or something happens to keep them at their current company. Then you have to wait until the next time they’re looking for a new role.” So, while it’s bad to make a snap judgment, speed is certainly of the essence.

“You have to be able to take interviews on the weekends, at night, and get good quick feedback so you can move on to the next person or the next step,” Morell explains. “People shouldn’t feel rushed. We’re not saying buy the first house. We’re saying spend two weekends going to 15 open houses. And when you’ve found the right house, don’t say ‘Well, I want one with slightly different windows.’ Then you’ll just keep looking for a house that doesn’t exist.”

At the same time, the founding team needs to sit down together and truly understand what it is about their company that’s so compelling. “Understand why someone would want to fill that role, what’s going to resonate, what the right alignment would look like,” Behnam says.

Landing this right fit has as much to do with the candidate’s experience as the hiring company’s, Morell says — and this is something that’s too often overlooked. “People remember experiences. Every touchpoint has to be a phenomenal experience. Worst case scenario, you don’t hire them but you end up with a great brand ambassador. You want them to leave saying, ‘Man, those guys really have it together.’ No matter what, they will talk about your company.” And you never know, this might be the reason the right candidate eventually walks through the door.

Landing the best hire is impossible if their experience hasn’t been near perfect end-to-end, Morell and Behnam agree. So frequently, they see companies rush to close a hire and think they can do it simply by being awesome at the end. Instead, “from the very beginning, every time you talk to them should be getting that person closer to saying yes.”

A Final Thought

“Whatever size, the one thing I’d say is if you want to hire a VP, be ready to make the hire. Don’t make a hire just to have a VP — that doesn’t make any sense,” Morell concludes. “I’ve struggled in the past with companies where there aren’t enough pain points to actually merit a hire, and they just end up creating unrealistic profiles of what they’re trying to attain.”