compumikeOriginally posted on the Triplebyte blog.
My co-worker Daniel is really into Emacs. It's his primary editor, his grocery list (via org-mode), and #4 of the 6 things he can't live without on his OkCupid profile. Other engineers in the office, however, prefer Vim (or Sublime Text, or even RubyMine). Naturally, this leads to war. But after months of salvos (VimGolf one-upmanship, gratuitous references to this Stack Overflow question) no clear victor has emerged.
So last week, I set out to see what the Triplebyte data has to say on the issue. Take this with a grain of salt (correlation does not imply causation), but there is a correlation between what editor an engineer uses, their language and OS, and their ability to pass programming interviews. And, according to our data, Vim users pass interviews at a rate ever-so-slightly higher than Emacs users! [1]
About the Data
At Triplebyte we evaluate hundreds of engineers every week. Each interview involves talking to one of our interviewers (like me!) over Google Hangouts. We record what language and editor each engineer uses. After our interview, we introduce the engineers to other companies (they skip the phone screens).
To gather the data I wanted, I pulled the editor / language data for every Triplebyte interview over the last year, as well as outcomes (pass/fail) for both the Triplebyte interviews and the interviews with companies the same engineers did through our platform. Here is the breakdown of editor use:
Editors and IDEs
It's not just an Emacs and Vi(m) world anymore (was it ever?):
The #1 most common editor we see used in interviews is Sublime Text, with Vim close behind.
Emacs represents a fairly small market share today at just about a quarter the userbase of Vim in our interviews. This nicely matches the 4:1 ratio of Google Search Trends for the two editors.
However, I'm really looking for signals of quality, not quantity. Which engineers are most likely to successfully pass our interview?
Vim takes the prize here, but PyCharm and Emacs are close behind. We've found that users of these editors tend to pass our interview at an above-average rate.
On the other end of the spectrum is Eclipse: it appears that someone using either Vim or Emacs is more than twice as likely to pass our technical interview as an Eclipse user.
I was curious to see what happened after our interview, so I looked at how likely each group was to pass their final onsite interviews and get a job offer at the companies we connected them with:
For the most part, this looks fairly even across editors, which tells us that even if Vim users are stronger as a population compared to Sublime users when it comes to Triplebyte's interview, we're still identifying the correct subset of Sublime users that will do well at the companies we work with.
I was surprised to find one huge outlier here: Eclipse! Oddly enough, it appears that even after we pre-screen Eclipse users for their programming skills, they still get offers from only half as many of the onsite interviews we send them to compared with the rest of our population. I don't have a good explanation for this correlation, and it's something I'll be looking into. (In any case, don't worry, we still happily accept and find jobs for many great Eclipse users too!)
Programming Languages
You can use whatever language you like in the first section of our technical interview. Today, a plurality of engineers are choosing to interview in Python:
Hey, what happened to PHP? As a former PHP developer myself, I found it interesting that it's now just a small drop in the “other” bucket. As one of the interviewers here, I've personally never seen anyone interview in PHP yet (although others have).
Python is also known for being beginner-friendly, so will an Eclipse-like trend play out here? Well, not quite:
In this case, we find that the average Ruby, Swift, and C# users tend to be stronger, with Python and Javascript in the middle of the pack.
As we dig into these numbers, it's worth noting that there's an unavoidable strong pairing between certain editors and certain languages: 90% of Eclipse or IntelliJ users also used Java, and 99% of PyCharm users used Python. One fascinating and statistically significant result I found is that a Java developer who used anything other than Eclipse was 47% more likely to pass our interview compared to those who did use Eclipse.
Here's what happens after we select engineers to work with and send them to onsites:
There are no wild outliers here, but let's look at the C++ segment. While C++ programmers have the most challenging time passing Triplebyte's technical interview on average, the ones we choose to work with tend to have a relatively easier time getting offers at each onsite.
I'm not entirely sure how to explain this, but one hypothesis might just be to look at the overall supply and demand on each side of our platform. We see strong demand for talented systems and low-level programmers from the companies we work with. On the other side, we're constrained by a relatively small fraction of people applying using C++, multiplied by a relatively smaller fraction of those who we find talented enough to work with. Another thought is that perhaps more niche specializations might give more signal to our matching process. In any case, this tells me we should be looking for more talented C++ developers!
Operating Systems
If you've frequented a coffee shop in San Francisco in the past few years, it should be no surprise that the majority of engineers are interviewing on macOS today:
In this case, the most popular choice is also the most successful:
We found that Windows users struggle a bit more on average. In this category, I was surprised not to see Linux in the lead. As a former Linux desktop user myself, I'd have guessed that Linux users might have a wide range of technical knowledge related to their daily computing.
The trend continues in onsite interviews:
Curiously, the Apple fans we work with also seem to pass about 20% more of their onsite interviews than Windows or Linux users. (This reminded me of an old OkCupid blog post comparing data from their Apple and Android users.)
It's hard to know why exactly this happens, but there may be a hidden variable here. For example, while 60% of the people we interviewed used macOS, if you walk into any tech company in San Francisco you're inevitably going to count far more than 60% MacBooks. While our interviews are background-blind, having a few years of professional programming experience usually helps!
Conclusion
As an interviewer and engineer here, I thought it'd be cool to see what kind of data we had on the tools engineers are using today.
In our 2017 dataset, we found that an engineer writing Ruby in Vim on macOS has a better-than-average chance of passing our technical interview, while one writing C++ in Eclipse on Windows is on the other end of the scale. We don't explicitly use these indicators to decide who we work with, but this is a side product of data collected in our interview process.
At a higher level, we believe that the hiring process should be focused on discovering strengths, not uncovering weaknesses, and we've written extensively on How to Interview Engineers. Our best suggestion for companies is to allow candidates to use the tools that let them best show their personal strengths. In our case, this means we allow engineers to choose any language and environment of their choice, and do convenient interviews from home using screen sharing.
If you're an engineer and want to give our process a try, the first step is completing our multiple-choice programming quiz here.
If you're a company looking for engineers and would like to learn more about working with Triplebyte, you can get started here.
[1] To be clear, I don't think this has any predictive value for hiring, and I don't think engineers should change their editor based on our analysis. Emacs (and Eclipse) are great editors! ↩
