Technical Interview Performance by Editor, OS, and Language
triplebyte.com
triplebyte.com
Then again, it's rare that I interview anymore. I've done maybe two of those kinds of (pair programming, someone else's rig) interviews in the last ten years and neither was for a job that I really wanted after spending time with the devs conducting the interviews.
You want me to consult with you? Let me talk to your biz people to see if I'm the right person to solve your problems, and let the technical folk go over code I've written and we can deep dive on that if there is any question about my chops. Otherwise, fuck off. I'm too old to play "dance monkey".
I feel you here.
With becoming longer of tooth (in the trade) comes an understanding of how to contribute value via the multitude of avenues that are not just grinding out lines at the keyboard.
I am surprised over and over at the incandescently bright young people who can wax lyrical about monad transformers, but to whom it does not occur to model a domain properly before cutting code, or who lack basic schema design and/or SQL knowledge.
I am also in the boat of having vanishingly few technical interviews in recent years, but on a good deal of those occasions it has been obvious that the interviewer is just so not the person I should be talking to.
Monad Transformers FWIW are bread and butter in Haskell. So it a bit like going on about dependency injection. Not necessarily a sign of being stuck up or head in the clouds.
Microsoft has Haskell in R&D pushing ideas to F# which in turn pushes them to C#.
I do feel for you. Maybe we're about the same age and level of experience? It sucks to be put through some one size fits all hiring process and it sucks to constantly feel like more jr people are reinventing process and best practices every few years. But what are you going to do if you are a consultant? you have to be the boss or you have to play ball.
While teams do need leaders with experience they also need collaborative (vs combative) team players who can ride out the details and help improve hiring and work process in a positive and constructive way.
I think more experienced devs get jaded. It is a trap IMO and probably contributes to a lot of the ageism that is perceived and that really exists in the industry.
I have been in two of them, and they were not a pleasant experience overall.
Most senior engineers take some time to understand what's going before diving into somebody's else code and making changes left and right. We all have that experience where fixing a bug, the right way, starts causing/surfaces other problems in the system. I am talking as a dev that has shipped multiple apps, and some of them very successful.
In anyway, if it works for you, keep on doing it, but it will absolutely turn off the more experienced folks from your company.
I noticed that companies with rigid hiring processes usually also have rigid development processes and those are indeed highly toxic to experienced engineers. I've seen my share of companies whose senior engineers were quite below average, but since they have no other engineers to compare to they come out as rock stars!
This is in stark contrast to companies whose hiring processes are tuned towards more experienced folks.
I'm hardly a spring chicken, but despite watching the industry rediscover the same things over and over, thinking they are discovering something new, and despite the industry looking almost nothing like when I started, and despite all of the crazy broken ways things get done... I _still_ get goosebumps over how excited I am to be working on software.
Getting jaded is giving up. I'll give up when I'm dead.
But the interview process? I went basically 18 years without a serious interview on the strength of my reputation, and then I jumped head first into the modern SV interview process and suffice it to say I got extremely jaded extremely quickly.
BTW - if you are looking and interested in autonomous driving / NVIDIA, ping me on LinkedIn. Same user ID as on HN.
He rightfully doesn't want to spend 3-4 hours of his time on an artificial interview process. You can call that combative or jaded, I call it knowing what you are worth. Or just protecting your limited time on earth.
Seems like you ultimately do want monkeys who consistently put their own interest behind the company's. No surprise that older people are less willing to do this - its not because they are "jaded", its because they recognize that no one else will put their interests ahead of their own.
Its a 2-way interview though, so overall you both steer clear of each other - great.
Or maybe (as you say) a candidate will simply disagree with a potential employer's philosophy on interviews and drop out of the process for that reason alone. That's fine too, and if a company's process is so onerous that nobody worth hiring is willing to go through it then that's on them.
That said, your characterization of the parent as wanting "want monkeys who consistently put their own interest behind the company's" is missing a key third stakeholder besides the company and the candidate - and that's the other employees! In my experience, a company built around a strong collaborative working culture will have employees that a) want to see what it's like working through a problem with you, in some form, b) feel that an environment where every incoming candidate has to pass the same bar, regardless of pedigree, is fairer c) prefer working with others that exhibit the humility you characterize as being a 'monkey'.
Again, ymmv, choose the culture you want, etc...
My idea of a collaborative working culture is where
1. we trust each employee to do their job well enough that we do not have to look over their shoulder
2. we respect each employee enough that we value their time
3. we can work together with low friction
I would not characterize these positions as a lack of humility. The use of 'monkey' was in the sense of OPs use of "dance monkey" where candidates go through a ritualized process that tests little but mostly serves to preserve a social illusion of fairness. Pedigree is one thing, actual experience and references to back it up absolutely is a reason to "bend the bar" unless preserving the appearance of fairness is more important than actually evaluating skill.
But overall, thank god for free choice. These are very different value systems I think - manifesting as differences in interview style, but they are more fundamental political differences.
Pretty much sums it up - when you reach a point of having so much experience you know what has to be done, and how to do it, you'll inevitably take exception to others trying to tell you what to do and how to do it. At that point you could step up and be the person who communicates & reinforces that with the team. People need that, especially if you're the kind who's right often, and helps people.
What's awkward for a lot of engineers is of course the fact that getting into a leadership role inevitably takes you away from the kind of work that first made you get into the business. You have enough experience to know how things should go, but too much experience to continue being an "individual contributor" as they say.
Consulting seems like an interesting "middle path" in that you're an expert and a leader in a way, yet you might still get to do some actual work sometimes!
You have a year or two when you can do that with authority without being in the code day to day. After that, you're a fucking blowhard.
And your response to it was deftly passive aggressive. Kudos to you on mastering that tongue.
Good strategy. The "game" has gone on for so long, now -- and become such absurd caricature of itself (apparently without the participants being aware of this fact) -- that it brings to mind the closing line from WarGames:
"Strange game. The only winning move is not to play."
That's demonstrably false. If you're willing to put up with whiteboard tech interviews you have access to companies that enforce them but will pay very highly, regardless of what you consider their merit to be to hiring in general.
Unfortunately one does not, in general, get a job simply by being "willing to put up with" a whiteboarding session or two.
Leaving aside the whole issue of whether actually done properly (or even reasonably).
Again, presenting this neutrally: the quote about the only way to win being not to play is clearly false. If you choose to play (choose to go through the whiteboard circus), you might win (get paid a lot of money at a large and stable tech company). You also might not, but that possibility is there for any interview.
As to the core of what you're saying -- between the lines, the value proposition seems to come down to this: "Yeah, everyone knows by know those whiteboard sessions are mostly bullshit. But hey, if you're willing to swallow your pride and suck it up like everyone else, these companies might throw a little bit of money your way. Coz that's the name of the game, yo -- in case you haven't heard by now."
At some point one gets tired of basing one's life on considerations like that.
I'm 37 and being working as a coder since 21. I still need to apply to job ads and go through the tests. Not all have whiteboard but most have a puzzle. I don't begrudge 2 hours which is the usual time, but I'd probably reject whole day tests.
I don't tend to stay friends with people for the sake of "having connections" but usually just to be friends. That means I don't have a big network to draw upon. There are so many advertised opportunities I'd miss out on if I said only get jobs through a network.
Some of my work I get through my network, for sure. But I'm picky about what I work on which means that I will often do business development for myself and actively cold call/hustle to scare up work in the area I prefer to be in. Strange for a dev, I know, but it's definitely been worth it for me.
I also like to take a lot of time off between contracts to give myself time to pursue non-dev activities and finding the right balance is hard without actively developing my clients.
For instance I'm finishing up a contract right now that was a green field build-out of a brand new product using Elixir. The only way to find that stuff (unless you're lucky) is to go out and dig it up.
I presume with such an approach there is increased $/day meaning the breaks can be taken without any financial sacrifice as compared to the 9-5?
To make matters worse, my layout isn't installed by default in any OS other than GNU/Linux/BSD (it does come with Xorg now).
Unfortunately, it causes another problem: You can't use a keycode-based (as opposed to keysym) configuration without switching your keyboard back to qwerty.
Super interesting post, Mike! Not surprising that emacs/vim users tested better since use of those is likely correlated with having programmed for longer or having programmed from earlier in life before some of the other editors even existed. Would love to see it normalized by years programming. Also, insert joke about how using emacs or vim probably led to natural selection weeding out all but the most committed programmers or how the bad vim programmers didn't start their Triplebye interview because they're still trying to figure out how to toggle back to command mode to exit. :stuck_out_tongue_winking_eye:
I was surprised to see such a big difference between PyCharms and Sublime. Would love to see a language breakdown for generic text editors like Sublime and Atom and compare python interviewees who used Sublime to python interviewees who used PyCharms. I also wonder if there's some confounding variable since AFAIK there's no free version of PyCharms whereas there are free versions of Sublime and Atom, so an interviewee who has a PyCharms license likely uses it in a school or professional setting.
FYI, I think there's a free version of PyCharm (Community Edition).
You all know the rest of the story, but I'm one counterexample to the "how long have you been coding?" angle. I was meant to discover these editors even though I'm relatively young. (31 heh)
Is there a higher/lower success rate in ops/devops vs pure development?
For example vim is used extensively in ops/devops and emacs use there is quite rare. If there is a higher success rate in ops/devops and if vim is used more in ops/devops - could this influence the results shown?
Fascinating data - thanks for sharing.
I recently started using PyCharm CE over the last year and it really is a lovely IDE. I'm not surprised it works well in interviews, it makes it easy to get things done with minimum fuss.
(Although, to be honest, using the GUI version of GNU Emacs as a Notepad replacement is possible: Just ignore the keyboard commands and interact solely via the menu up top. You even get access to at least some neat major modes this way.
Still, Emacs isn't the default, so anyone using Emacs probably knows why, and won't be the kind of person to use it without getting more out of it.)
Depends, at the school I went to (less than five years ago), you start by learning emacs in the first month. A lot of people just stay with emacs for the remaining of the three years, because that's the first thing they learn.
I know that every SE degree includes a course on practices in which they go over version control, some diff tools, etc. but I wasn't aware of a school teaching vim/emacs. May I ask which school it is that is doing this?
But that was 15 years ago. I have no idea what they teach now.
Are you sure that's actual French?
I should note that the professor, Paul Eggert, is the maintainer of many GNU projects.
> Using Emacs and/or Vim is also a test of how much you care, primarily because they're not the default anymore
That's probably true, but I want to add that I used emacs/vim because I hung around with folks who looked down on other tools. So, it wasn't so much that I cared about my tools as it was about I wanted to fit in.
In terms of "caring," I care enough to learn literally all of the hotkeys available by default in vscode, including some I added and changed. I'm a f'n beast in that IDE :P
There's always a few things that drive you crazy of course, but helm+projectile, flymake, and such are just so damn useful over using vim's poor attempts at the same.
Or it's diminishing returns, and most of it is busywork (fiddling with settings) and the pride of customization -- as opposed to any real tremendous productivity gains.
And I say that as someone who've used Vi(m) for 15+ years in platforms ranging from SunOS (pre Solaris), to Linux, Windows and OS X. I've also fiddled with Emacs for a year or so before (in uni) but it never really stuck. I use ST3 and Vim (terminal sessions) now and don't find Vim offering any huge productivity boost.
For all the folklore it's just an editor with some nice ideas, and a lot of random stuff throw in stolen from this or that earlier environment (read how Joy came up with vi).
Really? I feel so much faster with Vim bindings. Watching other people try to edit text even with OS or other-editor bindings, let alone just selecting text with a mouse, is very frustrating.
I know other editors have bookmarks, and I imagine they might work as well as in Vim. But, besides Emacs, in what other editors can you 'delete everything up to the next `=`' or 'delete everything from the current cursor position to the start of the third word (to the right)'.
Gah – even cutting and pasting more than a line or two is so much easier in Vim than anything else.
And that's just editing text. Editing code is even harder for me! Trying to remove a 'word' in a long camelCase variable without being able to jump to a letter or a search pattern consists of hitting way too many arrow keys.
And how does being able to edit text a few seconds faster contribute to development speed and quality. Not a single bit. While it may let you feel cool, it really doesn't add anything to the bottom line.
This can only really be shown via examples. Let's say, a lazy person has created a 3000 line file containing the declarations of a datastructure, and you would like to modify the constructors to each have an additional shared parameter.
In order to add this parameter you could add some behaviour to the constructor definition.
With vim you have a new choice open to you: you can define a macro that to find and modify all of the constructors in your declaration file in 20 seconds.
Now you can choose how you would like the constructors to look, rather than the choice being made for you by the existing code.
There are of course other ways to bulk-update text files, but the advantage of learning vim is that it is context-free. The same commands used to modify C# are fine to use for modifying javascript or XML. (i.e. you don't have to use parsing libraries)
It gives the ability to edit large amounts of code rapidly.
Refactoring implies design, coming with the right structure, applying some design pattern, etc -- which is the slow part, and not affected by your choice of editor.
If we're talking about "automated refactorings" (of the trivial variety, e.g. extract method, etc) then an IDE is an even better candidate than Vim.
>With vim you have a new choice open to you: you can define a macro that to find and modify all of the constructors in your declaration file in 20 seconds.
Or you know, I can do the same in any other editor with macros. Or use a regex, with almost all editors support and do it in 5 seconds.
For me, vim is not my editor - It is a plugin that I install into all of my editors/ides (emacs, visual studio, etc.) so that the keybindings are familar and all of the normal tools (regex, macros, multiple clipboards) all work the same way.
I never felt typing speed or special text transformation shortcuts being my limiting factor.
At the end of a day, I'd have at most written 200 or 500 lines of code, and removed as much. Most days less.
And I don't write, re-edit and discard 10,000 lines in the process of crafting those new ones.
Speed of navigating between files (which is even better in ST than in Vim, vim, at least on the Mac, takes more 100s of ms to open a file, and NerdTree is crap) and finding the stuff I want matters more to me.
>I know other editors have bookmarks, and I imagine they might work as well as in Vim. But, besides Emacs, in what other editors can you 'delete everything up to the next `=`' or 'delete everything from the current cursor position to the start of the third word (to the right)'.
That a find at best a slight convenience of marginal utility.
My problem while coding is rarely "if I could delete until the next = more easily" but "what algorithm should I use", "What was the person who wrote this smoking" and "where's the documentation for this API".
I also don't buy the "but doing it with a shortcut/motor skills frees my mind from having to think about editing etc". As I've said, I've used Vim for 15+ years (still do, a decade after every time I'm in a shell). I've known all these shortcuts, used them, etc. Haven't seen it make any great difference in that respect. I'd take better code navigation, linting support, etc over text wrangling features any day.
(Besides I can do it just fine in ST anyway with Vintageous/Six/etc).
I don't think anyone argues that they can ship a piece of software noticeably faster using Vim or Emacs. Like you said, we're never limited by editing speed. But it's also true that you're not figuring out what code to write one character at a time. You spend most of your time looking at the code, figuring out what to do next. But once you've decided that, there is a little burst of activity during which the limiting factor actually is the motor skill of executing that desired action. In theory, all the little savings might add up, but again, I don't think anyone seriously argues that. I think it's much more that that microburst of activity felt less frustrating while they were doing it.
You don't because the purpose of all the actions you take while editing text is to not think about it. As a non-vim users, you have trained yourself to not think about the time you spent editing text. You want to move an object of options from one file to another file and move it to another object (something which I did moments ago), then the moment you come to the conclusion that you need to make that change, you start using your mouse, select the object, Ctrl-C, then move to the other file, Ctrl-V, and press the shortcut to auto format. But you don't think about it any more than you think about switching windows when moving from one app to the other.
In vim too I probably think about it no more than you do about your own text editing, but with vim, I don't press modifier keys, I don't reach out for mouse, and opening a file is the same speed as IntelliJ (an editor my coworkers use), using Ctrl-P, if not faster. Traversing files is much faster. Splitting screen and navigating is a billion times faster than what you can achieve on any of these editors. All without using the mouse or modifiers.
But these are only small benefits when compared to the biggest difference I've seen between Vim/Emacs users vs non-Vim/Emacs users. Take for instance, few minutes ago I was doing pair programming with my coworkers, and I noticed that there were thousands of productivity shortcuts that he wouldn't use, and he is not the only one, majority of non-vim using people I've met, fail to make use of these productivity enhancements.
* Pressing Ctrl-C/X without selecting text just cut/copy the whole line. * Pressing double click on a line selects the word, triple click selects the whole line (or paragraph in text) * Using Alt-Tab to switch windows * Ctrl-Backspace deletes the whole word/token * Ctrl+Arrow key for navigation * Split Screen, nearly all their editors support split screen functionality, but opening multiple files in multiple panes is like crazy person talk
Even with this minor experience I have with their editors, I look like a magician using their IDE. Not to mention how slow their editor is while trying to help me by auto-completing. I was done typing the name of the class before their auto-completion kicked in. You call that slight convenience of marginal utility, but to
My point is that, productivity enhancements is a desire people have, which makes these people to move towards Vim/Emacs. For the people who don't even use Ctrl-A to select the whole text, they fail to see what they're really capable of doing on their editor.
> As I've said, I've used Vim for 15+ years (still do, a decade after every time I'm in a shell). I've known all these shortcuts, used them, etc. Haven't seen it make any great difference in that respect.
Considering you complaint about speed of navigating between files, I highly doubt that you know "all these shortcuts", because even if you do, the philosophy of vim is to keep optimizing your workflow, so you create a shortcut for shortcuts. I think it would be a cool idea to hold competitions where two people are given a codebase, and their own completely configured editor on the same machine with same processes running and then they are asked to do a number of tasks, like open UserController.js, UserView.html, UserService.js and refactor a variable and change some data.
You can doubt it, but you'll be wrong :-) Note that I'm not speaking of navigating between already OPEN files (I know how to move between splits, maximize them, etc), but between files IN the project I'm working with. Vim just takes longer for those tasks and gives worse overview/feedback on which they are/where they are/etc, including with the relevant plugins.
>because even if you do, the philosophy of vim is to keep optimizing your workflow, so you create a shortcut for shortcuts
That might be the philosophy of Emacs, but it's hardly the "philosophy of vim". In fact most old-time vim users I know try to keep their customizations to the minimum, so they feel at home to whatever system/vim they're thrown at.
As I said, there are fast plugins for that task and slow plugins. Command-T and Ctrl-P are megafast because they are run as native binaries.
> I never felt typing speed or special text transformation shortcuts being my limiting factor. > > ... > > Speed of navigating between files (which is even better in ST than in Vim, vim, at least on the Mac, takes more 100s of ms to open a file, and NerdTree is crap) and finding the stuff I want matters more to me.
I wasn't making any claims about the benefits of Vim bindings for my job as a whole. But why stop there? It's probably also not the "limiting factor" for me as human being.
But it's definitely a limiting factor when I'm writing and editing text.
> I also don't buy the "but doing it with a shortcut/motor skills frees my mind from having to think about editing etc". As I've said, I've used Vim for 15+ years (still do, a decade after every time I'm in a shell). I've known all these shortcuts, used them, etc. Haven't seen it make any great difference in that respect. I'd take better code navigation, linting support, etc over text wrangling features any day. > > (Besides I can do it just fine in ST anyway with Vintageous/Six/etc).
Above is even more that's besides the point I was making. I'd take good code navigation over Vim key bindings too. I wasn't saying Vim key bindings are better than anything other than not having them.
And then you basically make my point for me:
> (Besides I can do it just fine in ST anyway with Vintageous/Six/etc).
I do that too; not in ST, but in the IDEs, etc. that I use (e.g. Visual Studio, SQL Server Management Studio, Google Chrome). I assume you do for the same reason I do, i.e. you like the keybindings. I also find the keybindings generally useful and faster than not having them.
I do agree with the gist of your comment that Vim itself isn't particularly great compared to tools with features like code navigation. I don't use any Vim plugins for example tho I've tried a few.
I still like vim, I just don't think that argument is very convincing.
On the other hand, being really fluent in an editor (once upon a time in Vi, now Emacs) keeps me in a flow state. I think of code and it appears on the screen without interruption of the deep thoughts.
Obviously this is available in emacs, but not vim without serious configuring, and not with everything staying within a nice single editor context.
Love all lisps, including elisp.
(use-package unfill
:ensure t
:defer t
:commands (unfill-region unfill-paragraph unfill-toggle)
:bind ("M-q" . unfill-toggle))(And that's also why you care about causation even if you are only looking for prediction, because you don't want your proxies to change their meaning as soon as you act on them.)
It is indeed full of correlation that can unmask interesting facts if studied further. And the GP complaint is meaningless if viewed from that angle, so I do really think he was talkinga bout causation.
Let's say you have 100 programmers. 20 of them "care". Of the 100, 5 use emacs. P(emacs) = 0.05. If all emacs users care, then P(emacs|care) = 0.25. If I wanted to model this phenomenon, I shouldn't ignore the correlation just because it's not perfectly causal.
The fallacy if any is assuming likeliness implies implication. This is commonly known as stereotyping, but it's perhaps more clearly described as generalizing a group stereotype to the entire group.
"stereotyping" is a floating signifier, and arguably in common usage doesn't merely signify hasty generalization.
Sure that's possible. And I realize that you're being hypothetical, but if that's how you're going to use Emacs, you're not going to see much gain in your productivity ;)
I started learning emacs some time back, but the learning curve (and fighting against Vim muscle memory) vs. needing to get work done for money soon convinced me to try again another day.
Mastering vim or becoming pretty good in vim, seems quite a bit more difficult.
I've given up on vim 5 times, until someone on HN suggested me vimtutor. It really taught me the value in good educational content.
But it's my go to for Haskell projects.
What I like best about vim is the startup speed and fewer reaches for the mouse. I do like using the mouse to select a large block of text to indent tho!
> insert joke about how using emacs or vim probably led to natural selection weeding out all but the most committed programmers or how the bad vim programmers
uh.... /joke/?
Hardly so, IMO. Plus - and I assume here that you are talking here mainly about developers (using programming languages) and maybe web designers (using CSS/HTML/etc.) - the complexity of vi(m) and emacs can hardly be called higher than those dev and design technologies, except, maybe, for the use of advanced editor features and scripting of those editors in VimScript or ELisp.
Anyway, FWIW, I had published this article (a vi quickstart tutorial) in Linux For You magazine some years ago, after I first wrote it for some Windows sysadmin friends of mine, who were given additional charge of a few HP Unix boxes. After using the tutorial, they told me it helped them to quickly come up to speed on the basics of using vi (the predecessor of vim).
The tutorial is available here; it is short and easy to digest:
https://gumroad.com/l/vi_quick
Edited for wording and typos.
Edit: And have always recommended such an approach to younger developers and aspiring ones, when they ask me for advice, which sometimes happens. Basically tell them to learn as much as you can and invest in your future (with study effort, not only money) for the long term, not just for quick immediate gains.
I really do not understand where this belief comes from. I learnt Emacs as a beginner in the early 2000's when I started university I'd never done any programming before (I'd also never used Unix before) our course was taught on Sun Solaris workstations and emacs I had many frustrations at the time I remember cursing Solaris and the stupid GUI it had (CDE) but never emacs - emacs just worked not sure what is supposed to be torturous about it.
Nearly 20 years later I still use Emacs daily it's one of the programs on my computer I probably "couldn't live without". Maybe at this point I'm just a hostage to my tools but i'm too old and too familiar with emacs to bother trying some other editor out - what do people use now sublime text? (if the article is accurate) I've never heard of it before I don't know anyone here at my work who uses it (I am an engineer not a software person though) could be a generational thing I'm in my mid 30's and one of the younger people at my work so maybe all the "cool" Mac kids use it or something...
It's not torturous to learn Emacs. But Emacs doesn't work like a standard Windows/Mac app, and today we have fewer professional developers willing to make it past the initial rejection phase of "wellp, this is weird".
And yes, today people use Sublime Text, something like Atom, IntelliJ, or -- for hipster cred -- vim. Emacs is apparently retro enough to be a total gonk for new users but advanced enough to not be redeemable for whuffie in hipster circles. Vim closes that gap.
This actually appeared on Hackernews a couple of times, which is how I found it. Money quotes:
"It was soon found that Emacs could be taught within minutes or an hour to those with no technical experience at all." (referring to TECO Emacs actually but applicable to Multics Emacs)
"Non-technical personnel have been observed to acquire enough knowledge of Lisp to extend Emacs for only this purpose."
"While ITS TECO code is quite baroque, and accessible to only a few experts, the extension language of Multics Emacs is sufficiently natural and simple to learn that non-technical personnel have successfully written and debugged non-trivial extensions."
I don't think there is any value in looking at this blog post and making any conclusions about which editor and/or scripting language to learn next. You might go interview at a .net shop and get the opposite result from the one that you wanted.
Disclaimer: I personally match their preferred technologies very closely, but work with some really smart people that write C#/F# in VSCode.
I've had a boss who brought in at least 60 people in (while I was there) for an unsupervised (unless you had a question) 60-90 minute test, where he'd leave them alone, let them google, etc, but would sit down and discuss what they did and why afterwards, and the amount of people who couldn't get past the first FizzBuzz question was pretty high.
Those that got past that, the number who didn't know the basics, like what an Interface was or how they worked (or how you're not supposed to write implementation code directly into them) was higher still.
And these are people who say they have 5-15 years of direct experience on their resumes.
I'm not a huge fan of tests and I do tend to struggle myself, but not that badly, and not when unsupervised.
I can understand performance anxiety being a factor.. but 60-90 minutes and still can't answer FizzBuzz?
In the MS world you can get a fair bit done via drag'n'drop coding, this is the "visual" part of visual studio. From there on their a few effects come into play. One is that they are always putting out fires so they look good to management who don't realize they are also the arsonist. After a certain point most competent people with more options won't go near their creations, or at least they won't stick around long if they do. These people fail into job security.
Also, never underestimate how little work you can get away with in a large organisation.
Anyone who uses MacOS (ie. can afford a MacBook Pro) to work on has to have enough money to buy one despite being in a job interview.
Most people who use vim and emacs are going to be older, more experienced engineers, and as such, have the resources the acquire the tools they need to succeed (time to learn, books, computer, paid IDE).
I see statistics like this and just think about all of the kids who didn't get a free ride through college and a MacBook given to them who aren't succeeding.
It would be interesting to see more data on things like age, experience, demographics, etc. Despite the "correlation isn't causation" mantra, there are only 27 different possible causal graphs, so with enough data maybe we could get to the bottom of this Best Editor business. :P
> Anyone who uses MacOS (ie. can afford a MacBook Pro) to work on has to have enough money to buy one despite being in a job interview.
> Most people who use vim and emacs are going to be older, more experienced engineers, and as such, have the resources the acquire the tools they need to succeed (time to learn, books, computer, paid IDE).
> I see statistics like this and just think about all of the kids who didn't get a free ride through college and a MacBook given to them who aren't succeeding.
I think having money and resources is definitely a deciding factor, but I don't think it has much to do with wealthy backgrounds or free rides through college. Like your third paragraph says, the bias seems to favour older, more experienced programmers. It makes sense to me that more experienced programmers would do better in interviews than kids fresh out of school, and that experience just ends up being reflected in the data as being able to afford expensive tools. Like the article says when it discusses the MacOS bias: "if you walk into any tech company in San Francisco you’re inevitably going to count far more than 60% MacBooks. While our interviews are background-blind, having a few years of professional programming experience usually helps!"
The initial programming quiz has code snippets only in Javascript, Ruby, and C.
The interviewers are usually more familiar with more modern technologies. Indeed "enterprise" (C# or Java) developers tend to do poorly at TripleByte, and Java is highly correlated with Eclipse usage.
TripleByte also recruits primarily for startups which tend to use certain technologies. I'd say the final stage offer data is so biased by the selection process along the way that it's almost impossible to draw any conclusions from it.
Also, there are some rising stars we couldn't include in this analysis, like Go: candidates writing Go during their interview with us did extraordinarily well on average, but we don't yet see enough of them to break them out of the "Other" language category.
People that are coming in with fluency in Go, Erlang, Haskell, etc... indicates that they are paying attention.
And, last year I passed several rounds of technical interviews with one of the top 5 companies in SV, so perhaps I support the statistics presented here (though I did not get to use my chosen editor for the interviews, so they did not know I am an emacs user).
Anyway the day I finally clicked on the extensibility of emacs. Which was as trivial as defining a function, then defining a menu (which is duh.. a list). And have the GUI [1] updated I lost the remaining bit of love I had for Eclipse.
I got 2 or 3 years of Eclipse JDT/PDT, all the talks about the power of plugins. It was nothing but pain.
Then that old emacs gets me the same thing like it's breakfast. It hit me too hard.
Lastly, I revived a win95 box, with Turbo Pascal. Another old and tiny blast from the past, and even though the 600KB IDE/compiler was amazing in most and so many regards. The fact that I was stuck with the default text editing features (no paragraph selection etc etc) made me feel what Emacs' freedom meant to me even on the tiniest instances.
[1] note that I'm a keyboard fetishist, so I don't use 99% of Emacs GUI and knew most keybindings, I was just trying for the sake of learning elisp and emacs architecture.
Really, not a jab at anyone, just that Vim and Emacs are more advanced tools and it takes more advanced users to use them with any gain. More advanced users also use IDEs for some of their nicer tooling - personally I'd kill without Resharper at work.
Hey, I use them badly in real life, too. No reason to make it sound so duplicitous.
Just like in web development, do feature detection, don't sniff for things which might loosely correlate with what you actually care about.
While this data is fascinating, it does lead me to consider what other unknown factors are at play here influencing these results.
Obviously it's just highlighting a single interesting correlation.
Wow someone is going to learn an editor thinking it will help he/she interview better?
I can't count how many people have assumed I'm good just because I can drop technical words intelligently into a conversation, so there's also going to be some lazy folks looking for vim/emacs usage and judging someone's abilities to be a bit higher because of that.
I've seen plenty of professors say they can tell (generally, that is a strong correlation) whether someone will succeed or fail using a single criteria. One example: When hearing a new word, does the student immediately look it up in a dictionary? That person will go far because they recognize and correct areas where they lack knowledge.
Mike Acton of Insomniac Games has one criteria. The only one he's seen actually result in long-term great employees: An insatiable desire to learn. When they hear something they don't know, they go home after work and begin reading. It's basically the same thing as the dictionary.
People who are willing to endure the onslaught that is vim or emacs, are willing to sit down and learn things that aren't easy. It doesn't make you a better programmer. It's just a hallmark of an already great programmer.
When he was a kid, Linus Torvolds used to write programs with a hex editor and directly writing the opcodes (instead of even using an assembler) into a file. That didn't make him a great programmer but the drive to do things like that when most would go "why bother", is what made him a great developer. It stands to reason that when presented with future obstacles, he applied the same gusto and problem solving skills instead of giving up.
> At Triplebyte we evaluate hundreds of engineers every week.
> I pulled the editor / language data for every Triplebyte interview over the last year.
Based on that, if we suppose "hundreds" means "low hundreds," this suggests a sample size of 5K-15K.
That's not a bad sample size for the total population.
But if 35% use Python and only 5% use Ruby, then you're comparing a sample of 1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby data is more shaky than the Python data, and thus it's not a fair comparison.
Same goes for a lot of the other metrics where the sample sizes vary.
~ Disclaimer: I'm no statistician (but I play one on Correlated.org).
Example: Say they hired 5 out of a 100 applicants who used Python, and 2 out of 20 applicants who used Ruby. That's a 5% success rate vs a 10% success rate, a 100% increase. I'd hardly call that a significant result though.
Disclaimer: I'm no statistician either
So basically this article says nothing. It needs more information to be relevant.
You can do a proper analysis fixing some of the variables. For example you can fix the language and compare how the different editors affect the selection process. But still you have to show other variables as the years of experience in the language.
And finally you have outlayers like who did an interview with you but was discarded because of living in a different country. But I guess I might have enter into the statistics.
Fuzzy-matching autocomplete for emacs: https://company-mode.github.io
If you are at all interested in trying something with all the bells and whistles turned on for you, check out http://spacemacs.org
Its wholely unrealistic to suppose that someone would spend hours to days learning emacs but not spend 1 minute adding a completion plugin.
For me not using completion works to keep me disciplined about the structure of code I work on. It also can make me vicious about arity and composability in code reviews. Fortunately most of those details are taken care of by design prior to implementation, so it's rarely an issue.
Yes, it's that simple
Agreed. Great start.
[make an] investment by going through our interview process, afterwards we can fast track you through to final interviews...O(n) process into an O(1) process
Makes sense. Use of complexity notation is transparently gratuitous, but I liked it, because it shows the intention to understand and connect. Traveling the world most people appreciate an honest attempt to speak to them in their language even if you're not fluent.
You'll have to take a coding quiz with questions in several different languages, as well as some design questions. The coding questions involve reading blocks of code, and answering questions about them.
Intuitively I hate the idea because I'm skeptical of how effective it is. Do you have non-anecdotal data to show how well your quiz correlates to specific goals?
After the quiz, you may be asked to complete a coding challenge...
What triggers that requirement? Also same question as previous question about correlation/validity.
Then, you will move on to our technical interview...
What do the tech interviews consist of? They can vary wildly across companies, interviewers, and formats.
...work history [is] meaningful but relying solely on [it] results in missing good programmers
This is a show stopper for me. If a company doesn't acknowledge that what people have accomplished with previous contributions, projects, development work is not just meaningful, but extremely important and a top criterion to find great people, then I don't believe they have the correct perspective. But hey, I can be wrong about things, I'll try to be open minded. Point me to solid data showing tests and quizzes are a better predictor than accomplishments and I'll have a look.
I can see how it could cause an issue with some employers, but I would consider a 42 year old person with only three years experience not to be an issue whatsoever, in fact I've hired people with that profile.
One example was a guy with a degree in Biology who had ended up in technical sales for DNA related products. However, this was when mobile apps were getting hot, and the guy had shown an uncanny ability to learn iOS and build a few small but respectable apps using Objective-C during whatever nights and weekends he could scrape together. Considering many full time devs at the time struggled with Objective-C it was an impressive accomplishment. It was impossible to fake because we could scroll to a random piece of source code and ask him to start explaining it, which besides verifying he did the work, gave valuable insights into his development thought process.
These concrete results, combined with probing discussions of how they came about, proved he was smart, had the ability ramp up on new tech quickly, could make some immediate contributions, and as a bonus that he was full of motivation. He worked out great, and to this day we maintain occasional friendly contact.
Point being "work history" can be looked at in many ways. I'm advocating an emphasis on specific results, contributions, and weighing the value of those, not really generic work history patterns.
Not specifically tailored to programmers, but I find this paper relevant with respect to hiring practices:
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2853669
An excerpt:
>Overall, the two combinations with the highest multivariate validity and utility for predicting job performance were GMA plus an integrity test (mean validity of .78) and GMA plus a structured interview (mean validity of .76). Similar results were obtained for these two combinations in the prediction of performance in job training programs. A further advantage of these two combinations is that they can be used for both entry level hiring and selection of experienced job applicants.
"Accomplishments" could probably be called a combination of "Biographical Data" and "Reference Checks" as described in that paper. Biodata correlates highly with GMA tests, so it's a lot easier to just give a GMA test then pour over descriptions of of past experiences. Reference checks are generally not useful today because of the fear of lawsuits associated with giving a reference that costs someone a job.
It seems that a general programming test combined with a structured interview has data showing it's a good predictor for on-the-job performance.
The issue I would have is it doesn't seem to preclude the possibility that placing more emphasis on specific accomplishments and other software development contributions, is as I suspect, an even better approach.
You mention biographical data and reference checks. I dug just a little further into related papers to get a better idea of what the authors meant by this. From what I can tell those concepts are different things from what we might guess, and wouldn't serve as a proxy to evaluate my conjecture (for example, on the meaning of biographical data: https://www.biz.uiowa.edu/faculty/fschmidt/meta-analysis/rot...).
Speaking of conjecture, I realize that's what I'm offering here. Whereas they have strong data to say, here's a couple of approaches that can be effective.
Anecdotally, I still believe better candidates could be chosen by closely evaluating the specific work a developer has done, what role they played on the teams, and the impact and value of the work directly connected to their contributions. Of course these can't work just by listing Git repos, the interview process has to explore and validate the history. Combining this with the interview process can allow great insights into a candidates thinking, because it allows discussions about real world, non-trivial work and problem solving.
Reducing the rate of both false positives and false negatives reduces the time and cost of hiring an engineer for companies.
The data we look at for this is our "onsite to offer rate", which is how often Triplebyte candidates get a job offer after each final interview we send them to. Ours is roughly double what the companies see on their own for people applying directly or referred by a friend. This means we're doing a good job at pre-screening for technical talent and pre-matching for what each company is looking for. If this "onsite to offer rate" wasn't substantially better than normal, companies wouldn't work with us -- and especially wouldn't let our candidates skip their resume and recruiter screens.
If someone has 3 years experience, there should be plenty of results to allow an in-depth look at what they've been working on. It wouldn't matter if someone 10 years senior had "more stuff", because it's not about quantity. I think it's important to take in the context of where that person is at in their career and what's realistic.
>>There are false negatives from rejecting great engineers who...can't communicate their experience well
Well, it's true this is a problem isn't it? Whatever you call it, it's part of a lot of us, part of who we are, and does sometimes result in unfair judgement. The problem is, I'm not sure TripleByte's process gets around this either. Part of their process is a dynamic interview that surely requires some communication skills. Pass that and no doubt even more communication skills will be needed at the employer interviews.
To me, it's probably that the questions are geared towards Python/Ruby, rather than C++. For example, parsing a CSV is one line of code in Python/Ruby by default, but if you're supposed to do this in C++, that alone can take 10 mins to write and test, etc.
If they really want to make their test/evaluation fair, and if they truly want to be inclusive in the languages, they should figure out why there is such disparity, it's probably with the questions they ask rather than the candidates. To me, looking at these numbers, it just means that were I ever to interview with them (I never will because I think they offer no benefits whatsoever, since I don't want to interview with them, and then with the external companies again when I can just go with the external companies the first time), I would just choose Python or Ruby to interview in, since the tests appear more geared to those languages.
This is just fodder for the endless tech variants of ford-vs-chevy debates.
The elephant in the room is that we don't even know if technical interview performance is a good selector. I'd be a lot more interested if this data included long-term results of actual job performance like employee and manager satisfaction, time in role, time to promotion, etc.
(Now, I'm not entirely clear on how Triplebyte is getting their data. Do they ask explicitly, or pull it from the UA header? People could, I suppose, be doing it from their work laptop. Or, most people just don't care to customize that much, and just use macOS because it is good enough for them.)
Meanwhile every time I try to put Linux of any flavor onto a macbook or a lenovo carbon or whatever, something gets fucky. Video drivers don't work, no webcam driver exists, the mouse glitches, the tilde key doesn't work, plugging into/out of an external monitor crashes it or causes all my windows to collapse into a single workspace, same issue for if the computer goes to sleep while connected to an external monitor, all sorts of weird things causes it to not sleep on laptop lid close and nukes my battery, battery life will drain excessively, arbitrarily, chrome will randomly cause the computer to slow WAY down, etc.
Ok, that was a rant, but TLDR - I hate macs but coding on one just removes all the distractions from programming and let me "just do my job."
(I am experimenting with windows as well right now because I love the Surface platform as a whiteboard and prototyper, so far WSL is great and I actually prefer it over linux because all the OS stuff "just works." Lots of weirdness around WSL still though)
I'm using Firefox, so can't comment on Chrome experience.
Just curious, what is your reason? Is it standard "Apple hate" on an ideological level, or is there a specific technical obstacle?
1. https://duckduckgo.com/?q=apple+site%3Ahttp%3A%2F%2Fmjg59.dr...
Yes.
I've worked in several environments where the development team was equally mixed between Linux machines and Macs, and they all shared the same tools, setups, scripts, etc without issue. Which I think is why developers like Macs so much.
At BT we brought or dev test and live from the same production run so they where identical down to the particular rev of the MB our sysadmin guy also would have liked to make sure that any disks we brought where all from the same batch plus spares :-)
On mine, I was plagued by "stuck pageflip" issues, issues with external monitors, resource allocation, battery life, etc. Furthermore, there was no webcam driver.
The day someone comes out with an actual mbp competitor is the day i switch, but thus far its not even close :/
A good pc set up with multiple monitors with a proper mouse and your preferred key board is much better for working on and is more cost-effective use of your $
The end result is you are likely stuck with a Windows PC (usually a horrible, custom built Windows image, too). If you're lucky, you might elect to get a Mac. And if you like Unix, the Mac is the better option.
This is the case with me. I'm stuck on a Ubuntu computer, and not hating it as I get used to it over time, because I inherited a really old macbook that wasn't updated to Snow Leopard. Since I couldn't find an update CD, and certainly couldn't afford a new Macbook Pro, I had to put Ubuntu on it and deal.
https://insights.stackoverflow.com/survey/2017#technology-pl...
The article calls this out in the first paragraph.
This surprised me. It might be because I'm not from the USA but in every company I've worked for on-site (including the current one), the overwhelming majority of developers used and preferred linux.
That aside, I'm glad python does well.
(Ruby being an example; Python also applies here, whereas the converse would be a hypothesis that C++ programmers are not less skilled but rather are judged more harshly in interviews).
So for instance, Vim doesn't provide much help when you mess up. If you type a variable name that doesn't exist, that's your fault (I'm sure with plugins you can get more). So do Vim users do better because they're nerdier, don't mind a steep learning curve, and like working in a terminal, or is it because Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"?
In this setup writing the same code in C++ (including the right STL headers) is a pain in the a.
I remember that when I selected C++ as my language for interviewing, I was strongly punished, because my interviewer didn't understand why it takes 10 minutes to do something relatively simple (that is just a few line of code in a higher level language). After that I switched to using Ruby in my interviews, and passed easily.
Start typing a work and press C-x C-[nolk]
* n for a word in open buffers
* o for "omnicomplete" which is quite similar to "intellisense"
* l for the entire line in open buffers
* k for the word from a dictionary
There are several others, but those are the main ones. As soon as you start a completion, you can use C-p and C-n to go up and down the list respectively.
> Vim doesn't provide much help when you mess up
Vim is designed for editing text. I don't know if a better editor (yet) suited to editing, rather than writing.
> Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"?
It can, so long as you have it set up to. Vim even has a spell checker.
Someone who has been doing nothing but C++ for years will probably not be able to "code golf" some data munging problem, say. By the time they get their function right for tokenizing a string on whitespace delimitation, the time is almost up. :)
Does the triplebyte hiring process not take into account experience and role? Don't they expect more for certain roles and a given experience?
You really don't have to be great at every section to pass the Triplebyte interview; you just have to show your personal areas of strength. And we give you lots of different opportunities to do that.
Of course, we will then match you with appropriate roles and avoid the ones that require web-specific skills. FYI, we already have robotics, financial, and game companies on our platform!
It was also nice that they had an editor available with serviceable Vim bindings. I think it was a big improvement over coding on a whiteboard.
Quite cool that they're willing to share fairly transparent performance metrics with us.
I was hoping for a quick section introspecting on correlation vs causation and what this says about their process as opposed to a semi-serious presentation of predictive power.
I can only assume a total lack of rigor and process if this is what they choose to talk about publicly.
Buying a more expensive Mac to use just as a terminal just seams an odd choice.
For example, Eclipse is the standard University Java course editor. Users that have switched to IntelliJ have probably had more time away from University and hence more experience. Is this the reason, or is it a "just so story"?
Based on my anecdotal evidence of interviewing Java devs, I'd suggest the problem is the opposite. While there are some excellent Java coders out there, there are also an awful lot of "experienced" devs, with fifteen years of experience and salary expectations to match, whose technical ability proves to be pretty miserable.
I'd be curious to know what kind of shop they are at TripleByte. Something tells me they might be Ruby coders using OSX all the time. Like attracts like, especially in interviews.
i.e. we should calculate P(pass | <emacs>) or P(pass | <c#>) and see how that compares to P(pass) as a whole.
P(pass | vim) / P(pass) = 1.38
While I do understand that this is just content marketing to drive traffic, it would be great if the following questions were handled with rigor.
Is editor usage related with age of the interviewer? If there is a strong co-relation between Eclipse and older interviewees this could very well be explained by ageism.
Is PyCharm ranking high in the rankings because of confounding with Python also being high?
As a tangential example, back when OSX market share started climbing a ~decade ago, it was a safe bet to say that the average OS X user was savvier than the average Windows user. This was true at a baseline regardless of what you think the "savvy" choice for an OS is: the simple fact that the sample of Windows users includes the large chunk of the population that didn't give any thought to their choice of computer (a sub-population that naturally is less savvy). This dynamic would hold true whether you think that savvy users would prefer Windows or OS X: by dint of being the default, the Windows-using population had its average savviness dragged down, a fact that has no bearing on the suitability of the OS for computer-literate users.
There could easily be a similar dynamic at play here: beginner developers are going to be more likely to be on a big GUI IDE, which would drag down the measured average quality of Eclipse users. This is entirely consistent with high-quality developers preferring Eclipse to Vim, or being split 50/50 between the two, etc etc. As you point out, confounders like age and experience are also likely contributors.
> It is possible that developers who code with vim and emacs are forced to focus on low-level details like method names while developers using IDE's can focus on getting stuff done.
FWIW, using Vim doesn't mean using raw Vim with no plugins. I used vim my entire time at Google and I was the most productive dev on most teams I was on. I'm pretty sure this would've been impossible without plugins at least including auto-complete, jumping to definitions, etc.
The data gives PyCharm and Visual Studio an "advantage", suggesting this isn't the case. Or are you suggesting that only Eclipse is used by beginners?
> I was the most productive dev on most teams I was on. I'm pretty sure this would've been impossible without plugins at least including auto-complete, jumping to definitions, etc.
Pretty sure this was unrelated to the editor you were using. All sane editors have these features you've listed, and I'm sure you would do equally as well with them, especially since being a productive developer doesn't mean you write the most code (usually).
I was just using a hypothetical example of how data can be confounded by factors that have nothing to do with the central implication that "better developers use vim/emacs". You're right that my particular example is weakened by the fact you describe, but there could be _further_ confounders: in my anecdotal experience, Eclipse is a lot more common as a beginner's IDE than any of the Jetbrains IDEs (everyone I know in college used Eclipse, while my first exposure to a Jetbrains IDE was a few years into my career, while at Google. FWIW, I prefer JB IDEs to Eclipse (and prefer vim to both!)).
> Pretty sure this was unrelated to the editor you were using. All sane editors have these features you've listed, and I'm sure you would do equally as well with them, especially since being a productive developer doesn't mean you write the most code (usually).
I think you've completely misunderstood my comment. I didn't say these things were related to the editor I was using. I said that the parent comment's implication that you can only be productive with a big IDE (as opposed to vim) is based on a misunderstanding of what vim is capable of. Obviously every sane editor has these features: they're pretty much table stakes for development IMO (which is the whole reason why I augmented my vim with them).
It says much more about Triplebyte than about developers, technologies or interviews.
The amount of open source big data projects, and large profits and companies relying on Java technology should clue you in that this is unfounded. Hell, half of the modern smartphone market is all about Java.
I don't think companies dislike Java devs. I'd also take an article from a company trying their hardest to commoditize engineers and turn them in to test scores with a grain of salt.
/Skills
}
I
* vim
:wq
:wq"
and then left the editor running.
[1] Yes yes, I'm aware that this example isn't perfect because it could mean different things depending on the language
Interviewees, however, tend to get flustered if they can't remember the method names. It's another rock on a pile that eventually leads to nervousness collapse, something that happens far too often, so after a few dozen interviews I've started stressing up-front that no, it doesn't matter, I'm not scoring you on that.
That, on its own, produced a measurable increase in average score.
That's pretty vague. I use the same strategy as the sibling post, ratcheting up difficulty until the candidate fails, so I usually have a lot to go on, but the exact criteria end up changing from one interview to the next depending on what the candidate did.
Occasionally I end up with not enough data. I try to give my feedback in the form of a confidence interval, but if it's mediocre candidate combined with not-stellar signal quality... figuring out how to score them can take a while.
When I interviewed at Google, a major part of the rejection feedback they gave me was "we felt like you weren't worried about whether your code would really run".
This of course doesn't mean that for real life situations, they're never the right tool, but if someone chooses C++ to solve a simple tree inversion or sliding window problem in an interview situation it probably means that they don't have sufficient familiarity with any language more appropriate to the situation.
As to penalising people who switch languages for problems, if you give guidance that you want to see a solution in a particular language, that's fair enough, in any other case it seems unfair and misguided. Perhaps you don't really agree with 'right tool for the right job' when it comes to languages?