Using the wrong data structure or a data structure in the wrong scenario can make the difference between an application which can serve 1000 users per node or 100. That impacts the number of servers we recommend deploying in our cluster and in turn the hardware and maintenance costs for a customer buying a license for our software.
An understanding of Big-O and algos is very important and any good software eng or team lead should refresh themselves on the topics regularly since these really can make the difference between a performant app and one which brings the host device to its knees.
"Why does it take 30 seconds to complete this seemingly simple request? Oh, it's making a database call for every element in the list, vs. efficiently sending a single request. I fixed it, and pull request submitted."
"What is this script taking so long? Oh, exec is being called on every command, so let's just pull that out and put it in a wrapper script."
"What performance tweak can we use to make this Rails app hellza faster, since we've maxed out low hanging fruit caching options? Rewriting this part in Elixir takes advantage of concurrency in a way that Ruby can't do all that well."
I guess real world discussions don't seem to incorporate Big O lingo.
These situations don't come up too often, but when they do, it becomes important to at least be able to solve them in some fashion in a non-high pressure situation.
I think in many positions it probably isn't necessary to have knowledge of how to implement from scratch -- yet again, if the company could find someone that gets it, they may produce overall more efficient solutions. Whether that matters to the company or not is another story -- but things like finishing some scheduled task in 10secs instead of 2hrs can be really useful.
I used to have a mindset like "these algorithm quizzes are BS!" Yet when I have learned more about them, it is amazing to see how inefficient things can be produced, without the foundational knowledge to process/store things efficiently
EDIT: All that being said, I think the usual software engineer interview format is terrible. It's like an intellectual hazing gauntlet or something..
Correction: all other things being equal, the person who does better on an algorithms/data structures pop quiz is more likely to be a recent college graduate who you can pay less and treat worse than an established developer who's forgotten most of this stuff and just looks it up when/as needed.
I still think someone who understands these and has studied them will be in better shape to understand when/how to use such things in situations on the job (even if they can't be reproduced in a pop quiz scenario by the person). Maybe those situations will matter to the company, and maybe they won't.
What you're talking about is primarily a function of experience: seeing various situations over time and learning what works well and what doesn't. But these questions are used, quiz-style, as a barrier to even entry-level positions. So even if I grant you the possibility of some effective criterion being developed from algorithm and data-structure questions, the fact would remain that hardly anyone (and more likely no-one) uses them as you advocate.
Getting these questions out of interviews and replacing them with metrics tailored to the desired job skills and responsibilities (rather than proxies for them, or even proxies for proxies for them, as is unfortunately usual in the industry), would be far more effective.
Or perhaps more bluntly: I know of no other field which A) drills students on low-level fundamental techniques in school/training, and B) largely replaces them with higher-level techniques on the job which discourage use and thus continued top-of-mind retention of the low-level techniques, and C) nonetheless requires instantaneous unassisted perfect recall of all details of any arbitrarily-chosen low-level technique as a mandatory qualification for employment even into late stages of the career.
Lawyers learn a lot about the law and its history and important cases, but they don't sit through pop quizzes of 15th-century English common-law rulings on every job interview for the rest of their lives, even though there probably are some such cases that would on occasion be useful to know. Doctors learn a lot of biology and chemistry, but if passing a closed-book organic-chem exam was a prerequisite for every job they'd take after med school we'd have a lot fewer employed doctors.
Only programmers do this to themselves. Only programmers relentlessly demand that it be preserved as a gatekeeping ritual. Only programmers insist that "well, it's fundamental so it must be useful sooner or later" is a sufficient justification for doing this.
It's time to stop doing this. It's time to stop supporting this. It's time to stop perpetuating this.
(and getting back to my initial comment: there's a reason why bootcamps now have a "how to pass code interviews" unit, and a reason why Cracking the Coding Interview sells so well, and it's because these things have stopped having any useful purpose -- if indeed they ever had useful purpose -- and now serve as artificial barriers and ways to perpetuate biases about background while claiming objective grounds to disqualify people the interviewer doesn't want to hire, thus making life miserable for both the hired, who are likely to be inexperienced and easy to take advantage of, and the unhired, who have to try again or give up and find another way to make a living)
The question then for me is, how does one assess if someone can use these things in practice to make a difference for the business? I don't have a good answer for that right now. If anyone here knows a nice approach to use to assess this type of skill, I'd like to know more (to stop perpetuating the usual 'algorithms gauntlet' approach).
IMO, algorithmic interviews are maybe a good way to filter for motivation rather than anything else. Candidates who are likely to learn all this are more hardworking and career-consciuos (placements correlated pretty strongly with how much you 'knew').
That said, there was this one interview I had where the interviewer asked me how to design a distributed log collection service. Basically you have GBs of log files being generated on hundreds of servers and you want them sorted and stored on a single central database. What sort algo will you use, why? How to reduce transmitted over the network etc.
It was a fun round and tested a lot of disparate concepts.
To my inexperienced self, this seems like a good format:
- First round: Ask stuff like fizz-buzz, trees, file I/O or string processing stuff. Basically ensure that they know how to code. Another way could be asking them to code the same question in functional, OOP and procedural styles
- Second round: Discuss some project done by the candidate or ask them to design something from scratch. URL shorteners, databases, REST apis are normal candidates. Spreadsheets, DSLs etc might be good to make things harder.
- Third round (If applicable) : Dig into any one specific domain most relevant to your company/team and make them architect* the whole thing.
Ofcourse, any system, once its established and well-known, can be gamed. Banking on open source projects or past achievements is then what remains.
* Very few entry levels jobs really require such skills but (IMO) it will help you get better engineers instead of coders.
It's that intuition which is valuable, not the implementing of algorithms by name.
There are far more storied careers than mine in HN, but this is not a career of someone that spent their days writing CRUD apps in visual basic: I worked on fun things. Some of that worked involved algorithms: many of them very complicated. However, since I left school, other than in interviews, I never had to touch a high percentage of the algorithms in that page. Let me go a section at a time:
- I have used graphs plenty of times, but I can't recall using any of those algorithms directly. Yes, not even BFS. - I had to do linked list-like operations when I was writing in C at the beginning of my career. Not a single time since. - Zero dynamic programming. Nada. - Not a single manual sort algorithm, not a single manual search algorithm. - There's been plenty of trees, but none of the operations covered there. They were either provided by the libraries underneath, or never came up. - Number theory? Nothing from that list. I have implemented HyperLogLog though. Just don't ask me to do it from memory. -Early in my career I had to deal with some bit manipulation. I've not had to touch it in years. -Not a single one of those string/array manipulation ops
So my answer to the grandparent is that you can have a long, fun, not CRUD app career doing fun things without having to implement those algorithms once, because they are done for you. The algorithms those jobs need in practice are often harder, but you don't have to have them memorized: Some you go look for papers that solve your problems, others you develop yourself (and be afraid of that one, as I have seen a mathematician come up with a 5 page proof for an algorithm that only did what we needed in a parallel universe where latency is zero)
What almost every professional programmer has to understand what an array, a list, a set, a tree and a map are, and to go check the performance problems of specific implementations if it matters at the time. Almost every other interesting thing I have done was only relevant a small percentage of the time, and I could look up.
Interviews ask the questions they do because they match what is taught in a small subset of CS classes. We could teach other algorithms in those: Some of the ones I had to use would fit in said classes, instead of the ones we have. They can be implemented in under an hour too. However, nobody asks for them in interviews, because the people that come up with interview questions haven't solved them before.
So my point is not that you can ignore data structures and algorithms: You'll use some no matter what. But there is no subset of algorithms harder than a loop that every programmer uses in a regular basis, or data structures that we manually implement. We just ask for things that come from those same CS classes because we have no idea of how to assess if someone is any good, and the algorithm classes were some of the harder ones in college, so we assume that if you have them memorized, you must be pretty good. And we assume wrong.
So I want someone who says "this is analogous a dynamic convex hull problem" or "this can be solved greedily by branch and bound" etc.
That knowledge has to be learned it can't be Googled. Mergesort or an A* implementation is googleable once you know you need them (which was the important bit).
For a programer in the industry, the ability to write simple "legible" code is very important. An ideal interview process should spend some time evaluating those skills of a candidate - but this is very very hard to measure.
But good familiarity with the basics of CS is also important to be able to perform many of these jobs. Here is a good blog post about this form of interview and the rationale behind it: http://www.goodmath.org/blog/2015/11/19/technical-interviews...