You don't need to work on hard problems (2020)
benkuhn.net
benkuhn.net
Before Hadley, tools for streamlining data manipulation workflows had largely fallen stagnant. Even the best languages and tools were a chore to use. People's data prep processes tended to be idiosyncratic and hard to understand, even though there was nothing essentially new or deep about what they were doing. Data prep was an ugly chore best left to interns and grad students. It was kept out of sight so the big wigs could focus their slide decks on more interesting things.
Traditional academic statisticians and computer scientists had a blind spot for this issue. They didn't consider it a hard problem, which is what academics are supposed to work on, and discouraged Hadley from working on it early in his academic career. But Hadley persisted because he knew it was an important problem. Streamlining time-consuming chores liberates an enormous amount of human capital for the most meaningful work. (Which in some cases can be improving the data still further, opening up new vistas of insight the big wigs otherwise never would have even been aware of!)
Hadley's approach has not only revolutionized R programming, but has also quietly influenced the whole modern analytics ecosystem. It's become too good to ignore. Even for those who do most of their work in another language, with tools like pandas that don't follow the tidy philosophy, some end up trying to structure their workflows like Hadley's tools do, because it's so intuitive and compelling [3].
[1] http://vita.had.co.nz/papers/tidy-data.pdf
[2] https://imstat.org/2014/09/04/data-science-how-is-it-differe...
[3] https://stmorse.github.io/journal/tidyverse-style-pandas.htm...
Especially in academia, many professors, who are wedded to particular models, go so far as to dismiss simplicity without sufficient consideration.
It was not always this way. Among the older philosophers, you can observe a strong affinity for simplicity. Figures like Pythagoras, Newton, Descartes, Archytas, and various Indian philosophers, such as Shankaracharya, strongly favored simplicity.
Even practical statesmen such as Han Fei, Lee Kuan Yew, Chanakya, and others were able to distill simple yet powerful truths from complex situations. This is central to leadership and the effective organization of human activities.
You can observe the same drive for simplicity in great engineering leaders like Rickover and Visveswaraya.
Given more time, I believe I could make a very strong case for the power of an affinity for simplicity and its correlation with greatness in any field of activity.
Simplicity is underrated (in practice).
The individuals you mention mastered complexity. What they did could be seen by us as simplicity retroactively. Something like Denning’s crane. Newton developed calculus, that was anything but simple at that time.
I agree to use simplicity to decide, inform, lead. But great things tend to emerge from thinkers that can think through complexity and not oversimplify their own thinking.
Re: Newton - I am not denying complexity. I am simply saying recognising basic truths, honoring them to the extent to which they deserve, making them the foundation for all further activities/undertaking. For ex - Rickover dealt with super complicated projects, but how he dealt with them is based on utterly simple principles; so simple that people would outright reject them, until they see the results he achieved. One could say the same regarding Lee Kuan Yew's work.
Easy to favor simplicity when the basics in your field hasn't been fully mapped out yet. Much harder to make a name for yourself when pretty much all the low-hanging (i.e. simple) fruit has been claimed already by people in the past. Most people probably feel they don't have much choice but to pursue more complex ideas nowadays as a result.
I've shown these ideas to many so-called well-educated college graduates, and they don't warm up to it. They say "processes" > "individual initiative". This is too old-school, too simple, blah, blah.
I think people will learn only when you put them in charge of a complex undertaking to appreciate what simplicity means in leadership of complex projects.
Partly this, and partly because average college professors are being compared to some of the greatest scholars of all time.
Figures like Einstein or Feynman were able to simply inherently difficult/complex problems well into the 20th century, and similar people are surely doing the same today.
But often a difficult problem needs to be understood the hard way before it can be simplified down to something that is easier to communicate.
You must be speaking of personal experience, because otherwise your comment is completely ridiculous. But if you are, you certainly have no personal experience with historical figures. So how can you judge a day-to-day interaction with the greatest hits of someone dead?
PS: It is always better to rely on older sources for illustrations, because they have lived their life fully and you have documentation of their lives + effects of their efforts. It is not a good idea to rely on living people to illustrate principles.
"The days of old have turned to dust" as it says in the Epic of Gilgamesh (ie one of the earliest surviving works of literature includes a moan than things are not as good as they used to be).
The past thinkers did not reason under a hanging "publish or perish" sword that current researchers do
Source: me. I was a researcher at a national research institution for 7 years.
But you clearly do have a biased picture on your head. That claim that current people value it less than ancient people is not only baseless, but quickly dismissable just by reading those genius you quote.
To meaningfully engage, you must study the history of education systems, how it evolved, what was the situation of scholars in the past and now. If you study those things a bit more, then my argument may make more sense to you.
When it comes to getting things to work, to make things happen, to understanding things... our major bottleneck is the "size of human skull", your cognitive limits.
Descartes gives the example of visualizing a chain made up of many links... at any point, you can only look at say, maximum 4-5 links, you can never see all the links at the same time. That is, in fact, it is impossible for a human to "see a whole chain"! You are "making it up", even with such a simple operation as "see". Therefore, due to such horrendous limits to cognition, you are forced to implement some sort of simplification mechanism.
So the context of discussion is (2) not (1). The people interested in making things happen in the world, to obtain correct/useful/functional/practical viewpoints, tend to gravitate towards simplicity by necessity. There is no other option.
It is possible for both things to be true.
1. complexity is bad and undesirable.
2. Complexity is unavoidable.
See also: reality has a surprising amount of detail, http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...
I think of it as something to put on my "life CV"; it might not be an astounding technical achievement, but I'll remember it and be glad I did it for my whole life.
Beyond the technical constraints, there's just the domain itself. I've focused my efforts in this domain of just digitizing paper processes. There's so much to know from a regulation standpoint and a quirky middle man chain standpoint.
Some of my proudest accomplishments in my career is just standing anything up that replaces paper even partially. I've been praised many times for just making something that works after so many have failed. None of what I did was technical amazing. A lot of what I did was just doing a lot of reading and talking to people and I'm amazed how many engineers weren't doing that.
I think at least for the NHS, making procurement more effficient would actually be a pretty impactful idea. It cost us more than a year's licence fee just to onboard each trust, as they're all different.
My wife was considered at risk for gestational diabetes but I’m not sure if she was directed towards an app.
What do you think about Badger Notes?
I'll share what I can. The app[0] we built is now owned by Huma, and is active in I think about half of all NHS trusts. You have to be attached to one of those trusts to gain access to it. I'm aware of Badger, but I never saw it in use, so I can't say much about it. I think they weren't quite a direct competitor as we were purely targeting gestational diabetes, and they are more of a general-purpose system. But perhaps they now have enough specialisation in gestational diabetes to be a decent tool in their own right.
[0] https://apps.apple.com/gb/app/gdm-health/id1421840751 - same name for Android Play Store
> I could try to solve an easy problem as quickly as possible
The "solve X as quickly as possible" is now the problem. Not just solve X. I And that sounds like a hard (therefore interesting) problem!
The combination wont magically become a "hard problem", except if we take "as quickly as possible" to mean achieving some optimal solving time, as opposed to just casually meaning "without adding stupid bloat and by doing a KISS solution".
Hard disagree with this one.
Summing all numbers from 1 to 100 is easy, summing 1 to a million is also easy, just time consuming. 1+2+3+4....
Coming up with (n*(1+n))/2 is not.
Also I think that the parent didn't mean "as quickly as possible" in the execution speed or algorithm sense, he meant in the clock time sense: getting it solved quickly and moving on.
So in their (and mine) formulation, the problem is easy by definition. It's not an "easy problem turned had because it has to be optimized for speed".
I thought that your objection was that finding a simple solution would be hard itself (which wouldn't be the case if the problem is easy as in "easy to solve").
[1] not that coming with a basic series summation formula is "hard" (except you're Gauss, and you're the first mathematician doing it, the rest just need to know how to search for it), but I'll accept it as hard for the sake of argument.
That’s exactly what the initial comment said:
> The "solve X as quickly as possible" is now the problem. Not just solve X. I And that sounds like a hard (therefore interesting) problem!
There were two separate points: 1) Finding a simple solution to a problem can be hard 2) Solving an ‘easy’ problem quickly enough can be hard
Too easy or too difficult sudokus arne't rewarding. The easy ones are basically just writing digits without thinking for a while (i.e. boilerplate work). The difficult ones are often frustrating and too much work for little progress. The sweetspot is something that offers some challenge but not so much that it's frustrating.
A simple solution to a simple problem will always be just boilerplate. It will be work that can often effectively be automated (But you can't try to automate it because then you have traded your simple problem for a more difficult problem of automation!). To me it sounds extremely boring and unrewarding.
I didn't quite understand the gist of this article. What is the left out bit of the title? "You don't need to work on hard problems to...."?
I immediately thought "be happy". I didn't think "be successful" (or similar).
I sometimes roll calculators with a gui and useless animations for things I don't need a calculator for. If i get to use it 1 time a year later it feels much better than it sensibly should. Now that I think about it they could generate code with very little extra effort and even smaller odds to be useful. The fun is there tho.
I don't think that's what the author means. Thought TFA is a little vague about this: "Instead, I learned it was possible—and fun—to optimize on other dimensions, or play a different game. Rather than competing for an A+ on a hard problem, I could try to solve an easy problem as quickly as possible (like Wave’s accounting), or find the easiest problem whose solution would be useful (like identifying Kenyan names), or hire a team to solve easy problems faster than I ever could myself".
To me "solve an easy problem as quickly as possible" doesn't imply "fast" as in program performance, but as delivery time. And this is further corroborated by line "hire a team to solve easy problems faster than I ever could myself".
But I don't live for solving problems. Let alone "hard" problems.
I'm a "maker" (though I kind of hate that word). I like building things. If I'm not developing software then I'm making other stuff. Woodworking, theatre, props, clothing. Software applications are another thing to make and produce. Making new things always comes with a healthy dose of problem solving, but I don't get out of bed in the morning thinking "boy I can't wait to solve problems today .. I hope someone hits me across the face with a real stumper that's going to make my work a living hell until I can figure it out!"
For me it's more about whether or not I enjoy what I'm building, and would use the end product or at least can see the value it provides to others. Not so much about how many "challenges" there are in building it.
If you're a musician, that means playing that most technically advanced piece flawlessly.
If you're an athlete, that means raw ability like speed, strength, etc.
If you're a mathematician, that means having a go at some unsolved/open problem.
If you're in the military, that means joining the special forces or becoming a fighter pilot.
As the author said, when you're young, you often only get judged on one or two objectives. When applying for college, that's GPA and SAT - so you will optimize for those two, and little else.
All this works out for young people, because if there's one thing they tend to have, it is time to grind. And lots of it.
One thing I've noticed with young and ambitious employees, is that many need constant feedback. If they don't get any direct feedback immediately, they feel left out in the cold, and don't know how to react.
Where I work, we often get inquiries from clients, and 80% of the time there's zero feedback. We produce reports / presentations / recommendations, ship to the clients, and that's it. IF we get something back, it is because they want to go forward with something, and need follow-up work.
For the ambitious types that have gotten feedback their whole lives, that can be a hard pill to swallow - it really takes some "unlearning" to get comfortable to work in that environment.
Even worse if the rewards aren't aligned with your performance (e.g. teaching jobs), or if you're not able to perform well because the type of work you do isn't what you excel in.
It's an increasingly cold world and some young people may legit get zero feedback outside of work. Ghosted on dating scene, ghosted by the remaining friends post graduation, and now feeling ignored at work.
Add on no third place to go to and how work is increasingly ceasing to be a second place with wfh and you just have a loneliness sandwich. They spend 18-20 years having people their age all around them and then it poofs up in smoke. I'm sure they want some kind of feedback in the one remaining place that is now taking up more and more of their time.
Im older, 30s but I felt that friend contact really fell off after moving away from SF former room mates and startup friends.
When those things change, friendships may cease to exist. People move away, and grow away from each others - it is a sad fact of life, but it is what it is.
I had friendships in college which I felt were going to last my entire life - you really felt close to those people. Struggling together in the reading halls and labs, going out to pubs in the evening, and just chilling with movies on the weekends.
But then life happened, people move away, and you lose contact. Maybe one part tries to keep in touch, but you with work, family, and what not - it is not always a two-way street.
Then you bump into each others, 5-10-15-20 years later, and the magic is gone. Or maybe not, I've bumped into people I haven't seen in ages and we picked up were we left things.
Same goes for work - or dare I say - especially for work.
In university, social life is non-existent. Dating? It's in the category of "don't even think about that".
So yup, there's a reason why most of us are utterly lonely.
Though in my generation ghosting was considered much more rude (or at least seemed less common), you at least had to try to politely deflect a message or phone call.
There's no relationship-as-a-service app despite the hype, relationships take work and you only get what you give.
edit: when we do get together we just sit, watch youtube or play video games and drink beer just like in college heh. Our meatspace relationships are like a time capsule more or less, in that respect it's sort of depressing haha.
> Ghosted on dating scene, ghosted by the remaining friends post graduation, and now feeling ignored at work.
> Add on no third place to go to and how work is increasingly ceasing to be a second place with wfh and you just have a loneliness sandwich.
And all around them, a social media full of hucksters and fake personas that should how successful they are. Build something, and get inundated by infinite liars that have shipped your idea in 30 days and earned a billion dollar MMR, blessed with capacity to work 80 hours a week and come back for more.
Meanwhile these new people just want to feel they have a mark, any mark at all, in the world. Ever self doubting. Before the next layoff.
--
The older I get, the more I blame the internet for most of our current societal ills. I am a Millennial, but a digital native no more.
Similar to the conclusion I got to in the past 5 years, I'm in my mid-30s by now and been steadily removing myself from being a "digital native" back into the real world.
It was immensely enlightening to grow up on the internet, see it appear and flourish, being taught by so many anonymous people on IRC chats, forums, blogs, etc. but now I can also see it's been transformed into something that's not helping me, it's not making me more fulfilled or happier. The information available online is still my bread and butter for learning hobbies, new skills, etc., but the "internet scene" as a whole is definitely something I've been slowly waning myself away from.
Being part of the real world, part of the life of locals around me, helping others to set up events and gatherings for real people (even when they're small for about 10-20 others) is much, much more fulfilling than being yet another voice shouting into the void of the internet.
The internet was always a bit of a cold place but there was some character, the communities' interactions were between some hundreds to thousands of people gathering at a place, nowadays most of it is an incessant feed of bullshit pushed to millions at a time, no sense of community, just bland posts and snippets that you don't care about. Curating that has become such a massive job that I stopped caring, I found the places I like and stuck to them.
Maybe I'm just getting old...
Don't worry, plenty of us "zoomers" are getting sick of it too.
We'd meet up at a bar, pizza place, etc. in groups and just shoot the shit. Hell, my roommate at the time got a visitor from the other side of the world, and they'd only been discussing stuff on some (niche) hobby chatroom.
I haven't really met too many like that in the past 10-12 years though. Many forums moved over to become Facebook groups/pages, and the excitement is kind of gone.
You know, I don't think we are. We are in our 30s, and the only generation that has grown right in the middle of the pre- and post-internet world.
We have seen how life was before, how socialising functioned, and we still adopted this world-changing invention with the fresh excitement and energy of a teenager.
I feel we might be the only ones to truly understand what we have gained. And what we have lost.
I watched a lot of star trek when I was young. At some point it became hard to go back and re-watch because of the advent of social media and the 'modern' web. Social media was so obviously going to be a massive part in everyone's lives that it felt weird that it wasn't present in our star trek future.
Sure, star trek had a world war 3 that would have knocked out current existing social media platforms, but it seems like the natural progression of technology. At some point you get networks, then you hook computers up to them, then you get muggle friendly UX, then social media. Because star trek was a product of its time, they didn't consider the inevitable social media future, and now the suspension of disbelief is too great.
However, then social media continued to exist. Now I feel the obvious natural technological progression is something like: networks, computers, easy UX, social media, large scale societal disruption due to social media, continued technological progression without social media because everyone agrees 'never again'.
Yeah, still a stretch. But it could work...
We didn't say "never again" when we first threw rocks at a rival tribe and saw the aftermath. We didn't say "never again" after millions of people died in world wars. It is beyond optimistic that one day we'll look at Twitter and say "no more of that."
What if the Great Filter is light-speed communication and dissemination of ideas? The odds of a physical lifeform able to deal with the implications of that discovery from day 1 are infinitesimal.
We did, that's why the League of Nations, the UN and the EU were created.
I'm not saying we're doing an incredibly good job, but at least some people are trying.
Really though, a lot of it can be blamed on the advent of the internet. Which sucks, because the internet is an awesome tool.
The office never was my "second place" as a junior; to me this concept is very much a myth. I spent most of my time in my cubicle or in a meeting room and very little time with people my age. Most of my work got done when I was home and undistracted. I stayed up late coding and was perpetually trying to catch up on sleep. When my dog was in the process of slowly dying I worried every day that I'd come home to her corpse and that she'd die alone. I got lucky she died on a Saturday. Now that I work from home I spend a lot more time with people my age and those problems I faced that seemed to be part of every day life that I just needed to "deal with" as consequences to corporate life have evaporated out of thin air.
Illustrated (not my video): https://www.youtube.com/watch?v=3gqF_bkP6N0
People think pop music is "easy" because it's simple. In reality, it's accessible, and that's hard. Making something hundreds of millions of people can enjoy means keeping it simple and repetitive without being boring.
I can make wild melodies and exotic song structures all day. Easy, fun. Seems hard and praiseworthy if you don't know much about making music. Making one 8-16 bar loop stretch out for three minutes without putting you to sleep...that's hard.
I do believe most engineers experience this maturing when they get access to leverage in the workplace, such as going from being an individual contributor to a leader or mentor.
Try sales for a while, and you will realise that no customer has a neatly defined problem for you to solve. They have real complex problems, which you may be able to shoehorn your general solution into. But even then, there is always an alternative solution for the customer which may be simpler or cheaper, such as just hiring an intern to do the job your software can do.
But sometimes the customer doesn't want the alternative solution for various reasons, some ridiculous, a few valid.
Anyway my 2 cents and YMMV, it's not all doom and gloom.
Both research and business related problems are interesting. You don't have to swing between two extremes every time you have an epiphany.
Just my two cents. Nice article!
In actuality I do the same things over and over again and if it's something I haven't done then probably 10s of thousands or 100s of thousands have done it before. I certainly need a search engine to be able to work so it's obvious that I'm copying more than anything.
I’ve always been on the other side of the coin, programming has always been a means to and end, not an end itself.
This point of view is actually validating to hear. Now, on my own time, I do enjoy programming and even dabbling in things like gamedev, but at the same time I don't seek out difficult problems in particular - it's more about the end result.
When it comes to professional work, I have an even more conservative perspective: to just pick well known tools to solve problems in ways that have been proven to work over the years, ideally without blazing new trails and working on stuff nobody has ever done before.
In regards to technologies, that also means lots of Debian or Ubuntu, relational databases and boring back end languages like .NET or Java, with whatever is stable enough for the front end, but then again, the people shopping or submitting forms or whatever won't necessarily care whether you use NoSQL, serverless functions, or something else entirely.
I'm surprised that for once this kind of perspective is on HN. The world is absolutely full of low hanging fruit problems that need someone smarter than the junk you usually see. Nobody wants naive solutions, overabstraction, or premature optimization. The end user can definitely tell the difference even if they don't know how to say it.
Not necessarily. Sometimes the naive or not optimal works just fine, especially with such powerful hardware. They don't care if a web app is made in electron if it works and is snappy (most users aren't opening up their task manager to measure cpu load).
Sounds like a good a goal as any.
There were so many interesting problems and theories in school that you never see as a regular developer, often I think that I have a job that any person could do with a few months of training (does this make me a narcissist? Is the complain here just that my job is unglamorous or is wasting your skills a legit concern?). I guess I haven't matured enough, or maybe I'm not really sold into the impact my company is making.
I wonder if it’s similar to how a musician can’t comprehend how I can barely hear a beat.
Most people only say they want to do something because it is cool or earns money.
But then they fail to put in effort and without putting effort no one can really learn.
I use TDD, a list of projects that gradually increase in difficulty, multiple languages, and solo practice on tools like CodeWars. After they are comfortable with basics I'll have them build a personal project of their own design. Between CodeWars and TDD they've now gotten used to getting dopamine hits from programming. Add in intrinsic motivation on a personal project with a full stack website and they are glued to the editor. Time flies as they rapidly achieve a decent level of skill at reading and writing code. By the time they are wrapping up that personal project, as long as they've kept up with algorithm practice they're ready to interview.
Some have taken longer, usually about a third take a break and circle back around later. Some do take more like 1000 hours to feel confident. Sometimes I end up hiring them myself for contract work to help them build a resume and help me with a project. But on average it's only about 500 hours to teach the fundamentals of unit testing, algorithms, data structures, functional programming, SQL, Typescript, React, and a backend language like Python or C#. They'll not be a master of any of them, but know enough to build a website with persistence and use tools like Google and ChatGPT to get answers when stuck.
The "simple trick" is just making them write absolute tons of code, every day, for months. Then making them read other's code to learn new tricks. It's not easy, I'll work them like a rented mule but they'll come out way over qualified for an entry level position. By the time they're interviewing most will have written 10-20k lines of logic, often grinding a problem a half dozen ways to really see the pros and cons of various solutions.
I think most people greatly overestimate how much code the average CS graduate actually writes for their degree. I've interviewed hundreds of new grads who've written maybe a few hundred lines of code total. Sure they know the theory of computer science, but aren't prepared at all to be software engineers.
I'm not saying the bootcamp model is better, generally those students only know basics of copy paste. This is entirely the fault of the bootcamp instructors who are just trying to churn out graduates.
There just isn't much effort in actually teaching software engineering. Bootcamps have students do some basics and stop way before they're ready. Universities just have them study theory. Rarely is anyone teaching how to read and write code so fluently the students are dreaming in code. But this is extremely effective.
It's not cheap though, I do it as a charity. I've recently started my own business, so as of this summer I'm able to actually pay them which is epic. Still the program as a whole operates at a loss. So I do it because I want to pay it forward.
I occasionally think that too, but then I remember trying to introduce programming to people who couldn't even figure out how to save a text file.
> After two weeks, I come up with some of my own tweaks that make the algorithm work a bit better. I happily add “built a state-of-the-art library for numerical integration, with novel improvements on the best techniques in the academic literature” to my resume.
Ok so he's a liar. He made something "work a bit better" and then claimed to build the whole thing. Two weeks of work. Talk about resume padding.
> After declaring, I finally get assigned an adviser who doesn’t tell me to take easier courses.
Where I went, you barely got any such advice. P R I V I L E G E.
> Math classes haven’t saved me from getting bored of college.
Links to an another article "Why I left Harvard early" oh FFS
> directly trying to improve the world, at Wave.
As opposed to those lesser folks who aren't "directly" trying to improve the world, or just not improving the world at all. Just trying to get by and live a decent life.
> For some reason, a lot of smart college students end up with the idea that “solving hard technical problems” is the best thing they can do with their life
Well it's better than fucking up the world at least.
> most graduates of elite schools—including me—
Yes mr elite.
> my root goal was to use my skills to get the most possible leverage on improving the world.
I'm reminded of the great "Make the World a better place" skit from Silicon Valley.
> Thanks to Eve Bigaj, Alexey Guzey, Jeff Kaufman, Dan Luu, Lincoln Quirk, and Yuri Vishnevsky for reading a draft of this post.
Someone actually didn't catch the lameness of the post.
> Well it's better than fucking up the world at least.
I think there's a good case to be made that over-focus on narrowly defined "hard problems" is one of the biggest drivers of existential risk we face
I rather prefer to talk about elegance, thus I am likely neither an amateur nor a professional, but rather suited to academia. :-)
(I really liked my graduate studies, but was more suited for finding deep connections in research problems than for academic politics, which is why I left academia)