Will deep understanding still be valuable?
ericsink.com
ericsink.com
On the far left you have master generalists with incredible breadth of understanding and the ability to summarize and strategize. These people are highly valued and often land in architecture or leadership roles.
Then in the middle you have the specialists. The Java developers and the Ops people. They’re less valuable because there are so many of them, and they’re somewhat interchangeable.
But then on the far right you’ve got super-specialists. They are masters of their niche - world-class experts with deep understanding who are sought after as the authority in their domain. These folk are super-valuable too.
AI will probably eat the ones in the middle first.
Most roles that are run-of-the-mill “knows framework X” just enough to support naive clients or actually high/value teams, will indeed go to zero.
Which is amusing because 90% of the “AI” companies being started are run by people with no actual expertise, who probably are the most replaceable parts of deploying actual high/end systems.
These days I just do it for a living mostly. CRUD code, arbitrary complex systems because of governance/compliance. It is all a bit boring. It is a how I make a good bundle of money.
On the side I build custom motorcycles. Older ones... I buy an engine that is as mechanical as possible. Easy to fix/maintain etc. Then I do metal fabrication and build a bike around it. For hours I can tinkering with this. Just as I did over 25+ years ago with my computers.
what I wat to say is you still need people to do this so they can dive deep into the abstractions. Bit how people are able to drive their car, maybe know highlevel how the engine works. But are not able to build or maintain one :)
Progress is made by most people not needing deep understanding. There's only so much attention to go around. If people really needed to understand most of the things they dealt with, little would get done.
He doesn't mean deep understanding of the subject matter. He means detailed knowledge about the inner workings. Those are different. For his database example, deep understanding would involve understanding third normal form, the history of databases from ISAM to CODASYL through relational databases to large key-value stores, with a side trip into eventually-consistent systems. Understanding the inner workings is knowing how an InnoDB table in MySQL is represented on disk.
Besides, in the same way that being an engineer with no people skills isn’t a desirable thing, there are plenty of problems with knowing one’s business domain only at the surface-level:
1. It’s always these types of people who end up pushing for unrealistic, and many times, senseless engineering goals from management positions so that they waste a lot of the company’s time and money just by existing in it.
2. It’s these types of people who prioritize profits at all costs, even if it could endanger a customer’s life or data security.
3. These types of people are very easily replaceable, because people are by default not inclined to learn about the world around them, and hence they’re dime a dozen.
No this is still the case, because even folks who understand deeply can only understand a few things deeply. I recently spent several years working in battery tech. The field could be years ahead of where it is now if so many of the field's most pressing problems didn't require researchers to deeply understand programming as well as their field.
I didn't think people actually confused the theatrical performance with the progress of Boltzmann or Newton. Come on!
That is, the deep understanding of an abstraction is required so that you can confidently use it without having to worry about the possible side-effects that those with shallow understanding inevitably introduce.
I think we primarily make progress as engineers (and a species) by making better abstractions.
Yes there are some leaky abstractions, SQL is a great example. But there are plenty of systems that don't leak (e.g. harddrives with constantly dying sectors), and the more you think about all the abstractions the more you have to admit that nobody could get their job done if they had to understand every bit of the computer and the cloud platform and the software.
I think progress is made by standardizing on the right abstractions. What the purest and best form of progress is is something like half of the internet being wordpress. Progress is building tremendous things with the least amount of learning expense necessary.
AI will help get us there. Some day soon AI will write queries and it'll do it better than most companies need before they're profitable.
The software that has brought new things to the table has mostly been built close to the hardware, far away from the apex of the software stack. Stuff like redis, the recent AI boom, none of that is built in wasm or electron. The genuinely impressive leaps forward haven't been made by gluing together APIs or python calls.
From a personal experience, whenever I've built something genuinely new or impressive, it's always been by combining high and low levels of abstraction. If you're limited by what you can do with e.g. a DBMS, there are things a computer could do that you simply don't have access to. If all you understand is the abstraction, you probably won't even see that. With a deeper understanding of the stack, you can bypass that limitation and in some sense work miracles.
Having access to higher levels of abstraction is also important though. Being stuck building everything from the ground up would be just as limiting as being unable to lower the abstraction level when necessary.
Abstractions are produced and consumed. You can make your abstractions better, but that doesn't mean you won't have to deal with leaky abstractions done by others.
Also, having depth in understanding is not only good in cases of leaky abstractions. It's also good to have a better breadth of options for what's out there. If layer B is built on layer A, but you just know B, you might end up building layer C on top of B that's very much like A, but worse in bloat and convolution. Like making a server that serves requests passed as files via NFS because you didn't know TCP sockets existed.
– Alfred North Whitehead
Until they do. Had to deal with this recently. Not easy.
I got physically unwell when I read "the purest and best form of progress is is something like half of the internet being wordpress". That's like saying standardizing on crystal meth is progress.
I get the general sentiment, but I wouldn't confuse "popularity" with "progress".
I have a very hard time convincing myself I could be as effective a programmer (and learner) without all the "trivia" I accumulated. And this is coming from someone that dislikes trivia with an - at best - average memory.
I think this could be a fundamental problem with intelligence in general. You cannot just build towers of abstraction and live on the highest plane only. All layers co-operate and mingle chaoticly. Every little seemingly random patch of pixels in your field of vision could correlate highly with the highest of abstract concepts you are capable of mentating. Knowing what is "trivia" and what is not is itself too a very hard problem.
I have to be honest I never heard of Stack Canaries before - it seems like a fairly specific term of art and I would have a hard time thinking of it as "common trivia"
In contrast, stack canaries are potentially relevant to a large number of developers. They are also accessible: many developers should be able to figure out the details on their own once they have heard of the idea.
Imo stack canaries is esoteric, at least relative to myself. On reading the first google result, I can understand the jist of the intent, but it would certainly take me time to understand the details of how it works, and to come up with an implementation. And if I didn't see it mentioned in this thread I never would have even known to search for it, and probably would have had to dig through a lot of domain knowledge to even know it's something that exists. That's the same way something can be esoteric, even if it's written about and available to everyone with access to a university library, because anyone without domain knowledge would not even know it exists or why it's relevant.
And I'm sure you can agree that if Google did not exist, probably stack canaries would be unknown and inscrutable to a vast, vast majority of developers. So if we go with your definition, the invention of Google certainly greatly reduced the amount of esoteric knowledge in existence, by making information easier to access in general. So I would argue that generative AI further reduces the scope of which information could be considered generally inaccessible,
Other topics may be difficult and advanced, but they are not esoteric, because they are more universal. You don't need a specific context to understand them, only the basic expertise.
The internet turned much esoteric knowledge into common trivia by making information more accessible. I don't think AI will have a similar effect. At least not the public models that are mostly trained using information that is already accessible. Large organizations may achieve something with internal models, but in many cases, esoteric knowledge is esoteric because it's implicit.
1. Sufficiently obscure technical details in some field of research. There are several high-quality sources, but each of them uses its own terminology with subtly different definitions. Many immediate consequences are also left unsaid, because you can't say everything if you want to keep the paper readable. Anyone with sufficient background should see them anyway.
2. Word etymologies. There are often many competing explanations with varying degrees of plausibility and popularity, and many sources with varying degrees of credibility. The things the LLM hallucinates are often quite interesting.
3. Questions with well-defined answers that are overshadowed by something far more popular. Like, for example, what did Caesar do on the night before crossing Rubicon.
The way I experience AI is that it's like a super-powered search. It's like having an undergrad with limited knowledge read everything and come back to you with an answer.
Based on their own limited knowledge on complex topics, they're not always going to come back to you with the correct answer, but it gives you a heck of a better starting point than if you were to search for it yourself from scratch.
So what that means regarding esoteric knowledge is that the specific detail you were looking for is no longer hidden beneath 5 layers of obscura - you at least get pointed to the general area of the solution, and then maybe you have to read a paper or two to get the details you need. And instead of having to read 5 other papers to understand that one, you can ask questions and get decent answers about the terms you don't understand etc.
So imo it becomes less valuable to yourself be an encyclopedia of esoteric knowledge, since that knowledge becomes more accessible. It's like how being an expert on any road system become much less valuable with the advent of GPS.
> Last week I was trying to confirm that the CIL castclass
This person is working in the Microsoft .NET ecosystem, in which the "you don't need to know that" attitude is prevalent. The whole ecosystem is built around it, and I'm not for a second saying that is a bad thing. .NET technologies are powerful and effective.
However, like the author I didn't find myself comfortable in that world. I tend to gravitate towards a certain kind of problem and my problem solving approach relies on fully understanding what is going on. While .NET Core (now just .NET) is open source, the don't-look-under-the-hood mentality is built into it in many ways, and I found my way of thinking to be at odds with it. After a few years of Windows .NET work I eventually moved to a C++/Python/Linux role and am a lot happier. I don't think the employer I was with for those years should have gone with the latter stack (but I emphatically don't think my new employer should have gone with the former - it's a different problem domain). It's not a criticism. I just feel more at home doing this kind of development.
In short, if you find yourself reading the 30,000 line C++ file containing the .NET garbage collector implementation more than once, you may not be working in a technical environment best suited to your mental wellbeing.
But an AI buddy should make it way easier to get to deep understanding imo. People are naturally curious and will dive into topics that pique interest/joy.
It might come to you as a shocker: businesses rarely care about the "how does it work" - illusion is all that matters for most businesses, unfortunately.
The developer is the abstraction maintaining that illusion - be it by juggling a legacy codebase or a broken implementation, or a great one.
Then when you are given something hard to quickly learn, e.g. say you need to go from zero to Kubernetes pro in a week, doing so should be easier. Or even learning the bits that are worth learning to optimize that week so you can be most productive after it, will be easier.
It is a different argument than the one about avoiding leaky abstractions.
It doesn't matter if someone thinks they're special when an algorithm can and does take their job by spitting out disgusting Javascript that happens to accomplish business needs well enough to not hire the human. Business value is all businesses care about.
Those that need something more bespoke, that fits them better, that isn't disposable because the problems it's trying to solve aren't disposable... they might still need the tailor.
I know a few human programmers who would make really good LLMs. And they seem unable to come up with non-absurd code at all. If you define the specs with enough details that they haven't screwed it up, then you're at a point where you've basically already written the software anyway. Should I be worried that robot programmers will do any better?
Deep understanding of Nature? As valuable as ever.
"How does it work" rarely brings value.
It's true. Business doesn't need to care or know about how btree indices works in RDBMs, they just care whether there reports are generated fast enough.
> I wrestle with the whole abstraction question, and I'm still not sure where a good engineer these days should draw the line — the one below which it's all just magic.
> We all have that line somewhere; I encountered mine in school, back in my semiconductors course. We were (ostensibly) learning how semiconductors work, but it was perfectly clear to me that this was just magic, and I was never going to understand it — in part, because I didn't want to make what was clearly going to be a HUGE effort to get it. There was all this crap about K-spaces and lattices and who knows what else. Maybe they'd listed the prerequisites wrong, but I did not have the math foundations to understand what was going on.
> It didn't help that my professor was 100% senile, either. Once a week he'd break into his senile rant about what an Amazing Device the Human Eye is, totally forgetting that he'd shared this insight with us every week since the quarter started. We'd all recite the story along with him, sometimes jumping ahead, which he senile-ly mistook for us being just as big fans of the Human Eye as an Amazing Instrument as he was. It was really damned hard to concentrate in that class, and at the end of the quarter, my Human Eye was Amazed to see I'd almost failed it, which signalled clearly that it was Time to Change Majors. I'd found that I love programming and detest hardware, so staying a Computer Engineering major probably wouldn't be too wise.
> So I transferred into the UW's Computer Science department, which meant switching from Engineering to Arts & Sciences, which meant taking a bunch of extra foreign-language and froo-froo social anthropology classes, where I got to watch films of dolphins counting to five through their blow-holes. Which is about how smart I felt after that semiconductors class. I felt like I could have made it as a dolphin.
> It took me an extra year, but I finished in CS instead of CE, which allowed me to conveniently pretend that computer hardware, starting somewhere down at the level of the doping barrier in a silicon semiconductor, is just pure magic. Parts of the hardware above that level are still a little murky, but I have a pretty good idea how computers can be made by assembling successively larger components together from smaller ones. Given time and a gun to my head, I could probably start with any set of smaller components and derive how to build higher-level abstractions: transistors into logic gates, logic gates into larger assemblies that do arithmetic and boolean computing, those into state machines and ALUs, and so on. I'd suck at it, but I get the general idea, so it's not really "magic".
> It's magic when you have absolutely no frigging clue how it works, not the first clue. It just works. That's how it is for me and semiconductors.
> But I'm OK with that. I'm OK with treating semiconductors as "atoms" (or axioms, anyway) in our little programming universe. Chips are like molecules, CPUs are compounds, and Von Neumann machines can be bricks and mortar. Well, I'm sure you could come up with a way better metaphor. Whatever.
> The point is, the computers and networks and power supplies and so on are all amazingly complex under the hood, but usually I can just pretend they work a certain way, and not worry about how, any more than I have to worry about my Amazing Eye in order to pour myself another glass of fine wine, which I will doubtless use to polish my keyboard if I don't finish this blog entry soon.