Dilution of expertise in the rise and fall of collective innovation
nature.com
nature.com
[0] at least: the classics which still have attention drawn to them today
[1] to be fair, they didn't have much literature from which to draw — maybe a fairer comparison would be "references as percentage of literature extant"?
I don't have any proof to disagree with you, not even an anecdote, but be cautious of making claims such as these without data. Now will almost always seem uninteresting and pedestrian while the past will seem mysterious and interesting.
Very pareto principle/sturgeons law there.
This feels like an example of our trigger happy use of biases to dismiss opinions we disagree with. What perspective, experience, or evidence makes you think survivorship bias is at play here?
In the comment you’re replying to, for example, the rate of un-abandoned RFC started ~20%, and decreased to ~5-10% over the period of GP’s review. That seems lije a very significant change to me, which is hidden by the surprise that the original number wasn’t higher.
- First, I wasn't trying to dismiss any opinions by bringing up survivorship bias, I brought it up because I think it's valuable to have a term to describe the general and probably bi-directional relationship between how notable something is and how well it's known many years after creation - I'd argue that the original comment's footnote [0] talking about "classic" papers is explicitly bringing survivorship, if not the survivorship bias, into the conversation.
Honestly, at this point I've kinda lost track of what I wanted to clarify here, so - I apologize for the confusion and, if interpreted as such, the offensive posting.
Have a good day.
it’s only seminal papers that involve multiple ideas together, and even those have a lot of mathematics involved to make them stand up to scrutiny tests.
I’d even go as far as to argue that if you took any of the highly cited papers of the last 5 years in ML, 20 years or so back, they’d be even more groundbreaking than that time’s seminal work.
1. https://www.fourmilab.ch/etexts/einstein/specrel/specrel.pdf
I think the straight BS->PhD pipeline has introduced an "academic" bias, in the bad sense of the word. I also think that the dominance of Linux, Windows, etc have put blinkers on our research.
In any case, you can trace the evolution I describe in the journal https://onlinelibrary.wiley.com/journal/1097024x which has been around for decades.
That is extremely interesting. I always thought that the debugging tools we have are cool... but we could do _so much more_ to observe the systems we develop. And the scripting languages in most of the embedded debuggers I had worked with sucked.
To get us to the state of the art debugging research in the 90s:
- FOL scripting trap on every variable, function, etc. essentially you need this "monitor" or supervisor behind the scenes observing the total state of the system _as it exists in source code, not in hex_. - reversible programs that can go forward and backward in program state & history. Not sure if the 90s research had a full "tree" of possible executions you could walk or if it was linear. - if you attach something like a Lamport state matrix with the a transport layer you can move to a parallel/concurrent debugger
OTel and friends are essentially trying to build points 2 and 3 there, but its very mushy, since you're not actually capturing a full system state as part of the system, but whatever the coders chose to put in the system capture.
I remember in 2012, 2013 or so, Green River came out with a single core reversible debugger. gdb had one too in '09, or so.
Some cool information in an old SO thread: https://stackoverflow.com/q/1470434/26227
Isn’t this at odds with highly optimized AOT code? I suspect that’s the ultimate blocker but certainly you have this today for any kind of JIT/Interpreted language.
EBF does kind of let you get some of it back, but ultimately it turns out that writing code to debug other code is really complex and you experience rapidly diminishing returns for most bugs (except the super gnarly ones) because a little bit of state and system’s understanding proves sufficient most of the time.
I’m sure you’re aware but GDB has reversible debugging via rr. It’s still hard to get all the way back to source if you’re debugging an optimized build of your code. And polyglot environments can be extra challenging unless special care is taken. I agree it’s a mess, but I think that’s because it’s actually extremely hard to meet competing goals. Do you see the reason why it’s so bad differently with your greater experience in the space?
It's locked behind the really expensive (like 10k or so? For I guess a seat a year?) Visual studio license and IIUC windows-only.
My experience was that C++/Java debuggers beat interpreted language debuggers shockingly well. So much so I still don't touch interpreted language debuggers today. I should check in sometime on Python soon.
Debug code has to, essentially, include debug symbols and tracking, or, alternatively, there's that supervisor monitoring all source-code relevant locations. Things will be slower ... on a conventional x86 chip.
I would suggest that its plausible today that a separate _core_ could be developed for internal system monitoring (think on-SoC JTAG and you get the idea) and that core could be dedicated to more OS functionality otherwise. Notice that this involves a different ISA.
The core tooling advancement problem I believe reduces to this:
> because a little bit of state and system’s understanding proves sufficient most of the time
After a decade being an industry professional, I would say most is 99.5% or greater. 5 out of 1000 bugs seems about right.
So the genuinely advanced tools only start to come into play in very difficult situations - with the implication that only groups with difficult situations will _pay_ for the advanced tooling - with the implication that there's virtually no commercial market, and open source developers have to do this on their own time. Which in turn seems to be generally decreasing IMO - I believe the problem is approximately (say, 90% of the way) isomorphic to "cost of housing market" discussions essentially (how do I spend 1-2 years being someone who can understand graduate-level CS/SWE papers and write high quality code, and then write this state-of-the-art tool that no one will pay for, and still have housing, food, make debt payments etc).
Polyglot environments have to have an element of a debuggable protocol baked in at a pretty deep level to really make it work at, say, the level of Visual Studio in 1999 or something. There's all sorts of issues there around non-aligned interests that shouldn't need to exist. For example, a simple polyglot problem is "how do I send structured data from one process to another in a standardized way". The standard Unix way is "line structured", which works until it doesn't, and then you're staring at writing your own marshal/unmarshal system, which is, definitionally, only your standard. If we don't have _that_ solved, then the social pressure to have a interop debugger between, e.g., Python and Ruby is ... well, take your own guess there.
I think there's absolutely space for a hardware-software codesign to occur which builds out a hardware system, a compiler qua LLVM, a standardized interpreter layer, interop protocols, etc, and to produce a developer system that is a quantum leap beyond what we have today. But that would be a decade long project and have a very long term payoff.
the tl;dr then, imo, is that profound advancement in the field only directly benefits a few, and then only sometimes; so the incentive problem is the basic issue.
I do have to admit though, that my first read of your post was wrong tough, I read it as "debuggers used in embedded" like JTAG stuff, OpenOCD, etc. but you mean it as, if I read this right "debuggers embedded in the runtime/program" stuff like Python, or GDB server running on system right?
Think something like an Arduino.
when i've done this i've found that most of the papers present trivial and wrongheaded approaches, just like most papers today, but they're still a lot more pleasant to read because they're much better written; however, it's much slower going because the terminology is pretty alien
they also have much shorter lists of references but of course they didn't have bibtex
[0] at least: the utensils which still have attention drawn to them today
[1] to be fair: it’s probably survivorship bias.
to say that it's diluting softly implies that it's getting lost, that we're running out of a specific 'innovation' when in fact, it's getting adopted, integrated into our regular lives. Of course it's initially imitated, this is part of how it's getting spread (copied). People will imitate it so to learn what the 'innovation' (as a noun) actually is.
The difference being that the architect today has problems reasoning about problems, and can’t advise much. They’d tell you what the cargo cult says.
Jonathan Blow's 'collapse of civilization' talk goes into this quite a bit.
When there was extreme levels of connectivity and copying between traders collective returns went down because diversity of trading strategies went down, when there was too little connectivity the same thing happened as optimal strategies could not spread effectively.
The maximum returns were achieved somewhere in the middle when there was both room for individual new strategies to emerge but enough connectivity for good strategies to spread avoiding both a sort of herd dynamic and isolation.
It seems like their basic framework (ironically) could be extended to include other things, like incentives (eg for apparent production of "ideas").
One thing about these models is that they don't necessarily get into why things start to be imitated. It seems like the ability to imitate would be related to the spread of the ideas? That is, even if there's a valid idea if the imitators can't or won't imitate it, it won't be imitated. I guess it speaks to whether ideas have value outside of the social milleu they are presented in at a particular point in time. This might seem self evident, but then you have to think about the diffusion dynamics being a property of ideas vs imitators. The authors discuss this in terms of experts not being able to produce novel ideas fast enough, but how would your distinguish that from imitators not understanding how to imitate novel ideas?
so why are ideas imitated? it's human nature.
and not only humans, other apes are known to do this "monkey see, monkey do"
there was an interesting experiment in that human children, as opposed to chimpanzees, would imitate the nonsensical aspects of behavior, making it as though chimpanzees are more 'rational' or more 'logical' than human kids.
Much like Little's Law, this simple and retrospectively obvious observation has wide-ranging consequences. Any organization that aspires to eliteness is thus necessarily selective.