Comment by gandalfgeek | Hacker News Reader

Comment by gandalfgeek | Hacker News Reader

If you don't want to read the full thing, Mollick's substack has more accessible summary of this paper:

https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the...

There are two sides to this thing. The first author of this paper has also written another paper titles "Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment". Abstract:

"As AI quality increases, humans have fewer incentives to exert effort and remain attentive, allowing the AI to substitute, rather than augment their performance... I found that subjects with higher quality AI were less accurate in their assessments of job applications than subjects with lower quality AI. On average, recruiters receiving lower quality AI exerted more effort and spent more time evaluating the resumes, and were less likely to automatically select the AI-recommended candidate. The recruiters collaborating with low-quality AI learned to interact better with their assigned AI and improved their performance. Crucially, these effects were driven by more experienced recruiters. Overall, the results show that maximizing human/AI performance may require lower quality AI, depending on the effort, learning, and skillset of the humans involved."

https://static1.squarespace.com/static/604b23e38c22a96e9c788...

While this is great information, I wish more could be said about Centaurs & Cyborgs, the title of the piece.

There's a brief example of a type of task that each approach excels at, but I'd definitely like more. The crux of this piece is that our human judgment of when/how to use AI tools is the most important factor in work quality when we're on the edge of the frontier. But there's no analysis of whether centaurs or cyborgs did better at those outside-the-frontier tasks; it's not clear why those categories are even mentioned since they appear to have no relevance to the preceding research results. And as the article mentions, the frontier is "invisible" (or at least difficult to see); learning to detect tasks that are in, on, or outside of the frontier seems like an immensely important skill. (I also realize it may become completely obsolete in <5 years as the frontier expands exponentially.)

I understand the goal of this research was not to find these "edges" and to determine how we can improve our judgment about when & how to use AI. But after reading these strong results, that's definitely the only thing I'm interested in. I use AI almost not at all in my work, web development. It has been most useful to me in getting me unstuck from a thorny under-documented problem. Over a year after ChatGPT released, I still don't know if modern LLMs can actually be a force multiplier for my work or if I'm correctly judging that they're not appropriate for the majority of my tasks. The latter seems increasingly unlikely as capabilities advance.

I think this a fascinating, yet unsurprising finding. We’ve known for a long time that automation leads to skillset atrophy on the part of the user. Perhaps most famously, in aviation where some automated systems make complicated procedures so easy that pilots effectively lose the skills to do the proper procedure themselves.

The discussions I hope are happening is the prospect of intentional capability limits of AI in critical industries/areas where humans must absolutely and always intervene. It seems analogous to having a superpower without knowing how to control it, creating a potentially deadly situation at best.

Then perhaps more long term at the generational scale, will humans cognitively devolve to a point where AI is making all of our decisions for us? Essentially we make ourselves obsolete and eventually we’re reclassified as “regular/inferior” species of animals. Profit and return on investments here cannot get in the way of handling these details with great oversight and responsibly.

That opines interestingly on only allowing full self driving cars on pre-approved, high-assurance routes.

It either needs to be 100% or (bad).

Anywhere in the grey zone, and you retard human performance while still occasionally needing it.

This is a particularly relevant use-case for sure. I think we will see a huge skillset disparity between those of us who grew up driving a regular car versus the incoming generations who will grow up with cars that drive themselves. Whether that ends up being a big deal or not, it’s hard to say.

For some more reading material on this subject, see Ironies of Automation[1].

[1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...

If you don't need a skill, you won't practice it, and it will atrophy. This frees you to learn different things. Like how most software engineers today don't know computer engineering (hardware) but they can write more complex applications.

That is if the AI fully replaces the need for a skill. What happens if it mostly replaces it, enough so the skill atrophies, but not enough that the user never needs the skill?

Good point. That sounds like a temporary product problem, like a level 2-3 self-driving car, which still needs you to pay attention.

I think you also need to consider whether you can afford not to have the skill; what would happen if the AI were taken away, or malfunctioned? Airplane pilots are an example of this. In that case you simply have to learn the skill as if the AI will not be there.

> That sounds like a temporary product problem

The jury is still out on how "temporary" of a problem that will be. Improvements have been made but nobody really knows how to get an LLM, for example, to just say "I don't know".

And this is before people start poisoning the well. Google search worked better in a world without Google search. I suspect many AI systems will run into similar issues.

> Google search worked better in a world without Google search.

Oooh, I really like that. Very succinct and topical statement of Goodhart's Law.

Oh, you mean like self-driving cars?

Depends on what you mean by computer engineering but they should definitely have a good mental model of how computers work. Otherwise your complex applications will have trouble scaling. That can matter less in many cases but early architecture mistakes can easily put you down an evolutionary dead end forcing you to experience exponentially rising maintenance costs to add new features / fix bugs or rewrite the thing by someone who understands computer architecture better (+ take advantage of a better understanding the problem domain)

I see what you mean but there's a huge gap between a "good enough" mental model of how computers work and the actual in-depth details that GP means and that have been successfully abstracted away from us. For example: the amount of cases when I have needed to know about the details of how the transistors in RAM chips are laid out have been extremely minimal. Nor have I ever had to worry about the encoding scheme used by the transceivers for the optical fibers that my TCP packets move across.

Obviously there jobs out there that do have to worry about these things (mostly at companies like TSMC and Cisco), but in practice I doubt regular engineers at even the largest scale software companies have to worry about stuff like that.

But that is the exact problem with AI - 99% of the time it's good, so your skills atrophy, but then there's that one percent of the time it decides the reflective side of that 18-wheeler is the sun and drives you into it. That's very different from, say, a compiler, which as a programmer I completely trust to turn my source code into machine code accurately.

> completely trust to turn my source code into machine code accurately

Say hello to undefined behavior :)

Appropriately enough, much of the reason undefined behavior is a consistent problem is exactly the same—it can do something reasonable 99% of the time and then totally screw you over out of nowhere.

Undefined behavior is a defined thing, you know about it so you can avoid it if you want, or look up what the compiler you use does in that case to see if it is useful to you. No such thing for a black box AI model.

You're just mirroring the parent example. Prior to AI (or even computers), this was a problem.

I used to be a more conventional engineering guy. We do a ton of calculus to get our degree. My first job - you just didn't need calculus (although all the associated courses in school used it to develop the theory). As a result, the one offs (less than once a year), when a problem did come up where calculus knowledge was helpful, they couldn't do the task and would go to that one guy (me) who still remembered calculus.

BTW, I guarantee all the employees there got A's on their calculus courses.

> You're just mirroring the parent example. Prior to AI (or even computers), this was a problem.

Not at all, and your argument isn't the same thing.

The problem with current AI, which is very different from past forms of automation, is that, essentially, its failure mode is undefined. Past forms of automation, for example, did not "hallucinate" seemingly correct but false answers. Even taking your own example, it was obvious to your colleagues that they didn't know how to get the answer. So they did the correct thing - they went to someone who knew how to do it. What they did not do, and which is what most current AI solutions will do, is just "wing it" with a solution that is wrong but looks correct to those without more expertise.

“Since when is failure mode being undefined a new thing?”

- C programmer

This is the same bad argument made in a sibling comment.

Undefined behavior in C is actually strictly defined: if you want to avoid undefined behavior, don't do stuff the spec specifically says results in undefined behavior, like deference a null pointer.

Writing up a set of rules to guard against undefined behavior (which is a primary job of a C developer) is simply not possible with an LLM.

This a nice, tight statement of the big picture.

This isn't new, like since LLMs were born, either. In the aviation community, and probably many other domains, there has been a running debate for decades about the pros and cons of introducing more automation into the cockpit and what still should or shouldn't be taught and practiced in training (to be able to effectively recognize and handle the edge cases where things fail).

And in military aviation, this isn't just a question of safety; it's about productivity just like it is for knowledge workers. If an aircraft can fly itself most of the time, the crew can do more tactical decision-making and use more sensors and weapons. Instead of just flying their own aircraft, the crew can also control a number of unmanned aircraft with which they're teamed up.

We've changed the URL from that to https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321. Thanks!

(In cases like this, it's nice to have both the readable summary and the link to the paper.)

Reply on news.ycombinator.com