Field experimental evidence of AI on knowledge worker productivity and quality

There's a brief example of a type of task that each approach excels at, but I'd definitely like more. The crux of this piece is that our human judgment of when/how to use AI tools is the most important factor in work quality when we're on the edge of the frontier. But there's no analysis of whether centaurs or cyborgs did better at those outside-the-frontier tasks; it's not clear why those categories are even mentioned since they appear to have no relevance to the preceding research results. And as the article mentions, the frontier is "invisible" (or at least difficult to see); learning to detect tasks that are in, on, or outside of the frontier seems like an immensely important skill. (I also realize it may become completely obsolete in <5 years as the frontier expands exponentially.)

I understand the goal of this research was not to find these "edges" and to determine how we can improve our judgment about when & how to use AI. But after reading these strong results, that's definitely the only thing I'm interested in. I use AI almost not at all in my work, web development. It has been most useful to me in getting me unstuck from a thorny under-documented problem. Over a year after ChatGPT released, I still don't know if modern LLMs can actually be a force multiplier for my work or if I'm correctly judging that they're not appropriate for the majority of my tasks. The latter seems increasingly unlikely as capabilities advance.

(In cases like this, it's nice to have both the readable summary and the link to the paper.)

The discussions I hope are happening is the prospect of intentional capability limits of AI in critical industries/areas where humans must absolutely and always intervene. It seems analogous to having a superpower without knowing how to control it, creating a potentially deadly situation at best.

Then perhaps more long term at the generational scale, will humans cognitively devolve to a point where AI is making all of our decisions for us? Essentially we make ourselves obsolete and eventually we’re reclassified as “regular/inferior” species of animals. Profit and return on investments here cannot get in the way of handling these details with great oversight and responsibly.

I've been thinking about this a lot lately. I'm a data analyst and all these "give GPT you data warehouse schema, let it generate SQL to answer user's queries" products completely miss the point. An analyst has value as a curator of organizational knowledge, not translator from "business" to SQL. Things like knowing that when a product manager asks us for revenue/gmv, we exclude canceled orders, but include purchases made with bonus currency or promo codes.

Things like this are not documented and are decided during meetings and in Slack chats. So my idea is that in order to make LLMs truly useful, we'll create "hybrid" programming languages that are half-written by humans, and the part that's written by LLMs when translating from "human" language is simple enough for LLMs to do it reliably. I even made some weekend-prototypes with pretty interesting results.

I'm not so sure about this. A lot of work in and for especially large organizations is writing text that often doesn't even get read, or is at most just glanced at. LLMs are great at writing useless text.

Even more common is getting shell/UNIX commands to do what I want. I don't have all the standard UNIX tools (cut, etc) in my head, and I definitely don't keep all the command line options in my head. GPT4 tells me what I need, and it's much quicker to confirm it via the --help or man pages than craft it myself by reading those pages.

Not something I relished spending days on. Instead I spent thirty seconds and our customers are getting a major update shipped a hell of a lot faster and I can get back to bigger things.

I had it write me a handful of sticky functions. The thought of using a search engine for all of that brought tears to my eyes.

Just searching simple things anymore is gut wrenching.

That was just for tasks not suited to GPT-4 (and I assume, LLMs in general).

> "For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI."

For tasks suited to GPT-4, consultants produced 40% higher quality results by leveraging GPT-4.

My take on that is that people who have at least some understanding of how LLMs work will be at a significant advantage, while people who think LLMs "think" will misuse LLMs and get predictably worse results.

43% better it made below average people, 17% better it made above average people.

Same paragraph.

Consultant logic at its finest.

The reason being that the source of material for GPT-4 is rotting underneath. As the internet gets ruined by AI garbage, the quality of output of AI will get worse, not better.

edit: This is a modern day tower of Babel event. We had a magnificent resource and then polluted it with garbage and now it's quickly becoming useless.

It's really good at spitting out code in common, familiar patterns, but the failure rate is very high for anything a little unusual.

They wake up every morning hell bent on maintaining existing customer accounts, growing existing accounts, and sometimes adding a new account. They maintain a professional demeanor, with just the right kind of smalltalk. Finally, they produce a report that justifies whatever was the decision that the CEO already made before he engaged the consultants

Thats pretty much the whole job

What I do is pretty similar to what I did before in my CTO roles, just a bit more strategy and less execution. Clients come to me with questions and problems and I help them solve them based on my experience. If you've got multiple C levels, middle management and a board, you're in a similarly consulting position also as a full time CTO from what I've seen.

I don't go anywhere _near_ LLMs for this, I'd find it somewhere between disrespectful and unethical. They're paying for _my_ expertise. I can imagine it could make a decent alternative to Google and Wikipedia for research purposes, but I'd have to double check it all anyway. I don't see how it'd make my work any easier.

On HN, they get a bad rap from the "i'm smart, I know everything, worker bee crowd" who think business is just slinging JS code and product updates. Consultants are often brought into those people orgs to deal with / or around, valuable know-it-alls, fiefdoms, poor vision / strategy, or politics that are holding the company back. They're often cited as "just doing reports", when in fact consultants sometimes stay out of implementation to make sure the owning team owns & implements the solution.

Consultants often provide the hiring manager a shield on dicey projects or risky outcomes. Where if it goes wrong, the manager can say "it was those consultants".

Teams are brought in as trained, skilled, up to date staff the company cannot hire fire, or shouldn't due to duration of need, or skillset

Sometimes they're brought in because the politics of the internal company lead to stalemates, poor strategy, etc.

Often at the higher levels, they're brought in due to focusing on a speciality, market, or vertical to have large experience that isn't possible to get in house.

One's experience with consultants frames their opinion. I've only worked with very high quality teams in the past that provide a healthy balance of vision, strategy, implementation, etc

I guess they have the advantage of experience doing these things over and over again for similar organizations. It's also an outside perspective, which has both advantages and disadvantages. In the conversations that I was invited into, we spent most of our time just explaining things that any mid-level member of the organization would know, trying to get the RAND people up-to-speed, so that seemed wasteful. But the military is a huge bureaucracy dominated by people who've climbed up the ranks for 30-40 years, so there isn't a lot of fresh outside thinking, and it seems like it could be valuable to inject some.

It is a problem though, because frankly, although you might think that moving to a more modern way of working is a simple fix, we have found that your employees are actually your enemies and are plotting to kill you and your family. You might think that you depend on them to get work done, but we can fix that for you and get our much more skilled and low paid workers to do the job instead.

You will find out that if you allow us to take the required steps to prevent your family from rape and murder your shareholders will get a bump as well! And of course you will as well, you can expect your bonus to at least triple, and you are worth it for sure.

We have already set up the culling stations filled with whirling blades to deal with the scum downstairs, if you say nothing at all we will act and slaughter them instantly. There is no guilt, only reward.

We love you.