The threat is comfortable drift toward not understanding what you're doing
ergosphere.blog
ergosphere.blog
And so the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs.
Which means we need people like Alice! We have to make space for people like Alice, and find a way to promote her over Bob, even though Bob may seem to be faster.
The article gestures at this but I don't think it comes down hard enough. It doesn't seem practical. But we have to find a way, or we're all going to be in deep trouble when the next generation doesn't know how to evaluate what the LLMs produce!
---
† "Useful" in this context means "helps you produce good science that benefits humanity".
Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical—but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least consider these sorts of measures. https://www.theatlantic.com/culture/archive/2025/09/ai-colle...
Not so in plenty of other countries. Hopefully US reverses the anti-science trend before it's too late
I saw interviews with young Americans on spring break and they were so utterly uninformed it was mind-blowing. Their priorities are getting drunk and getting laid while their country bombs a nation “into the stone ages”, according to their president. And it’s not their fault: they are the product of a media environment and education system designed for exactly this outcome.
Trump doesn't have to justify a single thing because the billionaires behind him know that every last bet is off and their very livelihoods are at risk, and his entire base of support up and down the chain are either complicit or fooled.
What the world does when they finally realize Democrats and Republicans are simply two sides of the vast apparatus suppressing the will of the people by any means necessary will be... spectacular.
I remember things very differently. Everyone cared about the Iraq war, gay-straight alliance was one of the most up and coming clubs, and political music was everywhere. Green Day had their big second wave with American Idiot, System Of A Down was on top of the world, Rock Against Bush was huge, anarcho-punk like Rise Against was getting big.
I'm not a teenager anymore obviously, so it's entirely possible I'm just missing it, but I've seen very little that compares to those sort of movements. On the other hand, most millennials I know are still wildly politically active.
If you compare that to the current Iran war, a majority of the population is already against it, however the current administration doesn't seem to care much about public opinion, and there doesn't seem to be much that the public can do about it.
Gen Z is conducting revolutions. https://en.wikipedia.org/wiki/2025_Nepalese_Gen_Z_protests
Tearing down entrenched establishments that don't work for the people is what people were voting for in 2016 and were disappointed he didn't do it. They're glad he's finally doing what he promised a decade ago.
That is definitely an opinion.
Schools must be quite busy and schizophrenic what with the completely ideologically inconsistent indoctrination they are apparently doing.
Not to mention judging engagement in spring break goers is kind of a major selection bias. Those who are upset about it wouldn't be in the mood to vacation.
It sounds entirely reasonable and moderate to me.
It's going to collapse regardless because of the replication crisis. You might as well tackle the hard problem and figure out how to integrate replication into acceptance, or the consensus publication is intended to represent is meaningless. This is true regardless of whether a human or a robot is performing the work.
You can access the full article at https://archive.is/zSJ13 (I know archive.is is kind of shady, but it works).
Only, replacing the guts of such a machine to contain a local LLM is damn easy today. Right now the battery mass required to power the device would be a giveaway, but inference is getting energetically cheaper.
> Colleges that are especially committed to maintaining this tech-free environment could require students to live on campus, so they can’t use AI tools at home undetected.
Just like my on-campus classmates never smoked weed or drank underage, I'm sure.
It's very hard to hide the fact that someone else did an assignment when you have to defend it in front your tutor and a small group of fellow students and it's next to impossible to pass a final viva without knowing and understanding what you are talking about.
The problem is we have all become addicted to cheap 'education' and a the traditional methods are expensive.
But I think the institutions and the students need to ask themselves what the university is for. Is it to hand out diplomas or is it there so that the students can learn? A student who only wants the diploma has an incentive to cheat, one who wants to learn does not because the only person cheated is themself.
> Another reason that a no-exceptions policy is important: If students with disabilities are permitted to use laptops and AI, a significant percentage of other students will most likely find a way to get the same allowances, rendering the ban useless. I witnessed this time and again when I was a professor—students without disabilities finding ways to use disability accommodations for their own benefit. Professors I know who are still in the classroom have told me that this remains a serious problem.
This would be a huge problem for students with severe and uncorrectable visual impairments. People with degenerative eye diseases already have to relearn how to do every single thing in their life over and over and over. What works for them today will inevitably fail, and they have to start over.
But physical impairments like this are also difficult to fake and easy to discern accurately. It's already the case that disability services at many universities only grants you accommodations that have something to do with your actual condition.
There are also some things that are just difficult to accommodate without technology. For instance, my sister physically cannot read paper. Paper is not capable of contrast ratios that work for her. The only things she can even sometimes read are OLED screens in dark mode, with absolutely black backgrounds; she requires an extremely high contrast ratio. She doesn't know braille (which most blind people don't, these days) because she was not blind as a little girl.
Committed cheaters will be able to cheat anyway; contemporary AI is great at OCR. You'll successfully punish honest disabled people with a policy like this but you won't stop serious cheaters.
I enjoyed my time, I made a lot of lifelong friends, and figured out how to live on my own. My buddies that enrolled in boot camp instead of college learned all those same skills, for free.
Education won’t be ruined or blemished my LLMs, the whole thing was a joke to begin with. The bit that ruined college was unlimited student loans… and all of our best and brightest folks running the colleges raping students for money. It’s pathetic, evil, and somehow espoused.
Magical times.
The problem is that this is treating the symptom rather than the cause. The symptom is that cheating for college admission and achievement is too effective. The cause is that college admission and achievement has become high stakes, and it absolutely should not be.
My 40-some-odd years on this planet tells me the answer is yes.
Sure there is. Its the formal education system that produced the college grad.
The proposal that everyone pay for college until they are in their 40s doesn’t seem viable.
If AI output remains unreliable then eventually enough companies will be burned and management will reinstate proper oversight. All while continuing to pay themselves on the back.
Other projects' success will be proportional to their number of Schwartz' and so it seems unlikely they disappear. But they may disappear for areas in which there is no immediate money.
I hope it will encourage people to think more about what they get out of the work, what doing the work does for them; I think that's a good thing.
There's a couple unfortunate truths going on all at the same time:
- People with money are trying to build the "perfect" business: SaaS without software eng headcount. 100% margin. 0 Capex. And finally near-0 opex and R&D cost. Or at least, they're trying to sell the idea of this to anyone who will buy. And unfortunately this is exactly what most investors want to hear, so they believe every word and throw money at it. This of course then extends to many other business and not just SaaS, but those have worse margins to start with so are less prone to the wildfire.
- People who used to code 15 years ago but don't now, see claude generating very plausible looking code. Given their job is now "C suite" or "director", they don't perceive any direct personal risk, so the smell test is passed and they're all on board, happily wreaking destruction along the way.
- People who are nominally software engineers but are bad at it are truly elevated 100x by claude. Unfortunately, if their starting point was close to 0, this isn't saying a lot. And if it was negative, it's now 100x as negative.
- People who are adjacent to software engineering, like PMs, especially if they dabble in coding on the side, suddenly also see they "can code" now.
Now of course, not all capital owners, CTOs, PMs, etc exhibit this. Probably not even most. But I can already name like 4 example per category above from people I know. And they're all impossible to explain any kind of nuance to right now. There's too many people and articles and blog posts telling them they're absolutely right.
We need some bust cycle. Then maybe we can have a productive discussion of how we can leverage LLMs (we'll stop calling it "AI"...) to still do the team sport known as software engineering.
Because there's real productivity gains to be had here. Unfortunately, they don't replace everyone with AGI or allow people who don't know coding or software engineering to build actual working software, and they don't involve just letting claude code stochastically generate a startup for you.
I don't actually think the article refutes this. But the AI needs to be in the hands of someone who can review the code (or astrophysics paper), notice and understand issues, and tell the AI what changes to make. Rinse, repeat. It's still probably faster than writing all the code yourself (but that doesn't mean you can fire all your engineers).
The question is, how do you become the person who can effectively review AI code without actually writing code without an AI? I'd argue you basically can't.
I agree, and I'd go a step further:
You can be the absolute best coder in the world, the fastest and most accurate code reviewer ever to live, and AI still produces bad code so much faster than you can review that it will become a liability eventually no matter what
There is no amount of "LLM in a loop" "use a swarm of agents" or any other current trickery that fixes this because eventually some human needs to read and understand the code. All of it.
Any attempt to avoid reading and understanding the code means you have absolutely left the realm of quality software, no exceptions
Besides, it's not like QA ends at code analysis. In fact, that's pretty shallow for the field.
Was the LLM even useful for Schwartz, if it produced false output?
I have gained a lot of benefit using LLMs in conjunction with textbooks for studying. So, I think LLMs could help you become Schwartz.
Presumably you're also reading some kind of learning text about the Chinese language, so the sole source isn't just the LLM?
In my experience, asking an LLM to produce small examples of well-known things (or rather, things that are going to be talked about frequently in the training data, so generally basic or fundamental topics) tend to work fine, and is going to be at a level where you yourself can judge what's presented.
I think the real danger is when a person is prompting things they don't know how to verify for themselves, since then we're basically just rolling dice and hoping
Doesn’t sound like these tools should be used to write scientific papers for example and they seem to bamboozle people far more than help them.
I can only speak to the engineering context not academia, but I would expect there’s similar patterns of busywork. Even the blog authors admit to using AI. AI cant replace thinking but it can replace menial labor, which is prevalent even among “knowledge workers”.
I built a full stack app in Python+typescript where AI agents process 10k+ near-real-time decisions and executions per day.
I have never done full stack development and I would not have been able to do it without GitHub Copilot, but I have worked in IT (data) for 15 years including 6 in leadership. I have built many systems and teams from scratch, set up processes to ensure accuracy and minimize mistakes, and so on.
I have learned a ton about full stack development by asking the coding agent questions about the app, bouncing ideas off of it, planning together, and so on.
So yes, you need to have an idea of what you're doing if you want to build anything bigger than a cheap one shot throwaway project that sort of works, but brings no value and nobody is actually gonna use.
This is how it is right now, but at the same time AI coding agents have come an incredibly long way since 2022! I do think they will improve but it can't exactly know what you want to build. It's making an educated guess. An approximation of what you're asking it to do. You ask the same thing twice and it will have two slightly different results (assuming it's a big one shot).
This is the fundamental reality of LLMs, sort of like having a human walking (where we were before AI), a human using a car to get to places (where we are now) and FSD (this is future, look how long this took compared to the first cars).
What may well happen instead is that Bob publishes two papers. He then outcompetes Alice based on the insistence that others have on "publish or perish". Alice becomes unemployed and struggles, having been pushed out.
The person who puts the time and effort in doesn't just sit at the same level and they don't both just find decent employment. Competition happens and the authentic learning is considered a waste of time, which leads to real and often life threatening consequences (like being homeless after being unable to find employment).
This, I think, may be the more interesting bit. Steve Jobs anecdotally did caligraphy in school, which some would consider a waste of time, but Steve credited some of the stylistic choices to.
The question then becomes whether it will become an issue now or later. Having seen some of the output, I have no doubt that a lot can now be built by non-programmers ( including myself; I suppose I belong in the adjacent category ). The building blocks exist and as long as the problem was part of the initial training, odds are, LLM will help you build what you want.
It may not be perfect, safe, or optimized, but it may still be exactly what the user wanted to do. Now, the problems will start when those will, inevitably, move into production at big corps. In a sense, we have seen some interesting results of that in the past few weeks ( including accidental claude code release ).
In a grand scheme of things, not much is changing... except for speed of change. But are we quite ready for this?
The solution is relatively simple though - not sure the article suggests this as I only skimmed through:
Being good in your field doesn't only mean pushing articles but also being able to talk about them. I think academia should drift away from written form toward more spoken form, i.e. conferences.
What if, say, you can only publish something after presenting your work in person, answer questions, etc? The audience can be big or small, doesn't matter.
It would make publishing anything at all more expensive but maybe that's exactly what academia needs even irrespective of this AI craze?
My grad school friend who was a physicist would write his talk just before his conferences, and then submit the paper later. My experience in CS was totally backwards from that.
For so many workers, their companies just want them to produce bullshit. Their managers wouldn't frame it this way, but if their subordinates start producing work with strict intellectual rigor it's going to be an issue and the subordinates will hear about it.
So, you're not wrong. But the majority of LLM customers don't care and they just want to report success internally, and the product needs to be "just good enough." An LLM might produce a shitty webpage. So long as the page loads no on will ever notice or care that it's wrong in the way that a physics paper could be wrong.
Here's a competing thought experiment:
Jorge's Gym has a top notch body building program, which includes an extensive series of exercises that would-be body builders need to do over multiple years to complete the program. You enroll, and cleverly use a block and tackle system to complete all the exercises in weeks instead of years.
Did you get the intended results?
What your describing here
> reading the portions of those papers that you are interested in
Is a hybrid use of the technology which no one would argue against.
The question remains if students have the foresight to use the correct method and which results in the best learning outcomes.
For myself, I would have absolutely let LLMs summarise swaths of text and write my essays when I was at Uni. That's just me though. Maybe today's learners are better than I.
That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all, then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary.
Btw, note that in the thought experiment Bob isn't just delegating all the work to the LLM. He makes it summarise articles, extract important knowledge and clarify concepts. This is part of a process of learning, not being a passive consumer.
There are flowers that look & smell like female wasps well enough to fool male wasps into "mating" with them. But they don't fly off and lay wasp eggs afterwards.
But the whole point is that there is a significant difference between Schwartz and non-Schwartz, that only turns up after they start working for real, producing new work rather than rehashing established material, and it takes years to detect. By that time, Bob's forty.
It isn't a "sample problem" it's a process problem. By perpetually raising the stakes and focusing on metrics (e.g. grades, number of publications for students, graduation rates for schools) we've created and fallen into a Poe's law trap. Adding a new metric isn't likely to help.
What might help? Making the metrics harder to game (e.g. something like oral exams, early and often), more discerning (grade deflation), and moving the wrong-track consequences earlier (start holding people back in grade school, make failing to graduate high school easier, make getting into college harder, etc.), and change the cash-cow funding models to remove the perverse incentives.
We aren't likely to do any of these things.
No, this is impossible unless Bob is presenting at each weekly meeting simply the output of the LLM and feeding the tutor's feedback straight into it. For a total of 10 minutes work per week, and the tutor would notice straight away at least for the lack of progress.
No, the article specifies that Bob actually works with the LLM, doesn't just delegate. He asks the agent to summarise, to explain, and to help with bug fixing. You could easily argue that Bob, having such an AI tutor available 24/7, can develop understanding much faster. He certainly won't waste his time with small details of python syntax (though working with a "coding expert" will make his code much cleaner and more advanced).
So I settled on very incremental work. It's annoying cutting and pasting code blocks into the web interface while I'm working on my interface to Neovim, spent a whole day realizing I can't trust it to instrument neovim and don't want to learn enough lua to manage it. (I moved onto neovim from Emacs because I don't like elisp and gpt is even worse at working on my emacs setup than neovim, the end goal is my own editor in ruby but gpt damn sure can't understand that atm) But at least I'm pushing a real flywheel and not the brooms from Fantasia.
Then you can be writing the code, with the LLM doing the "boring" parts, in chunks small enough you can review them on the fly.
It's exactly the same for coding.
In the past, if you neglected your institutes and academia, what happened? A rival state got electricity/cars/nuclear weapons first and you would be SOL.
These days, what happens? They invent faster phones? Higher res video?
This take may be a bit hyperbolic but I find it's a good thought exercise prompt.
If you are a massive company that owns all of the knowledge and all of the technology needed to apply that knowledge, then you don't need Alice. You don't _want_ Alice. You want more Bobs. It looks better on the books.
Tale as old as time.
I thought I'd give Gemini a go. When I uploaded the 18-page PDF, it complained the output exceeded some limit. So I used pdftk to break it up into 4-page chunks, which seemed to work - the output looked very good and passed a couple of spot checks. But I don't trust these things as far as I can kick them.
There was a transaction column and a running balance column, so I did a quick check to see if every new balance equalled the previous one plus the transaction. And it almost always did. There were a couple of errors I put down to transcription errors. I was wrong. I eventually twigged that these errors only happened where I had split the PDFs. After tracking where the balance first went wrong, it became evident it had dropped chunks of lines, duplicated others, and misaligned the transaction and balance columns. It was complete rubbish, in other words.
So why did my balance check show so few errors? I put that down to it knowing what a good bank statement looked like. A good bank statement adds up. So it adjusted all the balances so it looked like a real bank statement. I also noticed these errors got more frequent in later pages. I tried splitting the PDF into single pages and loading them into the model one at a time. That didn't help much for the later pages, but the first one was usually good. So then I loaded each page into a fresh context, with a fresh prompt. If that didn't produce something that balanced, the second go always did.
I'm not sure it saved time over doing it manually in the end. It's a tired analogy now, but it's true: at their heart, these things are stochastic parrots. They almost never produce the same output twice when given the same input. Instead, they produce output that has a high probability of following the input tokens supplied. If there is only one correct output but the output is small enough, the odds are decent they will get it right. But once the size grows, the odds of it outputting complete crap become a near certainty.
This is one more argument to the only LLM trained to find "truth"
I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue.
To be honest, I’m looking at leaving software because the job has turned into a different sort of thing than what I signed up for.
So I think this article is partly right, Bob is not learning those skills which we used to require. But I think the market is going to stop valuing those skills, so it’s not really a _problem_, except for Bob’s own intellectual loss.
I don’t like it, but I’m trying to face up to it.
The problem arrises when Bob encounters a problem too complex or unique for agents to solve.
To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade meals because "those aren't going anywhere". Over the span of years, eventually a real cook will be able to make way better meals than anything you can buy at a grocery store.
The market will always value the exact things LLMs can not do, because if an LLM can do something, there is no reason to hire a person for that.
The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different).
This isn't guaranteed to play out, but it should be the default expectation until we actually see greatly diminishing outputs at the frontier of science, engineering, etc.
Calculators are deterministically correct given the right input. It does not require expert judgement on whether an answer they gave is reasonable or not.
As someone who uses LLMs all day for coding, and who regularly bumps against the boundaries of what they're capable of, that's very much not the case. The only reason I can use them effectively is because I know what good software looks like and when to drop down to more explicit instructions.
If I use a calculator to find a logarithm, and I know what a logarithm is, then the answer the calculator gives me is perfectly useful and 100% substitutable for what I would have found if I'd calculated the logarithm myself.
If I use Claude to "build a login page", it will definitely build me a login page. But there's a very real chance that what it generated contains a security issue. If I'm an experienced engineer I can take a quick look and validate whether it does or whether it doesn't, but if I'm not, I've introduced real risk to my application.
It's equivalent to asking your friend to pick you up, and they arrive in a big vs small car. Maybe you needed a big car because you were going to move furniture, or maybe you don't care, oops either way.
Calculators provide a deterministic solution to a well-defined task. LLMs don't.
It is not possible to be nearly as precise when describing a desired solution to an LLM, because natural languages are simply not capable of that level of precision... Which is the entire reason coding languages exist in the first place
Calculators are deterministic, but they are not necessarily correct. Consider 32-bit integer arithmetic:
30000000 * 1000 / 1000
30000000 / 1000 * 1000
Choosing a "better" language was not always an option, at least at the time. I was working with grad students who were managing huge datasets, sometimes for large simulations and sometimes from large surveys. They were using C. Some of the faculty may have used Fortran. C exposes you the vulgarities of the hardware, and I'm fairly certain Fortran does as well. They weren't going to use a calculator for those tasks, nor an interpreted language. Even if they wanted to choose another language, the choice of languages was limited by the machines they used. I've long since forgotten what the high performance cluster was running, but it wasn't Linux and it wasn't on Intel. They may have been able to license something like Mathematica for it, but that wasn't the type of computation they were doing.
That's not actually true. The HP-12C calculator is still the dominant calculator in business schools 45 years later precisely because it did take expert judgement to determine whether certain interest and amortization calculations were reasonable.
And yes yes you can pull up the quote or ask your AI, but they will be wrong. The quote is from Socrates reciting a "myth", as is pretty typical in a middle late dialogue like this.
But here, alas we can recognize the utter absurdity, that this just points out why writing can be bad, as Socrates does pose. Because you get guys 2000 years in future using you and misquoting you for their dumb cause! No more logos, only endless stochastic doxa. Truly a future of sophists!
Well, we still make people calculate manually for many years, and we still make people listen to lectures instead of just reading.
But will we still have people to go through years of manual coding? I guess in the future we will force them, at least if we want to keep people competent, just like the other things you mentioned. Currently you do that on the job, in the future people wont do that on the job so they will be expected to do it as a part of their education.
In a sense, I think you are right. We are currently going through a period of transition that values some skills and devalues others. The people who see huge productivity gains because they don't have to do the meaningless grunt work are enthusiastic about that. The people who did not come up with the tool are quick to point out pitfalls.
The thing is, the naysayers aren't wrong since the path we choose to follow will determine the outcome of using the technology. Using it to sift through papers to figure out what is worth reading in depth is useful. Using it to help us understand difficult points in a paper is useful. On the other hand, using it as a replacement for reading the papers is counterproductive. It is replacing what the author said with what a machine "thinks" an author said. That may get rid of unnecessary verbosity, but it is almost certainly stripping away necessary details as well.
My university days were spent studying astrophysics. It was long ago, but the struggles with technology handling data were similar. There were debates between older faculty who were fine with computers, as long as researchers were there to supervise the analysis every step of the way, and new faculty, who needed computers to take raw data to reduced results without human intervention. The reason was, as always, productivity. People could not handle the massive amounts of data being generated by the new generation of sensors or systematic large scale surveys if they had to intervene any step of the way. At a basic level, you couldn't figure out whether it was a garbage-in, garbage-out type scenario because no one had the time to look at the inputs. (I mean no time in an absolute sense. There was too much data.) At a deeper level, you couldn't even tell if the data processing steps were valid unless there was something obviously wrong with the data. Sure, the code looked fine. If the code did what we expected of it, mathematically, it would be fine. But there were occasions where I had to point out that the computer isn't working how they thought it was.
It was a debate in which both sides were right. You couldn't make scientific progress at a useful pace without sticking computers in the middle and without computers taking over the grunt work. On the other hand, the machine cannot be used as a replacement for the grunt work of understanding, may that involves reading papers or analyzing the code from the perspective of a computer scientist (rather than a mathematician).
Catching an LLM hallucinating often takes a basic understanding of what the answer should look like before asking the question.
I went to school the next day and told my teacher that the calculator says that 10+10 is 14, so why does she say it's 20?
So she showed me on her calculator. She pressed the hex button and explained why it was 14.
I think a major problem with people's usage of LLMs is that they stop at 10+10=14. They don't question it or ask someone (even the LLM) to explain the answer.
We had the same problem in the early days of calculators. Using a slide rule, you had to track the order of magnitude in your head; this habit let you spot a large class of errors (things that weren't even close to correct).
When calculators came on the scene, people who never used a slide rule would confidently accept answers that were wildly incorrect (example: a mole of ideal gas at STP is 22.4 liters. If you typo it as 2204, you get an answer that's off by roughly two orders of magnitude, say 0.0454 when it should be 4.46. Easy to spot if you know roughly what the answer should look like, but easy to miss if you don't).
The people who make the calculator analogy are already victims of the missing rung problem and they aren't even able to comprehend what they're lacking. That's where the future of LLM overuse will take us.
> “Most ingenious Theuth, one man has the ability to beget arts, but the ability to judge of their usefulness or harmfulness to their users belongs to another; [275a] and now you, who are the father of letters, have been led by your affection to ascribe to them a power the opposite of that which they really possess.
> "For this invention will produce forgetfulness in the minds of those who learn to use it, because they will not practice their memory. Their trust in writing, produced by external characters which are no part of themselves, will discourage the use of their own memory within them. You have invented an elixir not of memory, but of reminding; and you offer your pupils the appearance of wisdom, not true wisdom, for they will read many things without instruction and will therefore seem [275b] to know many things, when they are for the most part ignorant and hard to get along with, since they are not wise, but only appear wise."
Sounds to me like he was spot on.
Yes - specific faculties atrophied - I wouldn't dispute it. But the (most) relevant faculties for human flourishing change as a function of our tools and institutions.
> People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word.
If the conclusion now becomes “actually, Socrates was correct but it wasn’t that bad”, then why bring up Socrates in the first place?
As it happens, we generally don't let people use calculators while learning arithmetic. We make children spend years using pencil and paper to do what a calculator could in seconds.
But we might see a lot more specialization as a result
My point is that getting into the weeds of writing CRUD software is not the only way to gain the ability to write complex algorithms, or to debug complex issues, or do performance optimization. It's only common because the stuff you make on the journey used to be economically valuable
That’s the stuff that ai is eating. The stuff I’m talking about (scaling orgs, maintaining a project long term, deciding what features to build or not build etc) is stuff very hard for ai
agents might be better at it than people are, given the right structure
Example- Linus Torvalds, never worked for a company, made the original Linux while a grad student, and seems to be doing fine (I'm writing this message on a ThinkPad running Linux Mint). Or Bill Joy with BSD at Berkley, before his time at Sun. Or heck, why not go all the way back to Ken Thompson and Dennis Ritchie building Unix and C?
But there is also a more subtle thing, which is we're trending towards superintelligence with these AIs. At the point, Bob may discover that anything agents can't do, Alice can't do because she is limited by trying to think using soggy meat as opposed to a high-performance engineered thinking system. Not going to win that battle in the long term.
> The market will always value the exact things LLMs can not do, because if an LLM can do something, there is no reason to hire a person for that.
The market values bulldozers. Whether a human does actual work or not isn't particularly exciting to a market.
Do you have a solution for me? How does the market value things that don't yet exist in this brave new world?
It doesn't matter if Bob can be normal. There was no point to him being paid to be on the program.
From the article:
If you hand that process to a machine, you haven't accelerated science. You've removed the only part of it that anyone actually needed.
Yeah, I'm surprised at the number of people who read the article and came away with the conclusion that the program was designed to churn deliverables, and then they conclude that it doesn't matter if Bob can only function with an AI holding his hand, because he can still deliver.
That isn't the output of the program; the output is an Alice. That's the point of the program. They don't want the results generated by Alice, they want the final Alice.
> There's a common rebuttal to this, and I hear it constantly. "Just wait," people say. "In a few months, in a year, the models will be better. They won't hallucinate. They won't fake plots. The problems you're describing are temporary." I've been hearing "just wait" since 2023.
We're not trending towards superintelligence with these AIs. We're trending towards (and, in fact, have already reached) superintelligence with computers in general, but LLM agents are among the least capable known algorithms for the majority of tasks we get them to do. The problem, as it usually is, is that most people don't have access to the fruits of obscure research projects.
Untrained children write better code than the most sophisticated LLMs, without even noticing they're doing anything special.
I don't care how many terms you add to your Taylor series: your polynomial approximation of a sine wave is never going to be suitable for additive speech synthesis. Likewise, I don't care how good your predictive-text transformer model gets at instrumental NLP subtasks: it will never be a good programmer (except as far as it's a plagiarist). Just look at the Claude Code source code: if anyone's an expert in agentic AI development, it's the Claude people, and yet the codebase is utterly unmaintainable dogshit that shouldn't work and, on further inspection, doesn't work.
That's not to say that no computer program can write computer programs, but this computer program is well into the realm of diminishing returns.
The recent article where the AI companies are paying experts in the field to help train the models makes me wonder if they're also manually fixing a bunch of post-processing errors as they come up.
I’ll take that bet. How much money would you like to put on this, and we’ll have a neutral third party pick both the untrained child and the LLM.
Let me know.
It is (rightly) difficult to get hold of one uninvolved child, for safeguarding reasons, so it would be better to run it as a school (or interschool) competition, where multiple children may participate. For fairness, you may also provide multiple LLM participants (however you define that). The winner of the contest, as determined by the judge, would then determine the winner of the bet – unless the winning child had been trained, in which case we would fall back to the next-highest-ranked participant. The number of LLM candidates would be equal to the number of eligible children.
However, I don't see a good way to allow each child to pick a programming language and task, without leaving the competition results incomparable. So perhaps each child should be paired with an LLM, and the judge should determine which submission from each pair is better? But then if I only need one victory (to support my claim), this is clearly unfair. So each pair should be tested enough to determine whether they're consistently better than the LLM… but then we are demanding a lot of the child participants, for no real benefit to them.
If we can agree on a workable protocol, I can try to pull some strings and see if we can make this happen. I could use the money.
The article addresses this, because, well... no we aren't. Maybe we are. But it's far from clear that we're not moving toward a plateau in what these agents can do.
> Whether a human does actual work or not isn't particularly exciting to a market.
You seem to be convinced these AI agents will continue to improve without bound, so I think this is where the disconnect lies. Some of us (including the article author) are more skeptical. The market values work actually getting done. If the AIs have limits, and the humans driving them no longer have the capability to surpass those limits on their own, then people who have learned the hard way, without relying so much on an AI, will have an advantage in the market.
I already find myself getting lazy as a software developer, having an LLM verify my work, rather than going through the process of really thinking it through myself. I can feel that part of my skills atrophying. Now consider someone who has never developed those skills in the first place, because the LLM has done it for them. What happens when the LLM does a bad job of it? They'll have no idea. I still do, at least.
Maybe someday the AIs will be so capable that it won't matter. They'll be smarter and more through and be able to do more, and do it correctly, than even the most experienced person in the field. But I don't think that's even close to a certainty.
It is a debatable topic, and I agree with you that it's unclear whether we will hit the wall or not at some point. But one point I want to mention is that at the time when the AI agents were only conceived and the most popular type of """AI""" was LLM-based chatbot, it also seemed that we're approaching some kind of plateau in their performance. Then "agents" appeared, and this plateau, the wall we're likely to hit at some point, the boundary was pushed further. I don't know (who knows at all?) how far away we can push the boundaries, but who knows what comes next? Who knows, for example, when a completely new architecture different from Transformers will come out and be adopted everywhere, which will allow for something new? Future is uncertain. We may hit the wall this year, or we may not hit it in the next 10-20 years. It is, indeed, unclear.
P.S. I am well aware of all of the risks that agents brought. I'm speaking in terms of pure "maximum performance", so to speak.
do you have any evidence for that, though? Besides marketing claims, I mean.
If I would not type but speak this comment maybe 2 to 5 words would be wrong. For a human it is maybe 10% of that.
I wouldn't count on that because even if it happens, we don't know when it ill happen, and it's one of those things where how close it looks to be is no indication of how close it actually is. We could just as easily spend the next 100 years being 10 years away from agi. Just look at fusion power, self driving cars, etc.
Whatever models suck at, we can pour money into making them do better. It's very cut and dry. The squirrely bit is how that contributes to "general intelligence" and whether the models are progressing towards overall autonomy due to our changes. That mostly matters for the AGI mouthbreathers though, people doing actual work just care that the models have improved.
If Bob is going to spend $500 in tokens for something I can do for $50.
I think Bob is not going to stay long in lawn mowing market driving a bulldozer.
I would take that bet on the side of the wet meat. In the future, every AI will be an ad executive. At least the meat programming won't be preloaded to sell ads every N tokens.
I have literally never run into this in my career..challenges have always been something to help me grow.
As fewer know what good food tastes like, the entire market will enshitify towards lower and lower calibre food.
We already see this with, for example, fruits in cold climates. I've known people who have only ever bought them from the supermarket, then tried them at a farmers when they're in season for 2 weeks. The look of astonishment on their faces, at the flavour, is quite telling. They simply had no idea how dry, flavourless supermarket fruit is.
Nothing beats an apple picked just before you eat it.
(For reference, produce shipped to supermarkets is often picked, even locally, before being entirely ripe. It last longer, and handles shipping better, than a perfectly ripe fruit.)
The same will be true of LLMs. They're already out of "new things" to train on. I question that they'll ever learn new languages, who will they observe to train on? What does it matter if the code is unreadable by humans regardless?
And this is the real danger. Eventually, we'll have entire coding languages that are just weird, incomprehensible, tailored to LLMs, maybe even a language written by an LLM.
What then? Who will be able to decipher such gibberish?
Literally all true advancement will stop, for LLMs never invent, they only mimic.
If humans can prove that bespoke human code brings value, it'll stick around. I expect that the cases where this will be true will just gradually erode over time.
Value to who, and which type of value?
Will that defining value be purely economic in nature? While it be purely defined by mega-corps, and their perception of value? The market moves, the money flows to those which control its direction.
We already see it today, with some firms literally forcing people to use LLM coding tools. The stories abound, of simply being forced to use whatever it spits out. Value, is often designed by cost, and code maintainability 5 years later isn't an immediate, quarter-profit induced concern.
It feels like you're glossing over a lot of my point.
In terms of new coding languages, it's rare to see new coding languages gain any traction, rust is the only recent one I know, and it's had a larger backer behind it since its release 15 years ago. It was supported in house for almost a decade before even seeing the light of day.
Will that happen now? And created (or at least managed) by a human? If not, what would a new language look like? Would it be human maintainable? Understood?
To me, this goes right back to the classic "buying a car with the hood welded shut". No way to maintain, repair, or even evaluate the quality of the thing under you.
This happened a long time ago in the US. Drive through California's Central Valley sometime and sample the fruit sold fresh along the side of the road. It's a completely different experience than the version you get at Safeway.
Of course, that assumes a Bob with drive and agency. He could just as easily tell the AI to fix it without trying to stay in the loop.
Human nature says that Bob will skim over and trust the parts that he doesn't understand as long as he gets output that looks like he expects it to look, and that's extremely dangerous.
Once continuous learning is solved, I predict the problem addressed by TFA to become orders of magnitude bigger: What's the motivation for anyone to teach a person if an LLM can learn it much faster, will work for you forever, and won't take any sick days or consider changing careers?
The only reason we somewhat made it work is due to the interdependence between labor and capital. Once that's broken, the wheels will start falling off.
Which is more work, and less fun, than doing it myself. No thanks.
There will still be programming specialists in the future — we still have assembly experts and COBOL experts, after all. We just won’t need very many of them and the vast majority of software engineers will use higher-level tools.
Countries with proper human rights and labor laws don't do that.
The problem that the author describes is real. I have run into it hundreds of times now. I will know how to do something, I tell AI to do it, the AI does not actually know how to do it at a fundamental level and will create fake tests to prove that it is done, and you check the work and it is wrong.
You can describe to the AI to do X at a very high-level but if you don't know how to check the outcome then the AI isn't going to be useful.
The story about the cook is 100% right. McDonald's doesn't have "chefs", they have factory workers who assemble food. The argument with AI is that working in McDonald's means you are able to cook food as well as the best chef.
The issue with hiring is that companies won't be able to distinguish between AI-driven humans and people with knowledge until it is too late.
If you have knowledge and are using AI tools correctly (i.e. not trying to zero-shot work) then it is a huge multiplier. That the industry is moving towards agent-driven workflows indicates that the AI business is about selling fake expertise to the incompetent.
That's the true AI revolution: not the things it can accelerate, the things it can put in reach that you wouldn't countenance doing before.
It’s actually worse than that: the AI will not stop and say ”too complex, try in a month with the next SOTA model”. Rather, it will give Bob a plausible looking solution that Bob cannot identify as right or wrong. If Bob is working on an instant feedback problem, it’s ok: he can flag it, try again, ask for help. But if the error can’t be detected immediately, it can come back with a vengeance in a year. Perhaps Bob has already gotten promoted by then, and Bobs replacement gets to deal with it. In either case, Bob cannot be trusted any more than the LLM itself.
When he said we need more time to do this properly, he was labelled slow. They pushed him to use AI all day long and told at the all hands that there will be programmers who use AI and those who don't will be left behind. So he said fuck doing it right for the project, let me do it right for myself.
Now he got his promotion, they will hire 3 people in a cheaper location to handle various issues that are coming up (product will always have bugs you see). Given his excellent speed of delivery, they will report to him.
Good for Bob.
It isn’t. Bob has a different problem: that there are millions of Bobs with access to the same tools, meaning the value of Bobs labor is commodity priced. That may be good for some Bobs and bad for others.
Or even sooner, when Bob’s internet connection is down, or he ran out of tokens, or has been banned from his favourite service, or the service is down, or he needs to solve a problem with a machine unable to run local models, or essentially any situation where he’s unable to use an LLM.
I think the key issue is whether Bob develops the ability to choose valuable things to do with agents and to judge whether the output is actually right.
That’s the open question to me: how people develop the judgment needed to direct and evaluate that output.
Namely, if you can't do it without the AI, you can't tell when it's given you plausible sounding bullshit.
So Bob just wasted everyone's time and money.
I know we're not supposed to say RTFA, but your comment really takes the cake.
But why wouldn't my comment apply to all science? You have to make experiments and gather evidence, otherwise it's not science, it's just academia.
I do think coding with local agents will keep improving to a good level but if deep thinking cloud tokens become too expensive you'll reach the limits of what your local, limited agent can do much more quickly (i.e. be even less able to do more complex work as other replies mention).
Even if inference was subsidized (afaik it isn't when paying through API calls, subscription plans indeed might have losses for heavy users, but that's how any subscription model typically work, it can still be profitable overall).
Models are still improving/getting cheaper, so that seems unlikely.
I could imagine that when the music stops, advancement of new frontier models slows or stops, but that doesn't remove any curent capabilities.
(And to be fair the way we duplicate efforts on building new frontier models looks indeed wasteful. Tho maybe we reach a point later where progress is no longer started from scratch)
There is no evidence for this. The claims that API is "profitable on inference" are all hearsay. Despite the fact that any AI executive could immediately dismiss the misconception by merely making a public statement beholden to SEC regulation, they don't.
> Models are still improving/getting cheaper
The diminishing returns have set in for quality, and for a while now that increased quality has come at the cost of massive increases in token burn, it's not getting cheaper.
Worse yet, we're in an energy crisis. Iran has threatened to strike critical oil infrastructure, and repairs would take years.
AI is going to get significantly more expensive, soon.
Can he? If he outsources all his thinking and understanding to agents, can he then fix things he doesn't know how to fix without agents?
Any skill is practice first and foremost. If Bob has had no practice, what then?
Indeed. That's why Anthropic had to hire real engineers to make sure their vibe-coded shit doesn't consume 68GB of RAM. Because real world: https://x.com/jarredsumner/status/2026497606575398987
I’m not trying to argue that AI can do everything today. I acknowledge that there are many things that it is not good at.
If yes, then that's dangerously optimistic. If not, then we'll always need humans who have learned the "hard way" (the Alices, not the Bobs). But if LLMs make it impossible for Alices to come up in the field, we're screwed.
I personally haven't tried Claude Code because I can't install it on my PC. I'm starting to get the impression that they banned non Claude products from using their subscription, because their products are of such a poor quality that everyone is fleeing from them.
The question is: will agents improve to the point that even the most capable Alices will never be needed to solve problems? Maybe? Maybe not? I'm worried that they won't improve to that degree.
And even if they do, what is the purpose of humans in this world?
More importantly, what's gonna be the next stable category of remote-first jobs that a person with a tech-adjacent or tech-minded skillset can tack onto? That's all I care about, to be honest.
I may hate tech with a passion at times and be overly bullish on its future, but there's no replacing my past jobs which have graced me and many others with quality time around family, friends, nature and sports while off work.
Personally I’m looking at more physical domains, but it’s early days in my exploration. I think if I wanted to stick to remote work (which I have enjoyed since 2020), then the AI story would just keep playing out.
I’m also totally open to taking a big pay cut to do something I actually enjoy day to day, which I guess makes it easier.
(I'm also looking for local, personally satisfying work, in exchange for a pay cut. Early days, and I am finding the profession no longer commands quite the social cachet it once did, but I'm not foolish enough to fail to price for the buyer's market in which we now seek to sell our labor. Besides, everyone benefits from the occasional reminder to humility! "Memento mori" and all that.)
The more recent shift after December is mostly explained by people at my company catching up with the events that happened in December. And that’s more about drastically increased productivity expectations, layoffs, etc.
I’m also considering a self funded sabbatical. I could do it. What sort of thing have you been up to, any advice?
Imagination operates more freely and foolishness is less heavily ballasted, and any kind of emotional crap you've been keeping shoved to the side with the force of pressing obligations is likely to come out and start rearranging the metaphorical furniture. If you've got stuff like that, this will be a good opportunity to get to grips with it, whether you mean to or not. Prepare accordingly.
And finally, there's not too many more appealing social presentations in my experience than that deriving from the confident knowledge that, within reason at least, one has earned and is now deploying the privilege to do more or less whatever the hell one likes: not the confidence contingent on a fat wallet, but that inherent in having only those scheduled obligations one chooses, and also in understanding precisely the difference underlying that distinction. Very few people in this world have the skill to behave as if their time were entirely their own to command, and this makes a difference in deportment that others will notice and attend without necessarily knowing why. It is more subtle and far less brash than the confidence in wielding the name of an employer that everyone knows, but for like reasons it also has worth and durability which the other does not. Whether or not you keep it, the experience of having had it is about as unforgettable and as indescribable as the trick to riding a bike.
Thanks for the info! My last direct exposure to a frontier model was now almost twelve months ago, so I suppose I'll have to dedicate a few hours pretty soon.
Company started doling out Claude Code configs, everything is now cli/agentic AI harnessed and news about "90% of this company's code is now AI Generated" pop up every other day.
It seems the last frontier to breach before this was nailing agentic black boxes to not crap out during the first hour of work. After that, it's really been much smoother for those tools.
On the side, this might not have to do at all with your case, but the reason I personally keep putting off sabbaticals is that I feel it can severely compound my routine wrecking habits and I don't think I'd be too strong-willed to give it meaningful purpose. Not to mention the first point, i.e. it would 100% make my industry pessimism worse. I'd like to not bounce away from tech forever. Rather, figure what scratches the same itch I've been seeking since the start.
I'm all about big road trips, big adventures but I think the couch potato risk is all too real for me.
(1) Peace, Dan! I imply no substantial or material connection, only nascence within the same culture and enshrinement of the same desiderata, as you well know - and well know can't be gainsaid, or not in factual terms at least.
Much better take is to start establishing yourself as a slop wrangler. Lot of stupid money to be made from fools wanting to purify their slop.
Are you seriously arguing that the only thing that determines right from wrong is someone buying the thing? I mean that would explain most of the sickness that is neoliberalism currently infecting the US.
Aren't they currently propped up by investor money?
What happens when the investors realize the scam that it is and stop investing or start investing less...
Are Chinese model shops propped up by investor money? Is Google?
Open weights models are only 6 months behind SOTA. If new model development suddenly stopped, and today's SOTA models suddenly disappeared, we would still have access to capable agents.
But they would be outdated, right?
Would an agent that can only code in COBOL would be as useful today?
Outdated compared to what? In your counterfactual, VC funded agents don't exist anymore, no?
Your argument, if I understand it correctly, is that they might somehow go away entirely when VC funding dries up, when more realistically they'll probably at most become twice as expensive or regress half a year in performance.
Outdated compared to reality / humans, their knowledge cutoff is a year further behind every year they don't get updates. Humans continuously expands their knowledge, the models needs to keep up with that.
Nvidia chips get a lot cheaper
I’ve been reminded lately of a conversation I had with a guy at hacker space cafe around ten years ago in Berlin.
He had been working as a programmer for a significantly longer time than me. Long enough that for many years of his career, he had been programming in assembly.
He was lamenting that these days, software was written in higher level languages, and that more and more programmers no longer had the same level of knowledge about the lower level workings of computers. He had a valid point and I enjoyed talking to him.
I think about this now when I think about agentic coding. Perhaps over time most software development will be done without the knowledge of the higher level programming languages that we know today. There will still be people around that work in the higher level programming languages in the future, and are intimately familiar with the higher level languages just like today there are still people who work in assembly even if the percentage of people has gotten lower over time relative to those that don’t.
And just like there are areas where assembly is still required knowledge, I think there will be areas where knowledge of the programming languages we use today will remain necessary and vibe coding alone wont cut it. But the percentage of people working in high level languages will go down, relative to the number of people vibe coding and never even looking at the code that the LLM is writing.
I wonder how many assembly programmers got over it and retrained, versus moved on to do something totally different.
I find the agentic way of working simultaneously more exhausting and less stimulating. I don’t know if that’s something I’m going to get over, or whether this is the end of the line for me.
My mother actually started programming in octal. I don't remember her exact words, but she said something to the effect that her life got so much better when she got an assembler. I suspect that going from assembly to compilers was much the same - you no longer had to worry about register allocations and building stack frames.
LLMs are nothing like that. They are probabilistic systems at their very core. Sometimes you get garbage. Sometimes you win. Change a single character and you may get a completely different response. You can't easily build abstractions when the underlying system has so much randomness because you need to verify the output. And you can't verify the output if you have no idea what you are doing or what the output should look like.
LLMs don't make it impossible to do anything yourself, but they make it economically impractical to do so. In other words, you'll have to largely provide both your own funding and your own motivation for your education, unless we can somehow restructure society quickly enough to substitute both.
With assembly, we arguably got lucky: It turns out that high-level programming languages still require all the rigorous thinking necessary to structure a programmer's mind in ways that transfer to many adjacent tasks.
It's of course possible that the same is true for using LLMs, but at least personally, something feels substantially different about them. They exercise my "people management" muscle much more than my "puzzle solving" one, and wherever we're going, we'll probably still need some puzzle solvers too.
Please, not this pre-canned BS again!
Comparing abstractions to AI is an apples to oranges comparison. Abstractions are dependable due to being deterministic. When I write a function in C to return the factorial of a number, and then reuse it again and again from Java, I don't need a damn set of test cases in Java to verify that factorial of 5 is 120.
With LLMs, you do. They aren't an abstraction, and seeing this worn out, tired and routinely debunked comparison being presented in every bloody thread is wearing a little thin at this point.
We've seen this argument hundreds of times on this very site. Repeating it doesn't make it true.
The same is not true for LLM output. I can’t tell my manager I don’t know how to fix something in production the agent wrote. The equivalent analogy would be if we had to know both the high-level language _and_ assembly.
Moving up to an MMU and running Linux was a different (more abstract) world. Although since it was embedded, low-level functions might still be in both assembly and C if not the apps on top.
If not, you're changing learning to cook for Uber only meals.
And since the alternative is starving, Uber will boil the pot.
Don't give up your self sufficiency.
I agree totally with the sentiment, and I am concerned about my own skills atrophying.
There is absolutely nothing self-sufficient about computer hardware
"Self-sufficiency" arguments coming from tech nerds are so tiring.
We're already vulnerable to enshittification in so many areas, why increase the list? How does that work in my favor at all?
Assuming that by "at home" you mean using ordinary hardware, not something that costs as much as a car. Yes, very slowly, for simple tests. (Not proprietary models obviously, but quite capable ones nonetheless.) Not exactly viable for agentic coding that needs boatloads of tokens for the simplest things. But then you can run smaller local models that are still quite capable for many things.
It's not inherent, but it is reality unless folks stop giving up agency for convenience. I'm not holding my breath.
Are we net better off than if we didn't have cars and simply built public transport with walkable cities?
There is a vast range of scenarios in which being more or less independent from agents to perform cognitive tasks will be both desirable and necessary, at the individual, societal and economic level.
The question of how much territory we should give up to AI really is both philosophical and political. It isn’t going to be settled in mere one-sided arguments.
They’re not going to pay me to manually program because I find it more enjoyable, when they can get Bob to do twice as much for less.
This is why I say I don’t like it, but it is what it is.
But let's assume Bob continues to have an active role, because the people above him bought in to the hype and are convinced that "prompt engineer" is the job of the future. When things inevitably start falling apart because the Bobs of the world hit a wall and can't solve the problems that need to be solved (spoiler: this is already happening), what do we do? We need Alices to come in and fix it, but the market actively discourages the existence of Alice, so what happens when there are no more Alices left? Do we just give up and collectively forget how to do things beyond a basic level?
I have a feeling that, yes, we as a species are just going to forget how to do things beyond a certain level. We are going to forget how to write an innovative science paper. We are going to forget how to create websites that aren't giant, buggy piles of React spaghetti that make your browser tab eat 2GB of RAM. We've always been forgetting, really - there are many things that humans in the past knew how to do, but nobody knows how to do today, because that's what happens when the incentive goes missing for too long. Price and convenience often win over quality, to the point that quality stops being an option. This is a form of evolutionary regression, though, and negatively affects our quality of life in many ways. AI is massively accelerating this regression, and if we don't find some way to stop it, I believe our current way of life will be entirely unrecognizable in a few decades.
My point is that both Alice and Bob have a place in this world. In fact, Bob isn’t really doing much different from what a Pricipal Investigator is already doing today in a research context.
Those aren't mutually exclusive.
"People who do things" can do both, and doing the latter is a function of doing the former, so they tend to do the latter sufficiently well.
"People who prompt things" can only do the latter, and they routinely do it poorly.
Right, but what I don’t agree with here is the idea that this category of people will never be able to improve into the first category of people. The value of an experienced anything is that they realize there is a big chasm between something that works now and something that will continue to work long into the future.
I don’t agree that doing everything yourself manually is the only thing that can grant you that understanding, because I don’t think that understanding is domain-specific. It evolves naturally as soon as someone realizes that their list of unknown unknowns is FAR larger than their list of known anythings, and that the first step in attempting to solve a problem is to prune that list as far as you can get it while realizing you will never ever be able to reduce it to zero.
You can do that by spending two weeks to build a brick wall by hand, or you can do that by spending two weeks having your magical helpers build ten brick walls that eventually collapse. I don’t think the tools are some sort of fundamental threat to cognition, I think they’re - within this society - a fundamental threat to safety, because the relentless pursuit of profit means even those that realize those ten brick walls should never actually ever be used to hold anything up will find themselves pressured to put a roof on them and hope, pray, they hold.
And this isn’t an LLM-specific thing. The vast diverse space of building codes around the world proves this, and coincidentally, the countries with laxer building codes tend to get a lot more done a lot faster; and they also tend to deal with a big tragic collapse every now and then, which I suppose someone will file away as collateral somewhere.
This isn't true, a car mechanic never evolves into an engineer, a nurse never evolve into a doctor. A car mechanic can learn to do some tasks you normally need an engineer for and same with nurses, but they never build the entire core set of skills that separates engineers from mechanics and doctors from nurses.
There are maybe some exceptions to this, but those exceptions are so rare that it doesn't matter for this discussion. A few people still learning it properly wont save anything.
“Doesn’t generally happen” =/= “is literally impossible”. The word “never” should be used with care.
> A car mechanic can learn to do some tasks you normally need an engineer for and same with nurses
This statement can only make sense if you regard titles as something that’s imbued upon you, and until it is, you are incapable of performing the acts that someone who has earned that tile can perform. I’ll just say I fundamentally disagree with this notion on pretty much every conceivable level, and if that’s the belief system you subscribe to, that would also makes arguing about this any further pointless. But I might just be getting you wrong.
> There are maybe some exceptions to this, but those exceptions are so rare that it doesn't matter for this discussion. A few people still learning it properly wont save anything.
It is fine it was stupid of me to not go and fix the "never" earlier but instead corrected it at the end of my post. You are right it would be very dumb if you thought people never ever could make such a transition, but I wrote that as the weaker version of never. There is a reason there is a "never ever", since often in common language "never" doesn't mean never.
It did cause me to misread its point. The correction at the end read more like “sometimes a nurse can do some doctor stuff but that doesn’t mean anything.” You don’t seem to actually think that based on your correction, so fair enough.
The fundamental difference between the categories is that the first is filled with people who put the effort in to learning/understanding, and the second is filled with people who take the shortcut around learning/understanding.
Changing from the second category to the first is something that would require already being in the first.
Exactly! That’s my entire point. Because now you’re separating the categories by “is willing to put in effort” and “is not willing to put in effort” rather than by “has done the thing” and “hasn’t done the thing”.
I think the disagreement doesn’t lie in this concept, but rather in whether an LLM can be used by someone who’s willing to put in effort to assist them in doing so, rather than just having it do it for them. But as long as you understand what the thing you’re using it is for, you don’t have to understand how it works exactly. You can shift gears in a car without a physics degree.
No, you misunderstood here. People aren't saying "it is harder to learn in the future", the issue is "it will be harder to make sure that someone will learn in the future".
Currently you need an engineering degree and experience to do engineering work. However if in the future a lot of people get their degree and experience just by calling LLM for every problem, those engineers will not understand at all what they are doing. Before someone having that experience will have had solved a lot of problems manually on the job, that experience made them an expert. The same person solving those by calling an LLM and pasting in the answer will just as ignorant as someone with no experience.
Most such people today didn't wanna learn to be engineers out of curiosity, they just wanted a job. In the future all such people would use LLM and never learn. Those are the main parts of our workforce, so it is a scary prospect that in the future we cannot force them to learn things properly in the same way since LLM allows them to do basic tasks without learning.
If you argue there are plenty of people who learn for fun, then you would be wrong. Extremely few people learn enough in their own time to contribute meaningfully to for example mathematics, it isn't enough to matter. People learn those fundamentals primarily because they are forced to do it for a degree they need for a job, if they weren't forced to learn and pass tests they would happily go do the job without any knowledge or skills.
Code agents are great template generators and modifiers but for net new (innovative! work it‘s often barely usable without a ton of handholding or „non code generation coding“
Didn't PhD projects used to be about advancing the state of art?
Maybe we'll get back to that.
"Being able to deliver using AI" wasn't the point of the article. If it was the point, your comment would make sense.
The point of the program referred to in the article is not to deliver results, but to deliver an Alice. Delivering a Bob is a failure of the program.
Whether you think that a Bob+AI delivers the same results is not relevant to the point of the article, because the goal is not to deliver the results, it's to deliver an Alice.
That's irrelevant to the goal of the program - they care. Once they stop caring, they'd shut that program down.
Maybe it would be replaced with a new program that has the goal of delivering Bobs+AI, but what would be the point? I mean, the article explained in depth that there is no market for the results currently, so what would be the point of efficiently generating those results?
The market currently does not want the results, so replacing the current program with something that produces Bobs+AI would be for... what, exactly?
AI in software engineering is kept afloat by the bullshitters who jump on any new bandwagon because they are incompetent and need to distract from that. Managers like bullshit, so these people thrive for a couple of years until the next wave of bullshit is fashionable.
You're still working on intellectually stimulating programming problems. AI doesn't go all the way with any reliability, it just provides some assistance. You're still ultimately responsible for getting things right, even with key AI help.
I dread the flip side of this which is dealing with obtuse bullshit like trying to understand why Oracle ADF won’t render forms properly, or how to optimize some codebase with a lot of N+1 calls when there’s looming deadlines and the original devs never made it scalable, or needing to dig into undercommented legacy codebases or needing to work on 3-5 projects in parallel.
Agents iterating until those start working (at least cases that are testable) and taking some of the misery and dread away makes it so that I want to theatrically defenestrate myself less.
Not everyone has the circumstance to enjoy pleasant and mentally stimulating work that’s not a frustrating slog all the time - the projects that I actually like working on are the ones I pick for weekends, I can’t guarantee the same for the 9-5.
The problem with unlearning generic tools and relying on ones you rent by big corporations is that it is unreliable in the long term. The prices will be rising. The conditions will worsen. Oh nice that Bob made a thing using HammerAsAService™, but the terms of conditions (changing once a week) he accepted last week clearly say it belongs to the company now. Bob should be happy they are not suing him yet, but Bob isn't sure whether the thing that came out a month after was independently developed by that company or not just a clone of his work. Bob wishes he knew how to use a hammer.
Will the paid tools always tell their users how to use the free versions, and if not, how will the users learn to do it independently?
The same way any open-source infrastructure finds widespread use, I’d say. If you’re willing to put in the elbow grease, you can probably set it up yourself (maybe even with the help of one of the frontier, uh, hammers, in its free tier). Or there might be services that act as middlemen to make it all more convenient and cheaper. But the difference is that if Service X pisses you off, then there will be Services Y, Z, A, and B who sell the same service using the same open-source infrastructure, so you always have a choice.
If you don’t like GitHub, try Gitlab, Codeberg, Gitea, and so forth. Or Bitbucket or Azure DevOps. (Don’t actually, though.)
Let’s wait until they a business model that creates profit.
Most of them won’t go away, but many will become outdated or slow or enshittificated.
Imagine building your career based on the quality of google‘s search
Following the model of how startups have worked for the last 20 years or so, I expect agents to eventually be locked-down/nerfed/ad-infested for higher payments. We are enjoying the fruits of VC money at the moment and they are getting everyone addicted to agents. Eventually they need to turn a profit.
Not sure how this plays out, but I would hang on to any competencies you have for anyone (or business) that wants to stick around in software. Use agents strategically, but don't give up your ability to code/reason/document, etc. The only way I can see this working differently is that there are huge advances in efficiency and open-source models.
Alternatively you could find some other people to share the HW cost and run some larger models (like Kimi-K2.5 at 1.1T params).
That's definitely not an option for me :-D
True open LLMs could be a viable solution in the future, but only if they can be operated and sustained on a community basis. I have too little insights into the actual costs of running such models to judge whether this would be feasible. Then there is always the problem of how to deal with bad actors. This is all but trivial.
At the moment, I'd rather spend time working on sharpening my actual programming and thinking skills :) I actually enjoy the act of programming and see it as part of my creative expression. Fortunately, I don't code for a living (at least not directly), so nobody can tell me how to write my software.
He'll get things (papers, code, etc) which he can't evaluate. And the next round of agents will be trained on the slop produced by the previous ones. Both successive Bob's and successive agents will have less understanding.
This point is directly addressed in the paper: Bob will ultimately not be able to do the things Alice can, with or without agents, because he didn't build the necessary internal deep structure and understanding of the problem space.
And if Alice later on ends up being a better scientist (using agents!) than Bob will ever be, would you not say there was something lost to the world?
Learning needs a hill to climb, and somebody to actually climb it. Bob only learned how to press an elevator button.
I am in the same boat, but close enough to retirement that I'm less "scared" about it. For me I'm moving up the chain; not people management, but devoting a lot more of my time up the abstraction continuum. Looking a lot more at overall designs and code quality and managing specs and inputs and requirements.
I wrote some design docs past few days for a big project the team is embarking on. We never had that before, at least not in the level of detail (per time quantum) that I was able to produce. Used 2 models from 2 companies - one to write, one to review, and bounce between them until the 3 of us agree.
Honestly it didn't take any less time than I would have done it alone, but the level of detail was better, and covered more edge cases. Calling it a "win" right now. I still enjoy it, as most of the code I/we was/are writing is mostly fancy CRUD anyway, and doesn't have huge scaling problems to solve (and too few devs I feel are being honest about their work, here).
Some people treat toilet as magic hole where they throw stuff in flush and think it is fine.
If you throw garbage in you will at some point have problems.
We are in stage where people think it is fine to drop everything into LLM but then they will see the bill for usage and might be surprised that they burned money and the result was not exactly what they expected.
The economics and security model on full agents running in loops all day may come home to roost faster than expertise rot.
To the reader and the casual passerby, I ask: Do you have to work at this pace, in this manner? I understand completely that mandates and pressure from above may instill a primal fear to comply, but would you be willing to summon enough courage to talk to maybe one other person you think would be sympathetic to these feelings? If you have ever cared about quality outcomes, if for no other reason than the sake of personal fulfillment, would it not be worth it to firmly but politely refuse purely metrics-focused mandates?
It’s not for me. Being a middle manager, with all of the liability and none of the agency, is not what I want to do for a living. Telling a robot to generate mediocre web apps and SVGs of penguins on bicycles is a lousy job.
But he does things wrong.
Yes, but how does he know if it worked? If you have instant feedback, you can use LLMs and correct when things blow up. In fact, you can often try all options and see which works, which makes it ”easy” in terms of knowledge work. If you have delayed feedback, costly iterations, or multiple variables changing underneath you at all times, understanding is the only way.
That’s why building features and fixing bugs is easy, and system level technical decision making is hard. One has instant feedback, the other can take years. You could make the ”soon” argument, but even with better models, they’re still subject to training data, which is minimal for year+ delayed feedback and multivariate problems.
Also, the premise that it took each of them a year to do the project means Bob was slacking because he probably could've done it in less than a month.
Why not? Once the true cost of token generation is passed on to the end user and costs go up by 10 or 100 times, and once the honeymoon delusion of "oh wow I can just prompt the AI to write code" fades, there's a big question as to if what's left is worth it. If it isn't, agents will most certainly go away and all of this will be consigned to the "failed hype" bin along with cryptocurrency and "metaverse".
Now, you don't do thing and do other things when LLMs get stuck. There is no "given enough time I can do it".
I can't see how somebody would go solving slop bugs (slugs :)) in heavy AI generated codebase.
Hope, I'm wrong but that's somehing I personally encountered. Stay sharp.
But Bob can't do things with agents.
He can get a project from someone else and ask the agents to do that project. Then give the output of the agents back to that someone else, and that someone else reviews it, says why it's wrong, and sends it back. Bob feeds the review to the agents, gets something back, then gives the output back to that someone else who reviews it, etc. So,
1) The loop requires his advisor to know how to go about doing the thing.
2) Bob is absolutely unnecessary and should be discarded.
3) Alice will eventually be qualified to be an advisor.
edit: And the crisis that the article is really pointing out is that when the advisor is using the LLM (while Bob is driving an Uber), and his productivity goes way up because he's only handling the things only he can handle, what about Alice?
Let's say that pre-AI the advisor could either do the job in 2 months or assign it to Alice who could do it in 12 with a week of the advisor's supervision/review. Now, with the LLM, the advisor can do the job in 2 weeks without Alice. Before, Alice made barely any money and had no health insurance. After, Alice is also driving Uber.
Now the advisor has a heart attack and now the thing just can't be done. Also, Ubers become pretty much self-driving, so Bob and Alice are not only ignorant, but unemployed. They can't even afford to take an Uber.
How this plays out:
I use Claude to write some moderately complex code and raise a PR. Someone asks me to change something. I look at the review and think, yeah, that makes sense, I missed that and Claude missed that. The code works, but it's not quite right. I'll make some changes.
Except I can't.
For me, it turns out having decisions made for you and fed to you is not the same as making the decisions and moving the code from your brain to your hands yourself. Certainly every decision made was fine: I reviewed Claude's output, got it to ask questions, answered them, and it got everything right. I reviewed its code before I raised the PR. Everything looked fine within the bounds of my knowledge, and this review was simply something I didn't know about.
But I didn't make any of those decisions. And when I have to come back to the code to make updates - perhaps tomorrow - I have nothing to grab onto in my mind. Nothing is in my own mental cache. I know what decisions were made, but I merely checked them, I didn't decide them. I know where the code was written, but I merely verified it, I didn't write it.
And so I suffer an immediate and extreme slow-down, basically re-doing all of Claude's work in my mind to reach a point where I can make manual changes correctly.
But wait, I could just use Claude for this! But for now I don't, because I've seen this before. Just a few moments ago. Using Claude has just made it significantly slower when I need to use my own knowledge and skills.
I'm still figuring out whether this problem is transient (because this is a brand new system that I don't have years of experience with), or whether it will actually be a hard blocker to me using Claude long-term. Assuming I want to be at my new workplace for many years and be successful, it will cost me a lot in time and knowledge to NOT build the castle in the sky myself.
Then I spend time to read each file change and give feedback on things I'd do differently. Vastly saves me time and it's very close or even better than what I would have written.
If the result is something you can't explain than slow down and follow the steps it takes as they are taken.
> If the result is something you can't explain than slow down and follow the steps it takes as they are taken.
The problem is I can explain it. But it's rote and not malleable. I didn't do the work to prove it to myself. Its primary form is on the page, not in my head, as it were.
I still wrote bugs. I'd bet that my bugs/LoC has remained static if not decreased with AI usage.
What I do see is more bugs, because the LoC denominator has increased.
What I align myself towards is that becoming senior was never about knowing the entire standard library, it was about knowing when to use the standard library. I spent a decade building Taste by butting my head into walls. This new AI thing just requires more Taste. When to point Claude towards a bug report and tell it to auto-merge a PR and when to walk through code-gen function by function.
The AI can help with that too. Ask it "How would one think about this issue, to prove that what was done here is correct?" and it will come up with somethimg to help you ground that understanding intuitively.
Starting with week three the overall structure of the code base is done, but the actual implementation is lacking. Whenever I run out of tokens I just started programming by hand again. As you keep doing this, the code base becomes ever more familiar to you until you're at a point where you tear down the AI scaffolding in the places where it is lacking and keep it where it makes no difference.
It has always been extremely annoying to fight with people who mistake the ability of building or engaging with complicated systems (like your regex) with competency.
I work in building AI for a very complex application, and I used to be in the top 0.1% of Python programmers (by one metric) at my previous FAANG job, and Claude has completely removed any barriers I have between thinking and achieving. I have achieved internal SOTA for my company, alone, in 1 week, doing something that previously would have taken me months of work. Did I have to check that the AI did everything correctly? Sure. But I did that after saving months of implementation time so it was very worth it.
We're now in the age of being ideas-bound instead of implementation-bound.
Regex is the worst possible example you could have given. Seriously, how many people do you know who painstakingly hand-craft their own regexes as opposed to using one of the million tools out there that can work backwards from example inputs and outputs to generate a regex that satisfies the conditions?
It also doesn't help that Claude ends every recommendation with "Would you like me to go ahead and do that for you?" Eventually people get tired and it's all to easy to just nod and say "yes".
You can do this during the previous change phase of course. Just ask "How would one plan this change to the codebase? Could you explain in depth why?" If you're expected to be thoroughly familiar with that code, it makes no sense to skip that step.
Same with anything. You can read about how to meditate, cook, sew, whatever. But if you only read about something, your mental model is hollow and purely conceptual, having never had to interact with actual reality. Your brain has to work through the problems.
GP says that they have to come back tomorrow and edit the code to fix something. That's a verification step: if you can do that (even with some effort) you understand why the AI did what it did. This is not some completely new domain where what you wrote would apply very clearly, it's just a codebase that GP is supposed to be familiar with already!
In a living codebase you spent long stretches to learn how it works. It's like reading a book that doesn't match your taste, but you eventually need to understand and edit it, so you push through. That process is extremely valuable, you will get familiar with the codebase, you map it out in your head, you imagine big red alerts on the problematic stuff. Over time you become more and more efficient editing and refactoring the code.
The short term state of AI is pretty much outlined by you. You get a high level bug or task, you rephrase it into proper technical instructions and let a coding agent fill in the code. Yell a few times. Fix problems by hand.
But you are already "detached" from the codebase, you have to learn it the hard way. Each time your agent is too stupid. You are less efficient, at least in this phase. But your overall understanding of the codebase will degrade over time. Once the serious data corruption hits the company, it will take weeks to figure it out.
I think this psychological detachment can potentially play out really bad for the whole industry. If we get stuck for too long in this weird phase, the whole tech talent pool might implode. (Is anyone working on plumbing LLMs?)
And indeed running it through a few AI text detectors, like Pangram (not perfect, by any means, but a useful approximation), returns high probabilities.
It would have felt more honest if the author had included a disclaimer that it was at least part written with AI, especially given its length and subject matter.
It’s a well written, engaging article. Why would it need a disclaimer if they used AI on part of it? They’re not arguing against the usage of LLMs.
Maybe he won't be able to blog himself out of a wet paper bag tomorrow, but people seem to think he's a great thinker today. Isn't that all that matters?
(I tried faking some AI style.)
LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be chasing some error for two weeks, the LLM will very likely figure it out in matter of minutes, or not make it in the first place. Instructing it to validate intermediate results and to profile along the way can do magic.
The article is correct that Bob will not have understood anything, but if he wants to, he can spend the rest of the year trying to understand what the LLM has built for him, after verifying that the approach actually works in the first couple of weeks already. Even better, he can ask the LLM to train him to do the same if he wishes. Learn why things work the way they do, why something doesn't converge, etc.
Assuming that Bob is willing to do all that, he will progress way faster than Alice. LLMs won't take anything away if you are still willing to take the time to understand what it's actually building and why things are done that way.
5 years from now, Alice will be using LLMs just like Bob, or without a job if she refuses to, because the place will be full of Bobs, with or without understanding.
But do you actually understand it? The article argues exactly against this point - that you cannot understand the problems in the same way when letting agents do the initial work as you would when doing it without agents.
from the article: "you cannot learn physics by watching someone else do it. You have to pick up the pencil. You have to attempt the problem. You have to get it wrong, sit with the wrongness, and figure out where your reasoning broke. Reading the solution manual and nodding along feels like understanding. It is not understanding. Every student who has tried to coast through a problem set by reading the solutions and then bombed the exam knows this in their bones. We have centuries of accumulated pedagogical wisdom telling us that the attempt, including the failed attempt, is where the learning lives. And yet, somehow, when it comes to AI agents, we've collectively decided that maybe this time it's different. That maybe nodding at Claude's output is a substitute for doing the calculation yourself. It isn't. We knew that before LLMs existed. We seem to have forgotten it the moment they became convenient."
Your perspective is cut off. In the real world Bob is supposed to produce outcomes that work. If he moves on into the industry and keeps producing hallucinated, skewed, manipulated nonsense, then he will fall flat instantly. If he manages to survive unnoticed, he will become CEO. The latter rather unlikely.
This won’t affect everyone equally. Some Bob’s will nerd out and spend their free time learning, but other Bob’s won’t.
And it’s not strictly speaking university we’re talking about. The way we understand work is going to fundamentally change. And we’re not going to value the people who use LLMs to get 1x done.
But yes, university was always about how much work you put into it, and LLM’s are going to make that 10x more obvious.
The point is the Bob and Alice comparison is already a straw man, but I do squarely believe it’s the people with the best mental models and not the people who “get AI” who will win the new world. If you’re curious and good at developing mental models, you can learn “AI” in a week. But if you’re curious and good at developing mental models you’ve probably already lapped both Bob and Alice
Isn't this learning swimming by watching others explaining swimming? Bob would think he knows swimming, until he has to get into the water.
I don’t think we have good answers unfortunately. Im very happy to be able to get the exact tools I want for my specific niche and be able to run experiments in no time. But I also see that intellectually I do not engage at the same level. I can justify my reasoning for high level design decisions I tried to get the agent to follow, but if there is an issue or I need to justify an implementation decision that’s way messier. I had that experience a few times over the past year and every time I have to reverse engineer what the agent might have been doing before I can answer, or realizing I completely misunderstood a specific protocol because I didn’t have to actively engage with it
> Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.
No they won't be. They might be worse. They might be better. But they'll be very different.
And, like you said...
> Alice and Bob had the same year. One paper each.
No they won't. Alice would've taken a year. Bob would've taken a few days.
You've already covered why that might actually be OK, so I'll talk about the author's other error:
> This sounds idealistic until you think about what astrophysics actually is. Nobody's life depends on the precise value of the Hubble constant. No policy changes if the age of the Universe turns out to be 13.77 billion years instead of 13.79. Unlike medicine, where a cure for Alzheimer's would be invaluable regardless of whether a human or an AI discovered it, astrophysics has no clinical output. The results, in a strict practical sense, don't matter. What matters is the process of getting them: the development and application of methods, the training of minds, the creation of people who know how to think about hard problems. If you hand that process to a machine, you haven't accelerated science. You've removed the only part of it that anyone actually needed.
Keep asking why. why does the development of the application and methods, the training of minds matter?
the goal isn't abstract. The goal is still ultimately for the benefit of humanity, just like the cure for Alzheimer's.
Humanity learned physics, so we made rockets and now we have satellites, and the entire planet is connected with communication and information.
Humanity must continue to invest in astrophysics so that we do not get wiped out by a single rogue asteroid barreling through the cosmos, like the dinosaurs did.
Now i'm not saying that there isn't other benefit to making generically intelligent humans that know how to think. But at the end of the day, the purpose of astrophysics is no less existential than the purpose for developing medicine.
I want to know the age of the universe so that we can understand what created it, and if we can reverse entropy, and if there is anything beyond the universe. That is a quest for humanity that will take hundreds if not millions of years.
Speaking from personal experience, 99% people/students don't, and that is the problem.
I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction.
Similarly, I now produce 99% of my code using an agent. However, I still feel the need to thoroughly understand the code, in order to be able to catch the 1% of cases where it introduces a bug or does something suboptimally.
It's possible that in future, LLMs will get _so_ good that I don't feel the need to do this, in the same way that I don't think about the transistors my code is ultimately running on. When doing straightforward coding tasks, I think they're already there, but I think they aren't quite at that point when it comes to large distributed systems.
The problem is, they're nothing like transistors, and never will be. Those are simple. Work or don't, consistently, in an obvious, or easily testable, way.
LLM are more akin to biological things. Complex. Not well understood. Unpredictable behavior. To be safely useful, they need something like a lion tamer, except every individual LLM is its own unique species.
I like working on computers because it minimizes the amount of biological-like things I have to work with.
Perhaps a better analogy would be the Linux kernel. It's built by biological humans, and fallible ones at that. And yet, I don't feel the need to learn the intricacies of kernel internals, because it's reliable enough that it's essentially never the kernel's fault when my code doesn't work.
Every abstraction is leaky, its not like I have 1 in every 100 tickets I work on needs understanding of the existence of filesystem buffers, it's in the back of my mind, it's always there. I didn't read linux kernel source, but I know it's existence. LLM output doesn't have that.
It's trivial to say using an inadequate tool will have an inadequate result.
It's only an interesting claim to make if you are saying that there is no obtainable quality of the tool that can produce an adequate result (In this argument, the adequate result in question is a developer with an understanding of what they produce)
I do wonder where all the novel products produced by 10x devs who are now 100x with LLMs, the “idea guys” who can now produce products from whole clothe without having to hire pesky engineers.. where is the one-man 10 billion dollar startups, etc? We are 3-4 years into this mania and all I see on the other end of it is the LLMs themselves.
Why hasn’t anything gotten better?
The 10x dev doesn’t just set out to build a hello world app, ya know.
1) Stuff that was astonishingly not automated yet. I am talking about somebody opening up excel on one screen, and a website/pdf/whatever on the other.. and type stuff in to your excel sheet. So stuff where there wasn't any code involved previously, possibly due to diminishing returns of how adhoc it was to automate, skills mismatch, organizational politics or other reasons.
2) Lot of former BigData / crypto / SaaS guys who were in product/sales roles suddenly starting AI startups to help your company AI better. The product is facilitating the doing of AI.
So we can glue that together a bit faster, great.
What if we also stop producing new open source, frameworks, libraries, etc.
What about stories like Tailwind?
When people talk about the 'plateau of ability' agents are widely expected to reach at some point, I suspect a lot of it will boil down to skyrocketing costs and plummeting accuracy past a certain point of number of agents involved. This seems to me like a much harder limit than context windows or model sizes.
Things like Gas Town are exploring this in what you might call a reckless way; I'm sure there are plenty of more careful experiments being conducted.
What I think the ultimate measure of this new tech will be is, how simple of a question can a human put to an LLM group for how complex of a result, and how much will they have to pay for it? It seems obvious to me there is a significant plateau somewhere, it's just a question of exactly where. Things will probably be in flux for a few years before we have anything close to a good answer, and it will probably vary widely between different use cases.
Show me an llm that can sell my product and find market fit.
In reality llms are taking away profitable tools and keeping the revenue themselves.
If I told you the drink tastes bad, is an off putting color, comes in a small bottle, and is expensive you wouldn’t believe it would work. But Red Bull made billions.
People think they'll just have a personal bot out there buying airline tickets, hotel rooms, jeans, new phone, etc. Meanwhile as soon as you have agents like this out in the wild, the capital will flow to bad actors creating bots to game those bots.
The world is PvP unfortunately. There is more money to be made skimming agents trying to buy stuff than there is in getting people to pay for a personal assistant agent subscription.
It's like why a lot of ad-based stuff doesn't offer a premium option for people to pay to opt out (ex Youtube). The people who can afford to pay and avoid search/social media/etc advertising are exactly the people you can make a lot of money advertising to.
Are people just so addicted to doomscrolling or whatever that they just can’t spend a few minutes of their day doing some type of human activity?
That is - lacking a supercar was not the thing preventing you from getting to work faster.
> You do what your supervisor did for you, years ago: you give each of them a well-defined project. Something you know is solvable, because other people have solved adjacent versions of it. Something that would take you, personally, about a month or two. You expect it to take each student about a year ...
Is that how PhD projects are supposed to work? The supervisor is a subject matter expert and comes up with a well-defined achievable project for the student?
And there is everything in between.
This can lead to a lot of problems as I think in some fields, by some academics, the default assumption is the former, when it's really the latter. This leads to a kind of overattribution of contribution by senior faculty, or conversely, an underappreciation of less senior individuals. The tendency for senior faculty be listed last on papers, and therefore, for the first and last authors to accumulate credit, is a good example of how twisted this logic has become.
It's one tiny example of enormous problems with credit in academics (but also maybe far afield from your question).
Yet what did not change in this process is that it only made the production of the text more efficient; the act of writing, constructing a compelling narrative plot, and telling a story were not changed by this revolution.
Bad writers are still bad writers, good writers still have a superior understanding of how to construct a plot. The technological ability to produce text faster never really changed what we consider "good" and "bad" in terms of written literature; it just allow more people to produce it.
It is hard to tell if large language models can ever reach a state where it will have "good taste" (I suspect not). It will always reflect the taste and skill of the operator to some extent. Just because it allows you to produce more code faster does not mean it allows you to create a better product or better code. You still need to have good taste to create the structure of the product or codebase; you still have to understand the limitations of one architectural decision over another when the output is operationalized and run in production.
The AI industry is a lot of hype right now because they need you to believe that this is no longer relevant. That Garry Tan producing 37,000 LoC/day somehow equates to producing value. That a swarm of agents can produce a useful browser or kernel compiler.
Yet if you just peek behind the curtains at the Claude Code repo and see the pile of unresolved issues, regressions, missing features, half-baked features, and so on -- it seems plainly obvious that there are limitations because if Anthropic, with functionally unlimited tokens with frontier models, cannot use them to triage and fix their own product.
AI and coding agents are like the printing press in some ways. Yes, it takes some costs out of a labor intensive production process, but that doesn't mean that what is produced is of any value if the creator on the other end doesn't understand the structure of the plot and the underlying mechanics (be it of storytelling or system architecture).
I think the real danger is no longer caring about what you're doing. Yesterday I just pointed Claude at my static site generator and told it to clean it up. I wanted to care but... I didn't.
If someone claims to have "understood [middle school] algebra" but they aren't able to solve equations by themselves, you'd be skeptical. Of course past some point of familiarity it's simply faster to throw things into a CAS, but if you remove the initial manual struggle, then have you wired up your brain for understanding? There was a post on HN a few days back about how familiarity with a tool leads to a sense of "embodied understanding" [1], and I think the initial struggle is an intrinsic part of learning to get to the "unconscious competence' level.
Claude has been amazing for code review. Having someone to talk to about my own code is world changing for a solo developer like me.
Given the article, I don't think most people are using LLMs in a "tutor" fashion to learn how to do the homework, they're effectively having their homework done by the tutor.
I totally made Claude do my own homework on my static site generator project though. It's even added itself as coauthor. Haven't even read the commits yet. It was just ripping out broken code and eliminating dependencies. Nothing that even approaches the complexity of my programming language work.
Weak ownership, unclear direction, and "sure, I guess" reviews were survivable when output was slow. When changes came in one at a time, you could get away with not really deciding.
AI doesn't introduce a new failure mode. It puts pressure on the old one. The trickle becomes a firehose, and suddenly every gap is visible. Nobody quite owns the decision. Standards exist somewhere between tribal memory, wishful thinking, and coffee. And the question of whether something actually belongs gets deferred just long enough to merge it, but forces the answer without input.
The teams doing well with agentic workflows aren't typically using magic models. They've just done the uncomfortable work of deciding what they're building, how decisions are made, and who has the authority to say no.
AI is fine, it just removed another excuse for not having our act together. While we certainly can side-eye AI because of it, we own the problems. Well, not me. The other guy who quit before I started.
And it's not just a volume problem.
Mediocre devs previously couldn't complete a project by themselves and were forced to solicit help and receive feedback along the way.
When all managers care about is "shipping", development becomes a race to the bottom. Devs who used to collaborate are now competing. Whoever gets the slop into the codebase fastest, wins.
Academia doesn't want to produce astrophysics (or any field) scientists just so the people who became scientists can feel warm and fuzzy inside when looking at the stars, it wants to produce scientists who can produce useful results. Bob produced a useful result with the help of an agent, and learned how to do that, so Bob had, for all intents and purposes, the exact same output as Alice.
Well, unless you're saying that astrophysics as a field literally does not matter at all, no matter what results it produces, in which case, why are we bothering with it at all?
Once I realized that this white on black contrast was hurting my eyes, I decided to stop as I didn't want to see stripes for too long when looking away.
Some activity has outcomes that aren't strictly in the results.
This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, but with LLMs we seem to have forgotten that line of reasoning entirely.
LLMs, the way they typically get used, are solely to save time by handing over nearly the entire process. In that sense acuity can't remain intact, even less so improving over time.
Do you agree the different treatment is justified ? (Many do not). Or are you asking , so what if acuity is diminished so long as an LLM does the job equally well?
This is false. There absolutely are people that fall back on older tools when fancy tools fail. You will find such people in the military, in emergency services, in agriculture, generally in areas where getting the job done matters.
Perhaps you're unfamiliar.
They other week I finished putting holes in fence posts with a bit and brace as there was no fuel for the generator to run corded electric drills and the rechargable batteries were dead.
Ukrainians, and others, need to fall back on no GPS available strategies and have done so for a few years now.
etc.
It depends on the task though. If you are in a similar scenario as with your fence posts and want to edit computer programs, you can't. (Not even with xkcd's magnetic needle and a steady hand). ;-)
As technology marches on it seems inevitable that we will get increasingly large and frequent knowledge gaps. Otherwise progress would stop - we need the giant shoulders to stand on.
How many people in the world can recreate a ASML lithography machine vs how many people are surviving by doing something that requires that machine to exist?
Solar panels charging old thinkpads suddenly doesn't work, or are we reliant upon software that requires cloud services to function in your scenario?
> How many people in the world can recreate a ASML lithography machine ..
Some .. are you advocating the answer should be none, in your future?
It's your scenario, you said you had no batteries and had to drill holes by hand, I'm saying that's fine for drilling holes but if you have no batteries (or solar panels, those could charge your power tools as well so they're not in the scenario!) you can't alter computer code by hand.
Leaving computers aside, we can take a piano player - can play a piano but he might not know how to build one. A doctor can interpret X-rays but he can't build an X-ray machine.
And so on. We are living in the reality where many, many people are using tools they don't understand how to make already, and have done so for 100+ years.
>Some .. are you advocating the answer should be none, in your future?
I'm not advocating anything, I'm just saying that this is the reality already, it's not new with AI.
Ideally we'd forever keep a chain of key people who know how to make everything from atoms, but realistically that's a chain that will break. It's happened before, medieval dark ages for example.
> Mathematica has been able to do many integrals for decades and yet we still make students learn all the tricks to integrate by hand
But maybe, integrating by hand is still as big as ever in other parts of academia. Or were you thinking about high school? I'm fairly sure, that symbolic solving of integrals is treated as less important in education these days, than it was before digital computers, but I could be wrong. Mathematica's symbolic solve sure is very useful, but numeric solutions are what really makes the art of finding integrals much less relevant.
But to even *know* what is more useful, it is crucial to have walked the walk. Otherwise we will all end up with a bunch of people trying to reinvent the wheel, over and over again, like JavaScript "developers" who keep reinventing frameworks every six months.
> which nobody would buy for any other tool
I don't know about you, but I wasn't allowed to use calculators in my calculus classes precisely to learn the concepts properly. "Calculators are for those who know how to do it by hand" was something I heard a lot from my professors.
I feel like people tend to forget that among the many things LLMs can do these days, “using a search engine” is among them. In fact, they use them better than the majority of people do!
The conversation people think they’re having here and the conversation that actually needs to be had are two entirely different conversations.
> I don’t know about you, but I wasn’t allowed to use calculators in my calculus classes precisely to learn the concepts properly. “Calculators are for those who know how to do it by hand” was something I heard a lot from my professors.
Suppose I never learned how to derive a function. I don’t even know what a function is. I have no idea how to do make one, write one, or what it even does. So I start gathering knowledge:
- A function is some math that allows you to draw a picture of how a number develops if you do that math on it.
- A derivative is a function that you feed a function and a number into, and then it tells you something about what that function is doing to that number at that number.
- “What it’s doing” specifically means not the result of the math for that particular number, but the results for the immediate other numbers behind and in front of it.
- This can tell us about how the function works.
Now I go tell ClaudeGPTimini “hey, can you derive f(x) at 5 so that we can figure out where it came from and where it goes from there?”, and it gives me a result.
I’ve now ostensibly understood what a derivative does and what it’s used for, yet I have zero idea how to mathematically do it. Does that make any results I gain from this intuitive understanding any less valuable?
What I’ll give you is this: if I knew exactly how the math worked, then it would be far easier for me to instantly spot any errors ClaudeGPTimini produced. And the understanding of functions and derivatives outlined above may be simplistic in some places (intentionally so), in ways that may break it in certain edge cases. But that only matters if I take its output at face value. If I get a general understanding of something and run a test with it, I’ll generally have some sort of hypothesis of what kind of result I’m expecting, given that my understanding is correct. If I know that a lot of unknown unknowns exist around a thing I’m working with, then I also know that unexpected results, as well as expected ones, require more thorough verification. Science is what happens when you expect something, test something, and get a result - expected OR unexpected - and then systematically rule out that anything other than the thing you’re testing has had an effect on that result.
This is not a problem with LLMs. It’s a thing we should’ve started teaching in schools decades ago: how to understand that there are things you don’t understand. In my view, the vast majority of problems plaguing us as a species lies in this fundamental thing that far too many people are just never taught the concept of.
From a science standpoint, I'd say whatever "results" you got are completely worthless.
> I’ll generally have some sort of hypothesis of what kind of result I’m expecting, given that my understanding is correct
And how do you know if your understanding is correct, if you are only taking what the LLM gives to you and you are not able to verify independently?
> Science is what happens when you expect something, test something, and get a result.
Right, but has any LLM come up with any hypothesis on its own? Has any AI said "given all this literature that I read, I'd expect <insert something completely out of the training data space>?".
I'll leave these direct quotes from the comment as a hint:
> But that only matters if I take its output at face value. […] If I know that a lot of unknown unknowns exist around a thing I’m working with, then I also know that unexpected results, as well as expected ones, require more thorough verification.
I’m sure the students will manage. This is the exact same discussion we’ve all been through before, during the rise of Wikipedia, just wearing a new hat. The answer is “vet your sources, don’t trust unsourced claims.” The way they’ll develop the skills is the same way aspiring scientists and students have developed them throughout the entirety of human history’s vast corpus across time: by having good teachers teach them.
Here’s a very simple program I thought of from the top of my head in a minute or two. I’m sure people whose job it is to create educational content will be able to come up with something far better:
Design a small research project with as many LLM-tailored pitfalls as possible. It involves real measurements and real data, and the students may use their LLM to whichever extent they wish. Then, we compare results against the reference data, and find out all the myriads of ways in which LLMs can taint the data and the conclusions to be made from it, and then explore ways how to mitigate it.
Probably not perfect and nitpickable to oblivion, but also not the hardest mental exercise I’ve ever subjected myself to.
Science did fine in a world where information took years or decades to travel the globe, people thought diseases were spread by evil mojo and we had a grand total of four liquids circulating inside our bodies, and scientists saying the wrong things were actively hunted down and silenced. It got there. It’ll do fine in a world where you can semantically search every single written source model trainers could get their hands on _and_ ground the results with references to tangible sources using the same natural language query.
This was already a problem for Wikipedia (articles being written which upon further investigation were based on nothing but Wikipedia itself). With LLM themselves facilitating AI slop and plagiarism, this problem gets to a scale that it becomes impossible to control.
> I’m sure the students will manage.
The problem with your hubris is that you are not going to be the one solely facing the fallout when this blows up.
I’ll get back to you as soon as you make an actual point that’s based in some sort of precedent, some sort of data.
Basically, stop indignantly demanding that I contribute and start doing it yourself. I’m tired of having to spell the same thing out for you in entire paragraphs over and over only to have you refuse to even make an attempt at comprehension, cherry pick two lines, and proceed to add absolutely nothing substantial.
You on the other hand are the one "being sure" about how everything will be fine and of course there is no way for you to bring actual evidence because all you have is conjecture. So, given there is no way for you to back up your argument with evidence, the next best thing you can do is to put some Skin In The Game: can you back up with beliefs with actions? Are you willing to take any substantial risk in case your bet doesn't pay off?
The fact that you think “let the LLM guide you” is the argument I’ve been making tells me everything about how honestly you’ve engaged with it. I’m done here.
We're minting an entire generation of people completely dependent on VC funding. What happens if/when the AI companies fail to find a path to profitability and the VC funding dries up?
As of March 2026, OpenAI generates annual revenue exceeding $12 billion. However, the costs of running ChatGPT are around $17 billion a year.
Source: https://searchlab.nl/en/statistics/chatgpt-statistics-2026
Once they have to solve a novel problem that was not already solved for all intentes and purposes, Alice will be able to apply her skillset to that, whereas Bob will just run into a wall when the LLM starts producing garbage.
It seems to me that "high-skill human" > "LLM" > "low-skill human", the trap is that people with low levels of skills will see a fast improvement of their output, at the hidden cost of that slow build-up of skills that has a way higher ceiling.
Why is it a problem of the LLM if your test is unrelated to the performance you want?
I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.
Edit: let's look at a paper like Some Linear Transformations on Symmetric Functions Arising From a Formula of Thiel and Williams https://ecajournal.haifa.ac.il/Volume2023/ECA2023_S2A24.pdf and try and guess how many of trivial things were completely unneeded to write a paper like this.
If you send a kid to an elementary school, and they come back not having learned anything, do you blame the concept of elementary schools, or do you blame that particular school - perhaps a particular teacher _within_ that school?
There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.
You're conflating an algorithm in N or Z with inherent meaning.
Let's shift over to R: Expand e*pi to repeated addition.
Think exponentiation is "repeated multiplication"? Try 2^pi.
This sentence contains the entire point, and the easiest way to get there, as with many, many things, is to ask “why?”
No-ones becomes a good mathematician without first learning to write simple proofs, and then later on more complex proof. It's the very stuff of the field itself.
While we have a lot of abstractions that solve some subproblems, there still need to connect those solutions to solve the main problem. And there’s a point where this combination becomes its own technical challenge. And the skill that is needed is the same one as solving simpler problems with common algorithms.
At a certain point, higher level languages stop working. Performance, low level control of clocks and interrupts, etc.
I’m old enough dropping into assembly to be clever with the 8259 interrupt controller really was required. Programmers today? The vast majority don’t really understand how any of that works.
And honestly I still believe that hardware-up understanding is valuable. But is it necessary? Is it the most important thing for most programmers today?
When I step back this just reads like the same old “kids these days have it so easy, I had to walk to school uphill through the snow” thing.
Writing assembly is probably completely irrelevant. You should still know how programming language concepts map to basic operations though. Simple things like strict field offsets, calling conventions, function calls, dynamic linking, etc.
ffmpeg disagrees.
More broadly, though, it’s a logical step if you want to go from “here’s how PN junctions work” to “let’s run code on a microprocessor.” There was a game up here yesterday about building a GPU, in the same vein of nand2tetris, Turing Complete, etc. I find those quite fun, and if you wanted to do something like Ben Eater’s 8-bit computer, it would probably make sense to continue with assembly before going into C, and then a higher-level language.
Because we largely want people who have committed to tens of thousands of dollars of debt to feel sufficiently warm and fuzzy enough to promote the experience so that the business model doesn’t collapse.
It’s difficult to think anyone would end up truly regretting doing a course in astrophysics, or any of the liberal arts and sciences if they have a modicum of passion, but it’s very believable that a majority of them won’t go on to have a career in it, whatever it is, directly.
They’re probably more likely to gain employment on their data science skills, or whether core competencies they honed, or just the fact that they’ve proven they can learn highly abstract concepts, or whatever their field generalises to.
Most of the jobs are in not-highly-specific academic-outcome.
I will make an explicit, plausible, counterpoint: academia wants to produce understanding. This is, more or less, by definition, not possible with an AI directly (obviously AIs can be useful in the process).
Take GR as an example. The vast majority of the dynamical character of the theory is inaccessible to human beings. We study it because we wanted to understand it, and only secondarily because we had a concrete "result" we were trying to "achieve."
A person who cares only about results and not about understanding is barely a person, in my opinion.
The industrialization of academia hasn't even produced more results, it has produced more meaningless papers. Just like LLMs produce the 10.000th note taking app, which for the LLM psychosis afflicted is apparently enough.
The difficulty of passing the defence vary's wildly between Universities, departments and committees. Some are very serious affairs with a decent chance of failure while others are more of a show event for friends and family. Mine was more of the latter, but I doubt I would have passed that day if I had spend the previous years prompting instead of doing the grunt work.
The process you describe is a gate keeping exercise which will change to include llm judges at somepoint.
To be clear, I am not against alternative forms of education. Degrees are optional. But if you want a degree, there have to be exams and cheating has to be prevented.
How do we know that while the AI was writing python scripts that Bob wasn't reading more papers, getting more data and just overall doing more than Alice.
Maybe Bob is terrible at debugging python scripts while Alice is a pro at it?
Maybe Bob used his time to develop different skills that Alice couldn't dream of?
Maybe Bob will discover new techniques or ideas because he didn't follow the traditional research path that the established Researchers insist you follow?
Maybe Bob used the AI to learn even more because he had a customized tutor at his disposal?
Or maybe Bob just spent more time at the Pub with his friends.
Kids should start off with Commodore 64s, then get late 80’s or early 90’s Mac’s, then Windows 95, Debian and internet access (but only html). Finally, when they’re 18, be allowed an iPhone, Android and modern computing.
Parenting can’t prevent the use of LLMs in grad school, but a similar approach could be taken by grad departments: don’t allow LLMs for the first few years, and require pen and paper exams, as well as oral examinations for all research papers.
It's actually great since a lot of older technology is cheap and still readily available. My little ones love listening to old records, control the playback speed and hear the music go up in pitch if the RPMs are set too high. We look at the tracks on the vinyl under a microscope at talk about how the music is written on it that way. VHS an audio cassettes offer their own talking points.
For computers, we don't literally use a Commodore 64 but we run simpler, old software on new hardware. Mostly because a lot of newer education software is somehow also funded by injecting ads into the games (awful). But there is also some good "modern" educations software worth checking out. I highly recommend gcompris.net.
I don’t believe this. Totally plausible that someone would be able to produce passable work with LLMs at a similar pace to a curious and talented scientist. But if you, their advisor, are sitting down and talking with them every week? It’s obvious how much they care or understand, I can’t believe you wouldn’t be able to tell the difference between these students.
Very well said. I think people are about to realize how incredibly fortunate and exceptional it is to actually get paid, and in our industry very well, through a significant fraction of one's career while still "just" doing the grunt work, that arguably benefits the person doing it at least as much as the employer.
A stable paid demand for "first-year grad student level work" or the equivalent for a given industry is probably not the only possible way to maintain a steady supply of experts (there's always the option of immense amounts of student debt or public funding, after all), but it sure seems like a load-bearing one in so many industries and professions.
At the very least, such work being directly paid has the immense advantage of making artificially (often without any bad intentions!) created bullshit tasks that don't exercise actually relevant skillsets, or exercise the wrong ones, much easier to spot.
I'm sure this approach breaks down at the very frontiers of highly technical fields but... virtually all work, even work by educated professionals, happens outside that area anyway. On well-trodden ground, you can improve at supervising agents by doing things that test your ability to supervise agents.
Vs fields where there is not a reliable feedback path, or that feedback path is much more noisy.
This is solvable with harness engineering.
The model’s first try is never ready for human consumption. There needs to be automation (bespoke, a mix of code and prompt based hooks - which agents can build) to force the agent’s output back through itself to tell it to be more rigorous, search online for proof of its claims, etc etc. and not stop until every claim is verifiable.
No human should see the model’s output until it’s met these (again bespoke but not hand written) guardrails.
What I’m talking about doesn’t exist and really has no analogy yet, so you can think of it as a super advanced form of linting. It’s grounding, but also verification that the grounding links to the material, and refusal to accept the model’s work until it meets the bar.
We are asking models to dream (invent purely from their weights), and are surprised when their dreams, just like ours, have little relationship to reality. The current state of the art is going to look very naive in a few years’ time.
> But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding. Who know what buttons to press but not why those buttons exist. Who can get a paper through peer review but can't sit in a room with a colleague and explain, from the ground up, why the third term in their expansion has the sign that it does.
I have seen versions of this in the wild where a firm has gone through hard times and internally systems have lost all their original authors, and every subsequent generation of maintainers… being left with people in awe of the machine that hasn’t been maintained in a decade.
I interviewed a guy once that genuinely was proud of himself, volunteering the information to me as he described resolving a segfault in a live trading system by putting kill -9 in a cronjob. Ghastly.
I’ve recently started csprimer and whilst mentally stimulating I wonder if I’m not completely wasting my time.
If that comes to pass, we'll be rediscovering the same principles that biological evolution stumbled upon: the benefits of the imperfect "branch" or "successive limited comparison" approach of agentic behaviour, which perhaps favours heuristics (that clearly sometimes fail), interaction between imperfect collaborators with non-overlapping biases, etc etc
https://contraptions.venkateshrao.com/p/massed-muddler-intel...
> Lindblom’s paper identifies two patterns of agentic behavior, “root” (or rational-comprehensive) and “branch” (or successive limited comparisons), and argues that in complicated messy circumstances requiring coordinated action at scale, the way actually effective humans operate is the branch method, which looks like “muddling through” but gradually gets there, where the root method fails entirely.
I think this is a simplification, of course Bob relied on AI but they also used their own brain to think about the problem. Bob is not reducible to "a competent prompt engineer", if you think that just take any person who prompts unrelated to physics and ask them to do Bob's work.
In fact Bob might have a change to cover more mileage on the higher level of work while Alice does the same on the lower level. Which is better? It depends on how AI will evolve.
The article assumes the alternative to AI-assisted work is careful human work. I am not sure careful human work is all that good, or that it will scale well in the future. Better to rely on AI on top of careful human work.
My objection comes from remembering how senior devs review PRs ... "LGTM" .. it's pure vibes. If you are to seriously review a PR you have to run it, test it, check its edge cases, eval its performance - more work than making the PR itself. The entire history of software is littered with bugs that sailed through review because review is performative most of the time.
Anyone remember the verification crisis in science?
In my experience, doing these things with the right intentions can actually improve understanding faster than not using them. When studying physics I would sometimes get stuck on small details - e.g. what algebraic rule was used to get from Eq 2.1 to 2.2? what happens if this was d^2 instead of d^3 etc. Textbooks don't have space to answer all these small questions, but LLMs can, and help the student continue making progress.
Also, it seems hard to imagine that Alice and Bob's weekly updates would be indistinguishable if Bob didn't actually understand what he was working on.
[0] http://employees.oneonta.edu/blechmjb/JBpages/m360/Professio...
[1] https://s3.us-west-1.wasabisys.com/luminist/EB/A/Asimov%20-%...
https://boxobarks.leaflet.pub/3mj42airv3s2o#fingerprints-of-...
I've been having a lot of fun vibe coding little interactive data visualizations so when I present the feature to stakeholders they can fiddle with it and really understand how it relates to existing data. I saw the agent leave a comment regarding Cramer's rule and yeah its a bit unsettling that I forgot what that is and haven't bothered to look it up, but I can tell from the graphs that its doing the correct thing.
There's now a larger gap between me and the code, but the chasm between me and the stakeholders is getting smaller and so far that feels like an improvement.
A two-hour thesis defense isn't enough to uncover this, but a 40-hour deep probing examination by an AI might be. And the thesis committee gets a "highlight reel" of all the places the student fell short.
The general pattern is: "Suppose we change nothing but add extensive use of AI, look how everything falls apart." When in reality, science and education are complex adaptive systems that will change as much as needed to absorb the impact of AI.
To some extent, the reason models will get better is because companies will hire PhDs to train them on increasingly complex problems.
The problem is that more complex problems take longer to train, more time to test, require more compute, and are harder to verify. This is why “just make it bigger” is a losing proposition imo.
A lot of what I just said is also true for RLVR.
Most of what we call thinking is merely to justify beliefs that emotionally make us happy and is not creative per-se. I am making a distinction between "thinking" as we know it and "creative thinking" which is rare, and can see things in an unbiased manner breaking out of known categories. Arguably, at the PhD level, there needs to be a new ideas instead of remixing the existing ones.
To be fair to pedagogy, it's also being defeated by eons of human brain wiring that says: "People in stories are somewhat real, and the author is always real."
We are disarmed by how well these things mimic all of typical indicators of thought, and it takes work— work our brains can't really sustain—to not fall into assumptions that there's another humanoid bearing some of the responsibility for catching problems.
The risk is that civilization is over its skis because humans are lazy. Humans are always lazy. In science there’s a limit to bs because dependent works fail. In economics there’s a crash. In physics stuff breaks. Then there is a correction.
There’s a reason most people aren’t promoted to manager until they have years of experience under their belt. And now we’re expecting folks to be managers on day 1.
In academia, understanding is vital. The same for research.
But in production, results are what matters.
Alice would be a better researcher, but Bob would be a better producer. He knows how to wrangle the tools.
Each has its value. Many researchers develop marvelous ideas, but struggle to commercialize them, while production-oriented engineers, struggle to come up with the ideas.
You need both.
In production, there would be no "paper"; just some software/hardware product.
If there was a problem, that would be fairly obvious, with testing (we are going to be testing our products, right?).
I have been wrestling all morning, with an LLM. It keeps suggesting stuff that doesn't work, and I need to keep resetting the context.
I am often able to go in, and see what the issue is, but that's almost worthless. The most productive thing that I can do, is tell the LLM what is the problem, on the output end, and ask it to review and fix. I can highlight possible causes, but it often finds corner cases that I miss. I have to be careful not to be too dictatorial.
It's frustrating, as the LLM is like a junior programmer, but I can make suggestions that radically improve the result, and the total time is reduced drastically. I have gotten done, in about two hours, what might have taken all day.
I think the difference between the workflow you describe and the description of Bob's is that you have a pretty good idea of what a working solution would look like. In my reading if the article Bob does not.
In my opinion, the software developer analogue of Bob would be someone who would often reach for the LLM as it's nearby and easy. Maybe at first they would be careful about reading and vetting the model's output. Over time they might grow comfortable and overly confident with the model's output and pay less and less attention. As they take on more complex tasks they begin to understand less and less about the LLM tooling output but they don't really notice, it all looks good and tests are passing. Eventually we see a production problem, maybe even an outage. When we narrow down the issue to a PR with Bob's name on it and ask him how it led to the production issue, Bob tries to be helpful but struggles to understand his own PR.
For me, I started off with Machine Code, but these days, I have no idea what's going on, below the compiler. I know enough to be dangerous, looking at a stack dump, but that's about it. My days of hex keypads are long gone.
I still get stuff done, that I could only dream of, back then. LLMs are really just another step on the evolutionary ladder.
I'm old enough to remember when calculators were banned in the classroom, for just about exactly the same reasons that people are complaining about LLMs.
Anyway, the die is cast. We'll have to see how things turn out.
yea, there are multiple parts to education. 1) teach skills useful to the economy 2) teach the theories of the subject, and finally 3) tweak existing theories and create new ones. An electrician can fix problems without understanding theory of electromagnetism. These are the trades folks. A EE college graduate has presumably understood some theory, and can apply it in different useful ways. These are the engineers. Finally, there are folks who not only understand the theory of the craft, but can tweak it creatively for the future. These are the researchers.
Bob better fits as a trades-person or engineer whereas Alice fits better as a researcher.
As long as that’s the case, those that create crap will thrive.
Pretty basic, and long predates LLMs.
Acting like workers have a meaningful choice where their director had to spend $30million on SalesForce licenses when people had no say in the matter just further proves that the last frontier for democracy will be in the private sector where we can liberate the workers against their tyrants and actually run the companies effectively rather than relying on the government to kneecap their competitors or straight up ignoring the law to the detriment of society + the planet.
This is going to get worse, and eventually cause disastrous damage unless we do something about it, as we risk losing human institutional memory across just about every domain, and end up as child-like supplicants to the machines.
But as the article says, this is a people problem, not a machine problem.
i.e. science.
If an AI is capable of producing an elegant solution with fewer levels of abstraction it could be possible that we end up drifting towards having a better understanding of what's going on.
I don't know if I mind.
Example. This paragraph, to me, has a eerily perfect rhythm. The ending sentence perfectly delivers the twist. Like, why would you write in perfect prose an argument piece in the science realm?
> Unlike Alice, who spent the year reading papers with a pencil in hand, scribbling notes in the margins, getting confused, re-reading, looking things up, and slowly assembling a working understanding of her corner of the field, Bob has been using an AI agent. When his supervisor sent him a paper to read, Bob asked the agent to summarize it. When he needed to understand a new statistical method, he asked the agent to explain it. When his Python code broke, the agent debugged it. When the agent's fix introduced a new bug, it debugged that too. When it came time to write the paper, the agent wrote it. Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.
If you could, why wouldn't you? LLM witch-hunts over every halfway competent writer are becoming quite tiresome
Again, I appreciate the article very much and I'm glad the other comments are on the article's content.
LLM speak. But the rest of that quote doesn't look LLM-generated, it's too fiddly and complex of an argument. I think this was edited with AI, but the underlying argument at least is human.
It had great success, now when I propose to them to use some model to do something, they tends to avoid.
The author is a bit naive here:
1. Society only progresses when people are specialised and can delegate their thinking
2. Specialisation has been happening for millenia. Agriculture allowed people to become specialised due to abundance of food
3. We accept delegation of thinking in every part of life. A manager delegates thinking to their subordinates. I delegate some thinking to my accountant
4. People will eventually get the hang of using AI to do the optimum amount of delegation such that they still retain what is necessary and delegate what is not necessary. People who don't do this optimally will get outcompeted
The author just focuses on some local problems like skill atrophy but does not see the larger picture and how specific pattern has been repeating a lot in humanity's history.
> It is a profoundly erroneous truism ... that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them.
> 4. People will eventually get the hang of using AI to do the optimum amount of delegation such that they still retain what is necessary and delegate what is not necessary. People who don't do this optimally will get outcompeted
Then they’ll be at the mercy of the online service availability and the company themselves. Also there’s the non deterministic result. I can delegate my understanding of some problems to a library, a software, a framework, because their operation are deterministic. Not so with LLMs.
1. outsourcing and offshoring (non deterministic, easy to disrupt)
2. cloud computing (mercy of the online service availability)
we had the same dilemma.
Outsource exactly what you think is not critical to the business. Offshore enough so that you gain good talent across the globe. Use cloud computing so that your company does not spend time working on solving problems that have already been solved. Assess what skills are required and what aren't - an e-commerce company doesn't need deep expertise in linux and postgres.
Companies that do this well outcompete other companies that obsess over details that are not core to their value proposition. This is how modern startups work: it is in finding that critical balance of buying products externally vs building only the crucial skills internally.
The author describes specifically how specialists are produced and how AI undermines their production.
No, we won't get more and more specialists literally "the same way" as the agricultural revolution. You need to be much more specific about how we'll get more specialists under the incentive structure created by AI, otherwise this sounds like some kind of religious faith in AI and progress.
People do more things with AI.
More things = more inventions = the field growing.
The field grows and people become specialists on what used to be a small or trivial.
A mathematician in 1500's wouldn't think algebraic topology would be a specialisation.
How about addressing astrophysics specifically. What are you claiming about it? Are you claiming that in the future, we won't need astrophysicists at all, AI can do all of our astrophysics for us, freeing humans to specialize in... other subjects?
And doesn't the same problem exist for Linux kernel specialists? Why even become a Linux kernel specialists when AI can write your source code for you?
> people become specialists
This is precisely what is in question.
> A mathematician in 1500's wouldn't think algebraic topology would be a specialisation.
The specific subjects have changed over time, but the production of specialist mathematicians hasn't really changed. It takes hard work, grunt work, struggling, making mistakes and learning from them, as well as expert supervision. The problem with AI is that it encourages and incentivizes intellectual laziness, the opposite of what is required to produce specialists.
A related problem: LLMs have been trained with papers written and supervised by Alice-type specialists. There's a common claim that LLMs will hallucinate less in the future, but I think that LLMs will hallucinate more in the future, when specialty fields become dominated by Bob-type "specialists" who have a harder time distinguishing fact from fiction. When LLMs have to train on material produced by earlier versions of LLMs, the quality trend will go down, not up.
Let's take the example of economics. Economists use ideas in Mathematics like integrals, statistics, PDE's and so on. They know that these concepts exist. They know how to apply them. They don't know these concepts deep enough to make progress here.
Do you think that Economists should deeply learn integrals, PDE's, Functional Analysis and Differential Geometry and all other concepts they use? Or do you think its better for them to focus just on their specific domain while learning just enough from other domains?
You keep coming back to AI replacing mathematicians. I'm not making that claim. I'm not saying Linux kernel specialists will be replaced by AI. I'm simply claiming that not everyone needs to be Linux Kernel specialists. This is precisely what AI is allowing: it automates things I don't need to know deeply so that I can focus on things I do need to understand deeply.
This is an uninteresting and indeed silly claim, because nobody has ever asserted the opposite.
The point is that society needs some Linux kernel specialists, and some astrophysicists, but AI is undermining their production.
> This is precisely what AI is allowing: it automates things I don't need to know deeply so that I can focus on things I do need to understand deeply.
The submitted article is about how AI is automating the things that a specialist does need to understand deeply. It's about so-called astrophysicists using AI to produce astrophysics papers, not about how non-astrophysicists use AI to produce astrophysics papers so that they can focus on whatever their non-astrophysics specialty may be, if they have any specialty.
> Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small.
If we both agree that an astrophysicist may not need to understand things (even in their own domain) to make progress then we are in agreement. Not all the things a researcher works on while writing their paper is useful or necessarily done by them manually. In such cases it becomes necessary to let LLM take over.
> > Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe
The article author and I share a love of Frank Herbert, God Emperor of Dune, and the quote in question. Nonetheless, it's a mistake to focus on this quote rather than on the rest of the article. The quote is nothing more than a nice literary reference; it's not central to the argument.
The character who spoke the quote is a magically prescient human-sandworm hybrid, thousands of years old, speaking to his distant relative who was specially bred by him to be invisible to the magical prescience, so let's take the quote with a grain of... sand. ;-)
> If we both agree that an astrophysicist may not need to understand things (even in their own domain) to make progress then we are in agreement.
Your parenthetical remark is actually the main problem!
A combination of beancounters running the show and the old, experienced engineers dying, retiring, and going through buyouts has pretty much left things in a pretty sad state.
Its of course devious, exactly some of our styles :)
Give AI to VCs to use for all their domain stuff....
They than make wrong investment decisions based on AI wrong info and get killed in the market....
Market ends up killing AI outright....problem solved temporarily
It'll be valuable to refer back to this in 5-10 years.
Archived here so we can always come back to this: https://archive.md/MZtsI
The Aenid, Virgil, Dryden translation.
One thing I've seen asserted:
> What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days... The equations seemed right... Then Schwartz read it, and it was wrong... It faked results. It invented coefficients...
The argument that AI output isn't good enough is somewhat in opposition to the idea that we need to worry about folks losing or never gaining skills/knowledge.
There are ways around this:
"It's only evident to experts and there won't be experts if students don't learn"
But at the end of the day, in the long run, the ideas and results that last are the ones that work. By work, I mean ones that strictly improve outcomes (all outputs are the same with at least one better). This is because, with respect to technological progress, humans are pretty well modeled as just a slightly better than random search for optimal decisioning where we tend to not go backwards permanently.
All that to say that, at times, AI is one of the many things that we've come up with that is wrong. At times, it's right. If it helps on aggregate, we'll probably adopt it permanently, until we find something strictly better.
> Take away the agent, and Bob is still a first-year student who hasn't started yet.
This may be true, but I can see almost no conceivable word where the agent will be taken away. I think we should evaluate Bob's ability based on what he can do with an agent, not without, and here he seems to be doing quite well.
> I've been hearing "just wait" since 2023.
On almost any timeline, this is very short. Given the fact that we have already arrived at models able to almost build complete computer programs based on a single prompt, and solve frontier level math problems, I think any framework that relies on humans continuing to have an edge over LLMs in the medium term may be built on shaky grounds.
Two very interesting questions today in this vein for me are:
- Is the best way to teach complex topics to students today to have them carry out simple tasks?
The author acknowledges that the difference between Bob and Alice only materializes at a very high level, basically when Alice becomes a PI of her own. If we were solely focused on teaching thinking at this level (with access to LLMs), how would we frame the educational path? It may look exactly like it does now, but it could also look very differently.
- Is there inherent value in humans learning specific skills?
If we get to a stage where LLMs can carry out most/all intellectual tasks better than humans, do we still want humans to learn these skills? My belief is yes, but I am frankly not sure how to motivate this answer.
LLM access is a paid service. HN concerns itself with inequality constantly and it's not inconceivable that some individuals get ahead because they can afford to pay for more tokens and better models than those who are poorer.
The way I think about this is : We can't catch the hallucinations that we don't know are hallucinations.
Do you lack fundamental understand of those apps you built that are still in use? Did you lack understanding of their workings when you built them?
However, at one point in my career, I was frustrated with limitations in a language (Fortran II) and my curiosity got the better of me and I studied compilers thoroughly.
This led to a new job and the understanding of many new useful programming concepts. Very rewarding.
But if you are curious, studying compilers, maybe even writing a new one, will give you tools to do other things.
While working with LLMs, much of my experience gives me new ideas to push the LLM to explore.
This article first says that you give juniors well-defined projects and let them take a long time because the process is the product. Then goes on to lament the fact that they will no longer have to debug Python code, as if debugging python code is the point of it all. The thing that LLMs can't yet do is pick a high-level direction for a novel problem and iterate until the correct solution is reached. They absolutely can and do iterate until a solution is reached, but it's not necessarily correct. Previously, guiding the direction was the job of the professor. Now, in a smaller sense, the grad student needs to be guiding the direction and validating the details, rather than implementing the details with the professor guiding the direction. This is an improvement - everybody levels up.
I also disagree with the premise that the primary product of astrophysics is scientists. Like any advanced science it requires a lot of scientists to make the breakthroughs that trickle down into technology that improves everyday life, but those breakthroughs would be impossible otherwise. Gauss discovered the normal distribution while trying to understand the measurement error of his telescope. Without general relativity we would not have GPS or precision timekeeping. It uncovers the rules that will allow us to travel interplanetary. Understanding the composition and behavior of stars informs nuclear physics, reactor design, and solar panel design. The computation systems used by advanced science prototyped many commercial advances in computing (HPC, cluster computing, AI itself).
So not only are we developing the tools to improve our understanding of the universe faster, we're leveling everybody up. Students will take on the role of professors (badly, at first, but are professors good at first? probably not, they need time to learn under the guidance of other faculty). professors will take on the role of directors. Everybody's scope will widen because the tiny details will be handled by AI, but the big picture will still be in the domain of humans.
You have a good point, but I would argue that debugging itself is a foundational skill. Like imagine Sherlock Holmes being able to use any modern crime-fighting technology, and using it extensively. If Sherlock is not using his deductive reasoning, then he's not a 'detective'. He's just some schmuck who has a cool device to find the right/wrong person to arrest.
Debugging is "problem-solving" in a specific domain. Sure, if the problem is solved, then I guess that's the point of it all and you don't have to solve the problem. But we're all looking towards a world in which people have to solve problems, but their only problem-solving skill is trying to get an AI to find someone to arrest. We need more Sherlocks to use their minds to get to the bottom of things, not more idiot cops who arrest the wrong person because the AI told them to.
- Caddyshack
It's why I push for a hybrid mentor-apprentice model. We need to actively cultivate the next generation of "Schwartzes" with hands-on, critical thinking before throwing them into LLM-driven environments. The current incentive structure, as conception points out, isn't set up for this, but it's crucial if we want to avoid building on sand.
If I could make a rocket that could accelerate at 3 Gs for 10 years, how long would it take to travel from Earth to Alpha Centauri by accelerating at 3 Gs for half the time, then decelerating at 3 Gs for half the time?
Hint: They don't all get it right. Some of them never got it right after hints, corrections, etc.
This kind of follow-the-leader kind of "thinking" is probably a requirement. The amount of expertise it would require to understand and decide about things in our daily life is overwhelming. Do you fix your own car, decide each day how to travel, get food and understand how all that works? No.
So what is the problem? The problem is that if you follow the leader and the leader has an agenda that differs from your agenda. Do you really think that with Jeff Bezos being a (the?) major investor in Washington Post has anything to do with Democraccy? You know as in the WAPO slogan "Democracy dies in the Dark".
Does Jeff have an agenda that differs from yours? Yes. NYT? Yes. Hacker news? Yes. Google? Yes. We now live in a world so filled with propaganda that it makes no difference whether something is AI. We all "follow". Or not.
The bit they are missing, IMHO, is that if LLMs keep getting better, doing the steering-the-LLM version keeps getting easier, and being an LLM-using-expert rapidly drops to a value of zero. Like literally fucking nothing. If anyone can do it easily with an LLM, why would anyone pay you anything? Why would they care? You might as well be a teenager at a fast food job.
In this scenario, only actually understanding will be any kind of differentiator at all. And if that scenario doesn't come to pass, it will still be a far better differentiator to have a clue.
When I was fresh out of undergrad, joining a new lab, I followed a similar arc. I made mistakes, I took the wrong lessons from grad student code that came before mine, I used the wrong plotting libraries, I hijacked python's module import logic to embed a new language in its bytecode. These were all avoidable mistakes and I didn't learn anything except that I should have asked for help. Others in my lab, who were less self-reliant, asked for and got help avoiding the kinds of mistakes I confidently made.
With 15 more years of experience, I can see in hindsight that I should have asked for help more frequently because I spent more time learning what not to do than learning the right things.
If I had Claude Code, would I have made the same mistakes? Absolutely not! Would I have asked it to summarize research papers for me and to essentially think for me? Absolutely not!
My mother, an English professor, levies similar accusations about the students of today, and how they let models think for them. It's genuinely concerning, of course, but I can't help but think that this phenomenon occurs because learning institutions have not adjusted to the new technology.
If the goal is to produce scientists, PIs are going to need to stop complaining and figure out how to produce scientists who learn the skills that I did even when LLMs are available. Frankly I don't see how LLMs are different from asking other lab members for help, except that LLMs have infinite patience and don't have their own research that needs doing.
The problem, and I think the article indirectly points at that, is that the next generation to come along won't learn to think for themselves first. So they will on average end up on the 'B' track rather than that they will be able to develop their intelligence. I see this happening with the kids my kids hang out with. They don't want to understand anything because the AI can do that for them, or so they believe. They don't see that if you don't learn to think about smaller problems that the larger ones will be completely out of reach.
We're collectively racing towards the last scene of "Up", justifying each milestone along the way.
I've used this analogy more times than I thought I ever would; no regrets.
We're entering a post-thought era where knowledge itself is devalued and, worse, _seeking_ that knowledge is discouraged.
There's no time to stumble through learning a codebase when Claude can ship features faster than you can think. There's no time to learn.
Pixar warned us. Asimov warned us. Orwell warned us. This phenomenon isn't new; we just now have the technology to finally execute on this vision.
-- William Carlos Williams (quoted in Steve Reich's "The Desert Music")One of the marks of an educated person is the ability to dispassionately think from first principles. It is not a sufficient criterion, but it is a necessary one. In this case, the basic questions we must ask are: what is education, and what is education for?
An instrumentalist view of education, the one that has claimed the soul of the modern university and primary education , tells us that education is about preparing for a career - preparing to be an economic actor - and about the effect you can have. In short, it is about practical power and economic utility.
Now, the power to be able to do good things, to be practically able, is a good thing as such, and indeed one does acquire facility during one’s education. (And I would argue schooling today isn’t great at practicality either.) But the practical, unlike the theoretical, is always about something else. It is never for its own sake. What this means is that there must be a terminus. You cannot have an infinite regress of practical ends, because the justification for any practical end is not found in itself. And if the primary proximate end of education is the career, then what distinguishes education from training? Nothing. What’s more, if you then ask what the purpose of a career is, you find it is about consumption. So education today is about enabling people to be consumers. You wish to be effective so you can be payed more so you can buy more crap. Pure nihilism.
True education is best captured by the classical liberal arts, which is to say the free arts. Human beings are intellectual and moral creatures. The purpose of education is to free a person to be more human, to free them to be able to reason effectively and competently for the sake of wisdom and for the sake of living wisely. In other words, it is about becoming what you ought to become as a human being in the most definitive sense.
What good does AI do you if you haven’t become a better version of yourself in the process? So AI writes a paper for you. So what? The purpose of the paper is not the paper, but the knowledge, understanding, and insight that results from writing it.
Interestingly, the text has a number of AI-like writing artifacts, e.g. frequent use of the pattern "The problem isn't X. The problem is Y." Unlike much of the typical slop I see, I read it to the end and found it insightful.
I think that's because the author worked with an AI exactly as he advocates, providing the deep thinking and leaving some of the routine exposition to the bot.
The framing of the essay around learning through "grunt work" is not deep, it's simply that this specific phrase appeared in two of the sources. Anything that looks like insight is plagiarised directly from the sources in some fashion. I've covered in my evidence the pivot phrases where direct summaries of the essay incorrectly appear to transition to the author's own ideas, but there are parts right through the essay that come from the sources. No deep thinking by the prompter required.
I certainly agree that the author should disclose the use of an AI, how much is human vs silicon, and clarify which ideas are his own and which are not. I've written to him to ask about that.
I don't see how my point about the erroneous date makes your point. He POSTED IT with that date, and only changed it AFTER someone pointed it out, then BLOCKED them.
If he did write it by hand though, that means he is admitting to plagiarising the sources, and also to framing that makes absolutely no sense, like saying Schwartz didn't say something that he literally did say. So, great job I guess!
I fully believe that he used only a general prompt, and anything in the essay that seems specific, Claude has mined from the sources. I am going to try to reverse engineer the prompt.
D. W. Hogg, "Why do we do astrophysics?", https://arxiv.org/abs/2602.10181, February 2026.
I think people are underestimating the level of experience and knowledge that’s required to prompt LLM’s. Not in the micro sense but in the macro. It seems so easy because it feels easy. But if you don’t have deep understanding of the domain, it will just feel impossible. The person next to you with domain experience will say “it’s so easy, look at This simple sentence I typed in”. And be like “it’s just a skill issue, why is everyone struggling so much” and not understand the years of accumulated wisdom or innate talent it took to type that simple sentence.
AI makes the easy things easy and the hard things harder, and more omnipresent.