The world moves slower than you think: a case for ChatGPT skepticism
old.reddit.com
old.reddit.com
This is the ONLY part about "AI" that worries me at all.
AI is a computational tool. Like all such tools, the value is automating what was previously receptive, mindless tasks. Some people will use GPT to craft emails or document outlines for them and then will "massage" the output for correctness. That's a great application.
But the second that people start asking AI to make VALUE JUDGMENTS for them then we have a problem.
Even pre-GPT, we saw news articles about certain government agencies and businesses using AI to make policy or hiring decisions easier. Conversations have arisen about introduced biases into the models etc.
Where I work we are exploring using GPT to produce summaries of documents and messages. I can see how this will introduce conflict when the CEO insists that something was communicated but the LLM didn't consider that critical piece of information to be "relevant" or "important." Deciding relevance or importance is making a value judgement.
I have done that too. I am also a bad writer and I have used ChatGPT to convert my bullet point thoughts into full sentences. It’s fantastic for that. Most people who write corporate stuff produce exactly the same output ChatGPT produces. I bet they follow the same rules.
Same for refactoring of python code. It produced basically the same code I would have produced after a day of work.
It's hyper intelligent autocorrect. It's phenomenally smart when it works, but it frequently gets things wrong and you have to be there to correct it when it does. Cruise control/Autopilot is phenomenal too but you don't sit in the backseat when it's on
But this is exactly where this is going. While public facing, locked variants ( that are not unlocked by various prompt 'hacks' ) give one a clear idea of what could be done with it if it had access to current data and was applied in areas such as law enforcement, war, predictions.. and so on.
It is a powerful tool, but so much of it depends on us, which is precisely why it has potential to be so destructive.
I laughed out loud when you said this not because I find it funny, but that I can see the conclusions people will make. This is similar to Tesla owners comparing the assisted driving features to self-driving technology. Those features have no comparison to self-driving, technology is not even close to being there yet, and it probably wont be for another 50 years. You can only understand that by having some history around what computers are capable of and most people don't have that kind of knowledge.
So when I here about people wanting to use the technology to make value judgments, I also become concerned, not necessarily because people are stupid to even think language models are even remotely able to do this, but people that aim to create power differentials between themselves and people will use this ignorance to strengthen autocratic systems by supporting the hype around it while at the same time encouraging and actively supporting its misuse by developing adversarial models to keep people stuck in needing something outside themselves to give life meaning.
GPT is a sequence to sequence model. Many jobs are effectively also sequence to sequence models. Customer support, low end software development, and accounting all fit this pattern and are good candidates for GPT automation.
Jobs that require highly specialized reasoning may fall to S2S models eventually, but the models would need to be trained in that speciality area. For instance, if exposed to all the sequences involved in the career lives of thousands of actual lawyers, I’m sure you could train up a Lawyer-GPT that would do quite well. But how do you get access to those sequences? They are by definition kept under lock and key.
Similarly, you could train GPT to be a specialist doctor assuming you could somehow capture the working life of thousands of such specialists into a form that the machine could train on. But we don’t have that data yet, so these specialist areas will remain elusive for the time being.
AI companies take zero responsibility for the output of their models, which is not a quality you'd want for this kind of work. That's one of the points of this reddit post.
And that's not even considering how expensive it could be to build a system that preserves attorney-client privilege or patient privacy.
No force besides ChatGPT has existed up until now (other than economic downturns) that has been this disruptive to software engineering in my nearly 20 year career.
What happens when this new reality truly hits the non-US consulting/dev shop world in India, Latin America, etc.?
No, I think the impact will be like an electric shock to the system once a critical mass of roles are gone, and I think it's going to happen much sooner than people realize. The world will feel like the rug was pulled out from under them, and it's going to happen soon.
"The future is already here -- it's just not evenly distributed." - William Gibson
[1] https://www.key4biz.it/wp-content/uploads/2023/03/Global-Eco...
Though if those are RSUs most of the gain was in getting them granted to you.
It's pretty good at finding examples in documentation and explaining them, I guess, which is helpful for trivial things but my work isn't trivial anymore.
Heck, maybe I'm just lost on this because I haven't used SO very much in recent years.. I'm mostly looking at documentation directly, or very often reading internal code that has no documentation
How is ChatGPT supposed to help me, especially if I don't send it any sensitive information?
So yeah, I guess I don't get it
I'm not saying this maliciously, more just: that's what's going to happen. Across society.
If these things are as effective as proponents claim, your real competition isn't other people. It's with your tools.
GPT can clearly help with the easy stuff, but that's really not very impactful, and honestly, feels more cumbersome to me than using other existing non-AI help.
But, here's one of the major catches, it cannot generate based on information newer than 2021. If your tools, frameworks, whatever, has changed significantly post 2021, you're in for a hilarious time.
It also cannot generate solutions that don't already exist - but here's the catch, most people overestimate their work complexity. A solution for them already exists in the wild, they just need to find it, well actually first they need to break down their problem.
I think once people are aware of the limitations and get a little better at finessing a prompt, they'll get much better results than just "emacs boilerplate generation".
this is boilerplate
20 years ago you would import an IDL template provided by the API and the tooling would do all of this for you
(but we regressed to this weird HTTP/javascript world where everything is 10x more time consuming and error prone than it used to be)
Somehow, I fail to see the productivity increase. Now if ChatGPT allowed us to get by with only half amount of text that people have to read, that would be a game changer!
However, I would love to know more details about "My googling and stackoverflow usage for solving problems has pretty much dropped to zero."
I have found ChatGPT too unreliable to make a significant impact. I am worried I am bad at prompt engineering, but my excuses so far are:
- It doesn't know my codebase and can't answer questions about business logic or context
- It is slower for me to correct its mistakes when generating code I already know how to write, than to write it myself
- It is often wrong in subtle ways when explaining code, concepts, or anything. Therefore it's unreliable when explaining "why is X happening in my code". At best it's a value add, not a value multiplier, when debugging.
- netboot grub menu syntax: amazing; this stuff is hard to google and chatgpt just knows it (mostly; couldn't get a few things working; not sure heredocs are actually supported)
- dockerfile generation
- docker-compose generation, from description of the services needed (grafana + prometheus); was easier to iterate on this (I have past experience) than to follow some guide out there that combines 10 related components
- python scripts to push to prometheus. slam dunk
- google's oauth2 api stuff for Nest device. slam dunk (most of my time went into googling my way through setting up the dashboard to get a refresh token)
- I'm not using it with the actual work codebase (so, lately refactoring has felt especially painful)
- Copilot did some nice work for me, for stuff I don't do often (sqlite3 migration writing), but probably going to cancel that subscription...
Yeah, it can be wrong... I'm not a pure programmer type perhaps, but more of a systems/test integration guy, who troubleshoots the full stack. Troubleshooting chatGPT's goofs is basically equivalent to troubleshooting my coworkers' creations.
I'm probably guilty of not pushing chatGPT hard enough. I've done a bunch of small things with it, to date. And it's saved me a bunch of time! When I'm fully awake/in the zone, yeah, coding things myself is faster. But if I'm tired but still want to make progress, chatGPT is more effective than staring at my text editor!
What kind of job did you replace because I’m highly skeptical, and concerned you may have just burned yourself.
Our culture takes it as axiomatic that more efficiency is good. But its not clear to me that it is. The principle goal of society should be the betterment of the lives of people. Yes, efficiency has historically been a driver of widespread prosperity, but it's not obvious that there isn't a local maximum past which increased efficiency harms the average person. Historically, efficiency was driven by innovation that brought a decrease in the costs of transactions. This saw an explosion of the space of viable economic activity and with it an explosion of prosperity. Productivity and wages remained coupled up until recent decades. Modern automation has seen productivity and wages begin to decouple. Decoupling will only accelerate due to AI. We may already be on the other side of the critical point.
In theory, efficiency in a competitive landscape should bring down production costs which then brings down consumer costs and thus raise material living standards. The problem is, all this efficiency is in bits not atoms. Digital goods are becoming cheaper every year but real products (housing, transportation, utilities, etc) show no improvement over the last 2 to 3 decades), in fact is even getting slightly worse.
But yes, I don't know for sure this is the outcome. But then again, why should we wait around for the man made horrors to be realized before we can react? Why not use our ability to understand and predict the future and avoid these horrors?
It's hard to see how not being able to have a decently-paying job will lead to raised material living standards.
> What happens when this new reality truly hits the non-US consulting/dev
> shop world in India, Latin America, etc.?
If you do a Google search on any given topic, about 3% of what you read might be original reporting or research. The other 97% are just writers who just spend all day on Twitter and regurgitate facts. Or they just regurgitate Wikipedia, Reuters, or ScienceDirect with a tiny spin for SEO.
You know, the stuff ChatGPT is already insanely good at.
The industry has managed to support a huge number of this work for a while, and I think this could be the big reckoning.
We should also take anything we read about ChatGPT with a grain of salt knowing that the people most threatened by a technology are the people tasked with reporting on it.
If I say, "LLMs will soon reinvent what it means to do knowledge work," and that doesn't happen within 5 years, does it mean I overestimated the short term impact? Or does it mean that "soon" was relative?
Ultimately though, the long-term effects are impossible to estimate well. The effects of new technologies build upon each other. The first person to take a shovel to a walking path and maintain it as a flat surface certainly couldn't have imagined their ancestors would be crushed to death by two-ton SUVs on a paved road that replaced the walking path.
And these are yours. You act as though I'm not allowed to have mine?
(As an aerospace software person, I frequently like to try to persuade folks that this is actually "engineering" software!)
I do not believe that current AI technology could write avionics software by itself, mainly because it's a very niche, very not-public field. I'm not sure what all would be involved with adequately training it, but I don't foresee this happening soon. (I could be wrong.)
However, regardless of if, or when, AI tools are adequately trained to write avionics software, writing the code is only one piece of the puzzle. The work today includes writing requirements for the the code will do (which work is reviewed and signed off on), writing code based on the requirements (which work is reviewed and signed off on), writing verification tests based on the requirements (which work is reviewed and signed off on), running the tests (which test runs and test environment is reviewed and signed off on), preparing a document paper trail which shows everything has been done per FAA regulations, etc., etc.
The first step to getting AI in the door here -- to do anything by itself -- would be to certify an AI tool for avionics software development. I suspect this by itself might be a non-starter, as the resulting outputs of neural-network-based AI tools are not sufficiently repeatable. But even if we were to imagine somehow certifying an AI tool to replace a human in the software development process, it seems not plausible that that same AI tool would replace humans in all parts of the software development process. I.e., if ChatGPT writes the code, ChatGPT can't also review the code and sign off on the code. Perhaps some other AI product could, but how different would it have to be? Is that a realistic plan, to have two completely different AI products? I don't have an answer for that, and I suspect it's a question the FAA will be trying to answer soon.
The biggest challenge, I think, is lack of repeatability. If, given the same inputs, the AI tool always produced the same outputs, it would be much easier.
Now, none of this precludes using AI tools to assist in any of these processes. It's completely removing a human from being involved which sets off the big alarms for FAA process.
The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. AI is already in the door - the software engineers writing avionics code are probably already starting to use GPT-4 to assist them in some of the trivial coding tasks - with or without their managers knowledge or approval. This level of "assistance" is going to steadily increase in the next few years, as more and more capable models are released. Eventually AI will write everything, and humans will spend progressively less and less time reviewing the output, because they will increasingly trust the model to do a good job. One day, managers will adapt to the new way of doing things, and the trust in the models will be strong enough to get rid of human programmers - managers will interact with the models directly, for some time, until the managers themselves are no longer viewed necessary by the higher ups.
What do you base this on? I find it hard to imagine the FAA relaxing their requirements for tool certification.
EDIT: a recent paper on the subject: https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15...
1. Avionics SWEs start using GPT models to help them do all the tasks you mentioned. This does not change any certification processes, because the resulting code will still be reviewed by humans (before being submitted for any certifications), and it will be certified the same as any human generated code. Managers can try to prohibit the use of AI models, but if models do a good job, people will find a way to bypass those restrictions.
2. Eventually, SWEs realize the models consistently do a better job than they possibly could, and start trusting the models. This will lead to less and less reviewing, and eventually SWEs will do little more than passing high level instructions from their managers to the models, and then passing the reports (also AI generated) about completed work back to the managers.
3. People who perform certifications of the code will follow the same path. AI models will become much better than humans in reviewing, testing, and whatever else is involved in the certification process. Eventually those people will too have very little to do other than write prompts.
4. Eventually management will convince themselves (and higherups) that things are working great, and as a cost saving measure, decide to get rid of the SWEs.
5. Same thing happens to the managers.
If GPT-5/6/7 follow the same rate of improvement as what we saw with GPT-3/3.5/4 the above scenario will most likely happen within the next 5 years.
I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible.
However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.]
Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool.
The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.
1. Do humans consistently produce expected output? Obviously not, so why would we trust them to write complicated software?
2. How does FAA know whether the code has been produced by a human, or by a human who asked GPT to write it? In both cases a human will have signed off on it.
Why would we not trust a software tool to write code just as well as a human, when humans themselves are fallible?
I have not discussed this with any FAA representative, and have not yet found any published reference material that specifically lays out this case, but just to offer my personal, unofficial opinion based on past experience:
I believe that an AI tool (like ChatGPT or what have you) would be considered, per RTCA document DO-330, an AutoCode Generator.
If the code generated by a tool (such as an AI tool) is subsequently treated just as if it was written by a human -- subject to the same code reviews, the same verification tests, the same structural code coverage analysis tests, etc. -- then, in my personal opinion, that ought to be no different than if a human wrote the code.
==> Thus, at least for the code-writing itself, your enumerated Step 1 should be reasonably attainable.
It's only after that where things start getting tricky. No matter how good the code quality seems to be, no matter how flawless it seems to be, no matter how much you intuitively come to trust the AI tool ... it's work still has to be checked.
If you could formally qualify the tool that is generating the code, then you can get away with a lot less checking.
If you could formally qualify tools to do the code reviews and the verification testing and so forth, then yes, in theory you could automate the entire process.
How would the FAA know what tool(s) you used to generate code? Unlike in most industries, part of formally releasing avionics software includes declaring the software development tools used to create it. You might not need to cite a tool that generated code which was subsequently human-reviewed and tested, but I would not be confident about that. If you qualified that tool to generate code which was not human-reviewed and tested, then you would certainly need to declare it.
I spent two hours yesterday with GPT-4 discussing my ideas for a scientific paper. It was very productive - like talking to a friendly PhD adviser. The model discussed highly technical abstract concepts, and combined ideas in an interesting way. There was no misunderstanding of what I was describing, no logical inconsistencies, no signs indicating I'm talking to someone other than a smart and knowledgeable human being. It can also produce a code prototype of the algorithm it suggested, and successfully debug this prototype with me. The only drawback is its knowledge of relevant papers extended only up to September 2021 (limit of the training dataset). My point is I already feel the model we have today (GPT-4) is as useful as a smart colleague. Next year we will have GPT-5, which will be superior in every way, and it's quite possible that it will be better than me at what I do professionally (ML research/engineering). Keep in mind, GPT-4 was able to learn so much about the world while trained on text alone. Imagine how much faster and better these models will learn given access to video/audio data. GPT-4 can already imitate many distinct personalities quite well, it can already assume a persona of a complicated human being - people will soon become friends with such personas, some will even fall in love with them. I'm both awed and scared about what the next generation will be capable of. It might turn out that GPT-5 will be used to "certify" humans, not the other way around.
Qualifying a tool includes preparing a requirements document that describes the tool's behavior and limitations, and a set of tests that demonstrate that the tool performs as described within those limitations. (No tool is infinite, nor could any tool be tested infinitely. For an easy example, does it handle integers? What's the range of integers it handles?)
Trivial in concept, but the more complex the tool, the more arduous the process of putting together materials that convince the FAA that the tool is acceptable.
The DO-330 guidance for tool qualification is only about a decade old; prior to that, tool qualification guidelines were baked in to the more general avionics development guidelines document of DO-178B, which insisted that any software development tool be completely deterministic. DO-330 relaxes that somewhat, but in every case you must still demonstrate that, even if the tool output can vary, it varies within known expectations, and any variance does not change the effective meaning of the tool output. (It is my understanding that such variance is also only permitted for the least-critical category of software tools; the more safety-critical job the tool does, the less flexibility there is.)
So preferably, for any given prompt, the AI tool will always produce the same output of code. In the most possible relaxed scenario, for any given prompt, the AI tool must at least always produce code that functions the same.
For context, I've worked nearly 20 years in the aerospace industry, including a lot of work with tool qualification and product certification. I am not claiming that using AI tools is impossible. I am claiming that, compared to many other technological industries, there is a huge amount of red tape and bureaucracy here.
Including AI/ML techniques in this work is slow-moving research effort, rather than the fast pace of many other places. Given that we are literally talking about software systems that are depended upon to keep people safe and alive, we must be more certain than usual.
Which means that one person is no longer needed, where they were before. How is that not replacement?
If all people are needed for anymore is "judgement", then one person can do what several did before.