Society's Technical Debt and Software's Gutenberg Moment
skventures.substack.com
skventures.substack.com
It seems to always just erase the file, because it declares `keep_emojis` to be a list of one element (`keep_emojis = ['¯\_(ツ)_/¯']`) and then goes USV-by-USV through the file to check if the encoded/decoded coded char is "in" keep_emojis. And there's basically no way that any single char can be. So the output is always empty.
If we wrote it like `keep_emojis = '¯\_(ツ)_/¯'` instead, then it would be a little closer, but in that case the program ends up preserving all non-ASCII characters (including all emojis), because these characters become the empty string after emoji.encode('ascii', 'ignore') and are therefore always "in" the keep_emojis string, so they never get replaced.
The explanation also doesn't make any sense, where it writes: "In the above code, the assumption is made that any ASCII character that cannot be encoded using UTF-8 is an emoji."
That's silly, because every ASCII character can be encoded in UTF-8. It also doesn't correspond to what the code is doing -- what it's really doing (I think) is testing for USVs that can't be encoded in ASCII, but then throwing the char away no matter what.
And, of course, the vast majority of non-ASCII coded characters in Unicode are not emoji, so this is not really a good way to satisfy the user's request. The roundtrip through ASCII seems like a bad idea -- if the user wants you to preserve some emojis while getting rid of most emojis, converting to ASCII (where there are no emojis) doesn't seem like it will be helpful.
Of course it's amazing that ChatGPT is already good enough to produce something close enough to fool venture capitalists into trusting it without trying the code, and these techniques will doubtless continue to improve.
They have no concept of truth, and so will gladly generate tokens that are "likely", but are at best clearly wrong, or worse look correct to a non-expert, but are subtly wrong.
With software at least you can run it and inspect the behavior for flaws. Or have the results be validated by an actual programmer for correctness. And hopefully the developer using the LLM understands the limitations, and understands how it can hallucinate incorrect but convincing results.
However even software engineers I know, who are aware of all of the above issues, will ask ChatGPT questions about fields they aren't familiar with and view the results as authoritative. It's the same effect as when you read a news article about a topic you're familiar with and can immediately see flaws, shortcuts and exaggerations; but then you turn the page to a topic you aren't an expert it and say to yourself "what an interesting article".
The example from the article is even worse, since the author asked for an explanation of the code, and it hallucinated a clear and convincing but completely wrong explanation. Again to a programmer it is obvious, but to a layperson it's actively deceptive.
I could easily imagine a scenario where some asks ChatGPT, what is the best combination of chemicals to clean mold from a ceramic surface, and it responding with "Bleach and Vinegar are the perfect way to clean a ceramic surface." and following up with an explanation like "Vinegar de-greases the surface allowing the the disinfectant power of the bleach to penetrate into the mold". Which all sounds reasonable, unless you know beforehand that those chemicals mixed produces toxic chlorine gas.
Now that was definitely a contrived example, but unless you know and constantly question the output of an LLM you could easily be misled. Especially if you are asking about topics outside of it's training data or with minimal training examples.
Maybe with enough training data or some combination of transformers with reinforcement learning that has a "truth" metric, hallucinating completely incorrect information can be reduced to an acceptable level. But at this point it seems intractable.
Now will it write correct tests? Maybe?
My personal feeling is that an LLM is insufficient for strategy and is not the only technology suitable for implementing the tactics. I think it makes more sense to treat each as a module in a system, and build different modules to compete against each other.
1 - They aren't optimized for truth, they are optimized for best appearing answers.
2 - What things do you know that you are willing to state on HN without being fear of being contradicted?
I don't understand the second question.
Restated thus:
The quickest way to falsify something is to state it as a fact here on HN.
>> This actually gets at the heart of the problem with the current batch of LLMs.
Here you are implying that the fault lies [solely] with LLMs.
>> They have no concept of truth, and so will gladly generate tokens that are "likely", but are at best clearly wrong, or worse look correct to a non-expert, but are subtly wrong.
Hear you are asserting that you have the means yourself to reach truth. This is extremely easy to do if you're speaking abstractly, but try executing that at the concrete level (as above) and it's pretty difficult to avoid imperfection.
Regarding the difficulty of truth, is the problem here entirely with LLMs, or is the problem with reality itself?
When ChatGPT (or anyone/anything) says something "is" true (and some people agree) someone else says it "is" not, how are we to decide which is correct?
Where does this "is" that people "are [only] perceiving" come from? Where, and what, "is" "reality"?
Is the universe equal/identical to reality? That's how a lot of people talk...but is it true?
INB4 "we could all be brains in jars", which is one of the most common responses to arise "purely by coincidence" when this topic is raised.
VC funding is centered around making high risk/high payout bets in quantity. If they took the time it would take a functioning prudent investor to carry out due diligence, the startups would have already used up their runway and died.
VCs trade risk for reward, and make it up on volume. Fooling a VC isn't a high bar in the eyes of this Midwesterner.
The people writing this kind of blog either don't have the ability to understand, or didn't bother to check, that they just fired their software development team and replaced with with the software equivalent of a cargo cult. Just because the code highlights in the IDE does not mean the revenue will come.
The crazy thing is that this is super common in many posts about LLMs! Here's another example that was recently on HN:
https://news.ycombinator.com/item?id=34473783 - as called out here https://buttondown.email/hillelwayne/archive/programming-ais...
What I think the author wanted was to remove ASCII emoticons, because there is no such thing as ASCII emoji. Another confusing thing is that good 'ole shruggie (which in this form I'd classify as an emoticon) is not pure ASCII, and relies on Unicode codepoint 0x30c4 for the smiley face, and a couple more besides.
The author did such a poor job of describing what they wanted, it's no surprise the generated program is nonsense. If anything, I'm now less convinced that non-technical people will be able to use conversational AI tools to usurp my hard-earned societal role as a technologist.
It correctly figured out the problem and fixed it - removing only unicode emojis, not all the unicode characters.
Then I told him: „but it still didn’t remove things like :) :/ xD” (not specyfying that I mean non-ascii emoticons, not emojis really).
It replied by saying I meant emoticons and writing regexp for those and other ascii emoticons.
I believe this will cause an increase on the trend of technical debt.
There are several talks/papers/blogs on lines of code being a cost/liability and not a product/asset. (this blog post seems to assume the inverse)
For that reason, I could see AI-generated code leading to more tech debt since its output is, as the authors put it, “cheap enough to waste”. People might get the idea to just have it crank out AI code snippets with no real bearing on what they actually do or no real care to refactor said snippets for reuse / readability until their house of cards collapses.
https://github.blog/2023-02-14-github-copilot-for-business-i...
So we get about 2-fold increase on code to debug and fix later.
If you can't create the code to begin with, what hope could you possibly have of supporting it?
Writing code, like offering a product/service in the economy, is game theory. After a period of LOC / copypasta increase, there will be more need for development refactors.
Personally, I’m trying to use LLMs to learn to upgrade my knowledge of abstractions and getting experience with refactoring using CodeMods /JsCodeShift.
Maybe the bots will start to get better at deleting, but generating more code is not necessarily a good thing.
One important aspect of software is that it's not much easier to read than rewrite the code. Currently AI generated code needs much reading and checking, and some rounds of chats to iterate. I am even not quite happy to read humans' code, nevertheless AI's.
Only when one day AI can code that needs little human intervention will the article mean something to me. But is that ever possible? The crux is that a software bug is not in the code, it's the mismatch between the code and our mental model. Programming language is a precise way for us to present the mental model. Without a program, how do we tell AI precisely of our mental model? By natural language we can never do that. Even we can do that, we're only repeating a program in a extremely unconvinent way.
I do agree that AI can help developers spend less time on boilerplate and creeping in documentations. It can greatly improve productivity of the software industry.
99% of the code that you can generate via ChatGPT can be already be found in the form of an open source library, already packaged and ready to use. The number of libraries to solve various problems continues to grow each day.
Many of the questions you can ask to ChatGPT, if not already present in a project documentation, have been already asked on Stack Overflow, mailing lists, etc. and indexed by search engines. Typing a well formulated question in a search engine may likely take you to a valid answer these days.
When taking that into consideration, that leaves us with a very different narrative. LLMs can make a difference, but software engineers have been already reusing solutions to save effort for many decades now.
There are already programming languages that are close to natural language such as SQL and BASIC. Did the release of those mean the end of software engineering as an occupation? no. Each time programming gets easier, the result is often more programmers.
Finally, there are many expert developers on freelancing sites such as Upwork that you can hire, are "generally intelligent" beyond ChatGPT, and perhaps will cost you less than paying for OpenAI tokens. Still, organizations choose to hire full-time software engineers.
Many people are already using automated tax preparation tools. Yet tax preparers are still needed.
There are already customer support chatbots. Yet people often want to talk to a representative.
A smarter versions of those will probably be game changers but for high-stakes situations you will still want to deal with a human expert.
If you have an infected limb that is at risk of being amputated, will you go see DoctorGPT? Hell no.
Chatbots for customer support never worked right, so it wasn’t a choice either. With gpt-powered bots, many - if not most will choose that over calling a helpline.
The job of lawyers and doctors are less predictable than software engineers? I don't believe that for a second.
If AI comes for SWE jobs it's coming for a hell of a lot more white collar jobs as well.
Generative AI throws the switch in reverse, because it's indifferent to standardization: people are formulating questions about data and how to process it in their native tongues, skipping the encoding process and the need to study documentation altogether. There isn't a point where it becomes, "oh, but then you hit a brick wall and have to learn the real stuff to solve your problem". GPT helps relatively less as you go deeper and further away from a SO-type answer, but it's a gradual decay.
That graph places software engineers in the top-right quadrant, suggesting that they are predictable and "grammatical". However, I believe there should be a distinction between writing code and engineering. I would not classify software engineering as entirely predictable. For instance, while you could use a script to handle emojis with ChatGPT's help, it would be challenging to use the same strategy for a solution involving large systems that have multiple failure points and span multiple regions. Moreover, even if such a system were successful, multiple redundancies would need to be in place to prevent large-scale disruptions caused by downstream failures.
On the other hand, the chart places executives and investors on the 'safer' side. However, I think that with the real-time information available to AI, investing as a 'job' could be at a higher risk of being replaced. The same could be true for many executive positions (assuming that we are comparing merit-based positions to those based on nepotism).
"Soft skill" jobs like executives and managers are definitely on the chopping block first.
Wouldn’t that be some delicious irony? Never gonna happen but it would be quite funny if it did.
I think the "ladder climbers" aren't going to be fully replaced, but their ranks are already getting thinned, too.
Someone should make an AI-generated crypto-influencer persona. Virtual YouTube, Twitter presence. Mint a new coin once per week, hawk it to the masses. I bet they could make decent money.
I am afraid that as long as humans are writing code there will be other humans in charge of ensuring their work is profitable. We won't be able to do away with executives just yet, unfortunately.
However, let's take a step back and consider a few basic arguments that this article makes - I suspect most here would only disagree with the very last.
Firstly, the demand for quality software is far greater than the current supply. If there were suddenly twice the number of mid-level software engineers, there would still be more than enough work for them.
Secondly, software engineering is a process that can be refined and created almost entirely without the interference of non-deterministic real-world systems like roads, weather, or courts. This makes it an ideal field for automation and AI to play a larger role.
Thirdly, a sufficiently intelligent computer could exponentially increase its own efficacy in performing purely-digital tasks -- basically an [intelligence explosion](https://www.lesswrong.com/tag/intelligence-explosion) but much softer and much much more achievable (how smart is a mid-level SWE, really?).
Finally, LLMs that get a ~100 on an IQ test are enough to start that cycle.
Perhaps you're super duper convinced that the last point is wrong. Perhaps you have strongly-held convictions about explainability, symbolic reasoning, higher level thinking, etc. But if you really sit down and think, what are the chances you're wrong? What are the chances that we get another leap or two in the next 1-5 years like we just got with DL transformers?
If you're feeling excited and anxious about the implications of this, you're not alone. I've found that it's a difficult topic to discuss with those close to me, especially if they're not familiar with the latest developments in AI. If you have thoughts on how to use our SWE experience to navigate this exciting but uncertain landscape while maintaining a sense of self-preservation and helping others as much as possible, I'd be interested in hearing your ideas.
Sell magazine subscriptions door to door?
The second point is wrong. You can automatically synthesize code from a specification (and you could already do so before LLMs, arguably even better with SMT solvers), but who is going to write the specification in the first place, buddy?
That's the problem that's never been solved, and it will never be solved because it can't be solved. It is not easier to write a specification in English than it is to write it in C++. It's like saying that math would become easier if we took away the burden of using mathematical notation. Not only it wouldn't get any easier, it would get significantly more difficult!
Lies are dangerous, because if you say too many of them, you start believing your own. "AI is like a human you can talk to", they said. Now they act as though that's true, and they talk to the AI, without realizing it's the same faithful mechanical slave it has always been, perfectly capable of carrying out nonsensical instructions to their nonsensical results.
And the instructions are necessarily nonsensical, because they insist on using an interface too wide, where saying something precise is all but impossible.
Yes, a.k.a. programming.
If you assume that you have a machine powerful enough to do that, then you have a machine powerful enough to do anything at all: you are assuming the thing you're trying to prove.
There exists no universe where a machine that can do whatever you want by being instructed verbally isn't also replacing the one instructing it...
In terms of AI taking ALL THE JOBS... Maybe it will happen? Maybe it won't. But what action can we take from all this doom-scrolling? Other than "we should all be VERY CONCERNED." Which is just a way for us to stand around wringing our hands.
My conclusion is much the same as always: I can't grow complacent. Which we've known for years. "The industry is changing." Of course it is. It's been changing for ~80 years. And if you go back to similar advances, you'll see similar arguments.
Will this time be different? Maybe. But until it does, I think I'll continue to learn new skills (not just new languages/frameworks), refine my problem solving and troubleshooting, and above all, refine my people skills.
But I think I'll avoid the doom-scrolling. Either AI is coming for ALL THE JOBS or it isn't, but even knowing that it is doesn't give me anything actionable to do about it.
I’m hopeful AI can help all of us really tackle legacy software - rebuild it to be simpler, more secure, well tested and easier to maintain and iterate.
I kind of view companies and governments as super intelligences unto themselves. Capable of doing things individuals are not. That is what AI will be competing with and/or augmenting.
Literally just last week I was showing several dev teams how to cut days out of their troubleshooting time by using an APM that can capture debug snapshots from production. Combine with source indexing during builds and code versioned with the Git commit id and you can jump to the line of code with the issue in about a minute. You’ll see the stack variables and everything. This is possible even in a distributed microservices monstrosity.
It takes like a day to wire this up.
“We’re too busy for that today, there’s an issue in production!”
But the reality we've discovered is that there's a constant gravitational force pulling most dev orgs to keep increasing complexity, forever, much faster than the actual product increases in complexity. We can avoid this at the individual level, maybe the team level, but at this point it feels unavoidable at the company or industry level
Though AI can help people understand and locate code snippet, which is very helpful to newcomers to a project.
I myself have often wondered if what doctors do couldn't be automated. You have the primary care physicians who seem to run on a loop of "listen to symptoms" - "order tests" - "prescribe medication" - "refer to specialist". And the specialists themselves seem to follow their own loops. Many surgeons for example, perform the exact same procedure day in/day out for decades. Surely well-trained robots, not prone to fatigue or loss of dexterity as they age, could produce superior outcomes.
But they haven't been replaced yet, just like I haven't, so I assume there is more going on there than what looks from the outside like repetitive manual labor.
Doesn't have to. It just has to make it easier for a developer to do, say, 1/5th the work to get the same results he gets today doing it alone.
This is like a tick above the fucking whitepapers that crypto frauds used to have to sell their coin.
The complexity inherent in software development reached a minimum in the era of Visual Basic and Borland's Delphi, and has rapidly increased since then in the abandoning of the Windows Desktop and the Win32 UI as a standard and fairly reliable platform for applications.
It was possible to build and deploy applications in a rapid enough fashion that agile just happened without prompting in many one-man development projects. In fact, many domain experts were able to craft their own reliable and usable (to them) applications that remain in use to this day. That world existed, and has largely been abandoned.
I suspect what AI tooling will enable in aggregate (if we're lucky) is a return to the levels of productivity we previously had, despite the multiple new layers of interfaces and abstractions inherent in the non-Win32-GUI world we now inhabit.
Thus, I feel if you want to see how AI/LLMs are going to effect the economy and our profession, look to the effects of the Win32 desktop on software in terms of size and scale of disruption. History doesn't repeat, but it does rhyme.
-- EDIT/APPEND --
in an era of "free" debt, where the interest is effectively zero, management learns to spend money like crazy, and let the debt stack up, because it is the efficient thing to do
When that management mindset is mis-applied to understanding when the programmers say "we've got tech debt we've got to pay off".... they incorrectly assume it has no cost in interest, and can be ignored, as with money debt...
except it does have interest.... loan shark levels of interest
I know ChatGPT's statistical plagiarizing can do better than this.
Graphs of complete imagination, not based on any data at all.
Comparisons of apples and exaggerated notions of oranges.
Ungrounded, magical claims about LLMs.
This is unhinged fantasy.
> Again, programming is a good example of a predictable domain, one created to produce the same outputs given the same inputs. If it doesn’t do that, that’s 99.9999% likely to be on you, not the language. Other domains are much less predictable, like equity investing, or psychiatry, or maybe, meteorology.
> Entrepreneur and publisher Tim O’Reilly has a nice phrase that is applicable at this point. He argues investors and entrepreneurs should “create more value than you capture.” The technology industry started out that way, but in recent years it has too often gone for the quick win, usually by running gambits from the financial services playbook.
This is just getting started. Large language models coupled to more structured systems are going to be very powerful agents.