ChatGPT: The end of new programming languages?

Possible - here's a counterpoint: ChatGPT will make it easier to make new programming languages because all you'd need to do is give an AI your docs, and it'll be able to figure out how to re-write things in the new language.

I'm not sure what I agree with, but I think it's interesting to think about.

My off the top of my head guess is it makes it easier to make new programming languages and have them be usable by people, but it reduces the value of new programming languages. So both things happen. We end up with more new programming languages because they're easier to make and use, but more of the usage concentrates on the existing languages and the ones that ChatGPT is best at. So both a bigger long tail, but more concentrated 'big winners'

This is my guess too. Programming languages currently have to make tradeoffs around cognitive load and the ability to fully reason about effects of statements. An AI assistant with stronger reasoning ability might allow the creation of more powerful languages than we're capable of using by ourselves.

(If this sounds dangerous to you -- to use a technology that humans aren't smart enough to understand ourselves -- I guess I'd agree.)

Yep. The next big programming language will be a variant of English, or other human language of choice.

You can give it a BNF grammar for the target language of choice, if you like. Not only is C/C++ the "new assembly," but every other high-level programming language can be thought of that way as well.

I asked the free version to make a basic grammar for SQL SELECT statements that is compatible with Oracle and DB2.

It replied chirpily that it could do that, in so many words, and output a grammar (which I didn't test) that solely parsed MySQL identifiers rather than the double-quoted kind. It wasn't precisely an invalid response, but it was clearly inappropriate for a human-like reply and showed what I consider a qualitative absence of consciousness.

I don't know why, but this seems to be a pattern with SQL - I have never asked it about MySQL, and indeed MySQL results tend to pollute my Google searches, but there may just not be enough other training material.

It's starting to aggravate me every time someone concedes it only is useful sometimes, and you have to check it, because it is starting to seem like unwarranted "praising with faint damns".

But ChatGPT didn't get good simply from reading docs.python.org. While I'll be the first to argue that it does things that resemble reasoning, it's limited context window means it doesn't sit there and stew on things during inference. It got good at Python during training because of the reams and reams of Python code out there.

The reason I say this is because with something with fewer examples but still with docs available online, specifically, AutoHotKey; ChatGPT's not nearly as good.

We'll get to where you're going, but we're not there quite just yet.

We're not far off. I'll often copy and paste docs examples in and it will work its magic pretty well

You’re copying and pasting the docs of the things for which it has seen millions of lines of code already. It’s not making the inferences off the docs.

OP is saying that without this initial bootstrap corpus, this won’t work well and hence people accustomed to this won’t adopt the language.

No. I'm pasting docs of stuff not used in the training data. Otherwise why would I paste the docs in?

Languages are mostly not designed or made to be generally better than preceding languages. They are usually designed to support a specific need at the time.

> the new language will be at a disadvantage even if it's intrinsically superior.

Languages are not usually compared like this.

No one seriously designing new languages is thinking "gee better not do that because of large language models prior training".

Even amongst existing languages, the level training among large language models isn't a deciding factor in writing production code more so than if the language is suitable for the problem domain.

My hot take: it’ll lead to a new level of programming language.

Right now we’ve got our low level stuff, machine code where you’re telling the CPU what to do. Then C etc: want to read a file? You’re manipulating pointers, reading bytes etc etc. Then we have higher level languages where we just call readFile(). The standard library “knows” what we want to do when we say “read file” via the code contained within it.

I can imagine AI enabling an extra layer. We’ll say “format this string as a US telephone number” and it’ll make “XXX-XXX-XXXX” out of it. But it isn’t going to anywhere near as exciting as the current hype wave makes it out to be because if we want to get unambiguous responses that don’t make unwanted assumptions our prompts are going to have to be really, really specific. The most efficient way of doing that will probably be via some kind of structured prompt language. So… another programming language. It might look very different from the languages we use today (or it might not!) but it’ll be a programming language all the same.

I’ve been in the camp of chatGPT being just a layer on top of google search - but I like your idea way more. A structured prompt language is a really interesting perspective.

this makes sense as well, we already see some form of proto languages forming in stable-diffusion based communities.

I’ve fed chatGPT the specs for some toy languages that it didn’t know and it generated code in them no problem but did a lot of filling in the blanks for what it thought would be in the standard libs.

Something more interesting to think about is what would a programming language designed FOR llms to use look like?

Would that be good feedback then? To implement those features into the library because ChatGPT expecting to be able to use them may be akin to it expecting a human to likely use them. Or not really?

In most cases it made perfect sense. One joke language had no for loop by design, and just a while type loop and chatGPT invented a for loop, but when I told it to stick only to the spec it completed the problems without an issue.

When web browsing was enabled it was able to learn these by itself.

English, of course. really, really clear English.

No, then we’d need to write an English compiler, I’m talking about an output language.

I've had the same thought.

On the bright side, this is not a new phenomenon. The number of "things" that a new programming language needs to provide that are outside of the language itself has been growing for a long time now. A new language already isn't going to find purchase if it doesn't have a package manager, a build system, a hosted package repository, a standardized pretty printer, a test runner, a documentation generator, a StackOverflow presence, an active community on both synchronous and asynchronous communication platforms, and IDE support. Needing a trained LLM "just" raises the bar for IDE support. However, unlike all of the other "things", this is something that can't really leverage prior art or standardized protocols.

Of course, even to this day there are some people who program without IDEs at all. In order to bootstrap their own LLM training data, new languages will need to fiercely compete for this ever-shrinking segment.

IMO this is right and wrong.

ChatGPT does not understand the hard parts of Rust I read somewhere (no evidence).

But in the longer term it’s more likely that programming will morph into something much more inherently linked to AI.

People said computers and computer games would be the death of board games, but instead board games have exploded in popularity, I think mainly cause it’s easier to create fabulous board games with computers.

New technologies tend to be overestimated in the short term and underestimated in the long term.

I think LLMs will interact with programming language design, and software design, in a way that will likely be significant but is currently very difficult to predict. Sort of a singularity moment for software, where we're not sure what comes next but it does seem likely significantly change.

From my own experience, writing software today doesn't that much different than it was 20 years ago using some java IDE from that era (Borland? maybe I'm remembering it better than it was). I can imagine what development will be like next year with increasing integration of copilot like tools, but I can't imagine with any confidence what software development will be like 10 years from now.

I too had fond memories of JBuilder until i tried to use it … just not good at all. As bad as eclipse was, it was still far superior to Borland’s products (20 years ago). Funny the rose coloured glasses we put on sometimes.

I think not much will change in short term.

I can't imagine every programming practitioner is going to switch to AI tools even if they get a lot better than they are now. There will be contrarian people making new languages and writing in them. Some of them might catch on and become popular.

Also, it seems that the AI tools don't suck that bad even for made-up toy languages. Maybe it'll struggle if the new language is using lots of unconventional ideas.

ChatGPT will not innovate for you. If I'm a researcher for new programming languages, I can't imagine it would be all that useful for me in the research part. Maybe in the implementation part. I certainly wouldn't stop research just because AI is suddenly a thing.

Also, it seems like finetuning is pretty cheap, I expect it might not be very difficult to add knowledge about a new programming language to an existing foundational model made for code. The tools to do this will get better and more accessible.

I think my hotter take is that new programming languages that are really nice for humans are perhaps also really nice for the AIs. So LLMs being a thing wouldn't necessarily change how you design programming languages.

And my even more hot take is that I think the impact of these tools is waaaaay overestimated for now.

If some absolutely, obviously amazing programming language is suddenly invented that is only usable with AI assistance that could change things. I am not holding my breath though.

This was an issue way before ChatGPT, because older languages have more online resources, more experienced developers, more libraries, and are more battle-tested and hardened. It’s why JavaScript and FORTRAN (!) are still alive today, and you rarely hear about new languages gaining mass adoption. The newest “production” languages today (Dart, Go, Rust, Swift, Kotlin) are all not actually new (some started being developed 10+ years ago), not as widely-used as their replacements (React, C, C++, Objective-C in Apple’s macOS and iOS, and Java), and this is despite being spearheaded by huge companies (Google, Mozilla, Apple, JetBrains) and a passionate community.

Still, the # of software devs is huge and increasing, and there are plenty of new indie projects made by people who are passionate about new languages and/or want to do things more fun and possibly faster rather than more reliable. Take Zig, a very early language which competes directly with Rust, C, D, Go, and more; but there are real projects being developed with Zig (e.g. Bun). These projects and the developers using niche languages are growing a community which others can use in the future, and writing code which an LLM can use; and while the communities aren’t growing nearly as fast as ones of larger languages, there’s probably an exponential dropoff of community/code growth + usability growth. More importantly, even if these niche languages never become popular (more likely than not), they’ll still be used by a passionate group who make projects 2x slower (since they have less resources/LLMs), but would otherwise be unmotivated and not make anything at all.

I think you wife is wrong for the right reason. Her observation is astute, but is essentially the same observation that network effects impact all programming languages. E.g. there are more JS libraries so people are more likely to use JS, etc.

But, ultimately, new products, programming languages included, succeed because they satisfy needs not serviced by the existing incumbents. And I don’t see that changing.

Spreadsheets are similar to ChatGPT.

I think most devs have a hard time grasping how revolutionary the spreadsheet is for non programmers for accounting, etc tasks. Yes they get abused, but its a huge powertool that solves a lot of tasks that previously required teams of programmers.

Looking back, we might wonder how spreadsheets impacted programming languages?

GPT-4 doesn't need training to learn a new language. You can just define it in the prompt.

But if training is used, I think you would want to define a full self-contained platform. That way you know the LLM knows everything it needs to translate the user's requirements into code.

A group like DeepMind might go as far as to integrate code execution into their system somehow. Maybe it's like an LLM but trained to iteratively write code for some self-contained platform.

Or there could be a full platform designed so the specification fits well-within a context window.

Something like K or oK is very compact. You could give it a library to save data and display some UI stuff and define the whole thing in a context window maybe.

Here's a different hot take: GPT/LLM/AI will eventually make all languages obsolete. You tell it what you want and it'll just output machine code directly.

See, this is the sort of hot take that you make if you haven't done either law or philosophy.

Using natural language, it is incredibly difficult to make unambiguous statements. For most of the twentieth century, the problem was regarded as so severe that analytic philosophers would encode their most important points in symbolic logic. And it's not like we have one symbolic logic here; there are literally dozens of higher-order logics, each with their own syntax, and some of which are mutually incompatible.

Meanwhile, complex, difficult-to-master conlangs such as Ikthuil and Lojban have flourished in pockets of the philosopherse simply because they provide, essentially, the kind of clarity that any basic programming language provides.

I can easily imagine AI/ML inertia, as OP wonders about, and I can also easily imagine forms of e.g. Lojban, Ikthuil, or a mishmash of several higher-order symbolic logics becoming the basis for a new ML-informed programming language, but if your imagination has you talking to the computer the way you'd talk to a subordinate, in English or some other natural language, then do I have a monkey's paw for you.

Indeed it's hard to make unambiguous statements. But somehow business leaders do, and subordinates are able to make reasonable assumptions, ask clarifying questions and produce something that approximates what the business requested. Perhaps riddled with bugs and way past deadline, but ultimately something that produces value.

If you don't think that's ever going to be possible with AI (notice how I used the word "eventually" in my post) then I don't know what to say to you.

A big reason we have programming languages is because natural language is often ambiguous. ChatGPT does nothing to alter that.

It's like the joke:

The programmer's wife said: "Please go to the store and buy a carton of milk and if they have eggs, get a dozen."

He came back with 12 cartons of milk.

She said, "why did you buy twelve cartons of milk?"

He responds "they had eggs"

(there are rare features like this in some parsers, like dangling else, but they are considered design problems)

The executives deciding what software gets built are not writing code. The application goes from 0 LoC to production-ready machine code via only natural language instructions. Currently the intermediary processor are human developers, humans are adept at grasping what is actually intended and asking clarifying questions - but there's no reason this couldn't be done by a ChatGPT-like machine in the future.

There's a lot of reasons ChatGPT could neither ask clarifying questions nor get any examples of the clarifying questions nor interpret all the levels of code and purposes of code-that-was-never-written-before.

Language does not handle most business process nor efficiency goal and even security restrictions. People and logic do. Little of that language (!) is in the written word. It is done mostly by failures and historic-obvious-cultural practice.

Emulation does not mean 'comprehension' of all the purposes involved in craftsmanship of handling data for business or for even computation.

Most AI-solved problems are search reduction for spoon fed problem areas of data.

Asking clarifying questions is not always socially acceptable! Then what?

Some random things that seem difficult to get an AI to deal with:

1) Things that people expect you to know by observation, but not be told and not to say.

2) Things that people expect you to know/adhere to, in spite of being told something different.

3) Strategically creating the impression that you do or do not have the ability to change your course of action.

The stuff you're describing are things GPT-4 is already adept at, much more so than writing code.

Please go to the store and buy a carton of milk. And if they have eggs, get a dozen.

Punctuation problem.

No that's still ambiguous.

Have you tried GPT-4? It asks clarifying questions when a request is unclear.

I tried to trick it into running an infinite loop by giving it an obfuscated recursion for "generating squares" and overriding the exception that stops it.

It informed me that the action would infinite loop (spotted it better than me) and suggested that I didn't need such a complicated structure to calculate squares. Then it proposed a tessaract formula that "is bound to the laws of physics".

My complain isn't with ChatGPT, but with natural language. Even if GPT understands what you want, this interpretation needs to be stable over time and there's just no guarantee of that, and further, if you're actually programming in natural language as the original post suggested, it's not immediately possible to inspect how the language is interpreted.

This is a problem that humans struggle with too.

The two ways you typically end up getting around this problem with a programming language; a subset of natural language that uses very short and terse sentences with a simple and limited grammar;

  BUY MILK(1).
  IF EGGS EXIST THEN
    BUY EGGS(12)
  END IF

... or extremely laborious clarifications on the exact sense in which everything is meant, as seen in legal text which needs to be immune sophistry and various adversarial (mis)interpretations. e.g.

    Locate, select, and acquire, through lawful means, one (1) carton of milk, being of a commercially reasonable grade, suitable for human consumption, and free from any defects that would render it unsuitable for its intended purpose; and
    In the event the designated commercial establishment has available for sale, (12) one dozen eggs, the Designated Acquirer is hereby further empowered to purchase the aforementioned eggs, subject to them being in compliance with the prevailing quality standards for eggs.

So it’s all but impossible to check if the unknowable AI model has actually output the logic you want? And you just plug that untrusted machine code directly into the metal?

I knew the AI hype machine was real but I didn’t realise it had replaced decades of computing security already.

If you read my post carefully, you'll see I call it a "hot take", which should indicate it's a bit tongue in cheek. But more importantly, you'll notice that I used the word "eventually", with implies that I recognise that these capabilities are currently beyond us, without putting a timeline on when I think these capabilities could arise.

That said, it doesn't seem to be outside the realm of possibility. Which is why I said it.

Considering how much bad and non-idiomatic code is out there for many popular languages, this could actually be an advantage for a new language.

The AI will find gaps in the efficiency of asking AI system links to do things and will ‘prompt’ the need for a new programming language.

ChatGPT will be the end of lots of things for anybody that solely relies on it to get stuff done.

like alphago, machine could play in playground, so that it could improve it self.

Interesting thought

Gpt4/ GPT3.5 ?