ChatGPT and Code Interpreter = Magic
andrewmayneblog.wordpress.com
andrewmayneblog.wordpress.com
For instance:
1. The description it gives of how to generate a Shepard tone is wrong; to make a Shepard tone, you need to modulate _frequency_, not (or not only) amplitude.
2. The Shepard tone it generates is also wrong, but in a different way. For instance, there's no bass at the end matching the bass at the beginning, so it can't be looped to create the illusion of infinite ascending motion.
3. The "Game of Life QR code" isn't actually playing the Game of Life in a way that results in a QR code. It looks like it's starting with a QR code and playing the game of life, then time-reversing the result; so you see the Game of Life running "backward" until you get to the QR code as an initial state. I say "seems like" because I can't be confident it hasn't made any mistakes. This _may_ be what the author intended by "working backwards"? But I took that to mean that it should "work backwards" by first finding a state S that stepped to the QR code, then finding a state S' which stepped to S, etc; so that you'd have a run of the game of life ending in a QR code. This would involve actual search, and there are patterns which simply cannot be achieved this way (have no predecessors).
4. The planet orbit simulation seems to have all planets in circular orbits, rather than elliptical. For Earth this is probably unnoticeable, but Mars' orbit varies from 1.38 AU to 1.67 AU - quite noticeably elliptic - but appears circular in the "simulation".
etcetera, etcetera. There are also plenty of "obvious" glitches in the graphics simulations, but those concern me less - precisely because they're obvious.
You cannot even be correct with GPT, once it's in an error state you must start a new session. The chance of hallucinations rises rapidly.
It feels like it's trying to cover itself. Pretty weird stuff.
That he doesn't expect people to be doing what we call "prompt engineering" in 5 years. You'll simply ask for what you want and if there is ambiguity it can be sorted out interactively or it will just do the 'right' or obvious thing that we would expect.
Not to mention that there's still a large portion of the general public that just freezes up when it comes to anything technological. I've provided support to a number of different communities and even educated people can't do this type of thinking easily.
It's possible that we could eventually get to that point, but it would require some massive educational efforts and a culture shift, which would also require substantial investment without a clear road to profit, and I believe America lacks the political will or even ability to implement anything like this. Since we can't act in unison, it will be down to companies, and for them it makes more sense to solve the problem by hiring prompt engineers with the savings they get from cutting elsewhere instead of training all their existing teams on how to learn new ways of thinking.
tl;dr: Technologically he's probably right, but he's massively overestimating the public's ability to use interactive tools to get information.
> obvious thing that we would expect
Any time you rely on something being 'obvious', the public is going to somehow fuck it up.
This is also what programmers typically complain about when they say their PM isn't "technical" enough. What they usually mean is they don't understand the details of the business / domain well enough to meaningfully contribute to the organization / break up of the work.
To that end, I'd expect these language models to eventually improve at that task too, perhaps responding with choices or diagrams to help users learn how to ask what they actually want. That's in a sense what a prompt engineer is doing in this context. In that sense I think I agree w/ your assessment although I also think there's a lot of room for these AI's to handle that task, at least partially, as they improve.
The issue with language models in particular is that to use them effectively you have to fundamentally understand that you're translating human-brain-language into computer-language and understand that those are in fact two different things even if their delivery happens identically. For most coders, this jump happens semi-instinctively because they know how to test something to verify 'ah yes, this follows computer logic'. The general public thinks computers are wizardry. The other group of people who will be able to understand this are those who have thought deeply about language for whatever reason and therefore know that human-brain-language has qualities that extend beyond 'We use words'. This is also very hard to impart to novices; I have a Linguistics degree and studied a cumulative 10 years of 6 languages so I went through that knowledge transfer as well. They're both hard and one or the other is necessary to interface with language models effectively and will be for likely the next 10ish years just because providing a good technical solution will require the technical people to understand much more about the social side of communication and that will also take time.
Does it have something to do with grounding or implicit knowledge/ connotations and context?
An example of things I consider when interfacing with an LLM that derive from my linguistic knowledge:
* That the language it's trained on is thoroughly modern/confined to the speech of a particular place in time. Which means that I know any viewpoint I receive is not only modern but from a very specific time and place. The language that everybody uses is sort of like the water fish swim in in that most people don't think about it. Which means I know that if I ask it something about (to use an example that is a culture war issue) the history of racism, I know that the answer is being run through modern conceptions of racism and if I want historical views, I need to get those elsewhere.
* That which words are most commonly used relies on the social and economic status of the speaker as well as word properties like phonetics and phonology. This makes it much easier to pick and choose which vocabulary and sentence structures to use in order to 'sub-pick' the part of the training set you want answers to. Asking 'how to grow peppers' and 'what soil variables control how well a Capsicum annuum plant grows' are going to get you different answers.
* Related to this, the differences between spoken and written English on a large scale - one problem with the 'everybody can just use LLMs' idea is that the LLMs are trained on written English but the majority of people interfacing with them speak to them as though it were a verbal conversation. There are parts of verbal communication that don't exist in written communication, and knowing how to provide scaffolding to make a request make sense requires understanding the difference between the two.
* A basic knowledge of sociolinguistics is fantastically helpful for developing personae and/or picking out biases in the training data set. (Somewhat similar to how I can usually ID an American's media diet and political affiliation after a 5-10 minute casual conversation).
I think you are underestimating the pace of progress in the capability of the LLMs. It has infinite patience to query you for missing facts. Because it is not about the public's ability to use an interactive tool. It is about the LLM's capability to be a good focus group moderator/librarian/etc.
This role reversal is imminent from what it seems. The LLMs will be using us, rather than us using them.
LLMs are confidently wrong rather than inquisitive/have an interest in being right. They theoretically have infinite patience, but they really have no patience and accept input at face value.
I don't doubt that these can be improved, but I also don't doubt that some people will be better at interacting with LLMs than others, and that to get good at using an LLM will take onboarding time that organizations would prefer to centralize
Right now, you can't ask chatgpt to "think of a person, and the user will guess who it is". It can decide to think of something, but has nowhere to put that information. It should be put into the scratchpad.
Scratchpad could potentially be used for solving problems step by step too, much like how we consciously think through problems step by step. If it gets too complicated for us, the mental scratchpad we use is augmented by a physical notepad or other system.
You could also say that it enables an internal monologue.
Add an interpreter as per this article, and we're basically there.
Obviously LLM models cannot be compared one to one with our brains, but one thing is clear: despite them being executed by a computer, they are not the computer that is executing them and as such their ability to do operations that are trivial for the computer is not necessarily trivial for the language model.
Like whole industries sprung up around the various tools - email, search, notes, images, videos, databases. We're big on tools.
Yeah, pretty big on tools.
It's tools all the way down.
Another point, the apps that I wrote with GPT-4 were filling gaps in my platform and things that would never have come off the backburner due to the lack of ROI. I wonder how we can accelerate contributions to OSS projects. Knocking out new features, refactorings and technical debt 10x faster. Of course it has to be done with care but -- an army of willing AI contributors...
Commit times are plain text user generated content
Just throw generated commits in a private repository and you have as much history as you want
If a sycophant or recruiter or employer wants to use that metric as an indicator of anything, that is pure happenstance and completely on them
LLMs are impressive, but hacking on pre-existing complex codebases seems to be a current weakness.
If I'm your hiring manager, I won't be bothered by it, since I know you can use ChatGPT to solve problem, which is what the company needs. But I will still need to assess you more in detail, to ensure that you can develop / solving problem well without ChatGPT, in case somehow it's unusable or provide wrong answers.
One interesting question is: when people are using LLMs to generate software for their own use, what will they do for the server component of client-server use cases? Is "LLM, go register me an EC2 instance and configure it not to go over $20/mo, here's my credit card number" totally out of the question?
It's far from movie quality right now but there are a lot of people working on this sort of thing. People are also exploring text to 3D scene, model, and character generation pair that with text to animation which is also being worked on and we've at least taken the first steps towards that goal.
Have you actually tried this yourself? I have probably... 300+ ChatGPT chats/sessions past 30 days (aka, I actually use it).
I have 0 faith it would actually spit out anything error-free / working.
2. You don't have to find a good blog article that works for your specific purpose
3. You can ask for changes to the code like "translate this code to python" "I want to use Azure instead of AWS" "add a database so I can save player scores" etc. and it will both write the code and explain what it did.
4. You can ask for elaboration if you don't understand something, it can describe code or it can further explain written text.
5. It can explain why you're getting an error message after you change something and suggest a way to fix it.
7. If you have a feature you want to add but don't know how you can just ask it and it will show you how to implement the feature and how to integrate it into the existing code.
There's a lot more, too much to list but every one of these is far more useful than just cloning a repo and following an article.
If there was any point in doing it, people would’ve already done it 1000x, the LLM adds no value here.
It would be awesome if gpt4 could be made to edit the code and therefore, it didn’t have to regenerate it from scratch every time.
The regeneration also chews up a lot of the token limit, so it forgets crucial parts earlier on in the conversation.
The scariest thing is that... I did use Chatgpt to set up an aws cloud formation stack for a lambda function, and it gave me the right file to sync.
So... Yeah, it's not that many steps to a "text to (web|mobile) app generator" solution.
Uh.. I will get to work.
Until we see chatgpt figuring out how to fix issues like that, I'm not worried.
[0] https://iamondemand.com/blog/how-to-deploy-your-lambda-funct...
ChatGPT: Below is an AWS CloudFormation YAML template for creating an AWS Lambda function with 16GB of memory:
```yaml AWSTemplateFormatVersion: '2010-09-09' Description: 'AWS CloudFormation Lambda Function Template with 16GB Memory'
Resources: LambdaFunctionRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: LambdaExecutionPolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: 'arn:aws:logs:::*'
LambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
FunctionName: MyLambdaFunction
Runtime: python3.9
Role: !GetAtt LambdaFunctionRole.Arn
Handler: lambda_function.lambda_handler
MemorySize: 16384
Timeout: 300
Code:
ZipFile: |
import json
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda with 16GB memory!')
}This CloudFormation template creates an IAM role and a Lambda function with 16GB (16384MB) of memory. The Lambda function runtime is Python 3.9, but you can change this to another runtime if necessary. The code block contains a basic Python script for returning a simple message; you can replace this with your own script.
If you ask ChatGPT to generate a template for a lambda with 16 vCPUs, it will generate a template that allows a max of 16 concurrent invocations, which again is not what you asked it for, and unless you _know_ what reserved concurrenct executions are, you won't know that's what it's generated.
[0] https://docs.aws.amazon.com/lambda/latest/operatorguide/comp....
In here, it says that maximum memory is 10240mb, so that may not work:
Memory and computing power
Memory is the principal lever available to Lambda developers for controlling the performance of a function. You can configure the amount of memory allocated to a Lambda function, between 128 MB and 10,240 MB. The Lambda console defaults new functions to the smallest setting and many developers also choose 128 MB for their functions.
Its response:
ChatGPT: I apologize for the confusion earlier. You are correct that the maximum memory allocation for a Lambda function is 10,240 MB. Here is the revised CloudFormation YAML template with 10,240 MB memory allocation:
```yaml Resources: LambdaFunction: Type: AWS::Lambda::Function Properties: FunctionName: LambdaWith10GBMemory Handler: index.handler Runtime: nodejs14.x MemorySize: 10240 Timeout: 300 Role: !GetAtt 'LambdaRole.Arn' Code: S3Bucket: <S3_BUCKET_NAME> S3Key: <S3_KEY>
LambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: LambdaRoleWith10GBMemory
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: LambdaBasicExecution
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*Replace <S3_BUCKET_NAME> and <S3_KEY> with the appropriate values for your S3 bucket and object key that contain your Lambda function code. You may also adjust Runtime, Handler, and Timeout as needed.
===
Then I said:
What can we do to be able to use 16gb memory?
Then it said:
ChatGPT: To use 16 GB memory in a serverless context, you will need to consider other options as AWS Lambda currently supports up to 10,240 MB. Two alternatives are AWS Fargate or running a container with Amazon ECS.
Here's an example of using AWS Fargate with a task definition for a container with 16 GB of memory:
1. Create an ECS cluster:
aws ecs create-cluster --cluster-name MyFargateCluster
2. Create a task definition with 16 GB of memory (you can adjust the values as needed). Save the JSON below to a file called fargate-task-definition.json:
{ "family": "FargateTaskWith16GBMemory", "taskRoleArn": "<YOUR_TASK_ROLE_ARN>", "executionRoleArn": "<YOUR_EXECUTION_ROLE_ARN>", "networkMode": "awsvpc", "containerDefinitions": [ { "name": "MyContainer", "image": "<YOUR_DOCKER_IMAGE>", "cpu": 4096, "memory": 16384, "memoryReservation": 16384, "essential": true, "portMappings": [ { "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ] } ], "requiresCompatibilities": [ "FARGATE" ], "cpu": "4096", "memory": "16384" }
Replace <YOUR_TASK_ROLE_ARN>, <YOUR_EXECUTION_ROLE_ARN> and <YOUR_DOCKER_IMAGE> with appropriate values for your use case.
3. Register the task definition:
aws ecs register-task-definition --cli-input-json file://fargate-task-definition.json
4. Launch the Fargate task:
aws ecs run-task --cluster MyFargateCluster --launch-type FARGATE --task-definition FargateTaskWith16GBMemory --network-configuration "awsvpcConfiguration={subnets=[<YOUR_SUBNET_ID>],securityGroups=[<YOUR_SECURITY_GROUP_ID>]}"
Replace <YOUR_SUBNET_ID> and <YOUR_SECURITY_GROUP_ID> with the appropriate values for your VPC and security group. This will give you a Fargate container running with 16 GB of memory, and you can adjust the resources allocated to the container as needed.
====
I think that's quite a nice answer.
I think this proves the point though, that it's closer to Copilot - it can generate a bunch of boilerplate, but you the programmer still need to understand what it's generating to deploy it. The only difference is that instead of opening vscode and typing LambdaFunction, I ask chatgpt in english, but I still need to specify. _that's_ the part we're not seeing movement on yet.
> ? Is "LLM, go register me an EC2 instance and configure it not to go over $20/mo, here's my credit card number" totally out of the question?
I suspect we are _way_ closer to that than to having it respond to "make me a game like clash of clans".
But if a few people asking the right prompts is all you need, what happens the to other 50-100+ people a game like Clash of Clans would normally employ?
> The hard part of defining requirements is being specific in them
True but I don't think you're accounting for iteration. You know the "idea guy" cliche? Today, if/when this "idea guy" hires a team of devs from India to make his million-dollar idea into reality, his inability to write good requirements will lead to a useless app, and the idea guy asking for changes (that are also poorly described) until he runs out of money, right?
Now imagine that his "dev team" works for free and can turn around each new set of requirements in minutes rather than months. How long will it take him to learn how to good acceptance criteria? A day? A week? It's hard but not that hard. And a million or two other "idea guys" are doing the same thing, and sharing notes with each other.
> It's hard but not that hard.
It really is that hard. It's basically 90% of staff+ engineer's job.
I don't doubt that AI and ML in particular will absolutely change our industry, but as an engineer I see chatGPT as a tool like copilot, a debugger, or a linter. All those tools make me an order of magnitude more productive, but they're useless without the human making the decisions
We're talking about being a couple of years away from ChatGPT being able to copy and paste from tutorials here, and with my understanding of AI (I've done a few ML projects in the last decade, but nothing that stuck), that's the "easy" part. The impressive part of things like Mindjourney is the synthesis, and we're not really seeing that or any signs that it's coming IMO.
Nothing happens, and you don't. That's what I meant by "bad software". I'm reminded of a story Clay Shirky told about AT&T analysts from the mid-90s trying to understand the web hosting business[0]:
> The ATT guys had correctly understood that the income from $20-a-month customers wouldn’t pay for good web hosting. What they hadn’t understood, were in fact professionally incapable of understanding, was that the industry solution, circa 1996, was to offer hosting that wasn’t very good. This, for the ATT guys, wasn’t depressing so much as confusing... For a century, ATT’s culture had prized - insisted on - quality of service; they ran their own power grid to keep the dial-tone humming during blackouts. ATT, like most organizations, could not be good at the thing it was good at and good at the opposite thing at the same time. The web hosting business, because it followed the “Simplicity first, quality later” model, didn’t just present a new market, it required new cultural imperatives.
0: https://gwern.net/doc/economics/2010-04-01-shirky-thecollaps... (shirky's old site has been down for some time)
Let's say a simple single purpose web app such as Calendly. Why should I pay for such a service if I can just ask the LLM to create the entire app for me?
In TFA, the author let the LLM create an ics file. Why shouldn't the LLM also spawn a REST endpoint and serve stuff from it matching my prompt?
The more interesting question is, once non-technical people are able to auto-generate their own apps, what backend will they use and how will they handle auth? The easiest default option today is "cloud services, and 'log in with facebook' oauth" but we can hope for better...
Pls make website, all responsive like, w/ BIG pictures ooo, use my fav fonts, also fancy menus with whoosh on, load fast pls.
Thanks, Human
PS, no bugs :)
Which sucks because I want it to help me write multiplayer videogame net-code!!! It's too dang hard!!
Certainly a human with an understanding well enough to produce a working example in 30 seconds of thousands of different problems on GitHub would have the synthesized knowledge to figure out Netcode really quickly and easily for a complex game, how to structure it well, and even how to test and debug it just all immediately.
GPT-4 can't solve the easy, newer problems at CodeForces, but gets all the older hard problems.
Sure, I can make a Shepard tone for you.
>>> COMMMAND { "action": "run_python", "program": "import blah\n...." }
Re: token limit. A token can be a whole word and there can be thousands of them in the context window simultaneously. So it can look back quite a long way. GPT-4 is rumored to have a token limit in the tens of thousands, although the true number is apparently not public. So that's a lot of code and results.
That said, if you asked for a program that emitted a million line CSV file and then expected the AI to read it, it would indeed get very confused and go wrong because it'd lose track of what it was trying to do. But that's no different to asking a human to mentally digest a million line CSV.
I've tuned temperature, added a logit_bias to heavily prefer `{` token, this helped with the plain English v.s. JSON issue, but didn't help with hallucination. I guess I really need API access to {role: "tool"}.
Not try to read all of the lines
"Prompt engineering" ChatGPT English in this case is like a new higher order programming language. Prolog meets Ruby. A Haskell for mortals. No more counting parentheses or semicolons which turns away a lot of people from coding.
I guess that is magical, yeah.
Just consider all the prompts ever entered as just another set of data ;-)
I suppose the next intriguing step is for it to not just spew the boilerplate but pick on its own what are the best components to connect - hard to predict how far away that might be.
There are already working demos of ChatGPT writing incorrect code, running the code and seeing the incorrect output multiple times, and then iterating until the code worked. AI can write and rewrite infinitely, thousands of times a day, until it gets the desired result, learning and refining it's own process as it goes, and being fine tuned with human feedback.
This is especially true in something completely digital like programming where it's possible for the AI to run the code and use the output as its next prompt. At the moment it needs human feedback to know if the art it just generated is garbage, but the code it can test itself.
It does beg the question: How well does the LLM work on non-English languages?
He did mention translators, but I think he was talking about a more primitive incarnation.
I suspect that the Chinese have a version that works well with Chinese input, and maybe the Koreans and Japanese, but I'll bet one of the limitations is how much material is available to learn on, and the contractors needed to verify the training.
It sounds like training is a really expensive task, and may mean that for-profit corporations could get a lock on it.
I'm wondering if we'll be seeing "AI patent trolls," where you are sued, because your LLM has similar responses to someone else's. I see a whole industry springing up, around that (probably with AI lawyers).
The great thing about niche languages is that it's not cost effective to scammers to learn it for their purposes. That all changes now I suppose.
Well I haven't chatted to it much in Chinese, but I've asked extensively about Chinese, and that's probably one area where I get the most benefit. It can accurately translate, segment, romanize, and break down phrases; it usually does a pretty good job explaining why a given word was used, and is good at giving you example sentences using a given word or grammar pattern. I'd be surprised if it were significantly worse in Chinese than in English.
I have not used ChatGPT much but I see more and more uses that are not obvious ones (and usually only the obvious ones are addressed in articles or posts)
"If I want to remove plaster from walls, which tools should I use? What should I do with the debris afterwards?"
"Which kinds of cavity wall insulation exist? Will this cause moisture problems in my house? What can I do to detect or prevent these kinds of problems?"
"Is it useful to insulate interior walls in an old house? Or should I just focus on the exterior? Will interior wall insulation raise the EPC (energy performance) of my house?"
There are parts that doesn't really seem to follow the game of life indeed.
I'm still on the waiting list, or I would test it myself.
This morning I feel oddly compelled to play the fool so here are some near/medium term thoughts on where this may be going (worth less than what you paid for them):
1. The most important ChatGPT plugin is going to end up being the one that invokes itself recursively. The autoregression approach seems to be severely limiting what these models can do by limiting their ability to think without speaking. Although a few months ago I thought the obvious way to fix this was to train the model to emit special sort of "bracket" tokens that would be deleted by the driver once the inner thought completed, leaving only a sort of "result" section, the GPT-as-a-GPT-plugin effectively does the same thing
2. Whilst the first biggest win from the plugin will be "sub-thoughts", the next biggest will be training it how to dispatch multiple sub-thoughts in parallel. GPT already knows how to break a complex problem down into steps, but is still constrained by context window size and inference speed. Once it is taught how to split a problem up such that multiple independent inference sessions are able to work on it in parallel, it'll become feasible to make requests like "Build me a video game from scratch using Unreal Engine, set in the world of Harry Potter, about the adventures of a character named X" etc and it'll end up dispatching a massive tree of GPT sub-instances which end up working on the independent parts like character generation, writing the Unreal C++, prompting Midjourney and so on.
Parallel recursive LLMs are going to be much more awesome than current LLMs, and I mean that in both senses of the word (cool, awe-inspiring). In particular, this will allow us to pose questions like "How can we cure cancer?".
3. OpenAI need a desktop app, pronto. Whilst the cloud model can take you some way, the most valuable data is locked behind authentication screens. The cloud approach faces difficult institutional barriers, because data access inside organizations is oriented around granting permissions to individuals even when they work in teams. Giving a superbrain superuser access doesn't fit well with that, because there's no robust method to stop the AI immediately blabbing business secrets or PII to whoever tickles it in the right way. That's one reason why the current wave of AI startups is focused on open source technical docs and things. If ChatGPT is given tool access via a desktop app running on the end user's computer, it can access data using the same authentication tokens issued to individuals. This also neatly solves the question of who is accountability for mistakes: it's the user who runs the app.
4. Sandbox engineering is the new black.
> GPT-as-a-GPT-plugin
> Parallel recursive LLMs
Anyone else?
(edited to delete a post that was already answered by the other reply)
I’ve been thinking that taking this up a level is more a systems architecture problem. The core LLM model is so incredibly flexible and powerful that what I’m working on is the meta application of that tool and giving it the ability to use itself to solve complex problems in layers.
Hopefully that makes sense. I already have a fairly extensive and detailed systems architecture design.
I've been repeating this for a while now, I think OpenAI is 50% marketing, and a part of the rest is product.
GPT-enhanced code execution already exists (Langchain).
>sub-thoughts
Someone's trying to implement long-term memory : https://github.com/wawawario2/text-generation-webui
My blog audience is very non-technical so I write very broadly. We've been super busy with the launch of GPT-4 and Plugins (I produced the video content, found examples, briefed media on technical details, etc.) so I was only able to grab a few hours to put these demos together.
As far as the ChatGPT prompts go, I included a few, but they're just simple instructions. Unlike GPT 3.5 where I'd spend an hour or more getting the right instruction to do zero-shot app creation, GPT-4 just gets it.
After a stint in entertainment I realized that AI was where everything was heading. I took up programming and started studying AI.
docker run --rm python:3
Want more options? AWS lambda supports python. Cloud functions of various clouds (google, probably azure, etc) support python. Those are all sandboxes.
You can cross-compile python to wasm, and run it directly with javascript... or with any of a number of sandboxed wasm runtimes.
All that said, given the python packages they have available, my money is on a bunch of docker containers, not wasm or lambda or such.
Feel free to prove me wrong though. Give me a binary to run which will escape a container, configured only with docker flags I choose to set, but without gvisor.
If it's not a "secure sandbox", surely the above should be easy... But of course you won't be able to. "contained.af" (temporarily broken right now unfortunately) has been running for 5+ years with docker, no gvisor, and no one capturing the CTF flag.
This powers tio.run
Even though it's doing (most) of the programming and gluing stuff together, understanding the fundamental structures underneath is still pretty critical to really take advantage and push the boundaries.
It's still in development, but you can currently run bash and python from any markdown code blocks in the chat.
I'm working on getting a terminal emulator over the code blocks for better integration.
https://old.reddit.com/r/interactivefiction/comments/11z6p84...
Yesterday I went as far as implementing narrator/listener roles reversal. The player can now take over narration. As a result he must write what an IF system would print on the screen, and the IA replies by issuing commands, endorsing the role of the player.
This is unprecedented, all the more so 99% of the work is done by simply asking: "Let's play a session of interactive fiction".
I agree the comparison is a bit unfair since ChatGPT4 did not produce any code, so I would have to compare this against chatting with someone pretending to be an IF system.
But is it ? The most popular language used to write interactive is Inform7. Inform7 is a language that reads like english (a control subset of it) to program text based games the player interacts with using a very limited form of english, the limit being your capacity at handling textual structures using programmatic constructs.
Couldn't we program entirely using natural language this way (and maybe have ChatGPT writing its own Evals on the go) ? To me it looks like a new kind of programming that's mostly conversational rather than instructional. Instead of composing together contextless bricks to build towers of abstraction and striving for easily adaptable/maintainable code, which are consequences of the fact the concepts named in our codebase are influenced by the distributive aspects of meaning in real world ("hey, want to hear the last news ? Management has decided multiple users should be able to manage the same account"), our building material is some kind of primordial plasma of context out of which logical, cartesian, categorical forms can be enticed to emerge by priming the medium. This could become semiotic programming, in contrast to symbolic programming. Another aspect of Inform7 that's particularly interesting is the fact it is built upon a literate programing "web" engine. Seen in juxtaposition to its latent and broken homoiconic tendency and transposed to ChatGPT this makes for very interesting prompts where the lines drawn between documentation, code and programmed material gets blurred, as I explain using text (maybe to myself ?) how to imagine text using text.