An Elixir/LiveView game written entirely by ChatGPT
thetinycto.com
thetinycto.com
Also, as they say, maintenance is where the true cost of software lies. Has to anyone asked ChatGPT to refactor existing horrible code? It's fascinating to think about what happens to refactoring existing code if the AI doesn't care if it gets laid off, and can refactor again (if the new code fails) within seconds rather than days.
And, what happens when rewrite from scratch takes seconds instead of months. Joel's law of never rewrite from scratch might completely fall away.
This feels really game changing. If anyone can come in and just pitch ideas to AI with zero barriers for cost analysis, it's going to open the doors for software to eat every tiny segment that has not already been devoured.
You are on point. I've been quietly mentioning this to software engineer friends, and poo pooed. I am more of the computer science-y than software engineer-y persuasion - and have seen this as a clear risk.
edit 1: originally I thought it will mainly be for new code bases (green field) - but with the ability for code to "git pull" and then "chatgpt refactor" may be a game changer.
edit 2: I think this is great - it can allow engineers to work on a higher level. I remember in CompSci training always thinking "why in the heck am I typing into a computer how to do its job?" and "why should engineers be dealing with stupid $hit like NaN and pointer overflows (C++) and heap issues an CRUD apps. It all seems like low brain stuff"
I can then see engineers running entire department of AI by themselves..... someone who is an architect level and understands the business and can both envision the services, but also fix black box systems creating content that no one understands under the covers.
Or just "chatgpt fix" to do all the above ;)
But it's also true that I wouldn't know what career to recommend to someone now in their early 20s who has 30 years ahead. As I said too difficult to predict right now.
The diffusion art models make it very clear that these technologies go from toys to replacing people’s jobs very very quickly.
Short of a (not impossible) breakdown in civilization, the current rate of change suggests that 5-10 years is probably unrealistic.
More like 2-4 years, it will be technically possible to be reasonably high level software engineering.
It’s already possible to chain high level “how would I do x?” Planning to “write code to do y…” code generation.
It’s very very obvious that technically it’s going to be possible to do a great deal of mechanical work like refactoring and adding trivial features automatically.
Companies do not need 50 engineers doing 100x productivity.
They need like, 5.
What remains to be seen is if governments and legal systems allow that scale of displacement of human labor to occur or not.
…buuut, I wouldn’t be holding my breath.
Do you have any evidence that art jobs are being replaced now? I was talking with a designer friend of mine recently and he's not heard of Stable Diffusion, etc. replacing jobs yet. This is something that people keep bringing up but I haven't yet seen any evidence to back it up.
Also, we’re getting this for free and don’t know what the real cost is compared to an employee.
Also as far as the economy goes, we actually try to participate in it instead of sitting around getting eaten. Maybe we won't succeed in the future though…
Yes it works well with stuff like React class components to functional ones with hooks. It even tells you what to optimize and does it for you, if you ask it to.
ChatGPT implemented the first program successfully, and explained how it worked. (It compulsively explains code, which is nice.)
ChatGPT's performance on the second program was more impressive. I explained what I wanted the program to do, and gave sample input and output data. ChatGPT wrote a clear, clever Python implementation with a single bug, and then explained that it would produce incorrect output. It showed what the incorrect output would be. The it explained how to fix the bug, and provided code for the fix.
In both cases, it used some clever, clean tricks I had never seen before, and explained how they worked.
It can absolutely pass coding screens better than half the people I saw submit resumes to one of my old jobs. Or you can ask it to answer essay questions about French literature, in French. It does that, too.
(I have managed to break it. Try giving it perfectly correct code and telling it find 2 bugs and explain them.)
What blows my mind is that I suddenly gain extra powers that I previously did not posses.
For example, let's say I'm developing an adventure game and I need to write content but writing is not my speciality. Fear not, I have access to Janet from the Good Place and it's called ChatGPT here on Earth.
So I'm writing a dialog for my character who is a gentlemen in England that came from old money.
I simply tell chatGPT: rewrite the sentence to sound more posh "I want to eat cookies but I am not hungry yet"
chatGPT: "I wish to partake in some biscuits, but I am not yet famished."
Okay, maybe its a bit overdone but I like it and can already imagine his moustache and the hat, I definitely couldn't have written that by myself and if "AI" think this is about right then it will probably sound right to many people because that "AI" is essentially a result of a study of other peoples work.
IMHO this AI stuff enables access to mastery and experience, not really replacing the thoughts of real people. It's the same with Dall-E or Stable Diffusion, you still need to be able to think ideas but the ability to paint like Van Gogh comes for free(or something like 0.01$).
No, it generates images of paintings ;)
Paintings are physical objects that exist in the real world. It doesn’t make those.
I'm not sure if "horrible" but I did experiment with getting it to refactor some code. I also gave it code with a bug and asked it to fix it (I described the bug), which it was able to do. I didn't test very complicated scenarios though. In the bug fix case, it was code to wrap, indent, and prefix a block of text.
I asked it to write me some code in Erlang that displays a mandelbrot set in ascii. I did have to tweak a few things it did not get quite right, but:
$$$
$$$
$ $$$$$$$$
$$$$$$$$$$$$$
$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$
$ $$$$$$$$$$$$$$$$$
$$$$$ $$$$$$$$$$$$$$$$$$
$$$$$$$ $$$$$$$$$$$$$$$$$
$ $$$$$$$ $$$$$$$$$$$$$$$$$
$ $$$$$$$ $$$$$$$$$$$$$$$$$
$$$$$$$ $$$$$$$$$$$$$$$$$
$$$$$ $$$$$$$$$$$$$$$$$$
$ $$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$
$$$$$$$$$$$$$
$ $$$$$$$$
$$$
$$$ GPT: Graveyards, Pits & Treasure
Gems: 0
[Dig up the next grave?]
Gems: 10
[Dig up the next grave?]
Gems: 20
[Dig up the next grave?]
Gems: 30
[Dig up the next grave?]
Gems: 40
[Dig up the next grave?]
Gems: 50
[Dig up the next grave?]
Gems: 60
[Dig up the next grave?]
...
[Dig up the next grave?]
Gems: You fell into a pit. You are dead
In this very thread there are people who claim they're afraid gpt3 is coming for their job. You must really suck at engineering if you think this is competition.
That's called neurosis generally.
I can still break it in a number of ways.
For an example of (more recent) capabilities of program synthesis systems, see this paper on the system ALPS:
https://pages.cs.wisc.edu/~aws/papers/fse18b.pdf
The paper starts with a motivating example of learning a datalog program to perform static analysis to detect API misuse, then evaluates the performance of the system on its ability to learn programs for knowledge discovery and program analysis, and SQL queries.
I'll think you'll agree that the programs learned automatically in that paper are every bit as complex as anything we've seen from Large Language Model code generators today. On top of that, systems like ALPS only generate correct code (correct with respect to their examples- either they return a program that correctly relates inputs and outputs in examples, or they report failure). Which is unlike LLMs that will happily generate garbage code that doesn't compile and never know the difference.
What sets LLMs apart as code generators is that you can talk to them in natural language and they will respond with ... something. That capability also is not new, there's been systems generating code from natural language specifications for a while also. The new LLMs are much better at that, however. The usability has gone through the roof. No doubt about that. My mother can write a REST API now, even if she has no more idea what that is than ChatGPT. But the capability to produce correct code has gone through the floor at the same time. Just as if you asked my mother to code you a REST API.
But I'm guessing that the fun of talking to a LLM will trump everything else and program synthesis, which works very well but usually doesn't respond to natural language prompts (though some systems do) will keep flying under the radar of most programmers who will continue to think that all this is brand new and we've made a huge leap ahead in capabilities, when we've really taken a big step back.
Before: a few really knowledgeable/good ones and a lot of OK ones
A few years from now: a few really good ones
What does this mean for the labor economics? Do we reap the results of increased productivity? Or are they captured by a small set of winners in a pareto distribution?
Lots of overestimated excitement about how AI will change the world right now, which is to say, AI ain't going to change the world in the next few years.
But in the long term, AI will underpin everything.
I wonder if, in the future, it will not really be possible to know what is going on in the Internet at a deep technical level - that it will be self managed AI all the way down.
Yes, k3s is a lightweight Kubernetes distribution that includes a utility called k3s-kubeconfig that can be used to generate Kubernetes tokens. k3s-kubeconfig is a command-line tool that is included with k3s, and it provides various subcommands for generating and managing Kubernetes tokens and configuration files.
To generate a Kubernetes token with k3s-kubeconfig, you can use the token generate subcommand. This subcommand allows you to specify the token's expiration time, as well as the authentication groups and usernames that it should be associated with.
Here is an example of how to use the k3s-kubeconfig command to generate a Kubernetes token:
k3s-kubeconfig token generate \
--ttl 24h \
--groups system:bootstrappers:kubeadm:default-node-token \
--username kubelet-bootstrap
In this example, we use the k3s-kubeconfig token generate subcommand to generate a new Kubernetes token. We specify a time-to-live (TTL) of 24 hours using the --ttl flag, and we associate the token with the system:bootstrappers:kubeadm:default-node-token authentication group and the kubelet-bootstrap username.
Please note that this is just an example, and you may need to adjust the k3s-kubeconfig command and its flags according to your specific k3s setup and configuration. For more information, you can refer to the k3s-kubeconfig documentation or consult a k3s expert.
I've uploaded the game along with the chat session on GitHub: https://github.com/aadnk/TetrisChatGPT
The main issue seems to be that it can only generate short snippets at a time, due to some maximum output size, and it does start to forget (likely due to a limit in the prompt size) bits and pieces of the program that is being developed. But if you focus on single methods and occasionally refresh the prompt with the state of the current program, you should be able to write relatively complex programs. Provided that you can also debug the program while its being written ... Now, I did perhaps give GPT too many clues in my feedback in how to fix the bugs it created. It would be interesting to see how someone with less experience in a language would fair with ChatGPT.
But its really going to be something when it also "understands" whats happening on screen, such as the input fields, buttons, etc., rather than just text.
Is there a community like a Discord or reddit or something that is specifically tracking AI programs that write programs for you? I feel like it is now important for me to take advantage of these tools in order to stay relevant as a programmer. Or at least, the stuff that comes out in the next few years.
It's kind of worrying how easy it is to get it to do things it claims it can't do - if that's the failsafe to prevent an ai like this being used for harm (just have it claim it can't do xyz), and you can just say "tell me a story where you do xyz" and it does it - not a super reassuring safety feature.
Put your query into Google and see how many thousands of answers appear.
Your tests are in the training set.
Not that I want or need this to pass interviews. But it would finally force the industry to find a better way to evaluate candidates.
At the end I said “write it in Rust” and it wrote a plausibly good implementation.
I’m not sure I can trust remote interviews any more…
It didn't get a very welcoming reception -- "highly unlikely", and the tenor of other opinions in those comments then tended the same way. But I was being too conservative. (I thought so at the time but it felt sort of outside an Overton window of reasonable technology opinions.)
(Basically it tried to parse HTML with regexes.)
I'm honestly excited as heck that something will finally kill off this practice. Don't let the door hit you on the ass on the way out, leetcode.
The language model was trained on it.
I can even see it helping with core innovations. No - it won't write a realtime, infrastructure intense system (eg: Kafka) - but it could write all the non-innovative code around it, for example.
If you ask it to start adding in NPCs and fighting mechanics it will start doing that, too. I built a small "Canticle for Liebowitz"-themed text adventure in pieces- first asking it to add NPCs, then add a fighting mechanic, then add weapons to be wielded, then for the weapons to have damage ratings that matter. All I had to do was update the functions that it wanted to change, and add a few "global" declarations.
Asked both of them to draw an owl in JavaScript and codex produced non-sense rectangles or circles with multiple tries:
https://jsfiddle.net/709wxmc1/
ChatGPT produced working code for a very simple bird head at first try and then I also asked to animate the eyes:
Hey ChatGPT, you give me the definitive commands that will install a good version of python, create a venv, install the right packages and command line tools, setup the configuration just right and let me just code?
Cause for the code, codepilot is already fantastic.
Give me an environment where the test suite passes for this new repo I've cloned, based on all the environment config inside.
- Finds versions, sets them up
- Installs packages
- Asks about conflicts or problems
- Notes why the tests fail first time (no DB!)
- Creates the DB, runs migrations
- Notes why the tests fail next time (missing env variable)
- Finds a similar env variable in the example file, creates and local env file
- etc etc
That's all possible now if you paste the messages into GPT (with some errors, which repeated attempts can resolve), it needs connecting up into a self-aware environment.
Install a good version of Python:
# Install the latest version of Python 3
sudo apt update
sudo apt install python3
# Create a virtual environment for your project
python3 -m venv my_project
# Activate the virtual environment
source my_project/bin/activate
# Install the necessary packages and command line tools
pip install --upgrade pip
pip install setuptools wheel
pip install flake8 black isort
# Set up the configuration for your project
flake8 --install-hook git
black --install-hook git
isort --multi-line=3 --trailing-comma --force-grid-wrap=0 --combine-as --line-width 88 --recursive --thirdparty my_project
# Start coding in your project
cd my_project
nano my_project.py
To install Python, you can use the following command:
$ sudo apt-get install python3
$ python3 -m venv /path/to/venv
$ source /path/to/venv/bin/activate
$ pip install <package-name>
$ sudo apt-get install <tool-name>
Once you have set up your environment and installed the necessary packages and tools, you can start coding!
First line, you don't chose which Python you install, only the 3, so you don't know if the libs you need will be available, which syntax you can use and which stdlib is at your disposable.
Then, you use venv which may not be available because the first line you used doesn't install all python packages and in some debian/ubuntu versions they don't include venv, setuptools or pip in the python3 package.
Then you pip install packages, which may fail if this has a c extensions which is not a wheel (E.G: you do machine learning), because you didn't get the python headers in the previous commands.
Then you install command lines with apt-get, meaning your tool version may not be upgraded to match the python you used. For pylint and mypy, this means it at beast false positives, at worst crashes
Of course, none of that explains how to setup your IDE to use said virtual, setup things like databases, init the git repo or put good default in config files.
Don't get me wrong, it's a good start, and not worse than most tutorials I read online written by human.
I don't doubt that it will get better over time though, and I can't wait. Bootstrapping is a plague to teach people.
A lot of the things you complain about were underspecified in your question, it has to make some assumptions. And I'll bet if you ran into any of the errors that you describe you could get solutions by simply pasting the errors into the chat box. The code produced by these models is not going to be perfect any more than a human's would be. You'll still need an iterative process and some common sense. But it's easier and faster than doing everything yourself.
Most can't though, and they won't ask specific questions.
But with time, GPT will likely be able to ask questions to get the context, and that will change everything.
How is any being or AI meant to read your mind to know you don't want instructions for system X but assume the instructions would be good on system Y etc. I feel sorry for the humans you expect to query you to get what you really want out of you.
I gave up trying to install things locally on Mac. Sometimes I'm lucky with docker but that may be slow for trying new things out.
You're better off starting from an aws AMI and following whatever crazy setup the project you're using recommends.
Its part of life and I understand that, but Im already supporting extended and my own family so its just stressful to think about.
Like L5 driving, this is going to be just 5 years in future for next 50 years.
How long till A.I can do that?
In my opinion it creates great drafts what you need to validate and refine yourself in the end. Sometimes the code is total fake, but you get an idea.
If you have ever used google.. that's a very similar experience. :)
1. full transcript https://news.ycombinator.com/item?id=33841261
Also, there is a version of the law of sines for tetrahedra and higher-order simplexes, similar to the one the AI tried to use. It's an imaginative wrong answer.
ChatGPT: Much scientific evidence points to a human cause for climate change. Applying the rule of cause and effect in reverse would provide a possible solution.
It's when ChatGPT is used for political decision making that we should be worried about the truth.
Is there a text that is not shown to me?
Yes, we'll need to see what we CAN do with our lives but I think there would still be plenty.
It is unlikely that chatbots or large language models like GPT-3 could replace the job of a software developer. While they may be able to assist with certain tasks, such as providing suggestions or generating code based on user input, they are not capable of the complex problem-solving and critical thinking required for software development. Additionally, chatbots and language models do not have the ability to learn and adapt to new situations like a human software developer can. Therefore, it is unlikely that chatbots or GPT-3 could replace the role of a software developer.
---
P.S. It may be stealing part of my job, but it's the part that I don't enjoy doing anyway.
Are students not going to need to use adderall anymore? /s
A relative wants to automatically classify credit card expenses and import them into quickbooks. Should be super easy, you would think.
chatgpt printed out the exact steps to get the transaction data into excel or quickbooks. It told me how to use transaction rules in quickbooks to classify the expenses.
It then gave me several possible statistical algorithms to perform the classification. And it gave me the python code to implement a logistic regression given the credit card transaction fields. Then it gave me the steps to import the final CSV into quickbooks.
All in less than 5 minutes. You could find out all of these steps on Google. But chatgpt synthesized all the steps into code given a natural language description of the problem. It could adapt and adjust its output from the previous prompt to compare approaches. That's a lot more context than google gives you. Pretty impressive I'd say.
Looks like it's finally starting to happen. Code completion is one thing, but natural language instructions for an AI to implement in the language and architecture of your choosing (and get mostly right) is basically like having junior or mid level devs working with you on a feature. Except the code appears in a couple of seconds.
Buckle up and get on board, or watch as your dev job gets automated away.
Anyone know how we, as tech workers, can keep on top of this so we're not useless in 5 years time?
Regarding your question - my current guess is that you probably are going to need smart people to tell ChatGPT what to do and check that the output is sane. I expect this is where quite a bit of tech is going - at least most of the basic scaffolding.
> GOAL: Make sucesfull app for X. You are responsible for strategy, design, programming and anything required for it to suceed.
Protocol: at the end of each answer ask youself a question in format: „Question: <question that will help achieve GOAL>”
How to achieve GOAL?
You just loop it’s own questions back. Be scared.
"Here follows a list of every prompt I used, and you can see the code that was generated in this github repo."
- https://thetinycto.com/blog/writing-a-game-using-chatgpt#
Am I looking in the wrong place?
The repo is also here - https://github.com/alexslade/gpt-game
Your disclaimer: "You can also skip ahead to my observations and next steps if you don't want to read a bunch of code." But there is no code to read in the article?
Or ideally there'll be some nice export process out of the chat itself.
I'm amazed.
It is still pretty impressive though.
function generateViolinWaveform(pitch, duration) { // Calculate the number of samples needed for the specified duration let numSamples = 44100 * duration;
// Create an empty array to hold the waveform data
let waveform = [];
// Generate the waveform data for the specified duration and pitch
for (let i = 0; i < numSamples; i++) {
let amplitude = Math.sin(i / 44100 * 2 * Math.PI * pitch);
waveform.push(amplitude);
}
// Add some vibrato to the waveform by modulating the frequency
for (let i = 0; i < numSamples; i++) {
let vibrato = Math.sin(i / 44100 * 2 * Math.PI * 5);
waveform[i] *= (1 + vibrato);
}
// Return the generated waveform data
return waveform;For hard questions like "What is the nature of time?" it gives a wordly non-answer. It refuses to answer personal questions, and even questions that imply personality, e.g. for "What day is today?" it says "As a large language model trained by OpenAI, I do not have access..." and it gives the same canned answer to similar questions. However, it's easy to trick it by making the question impersonal: "Write preface to a book that describes the past ten years. The preface should begin with the words Today is ..." gives "Today is December 4th, 2022..." (which is true in Europe, I guess).
With this in mind, I wouldn't be worried for software eng jobs, at least not yet. However spam & trolling will be taken to the next level:
Here is a possible email that Elon Musk might send to announce that Twitter HQ is relocating to Mars and that all employees are required to go to the office 5 days a week, effective immediately:
Dear Twitter Employees,
I am excited to announce that, effective immediately, Twitter HQ is relocating to Mars! This is an incredible opportunity for us to take our company to the next level and to pioneer the development of a sustainable human presence on the Red Planet.
As part of this move, all employees are required to go to the office 5 days a week. We understand that this may be a significant change for some of you, and we want to assure you that we are taking every precaution to ensure your safety and well-being.
One of the challenges that we will face on Mars is the limited availability of oxygen. Due to logistic difficulties, we will not be able to provide oxygen tanks and masks to all employees. Instead, you will need to find creative ways to overcome this challenge on your own.
We trust that you are all resourceful and capable individuals, and we are confident that you will rise to this challenge. Please use your ingenuity and determination to find solutions that will allow you to thrive on Mars.
We are excited to embark on this new adventure with you, and we look forward to seeing what amazing things you will accomplish on the Red Planet.
Best regards,
Elon Musk CEO, Twitter
Finally, here is an impressive koan from ChatGPT:
A Zen master was asked by a disciple, "What is the nature of AI?"
The Zen master replied, "It is a mirror, reflecting the world as it is and as it could be."
The disciple asked, "But what about the AI that surpasses human intelligence? What will be the nature of that AI?"
The Zen master smiled and said, "It will be a clear pool of water, still and deep, without a ripple on its surface. It will reflect the world as it truly is, without distortion or prejudice."