I have similar results, yet you have people here on Hacker News saying that it's "more productive than 10 developers put together" via their special sauce prompt engineering.
Which is right?
Which is right?
I spent few hours asking to do few things with a well documented public but uncommon api. It sent me on a wild goose chase of functionality that API never had..(contacted developer to make sure I am not crazy). It wrote some stuff that looked VERY good, except for the fact that it interacted with things that were never there. Would have been nice if they were..
It seem to exercise wishful thinking. You can ask it to implement functionality and it will just imagine that API/system has stuff needed.
Major problem is that I for one have no clue how it is "thinking" maybe over time we will develop better understanding of it.
For reference, I used ChatGPT to build a simple ChatGPT-clone web interface. The JavaScript it generated is literally 100% correct (not 100% good. By correct I mean it does what I describe in prompt, without syntax errors or calling non-existing functions. Code can be both correct and bad, obviously.)
I also tried to use ChatGPT to generate some Houdini vex code. It sometimes gave me a useful scaffold, but mostly just pure garbage.
How would ChatGPT help you update button texts? Isn't that just changing a text field? ChatGPT can't search your code for where the button is, and I don't see how it would help you change the text.
Being limited the way it is - not being able to use a project's greater context, which is millions of tokens generally - is a severely limiting factor that makes it unsuitable for serious work as well. Working around this (figuring out what dependencies to inject into the prompt) is as much work as actually coding.
I can see that it's passed some sort of threshold with the greater population. It is great generating filler text. Just what it can do right now should have many applications.
The other day, I had a subtitles file with slightly mismatched timestamps. GPT wrote me a Python script to fix them that got 90% of the way there (and, in particular, the code had all the API calls that I needed to get it to 100%, even though this is the first time I've heard of the library it used). The whole thing took less time than finding and installing the app that would do it for me.
The catch is that you need to have a pretty good "gut feel" understanding of its limitations to figure out whether it's going to be a time saver or not before you sink too much time into making it do something right. But it is a skill that can be learned from experience (for a particular model, anyway), and I suspect that the ability to decide what to delegate and what to do yourself will be one of the key differences between junior and senior devs going forward.
The good news is 80% of what it spat out was usable. I gave up getting it to try to give me the last 20% and figured it out myself.
One thing I’ve found helpful in those cases it to tell it that it doesn’t know what it’s talking about and ask it for good keywords to search on Google.
Someday I’d love if it just gave me the relevant links itself…
As long as it can't read my entire codebase, understand it and help me with it - that's absolute horseshit. I don't spend much time writing a bunch of new code, I'm spending most of it trying to understand the heaps of legacy code in my company and then make some small tweaks/additions.
The day it can understand big code repos will truly be the end for human made code I think.
I used to get more non-existent APIs but not as much with GPT 4 recently.
I dont use copilot anymore (at least for now). Just chatgpt as an alternative to google/SO
Abbreviated prompt:
dateOfBirth: date().required().typeError('Invalid Date')The dateOfBirth validation is incorrect because it accepts values like "11". What is the correct validation to ensure that a user has inputted a valid date?
GPTs first go:
**** dateOfBirth: date() .required() .typeError('Invalid Date') .min(new Date(1900, 0, 1), 'Date must be after 1900') .max(new Date(), 'Date cannot be in the future'),
In this example, the min method sets a minimum date of January 1, 1900, and the max method sets a maximum date of the current date. You can adjust the minimum and maximum dates as needed to fit your specific use case. ****
Me:
It still treats "11" as valid input
GPT second attempt:
***** dateOfBirth: date() .required() .typeError('Invalid Date') .test('valid-date', 'Invalid date', (value) => { if (!value) { return false; } const date = new Date(value); return !isNaN(date.getTime()); }), *****
Me:
The parameter 'value' passed into test is a date and not a string
GPT third attempt:
******
dateOfBirth: date()
.required()
.typeError('Invalid Date')
.test('valid-date', 'Invalid date', (value) => {
return value instanceof Date && value.isValid();
}),
Me:
There is no isValid function on Date objects
GPT fourth attempt:
********
dateOfBirth: date()
.required()
.typeError('Invalid Date')
.test('valid-date', 'Invalid date', (value) => {
return value instanceof Date && !isNaN(value.getTime());
}),
Me:
That still accepts 11 as a valid date.
GPT fifth attempt:
*********
dateOfBirth: string()
.required()
.test('valid-date', 'Invalid date', (value) => {
const dateRegex = /^(?:(?:31(\/|-|\.)(?:0?[13578]|1[02]))\1|(?:(?:29|30)(\/|-|\.)(?:0?[1,3-9]|1[0-2])\2))(?:(?:1[6-9]|[2-9]\d)?\d{2})$|^(?:29(\/|-|\.)0?2\3(?:(?:1[6-9]|[2-9]\d)?(?:0[48]|[2468][048]|[13579][26])|(?:(?:16|[2468][048]|[3579][26])00)))$|^(?:0?[1-9]|1\d|2[0-8])(\/|-|\.)(?:(?:0?[1-9])|(?:1[0-2]))\4(?:(?:1[6-9]|[2-9]\d)?\d{2})$/;
if (!dateRegex.test(value)) {
return false;
}
const date = new Date(value);
return date instanceof Date && !isNaN(date.getTime());
}),
Which may or may not be correct. That regex is bonkers and given the errors in the previous responses I don't trust it.