Hacker News Reader

Comment by EMM_386 | Hacker News Reader

> but the second you start asking it to solve problems that aren't in its training database it gives you nonsense

I would disagree.

I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset.

When you inform it you are using a 3rd party UI component library, a 3rd party CSS library, and you need to alter the layout based on the current complex state of an Angular system, and it can provide HTML, CSS, and TypeScript code that works and is based on this unique arrangement in this system, I'd say that's reasoning on some level.

It's not copying and pasting from StackOverflow. It has to "understand" the scenario and implement it using my guidelines.

It will even take into account things like linting rules ("Never use the 'any' type") and it can do that too.

View on HN

It is quite literally copying and pasting from StackOverflow, and any other code that it has been trained on. I think you're underestimating exactly how much code there is out there, and the degree to which most of it is boilerplate.

Here's an example of what it can't do. I was recently messing around with the Plaid API, building sqlalchemy models for the Plaid API objects and defining relationships between them. After about 45 minutes of playing around, I got about 80% of the way to what I created without ChatGPT. It saved me a bunch of time googling around for documentation, but it gave me an absolute garbage password encryption implementation that was literally copied out of an intro tutorial I had read elsewhere, didn't even seem to recognize that there is a Password field type in sqlalchemy_utils, and omitted a bunch of key aspects of the model relationships. It also used field arguments that were either deprecated or not best practice for what I wanted to do. I also had to ask it to model every object individually, even though I specifically asked it to model all of the fields necessary to implement transactions.

Now I could go and ask it to specifically modify those things, but it doesn't "know" the best way of building this set of models. It gives me the statistically determined output of what it has consumed from all of the answered questions on stackexchange and tutorials it has been fed, and provides a halfway decent starting point, but to get from there to a working application requires actually understanding why you might build a model one way rather than another. You need to understand exactly what behavior you want to see when you delete a user, and what needs to happen to all of the related objects. You need to know that id columns should be UUIDs and not integers or strings, which is something you commonly see in tutorials but you would (hopefully) never see in a production application. All of this knowledge is the result of years of experience building applications and knowing not only what to do, but why you do it and in what context. ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly.

> It is quite literally copying and pasting from StackOverflow

Not necessarily.

They've already said that certain emergent properties as the models have scaled beyond certain thresholds are responsible for some of the abilities. The ability to code as well as some of them can has been included in this.

> ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly.

I agree that you need to know what you're doing to make use of them. I've been a SWE for over two decades, so I certainly know what to feed them (and what not to - such as proprietary information) to get a coherent answer. Yes, they need a lot of context on a given issue, without that the model can only guess at what you are looking for (variable names, UUIDs for primary keys, schemas, interface definitions, 3rd party components, etc). And they will occasionally suggest using deprecated methods or not the latest suggested approach, due to the cutoff date.

However, I stand by the opinion that if given the right inputs, they are capable of unique solutions to complex issues. The models are seemingly capable of taking points A and B from one StackOverflow post and combining that with C from the language documentation and D from a third-party vendors site and combining them into a coherent answer.

Yes, the language model is certainly capable of assembling text from different sources and getting an answer that tends to be mostly coherent, but that is exactly what LLMs are designed to do. To the LLM is doesn't actually matter that all of that content is from different sources, all that matters are the statistical relationships it has derived for the various tokens in your prompt. But those are not unique solutions, that are existing solutions that it has cobbled together. It's been demonstrated repeatedly that when prompted for information that ChatGPT has not been trained on, it fails badly. It also fails at a lot of higher order writing analysis prompts, like those found on English literature essay exams.

I've been a software engineer for almost two decades, and I actually use ChatGPT daily. It's a very helpful reference, but fails early and often on a lot of tasks. I generally compare it to a tool like create-react-app, except generalized across all languages and frameworks. A very powerful tool, but it's a statistical machine learning algorithm, not an AI. It doesn't understand your prompts and it isn't reasoning about them.

> I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset.

How do you know something similar isn't in its dataset?

I think we humans are a lot less original that we believe.

Reply on news.ycombinator.com