davebren

In effect the source code is being copied by the LLM. This is what it's designed to do. LLMs are a lossy statistical compression of their training data.

If you give it a prompt telling it to replicate a product that's in its training set then its optimal next token prediction output is going to be to a lossy copy of that product's source code.

Whatever LLM they used copied the source code. It took their prompt and filled in the blanks of the spec by copying from the closest matching open source project. This is just what a next token predictor is going to do if you tell it to replicate software that's in its training set since that is the optimal way to predict the next tokens.

Not popping doesn't seem possible to me, since all investment has been directed towards it, massive construction projects have been started, etc... The revenue needed to keep OpenAI and Anthropic afloat is insane and there's no way businesses are going to start paying the real price when there's no ROI compared to human expertise.

AGI is fundamentally impossible through data scaling like they tried to claim and achieving AGI is what all this depends on. The long tail problem will remain undefeated and the IPOs are a desperate move to get the cash needed to scale one last time.

They could have just kept improving the technology without all the psychosis and finding use cases and ways to make it more reliable but instead they bet everything on a language model becoming their slave god.

Slowly deflating would be nice, but I don't see how. In any case the economy is getting wrecked and any goodwill tech companies had with employees is gone after going completely adversarial towards them as soon as they had an opportunity to. The most profitable use case of gen-AI is still spams and scams.

Yes a test showing an LLM can reproduce an app in its training data and not an equivalent complexity app that is not in its training data is equivalent to proving the statement I made that having your codebase trained on will allow a competitor to copy your product.

And yes, you are obviously a fraud.

I have neural net applications I programmed entirely myself from 16 years ago and all I see from you are some python scripts so I guess I win this stupid game I wasn't even trying to play.

You haven't made a single technical point outside your bloviated claims to authority. I can safely assume you're a fraud because this is exactly how frauds speak. Actual scientists and engineers don't argue from authority they go and test the hypothesis for themselves, the fact that you balk at my suggestion to do this is amusing.

I said you can do a basic test because this is the best way to see it directly for yourself. It's very easy to do especially for an eminent machine learning visionary leader as yourself. You ask the LLM to produce two apps of similar complexity and technical challenge from a software perspective, one that is already in its training data and one that isn't and see which its more successful at. This isn't some controversial take, nor is it "unfalsifiable".

There's also hundreds of benchmarks demonstrating where the limitations are for LLMs, or you can study the progression of LLMs in mathematics and where the gains have been made and see that this also agrees with me. You can watch Chris Hay's videos demonstrating exactly how LLMs perform math layer by layer. Why is everyone using LLMs for search? Because it's an extremely efficient compression of all its training data. Did they figure out the Studio Ghibli art-style all on their own spontaneausly? No, they were trained on Studio Ghibli content. There's so many ways to come to this conclusion. But you seem to be too busy sniffing your own farts to be interested in learning anything about the field though.

I wasted 2k on a color e-ink monitor that ended up being pretty much unusable. Reviews said as much but the risk was worth it to me for the chance to spend my days looking at something that doesn't feel like a screen. I believe it's an "if you build it they will come" thing especially for anyone working on a computer all day.

First I never made any claims about myself or appealed to my authority, that was you. I still don't see any evidence of any technical expertise in ML or in programming anything more complicated than python scripts. Maybe you weren't aware but often on the internet people make false claims about their expertise on something. There's thousands of "Visionary leader driving ethical AI innovation and strategic growth" CEOs that know next to nothing about the actual technology.

Anyway like I said you can just test it out yourself and find out that I'm correct. Every skilled programmer already knows this and can predict what kind of complexity an LLM won't be able to handle. And anyone working on LLMs should know that they are completely dependent on their training data. The entire scaling hypothesis was based on this.

I don't use these tools, but wouldn't it be better to use them only after you do a manual review to see if they find anything you missed? Otherwise I could see reviewers getting false confidence and doing a less thorough review. This happens with seeing that unit tests pass.

You could just test it for yourself if you're actually interested in finding out (assuming you are a decent programmer, but your bio suggests you have no technical expertise in the area).

It has. LLMs being able to one-shot any higher-order complexity is entirely dependent on it already being in the training data.

You trust those?

I don't really use LLMs myself, but if someone wants to have any kind of software business then having the models trained on their products isn't ideal.

Who are you quoting?

Do you think that everyone using it and their employers are aware that they are giving their competition the ability to copy straight from their codebase when they ask it to replicate their product?

It's not theft in the same way a con artist convinces you to give him all your money. The people using it or their employers just don't realize any competitor will be able to ask the LLM to replicate their product and it will copy the codebase they uploaded to them.

One of the funniest things is how hard it was to get approval for a $100 software license but now people are being encouraged to burn thousands on tokens.

> "In its IPO filing, the company had said Cursor's access to developers' data, including coding requests and design decisions, could help improve its AI models such as Grok."

They're all stealing your IP and selling it back to your competitors in the form of tokens.

> "to practice what happens if a much more serious event occurs"

A pair of bolt cutters should do.

Uploading your IP to the biggest IP thieves in human history seems bad idk.

2. Eventually we'll get to where local models that don't have sycophancy and slot-machine mechanics trained into them will perform better.

This seems to be a good middle ground then. It allows for a way to prevent political projects getting grants under the guise of "scientific research", at least when they directly oppose the voters. I don't see any push to defund basic research, and if politicians start doing that there's at least a way for people to voice their disapproval through voting.

Aside from that, so much money was wasted on Alzheimer's research based on fraud.

Instead of programming something yourself now you'll be forced to program through the interface of someone that doesn't speak English putting what you say into an LLM. We've peaked.

Seems more like a materialist religious project to me.

> but there’s no denying that the barrier to entry has fallen significantly.

The barrier to entry to make slop is lower, but it's gotten much higher for developing the skill of programming. There was already an issue with a lack of mentorship and path for juniors when agile attempted to turn software engineers into assembly line workers, among other issues with the industry becoming hyper short-term focused.

Now you have educational barriers where students are competing with other students that are cheating with LLMs. There are psychological barriers with learned helplessness. The 100k lines of vibecoded slop produced hits a wall but they've gained no understanding of the code in the process or ability to make changes themselves. At the first job juniors and interns get they're being told not to take the time to learn and understand the problem they're working and instead they need to hit the LLM slot machine or risk getting fired.

> Clearly the theory that LLM's can't "extrapolate" is woefully incomplete at best (and most likely simply incorrect).

What example is there where an LLM has extrapolated? All I've seen is a data set so large and an extra decomposition process making it so interpolation feels like extrapolation if you don't look close enough.

> but a theory of why further advancements can't solve the deficiencies

How about LeCun's?

Newton did it at 23 and there would have been very few people with mathematical training. The LLM would be trained on the entirety of recorded human knowledge and mathematics up to that point, and would get to use a lot more energy so it still has a massive material advantage over young Isaac. Yet I don't believe calculus would magically appear in its response.

Synthetic training data is carefully crafted by humans. The rare geniuses of human history use a different magnitude and configuration of the same kind of human intelligence that posted a dad joke on a site that got scraped into the training set and repeated, convincing people that it is intelligent like humans.

> that you wouldn't otherwise dismiss as non-intelligent rehashing of the same tired patterns they always inhabit were those same actions attributed to LLMs?

Regardless of whether something's been done before people still come up with them on their own without directly copying or amalgamating several copies. Pretty much every skilled profession includes figuring things out on the fly through the use of general reasoning that doesn't involve pattern matching against millions of examples.

It sounds like you don't have the subjective experience of meaning that most humans do, so maybe that would explain why you don't think there is anything beyond associations. Maybe this is the core difference that's determining how people see LLMs.

I'm not sure how I would convey what meaning and understanding is to someone if they don't experience them. This is my poor attempt though: There can not just be associations there need to be "things" to associate between. Otherwise you have no ground, it is all map and no territory. Ultimately it would just be meaningless associations between meaningless symbols.

Ordinary humans do novel things all the time. Where do you think LLMs got all the training data that their responses come from?

What did you say that added to the discussion? I wasn't wondering at all. More compute time won't create new mathematics. To believe otherwise is to misunderstand the technology and there is no amount of hackernews votes that will change that.

> There’s no reason to believe that kind of guess-and-check is outside of the reach of LLMs

This doesn't make any sense, by their nature they can't "guess-and-check" things outside their training set.