Hacker News Reader

Comment by loveparade | Hacker News Reader

There has never been much secret sauce in the model itself. The secret sauce or competitive advantage has always been in the engineering that goes into the data collection, model training infrastructure, and lifecycle/debugging management of model training. As well as in the access to GPUs.

Yeah, with Deepseek the barrier to entry has become significantly lower now. That's good, and hopefully more competition will come. But it's not like it's a fundamental change of where the secret sauce is.

View on HN

I don't disagree, but the important point is that Deepseek showed that it's not just about CapEx, which is what the US firms were/are lining up to battle with.

In my opinion there is something qualitatively better about Deepseek in spite of its small size, even compared to o1-pro, that suggests a door has been opened.

GPUs are needed to rapidly iterate on ideas, train, evaluate, etc., but Deepseek has shown us that we are not yet in the phase where hardware CapEx guarantees victory. Imagine if Deeepseek hadn't been open sourced!

Let me qualify your statement... CapEx is what EXISTING US firms were/are lining up to battle with.

With R1 as inspiration/imperative, many new US startups will emerge who will be very strong. Can you feel a bunch of talent in limbo startups pivoting/re-energized now?

> Can you feel a bunch of talent in limbo startups pivoting/re-energized now?

True! It certainly should be, as there is a lot less reason to hitch one's wagon to one of the few big firms that can afford nation state scale GPU compute.

How do you know the CCP didn’t just help out with lots of compute and then tell the companies to lie about how much it cost to train the model?

Reagan did the same with Star Wars, in order to throw the USSR into exactly the same kind of competition hysteria and try to bankrupt it. And USA today is very much in debt as it is… seems like a similar move:

https://www.nytimes.com/1993/08/18/us/lies-and-rigged-star-w...

Back in the day there were a lot of things that appeared not to be about capex because the quality of the capital was improving so quickly. Computers became obsolete after a year or two. Then the major exponential trends finished running their course and computers stayed useful for longer. At that point, suddenly AWS popped up and it turned out computing was all about massive capital investments.

AI will be similar. In the fullness of time, for the major players it'll be all about capex. The question is really just what time horizon that equilibrium will form.

True but it is unknown how much of the capital will be used for training vs experimenting vs hosting vs talent.

The word you're looking for is copyright enfrignment.

That's the secret sause that every good model uses.

True, and likely lots of other data that should not have been used to train a model.

It is at this point hard to imagine a model that is good at reasoning that doesn't also have vast implicit "knowledge".

Humanity keeps running into copyright issues with every major leap in IT technology (photocopiers, tape cassettes, personal computers, internet, and now AI). I think it's about time for humanity to rethink their take on the unnatural restriction of information.

I personally hope that countries recognize copyright and patents for what they really are and abolish them. Countries that refuse to do so can play catch up.

This is based on a flawed view of how we humans behave. Without incentive no effort. This is also the reason why socialism has and always will fail. People who put massive effort in creating original content need to be able to earn the rewards.

The premise, that forgoing copyright would necessitate the forgoing of incentives and rewards, is one entirely of your own assertion and was not implied in my above comment. I agree that your assertion is flawed.

There can be, and are, incentives and rewards associated with sharing information without flawed artificial constraints like copyright.

Since all kinds of companies are getting a lot of money from the generative AI business, I think they can handle being sued for plagiarism if thats the content they produce.

since all models are treating human knowledge as copyright free (as they should) no this is not at all what this new Chinese model is about

Oh. Does that ethics framework also extend to art such as music, movies and software?

fires up BitTorrent

It will be interesting if a significant jurisdiction's copyright law is some day changed to treat LLM training as copying. In a lot of places, previous behaviour can't be retroactively outlawed[1]. So older LLMs will be much more capable than post-change ones.

[1] https://en.wikipedia.org/wiki/Ex_post_facto_law

Even if you can't be punished retroactively for previous behavior, continuing to benefit from it can be outlawed. In other words, it would be compatible from a legal perspective to ban the use of LLMs that were trained in violation of copyright law.

Given the political landscape I doubt that's going to happen, though.

The part where a python script ingested the books is not the infringing step, it's when they downloaded the books in the first place.

Reply on news.ycombinator.com