Hacker News Reader

Comment by metalcrow | Hacker News Reader

A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...

View on HN

> Do you think the current state of affairs is GOOD?

Chinese labs releasing open weights models is good.

All of these independent harnesses and model router services are good.

The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.

OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.

OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.

I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.

We only know about the stuff in https://www.anthropic.com/threat-intelligence-report-septemb... because it's not open source. I assure you the open source and Chinese models are being used in this same way.

I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.

> Criminals using LLMs does not make the LLM situation bad

If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.

> A billion people are going to die because you don't understand what is bad.

Lol. Lmfao, even.

https://en.wikipedia.org/wiki/Regulatory_capture

The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.

That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.

It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.

Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.

Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.

It’s quite possible for imperfect regulations to make a problem worse. See, for instance, sanctions intended to weaken China that ended up creating powerful Chinese competitors and reducing Western influence in their internal markets, bringing them closer to technological autonomy.

The sibling comment offers some good regulations that may actually reduce harms, but the kind of regulations offered there are not the ones that the “safety” people want, because they hurt profits.

Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.

But the proposal the article is reacting to is for literally none of that! It is quite literally the opposite, with its proposed measures applying only to frontier labs rather than grandfathering them. It doesn't say anything about open source training, self-hosting, open weights, or startups. It does not suggest a ban on Chinese models (just better enforcement of chip export controls).

What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.

Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.

The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start

What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.

Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.

To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.

> We don't have pocket nukes,

What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?

> and there's very little risk of random countries acquiring their own due to the amount of work involved.

And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.

North Korea got it with help from China, Israel got it with help from France. They didn't do it on their own.

And Pakistan with help from the Saudi's, and the US with help from an Australian and the Brits and Canadians

Do you have a point here?

The point is the risk is in counties being gifted or assisted in making it, not them building it on their own.

Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.

In contrast, rewind to the early 1800s and there is zero hope of a ban on the R&D of small arms being effective in the long run. The only thing it might maybe ensure is that no legitimate actors that fall under your jurisdiction are involved in it.

Basically I think that current trends point to an eventual situation where world changing research doesn't require anything more than consumer level compute. Pandora's box has already been opened.

> We don't have pocket nukes

Bit of a tangent but we do, actually. At least depending on if you consider an artillery shell to be pocket sized. https://en.wikipedia.org/wiki/Nuclear_artillery

This one is a bit long to fit in a pocket but you could certainly sling it over your back. https://en.wikipedia.org/wiki/W45_(nuclear_warhead)

> Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.

Isn't that true of AI as well? Data centers are very big and very expensive.

> Bit of a tangent but we do, actually

Sure but not like the movies, these don't destroy cities. But the metaphor isn't accurate anyway: nukes don't get worse. The actual idea here is preventing the frontier of AI from advancing.

You don't need a datacenter to train a 30B model. Further, the rapid trend of increasing efficiency and decreasing model size for a given level of capabilities means that what is possible to do without a datacenter will continue to increase. Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there. Ergo I expect such an approach to regulation would prove entirely self defeating.

Remember, as I mentioned earlier the human brain only consumes on the order of 20 watts and fits in a handbag. Would you have us destroy all chip fabs? Ban all biomedical and genetic research? How far are you imagining this butlerian jihad would go?

> Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there

That's an enormous presumption! You're saying that even in the theoretical case that frontier research is halted but efficiency isn't, we could do better then the frontier and reach world-changing AI in 30b parameters at home-scale labs?! If that's true then we can just give up now: the world as you know it is going to end in around a decade and billions are going to die, there's nothing we can do. But I don't think that's true. Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.

If you're right then I concede. It doesn't matter what we do, regulations or not. But if I'm right then regulation can do something and in theory help bring a better future.

> frontier research is halted but efficiency isn't

Why are you treating those as if they're separate things?

> Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.

This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.

The frontier labs build massive models because if you're competing and you have a lot of cash and brute force is a viable option then it's easy and predictable. But the fundamental research itself doesn't in general require scale (certainly not entire datacenters) and models at any given capability level keep shrinking.

I keep repeating myself at this point but the human brain is on the order of 20 watts. That's a fraction of a single datacenter GPU! So again, would you have us destroy all chip fabs and ban all biomedical research?

> Why are you treating those as if they're separate things?

Because they are? You haven't explained at all your belief that they are the same.

> This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.

So it's completely wrong but it's only what has happened so far? You're repeating yourself but you're not listening to my responding to your exact points. I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?" In the last post.

To both points, you appear to broadly lack technical understanding of the field. You are assuming that how things appear to you reflects how they are at a lower level.

A chronological review of the architectures of the best open models over the past 3 years should immediately make the problem clear to you. (If you really refuse to put in any effort then perhaps just dump this thread into a frontier model rather than responding.)

> I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?"

I don't see it? It's a rhetorical question to drive home the point about the human brain. Chip fabs and biomedical research are both viable pathways to more efficient hardware. Thanks to biology we know what's possible. If you outlaw something but don't effectively cut off all the pathways to it then someone will still do it if it's profitable.

Nukes don’t get “worse” because of physics, not because of laws. You can’t make a pocket Tsar Bomba even if there were no rules against proliferation.

Dial-a-yield nukes already provide a range of yields from tiny to massive, and this provides all the operational flexibility that is needed considering that nobody’s willing to actually launch them in anger. (And this reluctance again isn’t about laws, it’s about the military realities of what would happen across the world in the aftermath of even a single tactical detonation.)

Anti-proliferation agreements and laws only slow “rogue” states from rising to nuclear capability. The immense cost, complexity, technical difficulty, and danger already provide a sufficient cost to deter civilian development.

> Nukes don’t get “worse” because of physics, not because of laws.

Yes, that's my point, that's why the metaphor is wrong

Sorry, I was tired and misread the intent.

I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.

So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.

Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.

Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.

I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.

The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.

The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.

If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.

Giving the big boys a regulatory moat isnt going to do anything positive for the space.

I don't think giving them free reign is going to do anything positive for it either.

But which outcome is more likely with the US government?

You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe?

The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348

Reply on news.ycombinator.com