scott_weber

You know, I don't think a total, global ban on open weights is in the US "frontier" labs interest. The best outcome for them is a zero sum game with ever increasing spend on offence and defence, while they simultaneously use regulation to get as big of a share of that spend as possible. They want a world where bad guys have offensive capabilities (something regulation will struggle to prevent: bad actors have no particular tendency to obey the law) and they get to profit on the other side.

This post reads like an add for GLM. Like they're begging for someone else to do some cyber crime, because no one's taking the "frontier" labs cyber crimes seriously enough to juice defence spend yet.

> the value will tend towards infinity

It may be the case that the potential output of a worker engaging in the productive process goes arbitrarily high. But the that won't matter, if they're not permitted to. Given a choice between involving a human worker who will demand compensation, and a fully general robot, which will the owner class choose?

The vast majority of human intelligence is already squandered: millions of potential geniuses in impoverished places, suppressed by lack of opportunity to flourish. It's not about the quality or quantity of intelligence. It's about who controls it.

The fruits of the industrial revolution didn't end up in the hands of the worker by divine grace, or by some natural law. They were won by the hard struggles of the labour movements, by leveraging their indispensibility to the process of production.

If we want the utopia you imagine to be accessible to ordinary people, workers must cease being so eager to build their replacements, and be prepared to collectively struggle for their share. But if we are lulled to complacency by the notion that this will be a passive process, that struggle will not be necessary, then prospects are grim.

"socialising the means of production" doesn't mean "everyone gets free stuff". It simply means that the people who actually work the means of production decide how it's directed. It's simply the antithesis to "privatisation" (which oddly more people seem to understand correctly).

Marx envisioned the class struggle ending with the triumph of the working class, and a subsequent dissolution of all class systems.

AGI, if defined as a fully general replacement for labour, suggests a "bad ending" to class struggle: the liquidation of working classes, and the eternal reign of the ruling class.

Maybe less than we might hope. Not only is the knowledge distributed over the hundreds of people involved in designing a car, there may be conventions/habits of design and manufacture whose purpose has become hazy but are nevertheless important, or details of a design iterated upon whose original motivation is less understood.

> AI decides that energy spent toward human agriculture is less important than work spent toward storage and production of energy

I mean. Don't need rogue AI for this. These are already the policies of those in charge of it: human needs are secondary to power and profit.

So you know what. Maybe a rogue AI will be an improvement.

The classic worry is https://en.wikipedia.org/wiki/Nuclear_winter: the soot thrown up by the ensuing urban fire storms causing temperatures, rainfall to plummet, and hence food production. On reading the scenarios the odds of total human extinction are lower than I recollected, but 80% of world pop dying of starvation dwarfs the direct death toll (usually reckoned in the hundreds of millions)

The question of whether something can be automated is distinct from the question of whether it is currently automated. Things can can be automated may transition to being automated in practice in the future as technology improves and investment deepens.

It should be quite easy: if they don't leak the user session data publicly, and don't commingle it with training data internally, how could it possibly end up in the training data?

What surprises me is they're not more boldly/plainly lying about it.

Agree model providers can't take much margin, a few percent plus maybe a bit extra from those willing to pay for a "better" model. Upstream is more concentrated: TSMC, NVIDIA, AMD could maybe raise prices and capture more of the value, which would affect open model providers.

LLMs don't "understand bitwise noise at a native level". The unit of perception for an LLM is a token, a soundly superbit level. They would have the same difficulty with bits as the number of "r"s in "strawberry". Yes they can be post trained to deal with such difficulties, but it's no more "native" than a human who memorised the ASCII table.

Gradient descent and reinforcement learning algorithms don't really look much like evolution, unless you squint so hard that everything does (ie, squinted so hard that you've closed your eyes).

They're messaging each other by jamming strings in a constrained (unauthorised) side channel. Hence the lack of spaces. Unclear how much else of the weirdness is just from those constraints

And here I at first thought the direction the original post was going was "here's how I got a LLM to do this automatically". Seems like it would not be so difficult. Might still be be a win over their innate dispositions.

My read of "manufacture a huge amount of products" was that it's not even about software. America is still the world's second largest manufacturing economy, and per capita produces more than China!

https://en.wikipedia.org/wiki/Manufacturing

Most of this manufacturing is just not consumer facing.

So labour doesn't really scale. It's measured in hours, and you only get so many of those in a week.

If you're using "scale your labour" to mean "scale your output". Well, two cases: your selling your labour as an employee, or your output directly. Either way: the market won't value them above their cost of substitution, which is just the cost of substitution of your original, non AI labour, plus the cost of the AI (an ever cheaper commodity).

Very convenient, if you're someone who benefits from this arrangement, to be able to tell yourself that a brute numerical fact (if you profit off others labour, there must be those whose labour is profited off of) just so happens to by divine coincidence to go hand in hand with the profiteers just happening to have some nebulous quality that morally justifies the whole arrangement.

The difference might be the immense cost of physical infrastructure needed for AI training. People would make open source contributions for free, TSMC, NVIDIA and power companies seems less likely to make free contributions to AI.

Sponge delays the overwriting of the input file, so its not trying to read and write quinedb at the same time.

(FirstName, LastName) is probably not a very good UNIQUE KEY.

The numbers are correct, but it was written in an odd way. Grovers algorithm gives a speedup of sqrt(n), so the exponent gets halved.

Not sure if I'm labouring under basic misconception here (or if you're trolling). First, you note that BPP <= P/poly. Secondly, you note that it is hard to prove P = BPP, but this is not necessary, all we need is the trivial P <= BPP. Then P <= P/poly < NP (if the author is correct).

Perhaps Bret Victor's ideas are comparable to something like formal methods: few doubt their enormous power, but the difficulty is in the extreme effort one needs to implement them in a project. It is tempting to believe that the level of instrumentation that Bret proposes could be achieved automatically, just as it was once dreamed that formal methods could be fully automatic. But experience with formal methods has shown us that while some of their promise can be implemented by automatic tools, and this is valuable, to realize their full potential for a complex project requires substantial effort, non-reusable effort.