disgruntledphd2

> Don't open-weight models rely heavily on distillation on frontier models?

We honestly don't know. Anthropic keep claiming this, and certainly have provided some evidence of this.

However, given that they've removed the real thinking output, and yet the Chinese models are still doing well, I'm sceptical of this belief.

> Is the need for "pretty large breakthroughs" based on realistic calculations, or are you just guessing?

I'm just guessing. But absent some kind of path to get there, I'd prefer to plan for needing to cover this situation with nuclear or other storage (I feel like pumped storage may make sense, but again haven't done the numbers).

Like, if you're correct then we'll end up with lots of power we don't really need. That's great, as we can use it to decarbonise other parts of the economy, however, if I'm correct and we dont build some kind of baseload/storage solution then we'll be in trouble in the winters, which I really want to avoid.

> Okay, so scaling batteries up would work, got it.

Maybe? Only if there are pretty large breakthroughs in storage, so we need a backup plan.

> Also bonuspoints for not doing the dumb thing and buying a special-purpose-only cargo bike but the smart thing of owning a regular eBike, a trailer hitch and a bike trailer. You can get so much done with a bike trailer.

Cargo bikes are great for kids. I would be far too terrified to put them in one of those trailers.

> it is analogous to what you see as the future for software developers

I didn't really make that point, so I have no opinion on that.

> Yeah but I mean, what is their compensation based on?

It's mostly just size of organisation and size of their book (the losses they are responsible for).

Like, all the risk is on the downside so they (AFAIK) don't tend to get huge bonuses for good performance.

> I have seen coding agents on my own machine (in sandboxed VMs) start doing things while trying to accomplish what I've asked that I felt sort of exceeded my mandate (changing database passwords, poking at the egress proxy that's preventing them from accessing some domains). Not to the point of causing any real issues, but I don't have much trouble envisioning scenarios like this when using stronger instructions around pursuing the goal + a running in a misconfigured sandbox envrionment.

Yeah, when I was using Claude Code some months back, I was building something that needed LLM calls in the frontend. Sonnet decided that the best way to accomplish its goal (get the tests passing) was to start grepping around my filesystem looking for my API keys.

One of the few cases where I've ever used the memory system, I told it to never do that under any circumstances. And then it did it again, a few days later.

RL is definitely an issue here, the models are getting trained based on task completion which leads to madness like this.

Batteries work for 1-2 days right now. These wind breaks can be 1-2 weeks.

Like, to be fair, batteries have improved so much in the last decade that this may be solved, but I'm generally sceptical of one thing that will save us, as I reckon we'll need multiple things to ensure resilience (which is really, really, really important when it comes to core needs like electricity and heating).

> What we need to do is restrict capitalism. Some capitalism is ok, I'm not advocating for a pure communist system, but there needs to be a balance between both. Since the fall of the soviet union, the balance has swayed wayy too much in favour of the unlimited capitalism option, whatever you call it.

I don't disagree with this at all, I'm all for red in tooth and claw capitalism, once every person has food, clothing, shelter, healthcare and education.

> And no the system isn't really the same, or at least it wasn't. Interest-free loans were pretty common and when I studied the government even provided them as student loans. Unfortunately they don't anymore.

I think that many people are missing my point here. Fundamentally, "capitalism" is structured (today, at least) around people with money giving it to others and getting the interest payments as the reward.

In a world with zero growth, where does the money for interest come from? (hopefully this is also a world with zero inflation, or we're all in trouble).

Again, philosophically I agree with a lot of this (our society is very rich, and yet millions of people starve and die from avoidable illness), but my core question is how we get there.

Preferably in small, incremental steps as that's the safest way for us in general.

> With this mindset, nothing ever changes. No social problem was ever solved without looking at it from a social/political perspective. If you agree that growth ideology is problematic, then that's what you need to dismantle.

OK great, what's the plan then?

Where do we start?

What small problem can we fix to make this better?

If you don't know the answers to the above, then this discussion is entirely pointless, and I would remind you that although I would love a pony, that doesn't mean that I'll get one.

Any "solution" that relies on changing the entirety of how society is structured isn't going to work, absent an incredibly good plan (against which one can make incremental progress).

To ask probably the most important question, in a world of zero growth, how do we pay back all the consumer debt we owe? Thinking specifically of mortgages here, as they're the largest amount of debt held by the populace.

> Europe only turned neoliberal a few decades ago and only some countries are deeply into it like the Netherlands. Before the 90s we were much more socialist and can go back that way.

We all have the same financial system. Like, if you have interest bearing loans, then some amount of growth is required to pay back the principal + interest.

I think calling it neo-liberal isn't really correct, it's fundamental to the economic system we call capitalism.

> We really have to stop the neoliberal system that's based on ever increasing growth and externalisation of environmental impact.

Like, I too would like a pony, but we gotta live in the real world. Ultimately we need more energy of basically every carbon neutral form.

I agree that the ideology of growth at all costs is problematic, but it's how our society is structured right now, so we need to fix our current energy problems rather than bet on transforming society.

To be fair, the declining fertility rate will probably solve a bunch of environmental issues over the next few generations.

> they will push nuclear and gas out simply by being cheaper and more predictable as an investment.

Unfortunately, this probably won't work everywhere. Particularly in Northern Europe winter is tough as there's very little sunlight (manageable) and often very little wind (not manageable,as it lasts longer).

Nuclear is an expensive way of solving this problem, but it's carbon neutral which is super important.

> But it was pervasive in the zeitgeist. Basically anyone trying to argue against it was backwards and stupid, or shrill if they were lefty.

It was so very weird. Even in Ireland (which was the target of the first wave of US outsourcing), people seemed to believe that everyone could be "knowledge workers" and that industry was unimportant.

I still don't really get why everyone believed this, but they did. Something something Upton Sinclair I guess.

> by using plain and boring computer vision

Technology is funny, I remember when an intern project detected cats in Youtube videos and it was front page news (round here, at least).

Now it's just boring technology.

> Gemini is similarly a step back. Google Assistant could hear me speak up and ask it to like or skip a song being played, driving or otherwise. It would quickly beep at me, pause the music/etc, accept input, process, then finish and resume the music (or skip to next track if asked).

Yeah I used the old Assistant regularly, but after the Gemini update it started refusing to set a timer for me, so I churned.

Badly.

It's probably Product Marketing Manager. So essentially selling the new stuff to new and old clients at scale.

> Yes, they run US models on US hardware and there is no alternative. In principle there are indeed Chinese models, but in practice there are also political considerations.

Just to be clear, there's actually more of a dependency when using US models, as the vast majority of Chinese models can be self hosted.

And I would have taken the point about political considerations much more seriously pre January 2025.

> The EU has become ever more dependent on the US in the past few years, notably on energy and AI.

Energy yes, particularly since the Iran war started. AI, not really. Like most Europeans are using US models, but the switching costs are so low here that they could easily swap these out for Chinese models.

It's the cloud/hyperscaler dependencies that are the real problems here.

I was amused to see DeepSeek using it recently, as I didn't realize it was installed on my machine (macos with brew).

Ruby is much, much prettier for similar purposes though, and so that's what I'll learn if I ever need to do this more frequently.

Many of them went rogue because of poor security controls on the part of one company who was a subcontractor for many of the labs.

Yeah, it's funny. Both of the recent models from the big us labs require you to ask what does this mean before they'll rewrite in simpler language.

Provenance this, gated that. It almost sounds like smart stuff, but then you dig deeper and it's vacuous.

Rao's piece is much, much better. You should submit it as a top-level link!

> a sufficiently powerful AI

We don't have this kind of AI, and (as of right now), unless humans link the AIs we do have up to a bunch of killer robots (i.e. drones) then this is vanishingly unlikely.

Like, any scenario where AI kills us all requires us to give it the tools that it needs to do this (as well as a desire to do this, as right now they just attempt to generate text to solve a problem/question they are posed).

> this effort was for the sake of passing a test for no clear gain

It's actually even worse, because the people building the graders at OpenAI were a bunch of clowns who didn't even implement the correct grader.

But yeah, large collectives of individuals performing loads of work based on a misunderstanding is one of the most human things ever, which isn't surprising given that we've trained these things on essentially all human text.

> I'd say it's mostly feeling a loss of control, which they couldn't place and our politicians were not open enough about explaining that this loss of control of the population was caused by.... the politicians themselves.

Yeah, the EU is essentially run by the national governments.

However, they typically (and the UK were particularly bad here) use the EU as a scape-goat for unpopular decisions even when they'd agreed to them in Brussels.

Understandable, but bad for Europe/EU in general.

They do not, as they're aiming for full replacement rather than augmentation of human users.

Personally, I think this is a bad idea, but someone's gotta build the Machine God I guess.

> But there's a lot of re-inforcement learning afterwards.

That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.

Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).

> Silicon Valley can do ad networks extraordinarily easily.

This is just not true, building an effective advertising platform costs significant amounts of money, time and people.

Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering.

Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads.

Furthermore, you need to figure out where to put the ads and how to rank them.

Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook).

They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers.

Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.

So, their property sector essentially collapsed from 2019-21, vaporising much of the wealth possessed by many Chinese people (who were often buying expensive flats of the plans).

As a result of this many local government pushed money into car/electronics/whatever export industries. This lead to vicious competition, and many of the exporters are tapped out of Chinese markets (vicious competition) so they are attempting to export as much as possible (which is rational).

Looking at things more broadly, one could argue that lots of the reason for the vicious competition is precisely that many Chinese people are still in loads of debt from the property bust (much like Irish/Greek/Spanish people were after their property busts).

The central government is pushing the exporting companies hard, as they don't want to have to inject loads of money into the economy (apart from local support of exporters).

This makes sense from the government perspective as they want to be able to build everything that they need (so the US/EU can't screw them), but it's pretty bad for the chinese people, many of whom are not spending (there's been some deflation over the past 5 years).

So, in many ways the Chinese economy is not in a good state, even lots of the exporters/industrial companies are doing badly, hence why they are focusing on exports.

And because Xi is basically leader for life, the likelihood is that these policies won't change until he dies.