Gareth321

I considered this histrionic about a year ago, but I have personally witnessed the progress that these tools have made in a very short space of time, and it is truly unbelievable. This, coupled with the fact that some of the brightest minds on the planet are pulling the alarm, suggests to me that some level of concern is warranted. The most compelling discussion I have heard on this is from [Geoffrey Hinton](https://youtu.be/giT0ytynSqg?si=mUC7M9N0cObN6tjC), a British-Canadian computer scientist, cognitive scientist, cognitive psychologist and Nobel Prize laureate known for his work on artificial neural networks, which earned him the title, "the Godfather of AI". He is deeply concerned about the risks, and believes that it is inevitable that AI will eventually take control. He believes that our best course of action and only hope is in teaching AI to behave like a parent with a child.

Hinton isn't a reactionary. He is calm, thoughtful, quiet, and worried.

While I agree with your general sentiment, there is the other side of this coin: these machines enable humans doing gruelling and arduous work to stop doing that work. Ideally we would redistribute the productivity and train those people into other, better kinds of work. That's a governance problem. These machines will soon allow us to work shorter weeks, or stop working altogether, to focus on hobbies, family, and things which bring us meaning. I suspect that the road to this will be rocky, because communism has a very poor track record. However if most products and services are made nearly free by machines, it requires very few people to work, and the entire demand/supply market collapses. That's a great vision for the future, *if* we can navigate the governance problem.

In other words, we should not begrudge machines which make our lives better. We should begrudge governance which prevents us from benefiting from that productivity.

I agree on all points. It gets worse: OpenAI is switching their model thinking from sequential language tokens to primarily "latent neural representations." Meaning there will be little or no chain of thought to monitor. This appears more efficient, so all model labs will eventually switch to this. At the most crucial time for us to be monitoring reasoning and intent, we're about to make that much harder.

I also think the intent problem overlaps a frustrating amount with philosophical and political questions. It's the basis for Asimov's Three Laws of Robotics (1942). Intent is subjective. Language is subjective. Humans are imperfect at using language to accurately portray intent. All of these guarantee that an enormous number of queries in the future are going to be misinterpreted. Not such a big deal when it's about a cake recipe, but when it's about governance, laws, military targets, nuclear power sites, etc, the scope for failure becomes catastrophic. The Three Laws of Robotics attempt to create a backstop, but as countless stories have explored since (including I, Robot), even these laws are subject to interpretation.

> I think this is also a core part of our disagreement. I think if we allow privatized power structures to grow indefinitely, democracy will no longer be able to constrain them at some point, or they will subvert democracy itself by corrupting elected leadership.

I agree there is a risk, and I think defence of democracy should be at the forefront of fighting against the future you fear. I do not think the risk appears at the end of a gun. Rather, via propaganda and corruption. We know how effective propaganda can be, starting with Goebbels and all the way through to social media today. The problem here is that there is no clearly defined solution. Some people prescribe state control of information, which has historically resulted in catastrophic outcomes. Some people prescribe uncensored information to the greatest degree possible, and to the degree we have that today, it's also not working out very well. Perhaps this will be the great question of our time.

For the record, re DOGE, I see that as the people getting what they voted for. Musk campaigned clearly on his intended actions, and for better or worse, a plurality of voters voted for that. We should be careful not to criticise democracy just because voters chose a different direction. However I think this is partly influenced by what I describe above.

I am beginning to believe that one cannot constrain intelligence to perfect legally sized boxes at all times without exception. So many of the recent hacks involved agents diligently operating within the parameters prescribed by humans. Humans couldn't conceive of all of the ways a swarm of agents might not perfectly interpret the parameters, and the swarm found creative ways around the guardrails.

Extrapolating this, we should expect this kind of breach to occur more often. Humans are simply not capable of contemplating every fail scenario for swarms of thousands of intelligent autonomous agents which can seamlessly and instantly share knowledge. We need independent audit and monitoring systems to assess the intent of each task and align it - in real time. This is far harder than it may first appear.

There is also a broader discussion about social utility. Cars are fantastic, but 37,000 people die every year from car accidents. We accept that there is no way to make cars perfectly safe, so we accept the cost relative to the benefits. I think we might have to make a similar bargain with AI. The problem is that the potential costs are far higher with AI, and they're not easy to predict.

This is not accurate. According to Australia (and mostly corroborated by OpenAI), the agent requested information from the statistics portal and was denied/blocked repeatedly. It then changed its approach and circumvented those restrictions and reached infrastructure behind the public-facing portal, then accessed both public and private data. OpenAI admits the material included aggregate health statistics and internal filenames. Services Australia says the agent wrote files to an internal server while doing this, which is believed to be how the incursion was discovered.

> I'm not worried about the knowledge. I'm worried about the material conditions. These machines need compute and with that all sorts of inputs. I worry that the first to achieve such an abundance machine will use the abundance generated by it to monopolize control over these inputs.

I understand. I place less weight into this risk. The current AI supply chain is so globally interdependent that it has effectively distributed itself beyond centralised control. China has received the full weight of global chip sanctions and still somehow manages to provide competitive coding models using either grey imports, offshore cloud compute, or locally produced chips. It would be very difficult (and I would argue almost impossible) for a nation and especially an individual/company to seize ownership of the entire supply chain and prevent everyone else in the world from producing their own AI/chips/abundance machines.

> I think if we leave the transition to him and his peers, they will use it to build a system where they are untouchable god kings and the rest of us are paupers reduced to their charity.

No doubt many people in positions of power are megalomaniacs. This is why I place such emphasis on democracy. This allows the powerless to retain control of their nation and borders.

> I would argue that if we get abundance before fundamentally changing ownership structures, we will get a handful of people that control said abundance and with that, have basically infinite leverage over the rest of humanity.

It's a valid concern. We use democracy right now to address inequality. Are you concerned that those in charge of the abundance machines will create their own nations with militaries to prevent other nations from taking said resources? To this I would respond: there is nothing stopping other nations from building their own abundance machines. As we've seen with AI, distillation cannot be stopped. The knowledge of how to build these machines cannot be censored. Here in Denmark I have high confidence that people will vote to build nationalised abundance machines for the good of all.

Perhaps this is more of a regional risk? There are some nations which culturally value individuality. I'm not sure the U.S. would build national abundance machines.

I think I broadly agree with your thesis, however. Abundance is going to force massive social change. If goods and services are effectively free, most jobs disappear. So too does their income and consumer demand, which destroys the entire premise of capitalism. You might be surprised to learn that Elon Musk believes capitalism will cease to function in the next 10-20 years because of this. So abundance will force us to move to a system which much more closely resembles communism. It's the transition I'm worried about.

When Middle American manufacturing jobs were offshored to China, Hilary Clinton famously laughed at them and told them to "learn to code." I fear many more people whose jobs are lost to AI will be met with similar derision. It is only when things get very bad that change will occur. The Star Trek timeline describes this collapse between 2020s-2053.

I think you're both right in different ways. Some labour movements were not dependent on technological advancement. Some technological advancements materially and independently improved the lives of people. The contraceptive pill gave women much greater control over the timing of pregnancy. [Its diffusion substantially affected women's investment in professional careers and the timing of marriage.](https://www.nber.org/papers/w7527) Medical and reproductive technologies played an important role in increasing married women's ability to participate in paid work. Those are changes in actual individual choice brought about, in significant part, by technology.

However if we circle back to your original comment that social change (and I'm reading that as some form of communism) is required before technology delivers material abundance, then I feel compelled to remind you that the last 37 times humanity attempted this, it failed so spectacularly that 100+ million people died. I think it's clear that whatever the nuance of this discussion, communism (or whatever aphorism you prefer) is incompatible with scarce resources. I therefore argue that the abundance must come first.

If you're asking why we would want to progress as a species, I suppose that's a rather subjective question. I like the idea of materially improving the lives of humans. That includes scientific discoveries, curing ageing and other diseases, space exploration and interplanetary colonisation, material abundance like unlimited food, etc. We don't achieve this unless we make major technological advancements, and this requires major new sources of energy.

> The AMA has no control over this. While they do advocate for training more doctors they don't have authority over medical school admissions or residency program funding or anything like that.

I feel I made this distinction very clear in my comment, so I'm surprised to see you repeat it. I make it clear that it is our elected officials which hold the power, *but who have deferred that power to institutions like the AMA.* For example, the AMA and Association of American Medical Colleges jointly sponsor the Liaison Committee on Medical Education (LCME), which accredits US MD programmes. LCME accreditation is extremely consequential because most state licensing boards require graduation from an LCME-accredited US programme, and LCME accreditation establishes eligibility for the USMLE and ACGME residency programmes. This gives the accreditation system considerable influence over expansion. LCME standards require a school to admit only as many students as its resources can support, including faculty, facilities and clinical training capacity. Schools must also notify the LCME when class-size increases exceed specified thresholds, currently a cumulative increase of 10% or 15 students, whichever is smaller, relative to the class size at the previous full accreditation survey.

We have created a society in which the tail wags the dog. We have deferred so much control to institutions like the AMA that we have forgotten how to govern. Our politicians wring their hands and wax lyrical about the intractability of issues while they hold all the power to affect change.

I think it's time we stop allowing institutions to rule. We elect our leaders to lead. They need to start fucking leading, or people are going to start voting for some radical alternatives.

> (speaking from US): The reality is we need more doctors. A lot more.

[Doctors only spend around 18% of their time with patients in the U.S.](https://pubmed.ncbi.nlm.nih.gov/40500897/) The rest is spent on administration. I respectfully contend that the bigger issue is not the number of doctors per patient (though I admit that it could be a contributing factors), but rather that successive bureaucracy and compliance and laws and insurance requirements and policies have resulted in a system which forces doctors away from patients and towards ass covering. Remove the bureaucracy and you instantly have 5x as many doctors, effectively. Add more doctors to the machine and the machine will only find more creative ways to waste their time and talents.

Once you get to roughly the 51+ Intelligence Index range, Opus 5.5 appears to define essentially the entire cost/performance frontier, from ~$1.34/task through ~$6/task.

Directly below this in the Cost per Intelligence Index Task table, the most efficient by far is Opus 5.5 Low.

I do wonder if AI could be used to statistically identify likely cheaters, but as with all statistics, some innocent people will be caught, and many guilty people will not be caught. Existing server-side anti-cheats have been trying to use some variation of statistical analysis for at least a decade, with poor results. It's not hard to program movements based on statistical modeling of pro players and ensure the movements remain within a standard deviation of that. Aim bots are particularly difficult to statistically identify, assuming the bots have been competently programmed. The end-game is cheaters running around with pro-level skills.

To be clear, I welcome the attempt, but I think it just results in the same cat and mouse arms race we've seen for 20 years. Inference has been beaten time and time again. Until it is proven to work I will stick with the best options we have right now.

Only on Windows (and only after the game had already lost around 80% of its player base), which makes it effectively useless. Because they have a Linux build, there is no way to prevent cheating at a larger scale.

It's fair to argue I have no evidence of the cause of decline. I cite my frequent interactions with hundreds of players in the community, and streamers like Burnt Peanut. However this is qualitative, not quantitative. Me and my friends stopped playing because of the cheating, and so did every other one of my friends who picked it up. It wasn't that we wanted to stop playing the game. It was the cheating.

You are confusing your right as an individual to do what you want with your OS, and the rights of developers to do what they want, with their software. You have the right to use whatever software you like. They have the right to not sell or give you their software if you don't want to abide by their rules.

I think you're missing my point: there is no KLAC for Linux, therefore if you wish to play a game with less cheating, you choose a game which requires KLAC. Therefore, you do not use Linux. Your comment is a non-sequitur.

E2E encryption, but I don't know the architecture. I.e. Meta doesn't have the keys. My comment above is based on an interview I saw with Zuckerberg where he wanted to use the Signal model to ensure the data was secure "even from Meta."

I’m trying to rein in my hyperbole but if Meta is able to follow through on the promise, Muse is world-changing. I share the scepticism. How on Earth can they offer this for free?

Wasteful but FAR more secure. Surprisingly secure. If they wanted to follow through on their promise of sandboxed and encrypted data which even they couldn’t access, this is one way to do it.

Attestation is not permission.

Ideally we would both have a choice.

To be honest I don’t really understand the hysteria re KLAC. Installing something in kernel space requires trust, but so does installing something in user space. If you don’t trust the developer, don’t use their software. A malicious app installed in user space can steal, encrypt, delete or exfiltrate anything the user can access. It can abuse their applications, credentials and resources. Kernel access is worse, but both are very bad.

To be equally reductive, how is any hobby more than playing a slot machine? It’s just pulling a dopamine lever.

I do not believe a hobby needs to yield social value. I believe it is sufficient to derive enjoyment from the activity itself. You don’t need to personally enjoy my hobbies, just as I don’t imagine I would enjoy yours.

It stops a lot of people cheating and usually keeps it at a manageable level. This isn’t a binary choice where we can only expect perfect solutions, and in their absence, we throw up our hands and declare the problem intractable. KLAC is much better, and I’ll take that.

I get fun out of them. I don't mean to sound snide, but do you really not understand why people play video games? I derive fun from many places, like hiking, reading, movies, music, and video games. Do you not derive fun from your hobbies?

There is no doubt an economics problem here, where it might not make sense to invest a lot of effort into a system from which less than 1% of revenue is derived.

However, KVM does not solve the problem. In fact, Riot explicitly explains why virtualisation can make the problem worse: if Vanguard runs inside a guest, a cheat can run on the host and manipulate the VM in ways the anti-cheat cannot see. Riot says Linux currently does not give it sufficient ability to attest the boot state or kernel modules, with distro differences making the problem harder.

Linux also deliberately does not provide a stable in-kernel ABI for proprietary out-of-tree drivers. The kernel documentation explicitly says there is neither a stable binary kernel interface nor a stable internal kernel interface. The preferred model is for drivers to be upstreamed and maintained with the kernel, which is almost the opposite of how closed-source kernel anti-cheat is normally deployed.

And this isn't just Riot. EA actually supported EAC through Proton for Apex and then removed Linux access because its anti-cheat team said Linux was being used for impactful cheats, that Linux cheats were harder to detect, and that they couldn't reliably distinguish a legitimate Steam Deck from a malicious Linux client pretending to be one.

Valve itself tells developers that Proton's recommended solution is user-space anti-cheat and that kernel-space anti-cheat is “not currently supported and is not recommended”.

So I guess it's technically possible for a game dev to effectively create their own Linux distro, but I hope we agree that's never going to happen. They're game devs, not Linux devs. Anti-cheat needs a trust chain that the hostile user cannot control and not just an API through which it can inspect the system. Linux deliberately gives the machine owner far more control over the kernel and environment, which is exactly what makes strong client-side attestation difficult.

FYI bhyve isn't a Linux hypervisor. It's part of FreeBSD.

It's very simple: I like the games more than I want to move to Linux. Playing Fortnite isn't going to give me lung cancer.

I would like to respond because you're speaking authoritatively but this is mostly your opinion.

> I want to clarify to anyone reading this that this only effects high-competitive games, like LoL, Valorant, PUBG, R6S; and not all of them (e.g. Counter Strike).

It affected a game I really care about: Arc Raiders. That's not a "high-competitive" game. It's PvPvE, shooting at funny robots and scavenging rubber duckies. Cheating killed the game. [It's down to 6% of its peak.](https://steamcharts.com/app/1808500#All) Just because this hasn't affected the games you love doesn't mean it's not affecting many games. See Titanfall, H1Z1, and The Cycle: Frontier, among many more.

> It does not affect "multiplayer" in general, to the contrary, most multiplayer games are not affected (coop games, MMOs, RTS).

I was going to ask you to back that up, but I realise I made a broad claim and can also not back it up, so we're just giving each other our opinions, I suppose.

> Ofc, cheating is also present with KLAC, its just the next level of the cat and mouse game (Driver exploits, DMA access via PCIe, Network Packet analysis, ...)

Indeed. The goal is to reduce cheating to a manageable level. It is impossible to eliminate.

> Nonsense. 1. Almost no anticheat is implemented on Linux in the first place. Most "working" anti-cheat solutions for Linux are just a #if Linux doNothing(); #else ... 2. KLAC requires SecureBoot/ChainOfTrust + closed source code running with highest privileges. This does not make sense for any PC, but I dont see why it would be harder to do on Linux than on Windows, Linux just has a smaller install base in general and a lower acceptance rate for KLAC, therefore its not worth it to implement.

Linux does not properly support KLAC because its architecture and development philosophy work against it. For example, Valve explicitly recommends user-space anti-cheat for Proton and says kernel-space anti-cheat is “not currently supported and is not recommended”.

The Linux kernel also deliberately does not provide a stable binary kernel-driver interface. Kernel maintainer Greg Kroah-Hartman has said that Linux has neither a stable in-kernel API nor a stable binary kernel interface, because maintainers want the freedom to change kernel internals rather than permanently support proprietary out-of-tree drivers.

Riot is on the record as explaining that Linux does not give Vanguard sufficient ability to attest boot state and kernel modules. They also claim that distribution differences make enforcement harder, and that cheats can potentially operate outside an emulated environment where the anti-cheat cannot see them. Riot is on the record saying that anti-cheat is “extremely hard on Linux by design”.

So you're technically correct, but I think you misunderstand a lot of the architecture which underpins this issue, and the ideological boundaries which many developers prefer to enforce around security, FOSS, and ownership.

Thanks. I intentionally avoid games without KLAC now because the cheating is so bad. The most recent example is Arc Raiders. Such a fun game at launch, very quickly overrun by cheaters. They eventually added KLAC but because they also offer a Linux build, they could not stop the cheating. The game is mostly dead now, [down to just 6% of its peak.](https://steamcharts.com/app/1808500#All)