1,612 karma · joined September 12, 2013
> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.
[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).
Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.
Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.
Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.
In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
> The FernUniversität in Hagen is Germany’s only state distance-learning university, and its largest in terms of student numbers. Its flexible degree programs and continuing education options make higher education accessible to students in a variety of life situations. Its five faculties conduct a wide range of theoretical and applied research in their subject areas of Humanities and Social Sciences, Psychology, Mathematics and Computer Science, Economics and Business Administration, and Law.
I believe some exams have to be taken non-remote: https://www.fernuni-hagen.de/english/study/distance-learning...
And just as important: Help your friends and family to move as well, so they can have ad blockers, NewPipe etc. We need a critical mass of users invested in their freedom, otherwise its going to be crushed by malicious/dumb security measures of their banking apps, corporate greed ("oh, a simple misunderstanding, when you clicked 'buy' you rented a limited license. Did you not read the ToS?") and police overreach. It's a perpetual battle.
I agree, but would like to point out that sometimes this might not be what you want. I had a bike with a front motor and a cadence sensor that I used for my commute for a few years, and was very happy with the motor just pulling me along while I lazily pushed the pedals with ~zero torque, barely enough to let the bike know I'm there. Basically like pressing a throttle. If all you want is to get from A to B comfortably, I've found this to be superior to a torque sensor, and it was also dirt cheap.
The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.
The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.
[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.
[1] Structural inertia is the killer here. It will certainly not happen until the problem is huge enough.
[2] Exceptions can of course apply to numbers that are meant to primarily be cold called, like doctors offices. The callee possibly have to be specially trained to withstand this kind of attacks.
Do note, though, that it was the unbelievable irresponsibility of past operators that has spurred the anti-nuclear movement in the first place. See e.g. https://youtu.be/929B8sgOOTM?si=FttZr_MsbQ1hB4Nj&t=1664 from 27:44 to 31:35.
> 2022 was another consecutive year in which water levels of major European rivers – such as the Rhine, the Danube, and the Rhône – were dangerously low and the water temperatures very high. This caused severe problems for the operation of nuclear power plants across continental Europe. Energy companies in France, Germany, Switzerland, Belgium and elsewhere had to shut down their nuclear power plants partly or fully because there was not enough cooling water available or, more commonly, because the cooling water that was returned to the river became too warm (Barber, 2022; Limb, 2022; Miller and Vladkov, 2022). Environmental regulations, designed to protect the riverine flora and fauna as far as possible, stipulated that nuclear power plants were not allowed to release cooling water above a certain temperature (European Parliament and European Council, 2000; IKSR, 2022b). The resulting unplanned outages — and the efforts by nuclear operators to avoid such disruptions — highlight pressing concerns about the sustainability of nuclear energy, particularly its impact on river ecosystems in an increasingly warming world (see Fig. 1).
https://www.sciencedirect.com/science/article/pii/S030142152...
How much is possible there, is only constrained in our understanding of biology. How difficult that turns out to be for a super intelligence… who knows? If we are actually on the cusp of the AI singularity, the future is going to be weird and/or wonderful and/or horrible, but definitely unimaginable different than today.
My mind is kind of shooing away from this intuitively. Too hard to believe. My whole life experience has been living in a different world. But imagine if "we" could actually create human level intelligence, say, for the price of a 100k USD/EUR/GBP. It could only do knowledge work, of course, but it would easily pay for itself and thus be mass produced. What is the market cap for cheap knowledge work? I would be surprised if it is only a billion human-equivalents, given humans find new creative ways to pay for themselves all the time. That explosion alone is mind-boggling and it does not rely on super-intelligence.
All of this should make one point clear: At no other point in history has it been more important to have our power structures be aligned with the interests of society.
> Almost like there are aligned interests there rather than a purely adversarial relationship.
You might very well be the exception, but for something like 99% of marketing content that reaches me, our interests aren't aligned. First of all, they want to generate "needs" where there weren't any before and probably shouldn't be. A pizza ad produces the wish for unhealthy food. A fashion ad produces the wish for new clothes (even though I have enough) and probably even changes the societal dynamics of individual expression and personal style to be consumption oriented.
Second, even if I have a legitimate need for a solution, they still want me to buy their product, consume their media, give my attention to them. I, on the other hand, want to be informed by a neutral third party about the pros and cons of some product. Sure you can say "but unhappy customers are bad for us", but there are actually very few niches, where this signal is powerful enough to align incentives, because information and power asymmetry limit the customers understanding of product quality and their leverage to correct harmful market dynamics.
You can even go so far and have a public sub domain for each devices ( serialnumber.manufacturer.com ) which you only operate as a dumb proxy so that even the TLS certificates are negotiated end-to-end between the IoT device and Let's Encrypt. (The devices connect to your backend via Wireguard and you rate limit with their device individual key, whose public key you read out during the end-of-line production step.)
Hell, with today's browser heavy applications you can even run the whole slicer in the browser. Let the app be distributed via CDN so the code does not need to go through the proxy.
[1] In the case of non-battery operated and always or mostly on devices, like 3d printers at least.
> ugh sorry this was a bug with the 3rd party harness detection and how we pull git status into the system prompt
Claude wants to exercise control of how I use the "inclusive volume" that I purchased with my monthly subscription. This harms competition (someone else could write a more efficient or safer coding agent) and is generally not in the best interest of society. Why do we allow this?
This specific case is interesting, because it is so clear cut. There is no cross financing via ads, they already have the infrastructure to measure usage and even the infrastructure to bill extra usage. I also don't see how you can plausible make the argument that restricting usage to their blessed client is necessary for fair use or for the basic structure of their business model (this would be the standard argument for e.g. Youtube: Purposefully degrading the experience of their free client to not support background playback enables the subscription model).
Absolutely not. If people were advocating for ECC only, you would have a point. But this thread is about hybrids vs ML-KEM-only (for key exchange!). Everybody here wants to deploy the algorithm your favoring and wants to deploy it now, just not without a safety net.
Pro hybrid: Negligible performance impact (negligible for battery devices, negligible for data send over the wire (number of packets -> sub-discussion about specific circumstances, time on the air for cellular), negligible for speed, negligible code size increase), little implementation effort as every library already has ECC in it, ML-KEM is too new (yes actually old, but far less research interest, implementations new), conservative design choice
Pro ML-KEM only / produce a TLS RFC for non-hybrid ML-KEM: Reduction in complexity, reduction of transitions (non-hybrid is going to be the final state, so lets skip ahead already), lattice crypto is actually an old branch of cryptography (discussion over different metrics), NSA says its secure for government use, NSA stipulates use of non-hybrid and we want/need to be compatible, we want/need to have a well defined place to have a reference, if people are going to write an RFC to document non-hybrid ML-KEM let us at least have influence over what is written there, better performance (speed, data on the wire, number of packets in handshake, energy budget), actually the non-hybrid TLS connection is intended to be the inner one while the outer transport is secured with classic cryptography (or vice versa) so hybrids are a complete waste, for any interesting timeline ECC is broken anyway so it is a useless burden, we just want choice dammit, don't undermine the process dammit.
Pro hybrid only / don't produce a TLS RFC for non-hybrid ML-KEM: Let's not make it easy for people to choose wrongly by accident/incompetence/malice, actually no complexity reduction as implementations still need to implement hybrids to be compatible, TLS WG publishing something has weight and might sway others to consider non-hybrid ML-KEM, NSA might have pushed for non-hybrid ML-KEM because they believe only they can break it, don't care if US institutions are pushing for non-hybrid ML-KEM for weird internal political reasons, don't you see how this is all a ploy to weaken our crypto again?, don't undermine the process dammit.
Did I forget any important talking point? The TLS WG discussion is actually quite tiresome. For anybody new the party, here is a random pointer for a current thread: https://mailarchive.ietf.org/arch/msg/tls/7OGS_X1e-zG8O0eRJP...
Exactly in the way the succeeding sentence defines: "For both cases there are credible expert opinions that say the risk is incredibly overrated and credible expert opinions that say the risk is incredible underrated."
> when ECDH is nearly guaranteed to be broken in five years
Most of your argument (and that of many others pushing the contra-hybrid point) hinges on this. I don't think this position is justified. I believe there is significant risk for quantum attacks in the near term (and thus fully support the speedy adoption of hybrids), yes, but quite far away from certainty. Personally, I'd even say better than coin-flip is pushing it. I mean, look at what Scott Aaronson is writing on that matter:
"I also continue to profess ignorance of exactly how many years it will take to realize those principles in the lab, and of which hardware approach will get there first. […] This year [=2025] updated me in favor of taking more seriously the aggressive pronouncements—the “roadmaps”—of Google, Quantinuum, QuEra, PsiQuantum, and other companies about where they could be in 2028 or 2029." -- https://scottaaronson.blog/?p=9425
This is nothing like "nearly guaranteed" in five years.
> and Kyber is two decades old
But the implementations aren't and it's not been under heavy scrutiny for that long. One can very much make the point that we weren't that critical when elliptic curve cryptography entered the scene, but we do now have the luxury to have these heavily battle-tested primitives and implementations at our disposal, so why throw them out of the window so eagerly? Also an interesting comparison to elliptic curve cryptography is that it took until 2005 to get good key exchanges primitives and until 2011 to get good signature primitives (Curve25519, now known as X25519, and Ed25519 respectively) and mainstream availability of those took waaaay longer.
Coming back to this again, for second remark:
> when ECDH is nearly guaranteed to be broken in five years
Another important point is all quantum attack on ECDH will require inherently expensive equipment for the foreseeable future, see adgjlsfhk1's comment https://news.ycombinator.com/item?id=47665561 , whereas a stupid Kyber implementation error in a mainstream library can very likely end up being attackable by a Metasploit plugin. Our threat model should most definitely include nation state attackers prominently, but these are not at all the only attackers that we should focus on. There is still significant value in keeping out attackers that did not spend >100k$ on equipment.
> Yes, djb keeps making the same crankish complaint without any evidence or reason, that doesn't mean you have to repeat it uncritically.
I did not repeat it uncritically, I just happen to share his conclusion, even after months of following the pro and contra discussion. Also, how can you say he complains without reason? He has explained them at length, see https://cr.yp.to/2025/20250812-non-hybrid.pdf for example. Whether his methods of complaining are commendable or effective is another topic, though.
There is also a difference between closed ecosystems and systems that are composed of components by many different vendors and suppliers. If you are Google, securing the connection between data centers on different continents requires only trivial coordination. If you are an industrial IoT operator, you require dozens of suppliers to flock around a shared solution. And for comparison, in the space of operation technology ("OT"), there are still operators that choose RSA for new setups, because that is what they know best. Change happens in a glacial pace there.
If you worry about a >=1% risk of quantum attacks being available soon, you should also worry about a >=1% risk of the relatively new ML-KEM being broken soon. The risk profile is pretty comparable. For both cases there are credible expert opinions that say the risk is incredibly overrated and credible expert opinions that say the risk is incredible underrated.
Filippo has linked opinions that quantum attacks are right around the corner. People like Dan Bernstein (djb) are throwing all their weight to stress that anything but hybrids are irresponsible. I don't think there is anybody that says "hybrids are a bad idea", just people that want to make it easy to choose non-hybrid ML-KEM.
Re Oppenheimer: I know. My point was that he very much knew what his work was being used for, as should people working at xAI at the moment.
I don't know why this argument often pops up in these kinds of discussions. Approximately no one is judging people who have done their best effort to avoid doing harm. We are judging people who don't care in the first place.
This is definitely not universally true. E.g. photos are very cheaply printed on demand. Even on-demand books are printed at reasonable prices. Sure, mass production is cheaper (both for books and pictures), but the value difference of the individual product is high enough to bridge the price gap.
For cloth this area has found little exploration. TFA covers production at niche scale. If you would mass produce the looms to reduce the capital expense and heavily lean into customer value, e.g. individual fittings via 3d scans, as my sister comment proposes, or even just letting me customize my sweater with motive, color choice, garment etc., this could radically change the cost to value ratio. The company that has published TFA sells extremely bland apparel in a shop that looks just like any mass produced clothing shop and leaves all of the customer value of custom production on the table.
Last but not least: This "3d knitting" seems to need only a fraction of the labor of traditional sewed clothes. If textile production didn't default to underpaid labor under precarious working conditions in low income countries, it would probably already be cheaper.