A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site.
He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.
A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.
The concern here is not if an amateur attempt to make a reactor, hack a bank, bioengineer a medicine/poison is successful or not. Interactive and instructive access to some forms of knowledge used to come with discretion along side instruction.
Yes, perhaps your swearing at me in this context is a little hysterical
Step 1. Obtain pliers
Step 2. Obtain 300 discarded smoke detectors
Step 3. Start yanking!
You'd maybe get a sketchy but near-viable plan that could be dangerous if asked for a dirty bomb, but there the danger would more be the conventional explosives and not where to source radioisotopes, as it was already common knowledge that most residential smoke detectors contained americium until recently.
Fair enough, I misread your original comment.
The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.
If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"
Should the library where he read books about physics also be liable?
E.g. look at programming - people who don't know how what a compiler is, are making things that I could only make after a few years into my programming journey.
You obviously get the same results in chemistry or nuclear physics or whatever, the models are heavily trained on code in particular, but if there's a chance that we've reduced the ease of committing certain kinds of crime that were previously gate-kept by knowledge, we should know about it.
I bet the professional would be able to sate the kid's curiosity safely without creating excessive risks.
I've come across detailed instructions on how to synthesize sarin gas on the internet. Anyone who follows those instructions will probably die horribly. I still thought it was pretty interesting.
This is a pretty different argument though. The comment that started this thread was talking about LLMs making potentially dangerous knowledge more available to bad actors, now we're talking about LLMs giving personally harmful advice.
You asked:
>If he had the help of Claude at the time, how much more dangerous would his bumbling have been?
Probably less? Even if you removed all the guardrails from Claude it would've likely told him his reactor plan wouldn't work and that he would have a high chance of poisoning himself and the environment.
There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?
But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.
I'm sure China et al are already doing this.
For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.
It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.
So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression.
1. Suppression / elimination of white collar jobs
2. Negative cognitive effects, especially for young people
3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction.
4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates.
5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves.
6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.
> A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen.
As long as that "gift" requires me to call up Sam Altman's datacenter every time I want to do anything with that "superintelligence", it's not empowering, it's deepening the control.
That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.
Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.
Never let your age stop your curiosity.
But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)
I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.
So I painstakingly ground down some charcoal to fine dust and redid the same experiment. That gave a much more impressive boom, but no dust plume, which puzzled me until I learned about dust explosions.
I blame my dad though, he found the recipe online and printed it off at work to bring to me.
Let the kids play.
I tried this as a grownup because I finally managed to get my hands on saltpeter (could only dream of it when kid). Followed the instructions, mixed everything in correct ratios, lit it with great care and fanfare and... hiss fizzle. I was so disappointed! I think it came down to purity of ingredients and not enough surface area.
Point is, there are certain details of the process required to make it truly work, that are not readily known; in a similar way with nuclear energy, the theory is pretty well known but some nitty gritty details like the implosion or detonator design are not.
To a point. Plenty of people from previous generations with missing digits and hands thanks to play with commonly available fireworks of the area (Australia based, so no idea how common that remains in the US).
My own experiments from my youth also one time resulted in some shrapnel punching through a 5 inch thick concrete tile very close to someone’s head (thought we were safe behind said tiles).
Get involved with the kids blowing stuff up so the danger is within reasonable bounds.
https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...
It really isn't.
A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got.
The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.
"A key factor that made the reaction possible was that, at the time the reactor went critical 1.7 billion years ago, the fissile isotope 235U made up about 3.1% of the natural uranium, which is comparable to the amount used in some of today's reactors. [...] the current abundance of 235U in natural uranium is only 0.72%. A natural nuclear reactor is therefore no longer possible on Earth without heavy water or graphite."
Another fascinating detail from the article, due to our understanding of fission, we can get some incredible results:
"The concentrations of xenon isotopes, found trapped in mineral formations 2 billion years later, make it possible to calculate the specific time intervals of reactor operation: approximately 30 minutes of criticality followed by 2 hours and 30 minutes of cooling down"
He hoped to create a breeder reactor, but he was very far creating a working breeder reactor.
Also:
"EPA scientists believed that Hahn's life expectancy may have been shortened due to his exposure to radioactivity, particularly since he spent long periods in the small, enclosed shed with relatively large amounts of radioactive material and only minimal safety precautions, but he refused their recommendation that he be examined at the Enrico Fermi Nuclear Generating Station."
Kids, don't play with Americium.