Hacker News Reader

Comment by uptownfunk | Hacker News Reader

Very sad to see they did not make it open source. When you have a technology that has the potential to be a gateway for drug development, to the cures of new diseases, and instead you choose to make it closed, it is a very huge disservice to the community at large. Sure, release your own product alongside it, but making it closed source does not help the scientific community upon which all these innovations were built. Especially if you have lost a loved one to a disease which this technology will one day be able to create cures for, it is very disappointing.

View on HN

I suspect https://www.isomorphiclabs.com/ is the reason.

There are 3 basic ways to fund research.

- Taxes - most academic research

- begging - research charities

- profits - companies like Google.

Sometimes the lines get blurred - but I don't think you can expect Google to release as much of their work for free as people who are paid via central taxes.

Worth noting that they did release the AlphaFold 2 weights after a while. Milking an expensive discovery for a limited period should be considered laudable, unless you think tax funding of all research would be awesome and it's just a weird anomaly how the org producing these results was a tiny heterodox startup very recently.

> Worth noting that they did release the AlphaFold 2 weights after a while.

Yes - though I don't think Isomorphic labs existed at that point.

Obviously the real reason AlphaFold was possible was the huge tax payer funded effort running over decades to generate a diverse, high quality 3D structure dataset.

However that's why we put taxes into research - to spur innovation in a pre-competitive way - so that's fine.

What's not fine is any benefiting company avoiding paying any tax back on resulting profits - that's just free riding - and many of the big tech companies are, in my view, guilty.

The data is a treasure. It was just as available to others, before and after.

Yep not knocking alphafold - Alphafold showed it could be done[1], and others have subsequently followed.

However if all the tech companies hoover up all the profits and simultaneously avoid paying the appropriate level of taxes - then the cycle of innovation isn't sustainable. As well taxes paying for that pre-competitive data, they train the next generation of PhDs.

[1] There were other groups making progress with DL based structure prediction before Alphafold, but alphafold was a leap forward.

You know I can appreciate that. For some reason because it’s related to medicine, it feels insidious to keep it closed.

I'd be willing to bet that OpenFold has an implementation inside of a few months.

https://openfold.io/

Openness and collaboration could have far-reaching implications for public health and well-being but there are lots of aspects of being open

> the community at large

Which community? Not any I'm part of.

The closer it gets to enabling full drug discovery, the closer it also gets to enabling bioterrorism. Taking it to the extreme, if they had the theory of everything, I don't think I'd want it to be made available to the whole world as it is today.

On a related note, I highly recommend The Talos Principle 2, which really made me think about these questions.

Any organization/country that has the ability to use a tool like this to create a bio weapon is already sophisticated enough to do bioterrorism today.

Alright, but now picture this: it's now open to the masses, meaning an individual could probably even do it.

The risks don't exceed what is already out there. If someone wants to do damage, especially in America, there are more than enough ways they can do it already. The technology should be made free. I also wonder how much the claims are being exaggerated and are marketing-speak vs. real results. Is there any benchmark for this that they have published?

No. I love to be egalitarian as well but this AI thing really feels different. We didn't just invent a better plow or a more durable sword. We're working on making a better brain. I think social media shows us a pretty good slice of the average person and it's not great. Now imagine they can manipulate the smartest person in the world to do dangerous, dumb shit.

>we didnt just invent a better plow

But we invented metalworking. And if metal were for kings chairs only, we would still have no plows.

I hear you, but I don't think an individual can. If I gave you $20,000 and an open source 70% accurate protein folding model and told you to develop, mass produce, and develop a deployment mechanism for a highly infectious and deadly pathogen, I don't think you could make that happen. Nor do I think you could do it if you had a PhD in microbiology.

right now

You raise an extremely important point. It appears to me that most people do not understand the implications of your point.

Organized terrorism by groups is actually extremely rare. What is much less rare are mass shootings in the USA, by deranged individuals.

What would a psychopathic mass shooter type choose as a weapon if he not only had access to semi-automatic weapons, but now we added bio-weapons to the menu?

It seems very clear to me that when creating custom viruses becomes high school level knowledge, and the tools can be charged on a credit card, nuclear weapons will be relegated to the second most likely way that our human civilization will end.

I believe the two concepts being brought together here are the Law of Large Numbers, and the sudden ability for one single human to kill at least millions.

> It seems very clear to me that when creating custom viruses becomes high school level knowledge

That would be very bad indeed, but there is no path from AI to that. Making custom viruses is never going to be an easy task even if you had a magic machine that could explain the effects of adding any chemical to the mix. You still need to procure the chemicals and work with them in very careful ways, often for a long time, in a highly controlled environment. It's still biology lab work, even if you know exactly what you have to do.

Also, bioweapons already exist and have been used in a few conflicts, even as recently as WWII. They're terrifying in many ways, but are not really comparable to the horror of nuclear weapons hitting major cities.

> Making custom viruses is never going to be an easy task even if you had a magic machine that could explain the effects of adding any chemical to the mix. You still need to procure the chemicals and work with them in very careful ways, often for a long time, in a highly controlled environment. It's still biology lab work, even if you know exactly what you have to do.

You appear to be talking about today. I am referring to some point in the future.

If you extrapolate our technological progress out to the future, it certainly seems possible, at some point.

Not based on AI biochemistry simulators, at the very least.

> You still need to procure the chemicals and work with them in very careful ways, often for a long time, in a highly controlled environment. It's still biology lab work, even if you know exactly what you have to do.

You can get that as-a-Service, and I imagine that successes in computational biology will make mail-order protein synthesis broadly available. At that point, making a bioweapon or creating a grey goo (green goo) scenario will be a divide-and-conquer issue: how many pieces you need to procure independently from different facilities, so that no one suspects what you're doing until you mix them together and the world goes poof.

We know the principles of how to make very powerful and dangerous anorganic compounds today, with extreme precision. Do you see any chemistry-as-a-service products that sell to the general public? Is it easy to obtain the components and expertise to make sarin gas, a clearly existing and much simpler to synthesize substance than some hypothetical green goo bioweapon?

> Do you see any chemistry-as-a-service products that sell to the general public?

Sort of? Depends on how general you insist the general public to be. Never used one myself, but I used to lurk on nootropic and cognitive enhancement groups, and I recall some people claiming they managed to get experimental nootropics synthesized and sent from abroad, without any special license or access. And then there's all the lab supply companies - again, I never tried, but talking with people I never got the impression it's in any way restricted, other than being niche; I never heard them e.g. requiring a verified association with an university lab or something. Hell, back in high school, my classmate managed to get his hand on some uranium salts (half for chemistry nerdom, half for pure bragging rights), with zero problems.

> Is it easy to obtain the components and expertise to make sarin gas, a clearly existing and much simpler to synthesize substance than some hypothetical green goo bioweapon?

Given that I know for a fact that making several kinds of explosives and propellants is a bored middle-schooler level problem, I imagine sarin is also synthesizeable by a smart amateur. Fortunately, the intersection of being able to make it, and having a malicious reason for it, is vanishingly small. But I don't doubt that, should a terrorist group decide to use some of either, there's approximately nothing that can stop from cooking some up.

What makes me more nervous about potential biosafety issues in the future is that, well, sarin is only effective as far as the air circulation will carry it; pathogens have indefinite range.

The problem of producing bio weapons is not computational, it's physical in nature. Even if predictions from these tools become 100% accurate and encompassed 100% of the chemistry, you still need to actually do the manual steps to breed the bioagents. And, very importantly, you need to do so without getting you and your co-workers deadly ill long before finishing the thing. Which requires extremely sophisticated machinery.

Alternatively, you can go today in some of the poorer corners of the world, find some people with drug resistant tuberculosis, pay them a pittance to give you bodily fluids, and disperse those in a large crowd, say at a concert or similar. You'll get a good chunk of the effects of the worse possible bioterrorism.

These are very specific ideas you have..

These are ideas you should read and think about because intelligence agencies all over the world have been thinking about them for the past hundred years.

It’s very frustrating when people consider the possession of information equivalent to malice. It suggests that the right way to run society is to keep people stupid and harmless.

Harmless is enough. The smart ones will figure out that knowing shortens lives.

Okay, I'll live in the harmless society, you go live in the harmful society.

We all live in the harmful society already right? I’m not aware of where to find this harmless society.

Suffering of finite beings is inevitable. While a very worthwhile goal, creating a harmless civilization isn’t possible. There are some common sense things we should do to prevent harm like negative consequences (prison etc) for needlessly harming each other. However, locking up knowledge doesn’t make much sense to me.

I’d rather explore the bounds of this world than mindlessly collect my drip of Soma and live comatose. To me that sounds more harmful.

I don't care about people harming people incidentally. I also don't want to shut down knowledge. But there is "knowledge" and there are "materials" that everyone agrees must be controlled and limited, like high explosives and bioweapons. Then the question is if large AI weights are a kind of "knowledge" or a kind of "material", and IMO they're much closer to material despite being data.

> I’d rather explore the bounds of this world than mindlessly collect my drip of Soma and live comatose. To me that sounds more harmful.

This only once again demonstrates that winning a debate is entirely about getting to define the choice under consideration. To me, it's not about Soma, it's about "humanity survives" and "humanity goes extinct due to out-of-control superintelligence." I don't want to die, so I'm for AI regulation.

Remember, I also said stupid. Ima go live in the non-dummy society and you can do whatever.

To be honest, apart from the containment systems you mentioned, much of biology basic research doesn't actually need that much sophisticated kit for basic cloning and genetic manipulation.

A lot of the key reagents can just be bought - and BTW it's why code like Screepy exist ( https://edinburgh-genome-foundry.github.io/ ),.

I think the real thing that stops it - it not that you can't make stuff that kills people, but the problem of specificity - ie how do you stop it killing yourself.

>>kit for basic cloning and genetic manipulation.

A lot of the key reagents can just be bought<<

along with the cryo-fridges required to keep said reagents.

add about 10,000 usd to your purchase request.

Most reagents and kits for molecular cloning will be fine at -20°C.

Honestly, I wouldn't worry about bioterrorism as much as handling mishap. Stick new proteins into a bacteria the wrong way, don't wash hands thoroughly enough, and suddenly something is eating all the trees in the region, or whatnot.

Designing an effective, lethal pathogen - fast enough to do damage, but slow enough to not burn itself out - is hard. Accidentally making something ecologically damaging is probably much simpler, and I imagine the future holds plenty such localized minor ecophagy[0] event.

[0] - Yes, I totally just learned that term from https://en.wikipedia.org/wiki/Gray_goo a minute ago.

Why do you need AI for bioterrorism? There are plenty of well known biological organisms that can kill us today...

Oh please, like a terrorist cant fork over a couple bucks to do the bioterrorism. This excuse is utter bs, whether its applied to LLMs or to alphafold. The motivator is profit, not safety.

Why on Earth would any terrorist want to invest into this when you can just purchase guns and explosives much easier?

Because a pathogen could potentially sicken or kill many more people.

Reply on news.ycombinator.com