ESM3 – a multimodal generative protein language model

Show HN:

We trained a bidirectional masked generative model on protein sequence, structure and function. ESM3 work over tokenized representations of multiple modalities and can generate proteins with high fidelity and controllability. ESM3 further improves with feedback using alignment methods similar to Reinforcement Learning from Human Feedback (RLHF) applied in LLMs.

We have prompted ESM3 to generate fluorescent proteins with a chain of thought. Among the generations that we synthe- sized, we found a bright fluorescent protein at far distance (58% identity) from known fluorescent proteins. Similarly distant natural fluorescent proteins are separated by over five hundred million years of evolution.

- Check out our github repository: https://github.com/evolutionaryscale/esm

- Read the paper: https://www.evolutionaryscale.ai/papers/esm3-simulating-500-...

- And make your own proteins on Colab: https://colab.research.google.com/github/evolutionaryscale/e...

I’m surprised this isn’t making a bigger splash on here. Maybe people just have AI/LLM fatigue. But this sounds really incredible and has so much potential to lead to economically significant inventions in the near future.

Ever since deep convolutional NNs started doing well in molecular biology stuff (predicting small molecule drug effects from the structural descriptions of the molecules) around 2012 or so, I’ve been expecting there to be a big leap like this where we can basically dump in the massive databases that have been compiled over the last 30 years of gene sequences and protein structures/functional information, and have these powerful models figure out the hidden structure that is probably just too complicated for human minds to understand using “simple” math (obviously not that simple, I mean like partial differential equations) that assumes lots of symmetry.

It looks like the step up in power of Transformers is what was needed to make that next leap in explanatory power. Ironically, I feel like this is one of those cases where if you really had a model that worked well, you’d probably make more money by keeping it secret and using it to generate new and useful inventions (eg, bacteria that eat plastic, electricity generating algae, etc).