ProtTrans: Understanding the Language of Life Through Self-Supervised Learning

Fantastic paper all around, and truly amazing results. The supercomputers they used at Oak Ridge National Laboratory have a mind-blowing amount of computational power, enough to make anyone training machine learning models a bit envious.

Here are quotes from the abstract:

> we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids.

> The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores.

> Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences.

> We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%).

> For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches.

> Taken together, the results implied that pLMs learned some of the grammar of the language of life.