Diffusion Language Models Are Versatile Protein Learners

Tuesday May 28th, 4-5pm EST | Zaixiang Zheng, PhD (ByteDance Research)

Abstract: This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioners, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance.

Preprint: https://arxiv.org/abs/2402.18567

 

Zaixiang Zheng is a senior research scientist at ByteDance Research. His research interest lies in large-scale generative modeling and its applications in various real-world problems, especially in human language and AI for Science. At ByteDance Research, he is working on generative protein modeling & design affiliated with AI Drug Discovery Team. Prior to joining ByteDance, he received his Ph.D in computer science from Nanjing University. Homepage: https://zhengzx-nlp.github.io/