MBZUAI Institutional Repository

Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pretraining objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pretraining framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that (1) xTrimoPGLM substantially outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced three-dimensional structural prediction model that surpasses existing language model-based tools. (2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science. Trained weight for the xTrimoPGLM model, and downstream datasets are available at https://huggingface.co/biomap-research.

By abusing access to a well-trained classifier, model inversion (MI) attacks pose a significant threat as they can recover the original training data, leading to privacy leakage. Previous studies mitigated MI attacks by imposing regularization to reduce the dependency between input features and outputs during classifier training, a strategy known as unilateral dependency optimization. However, this strategy contradicts the objective of minimizing the supervised classification loss, which inherently seeks to maximize the dependency between input features and outputs. Consequently, there is a trade-off between improving the model's robustness against MI attacks and maintaining its classification performance. To address this issue, we propose the bilateral dependency optimization strategy (BiDO), a dual-objective approach that minimizes the dependency between input features and latent representations, while simultaneously maximizing the dependency between latent representations and labels. BiDO is remarkable for its privacy-preserving capabilities. However, models trained with BiDO exhibit diminished capabilities in out-of-distribution (OOD) detection compared to models trained with standard classification supervision. Given the open-world nature of deep learning systems, this limitation could lead to significant security risks, as encountering OOD inputs whose label spaces do not overlap with the in-distribution (ID) data used during training-is inevitable. To address this, we leverage readily available auxiliary OOD data to enhance the OOD detection performance of models trained with BiDO. This leads to the introduction of an upgraded framework, unknown-aware BiDO (BiDO+), which mitigates both privacy and security concerns. As a highlight, with comparable model utility, BiDO-HSIC+ reduces the FPR95 by 55.02% and enhances the AUCROC by 9.52% compared to BiDO-HSIC, while also providing superior MI robustness.

Background: “Financial toxicity” refers to the financial burden imposed by treatment costs on individuals with cancer, constituting a major barrier to achieving equitable cancer outcomes. Recent literature increasingly demonstrates the detrimental impacts of financial toxicity on quality of life (QOL) among individuals with cancer, including individuals who have undergone hematopoietic stem cell transplantation (HSCT). This study evaluates associations among treatment cost burden and various aspects of QOL following HSCT. Methods: Seven hundred one HSCT recipients completed a survey examining their biopsychosocial health one year following transplant. The survey included the Functional Assessment of Cancer Therapy – Bone Marrow Transplantation (FACT-BMT), a multifactorial measure of QOL specific to this population. Treatment cost burden endorsement was measured on a 5-item Likert scale. Hierarchical regression models were developed to assess the incremental effects of demographic characteristics (i.e., Block 1), clinical predictors (Block 2), and cost burden (Block 3) on physical, emotional, social, functional, BMT-specific, general, and composite QOL outcomes. Results: Significant model improvement was observed with the addition of clinical factors (ΔF(2,650) = 20.28, p < .001), and subsequently, treatment cost burden (ΔF(1,649) = 110.29, p < .001). In the final model, higher cost burden was associated with poorer physical (β = -0.323, p < .001), emotional (β = -0.301, p < .001), social (β = -0.250, p < .001), functional (β = -0.317, p < .001), BMT-specific (β = -0.341, p < .001), general (β = -0.377, p < .001), and composite QOL (β = -0.381, p < .001). Poorer performance score was associated with each QOL indicator (p < .001), with allogeneic transplant type associated with poorer functional (β = -0.001, p = .002), but higher emotional (β = 0.118, p = .002), wellbeing. Older age (β = 0.113, p = .003) and female sex predicted higher (β = 0.183, p < .001), while Hispanic ethnicity predicted poorer (β = -0.095, p = .010), social wellbeing. Female sex was associated with poorer QOL specific to BMT concerns (β = -0.118, p = .001). Conclusions: Higher treatment cost burden is associated with poorer overall QOL and its physical, emotional, social, functional, and BMT-specific components one year following HSCT, after controlling for demographic and clinical characteristics. This reflects a critical barrier to equitable cancer care, suggesting that financial toxicity may perpetuate preexisting inequities in QOL, treatment, disease, and survival outcomes that disproportionately impact the underserved. Future research should prioritize 1) better understanding relationships among complex indicators of financial toxicity, QOL, and their underpinning mechanisms and 2) developing solutions to mitigate financial toxicity of HSCT and overall cancer care.