banner of LMP department's DP&AI initiative

Zhu’s project details how deep learning is coming for the cancer cell

|
William Hoffman
6 minutes
Patients will benefit as pathology enters a new era

Dikgital pathology and artificial intelligence imageWhat is the difference between a cell and a bit? The answer is that a cell is the basic unit of life, and a bit is the basic unit of information.  They would seem to be worlds apart.  But those worlds are converging, and fast. 

The changing face of biomedical and clinical research is nowhere more apparent than in the rapid rise of computational biology and bioinformatics since completion of the Human Genome Project was announced in 2003.  No research laboratory bent on discovery can do without access to advanced computer hardware and software.  “Dry lab” scientists are becoming as essential as their wet bench counterparts.

With the rise of machine learning and artificial intelligence (AI), the cell’s precise deep-vision image and its precise biological function are coming into alignment.   Molecular imaging of tissue specimens is moving into the space normally reserved for wet bench testing and sequence analysis.  Plus the human body’s estimated 37 trillion cells, among them two trillion immune system lymphocytes that keep us alive, means massive datasets are needed if we’re to map ourselves the way powerful space telescopes map distant galaxies. That’s the goal of the Human Cell Atlas consortium.

Shijia Zhu photo

Computational biologist and LMP assistant professor Shijia Zhu and his research team are bringing the molecular and cellular imaging of cancer into sharp focus with their digital pathology project “Transferring and interpreting multi-modal deep learning models for pan-cancer histology and multi-omics integrative analysis.”

In their application, Zhu and his colleagues observe that the diagnosis of cancer is typically based on histopathology of tissue sections supplemented with multi-omics testing that may include genomics (DNA), transcriptomics (RNA), proteomics (proteins), and epigenomics (chemical modifications of DNA).   Integrating results from whole-slide histopathological images, ‘omics, and other technologies is now underway. 

“Artificial intelligence (AI) models can integrate complementary information and clinical context from diverse multimodal data sources to provide more accurate patient predictions,” they write, but note three major challenges for further biomedical AI model development that currently hinder the field:  the ability to generalize from limited data (transfer learning), collection of large, labelled training datasets, and human interpretability.  These limitations currently forestall or impede the direct use of existing large-scale AI models for small morphology-plus-omics datasets. They also make it difficult to interpret the underlying mechanism of tumor biology from AI models. 

Overcoming limitations

By overcoming these limitations, clinicians will be better able to classify and subclassify tumor types, find new therapeutic biomarkers, and better identify patients most likely to benefit from targeted therapy.  Although trained on multiple types of cancers, deep learning models will probably need to be developed for each type of cancer, Zhu said. 

Transfer learning in machine learning is the reuse of a pre-trained model from a previous problem on a new and related problem in order to boost performance and efficiency.  “Transfer learning is a kind of strategy of the new machine learning algorithm,” Zhu said in an interview.  “Models based on previous knowledge can be reused in new, similar application scenarios. Say, if you wanted your model to learn how to recognize a truck, then we could use a previous learning model for recognizing a car.  So, this is a very simple example of transfer learning.” 

In the biomedical research arena, sophisticated AI models trained on large datasets have been advanced that associate the pathological image, its elements and features, with omics data and other variables, he said. 

Because M Health Fairview has comparatively few relevant datasets, transfer learning will be used “to localize our more limited in-house datasets in existing pathology models built with massive data,” Zhu said.  Transfer learning will enable a “generalized capability” by synchronizing small in-house datasets with models such as PORPOISE (Pathology-Omics Research Platform for Integrated Survival Estimation), an interactive pan-cancer multimodal deep learning platform that integrates whole-slide histopathology imaging data and molecular omics profile data. Transfer learning will allow Zhu’s team to refine the parameters of its in-house dataset and forgo the need to build a larger dataset and the manual labelling and training that building such a dataset would require. 

The 'black box' of deep learning

The challenge of “human interpretability” for AI model development is associated with our incomprehension of how exactly AI works.  “Traditional deep learning is kind of like a black box,” Zhu said.  “We have data as input.  It goes into the model.  Then the output.  But we don’t know how it works.  People are now referring to multiple instance learning.  They use a term called ‘attention-based mechanisms’ in deep learning, a way of assigning different ‘attention’ weights or degrees of importance to different small inputs. It can be visualized as a heatmap overlaid on the pathology image, showing how each small area in the larger image contributes to the model of disease prognosis and potentially reflecting the biological relevance of each small area with the disease.” 

The terms of art are segmentation, partitioning, and extraction.  Powerful software like HoverNet segments and partitions cell nuclei from the most histopathologically relevant H&E slide images -- which marks the cell nuclei with more stain than the rest of the cell and thus are more easily delineated – and then extracts elements, features, structures, nodes, edges, and density from the compartment under scrutiny and analyzes them.  The final step is classification of the nucleus, which is associated with cell type, state, and health.  The sequential process enables the investigation of cell type composition and interactions between different cell types and accordingly assign varying degrees of importance to different parts of input data, then summing them up for an output that approximates the biological complexity of tumor microenvironment in play.  “The tumor microenvironment reflects the complex ecosystem surrounding a tumor,” Zhu said.  “Understanding it is crucial for uncovering the underlying mechanisms of cancers and developing new therapeutic strategies”

In their digital pathology project, Zhu and his team are also applying the powerful cell typing and improved interpretability of AI models in another exploration of a crucial component of the tumor microenvironment:  To detect the presence of tumor infiltrating lymphocytes (TILs) that cohabit with tumors, attack tumor cells, and influence response to immunotherapy (see “Giubellino’s team uses AI in a hunt for tumor-killing cell biomarkers”). 

“With nuclear segmentation and morphological information, deep learning allows us to define the cell type and its spatial distribution in the tumor microenvironment,” Zhu said.  “We will be able to see whether immune cells are close to the tumor.  This is an important predictor of overall survival and would be very informative for clinicians.”

“Transferring and interpreting multi-modal deep learning models for pan-cancer histology and multi-omics integrative analysis”

PI:  Shijia Zhu, PhD

Co-investigators:

  • Siddhartha Sen, MD PhD
  • Alessio Giubellino, MD PhD

  • Ryan Martinez, MD PhD

Here are the stories in our DP & AI series: