Projects

Xiangqi Bai, Hojoon Lee

Spatial imaging technologies can map the tumor microenvironment at single-cell resolution, but they remain costly and difficult to apply to large cancer cohorts. In contrast, H&E histopathology slides are widely available, but single-cell interpretation from these images is difficult to scale manually. To address this gap, we developed a transformer-based multimodal AI model that transfers spatial molecular labels onto H&E images and predicts single-cell cell types directly from histopathology.

We trained the model on colorectal cancer samples, including 40 multiplexed IHC slides and five Xenium spatial transcriptomic slides, covering approximately 34 million single cells. The model achieved 87.1% overall accuracy and a 96.1% macro-average AUROC. In an independent validation set of seven colorectal cancer samples with approximately 8 million cells, the model showed consistent performance, with 87.2% accuracy and a 95.0% macro-average AUROC. 

This framework enables automated and scalable single-cell-level cell type annotation from H&E images. It provides a quantitative foundation for studying immune–tumor interactions in colorectal cancer. By integrating H&E-derived cell type maps with tumor-specific genomic alterations from matched TCGA datasets, the model can support systematic analysis of key tumor microenvironment cell types, such as tumor-infiltrating lymphocytes, and their spatial cellular distributions. 

Additional Information:

https://aacrjournals.org/cancerres/article/86/7_Supplement/5491/778698