# HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator

Source: [HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator](<https://www.sota2.com/research/paper/68fa74f31e209131cea64c7f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:22:19.696Z

Authors: Chang Bian, Beth Philips, Tim Cootes, Martin Fergie

Published: 2024-03-27T12:24:20.000Z

## Abstract

Computational analysis of multiplexed immunofluorescence histology data is emerging as an important method for understanding the tumour micro-environment in cancer. This work presents HEMIT, a dataset designed for translating Hematoxylin and Eosin (H&E) sections to multiplex-immunohistochemistry (mIHC) images, featuring DAPI, CD3, and panCK markers. Distinctively, HEMIT's mIHC images are multi-component and cellular-level aligned with H&E, enriching supervised stain translation tasks. To our knowledge, HEMIT is the first publicly available cellular-level aligned dataset that enables H&E to multi-target mIHC image translation. This dataset provides the computer vision community with a valuable resource to develop novel computational methods which have the potential to gain new insights from H&E slide archives.   We also propose a new dual-branch generator architecture, using residual Convolutional Neural Networks (CNNs) and Swin Transformers which achieves better translation outcomes than other popular algorithms. When evaluated on HEMIT, it outperforms pix2pixHD, pix2pix, U-Net, and ResNet, achieving the highest overall score on key metrics including the Structural Similarity Index Measure (SSIM), Pearson correlation score (R), and Peak signal-to-noise Ratio (PSNR). Additionally, downstream analysis has been used to further validate the quality of the generated mIHC images. These results set a new benchmark in the field of stain translation tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multiplex Immunofluorescence Prediction on HEMIT (val & test)](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-hemit-val-and-test.md>): Multiplex Immunofluorescence Prediction; HEMIT (val & test); F1 Score: 66.3.

- [Multiplex Immunofluorescence Prediction on ORION (test)](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-orion-test.md>): Multiplex Immunofluorescence Prediction; ORION (test); F1: 36.

- [Multiplex Immunofluorescence Prediction on IMMUcan Dataset](<https://www.sota2.com/research/sota/multiplex-immunofluorescence-prediction-on-immucan-dataset.md>): Multiplex Immunofluorescence Prediction; IMMUcan Dataset; Pearson Correlation: 0.667.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.18501>)

[PDF](<https://arxiv.org/pdf/2403.18501.pdf>)
