Multi-scale Fully Convolutional Neural Networks for Histopathology Image Segmentation: from Nuclear Aberrations to the Global Tissue Architecture

Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations (approximate to O(0.1 mu m)) through cellular structures (approximate to O(10 mu m)) to the global tissue architecture ((sic) O(1mm)). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder fully-convolutional neural networks with deep fusion. We present a simple block for merging model paths with differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019: hepatocellular carcinoma segmentation; BACH 2020: breast cancer segmentation; CAMELYON 2016: metastasis detection in lymph nodes). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimisation. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder fully-convolutional neural networks to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales. (C) 2021 The Authors. Published by Elsevier B.V.