Text

Self-Supervised Fusion for Multi-Modal Medical Images via Contrastive Auto-Encoding and Convolutional Information Exchange

This paper proposes a self-supervised framework based on a contrastive auto-encoding and convolutional information exchange for multi-modal medical fusion tasks. It is well known that multi-modal medical images have the same and unique features, and information redundancy is easily led when source image features are extracted in pairs. Inspired by contrastive learning, this article constructs positive and negative results pairs and proposes a novel contrastive loss in an auto-encoder. The paired source images are considered as positive and negative results used to reconstruct the source images to avoid the information redundancy problem. This article proposes preserving both the global and local features based on prior knowledge by combining transformer and convolution neural networks in parallel as an auto-encoder. A contribution estimation model is adopted to fuse multi-modal medical images. In the contribution estimation stage, an information exchange block is designed to exchange the feature maps of source images in multi-kernel convolutions, and then the multi-convolutional features are utilized to estimate the best fusion contribution of the paired source images. Experiments demonstrate that our method gives the best results than other state-of-the-art fusion approaches.

Ketersediaan

Informasi Detail

Judul Seri

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE; Vol.18 No.1 February 2023

No. Panggil

-

Penerbit

: .,

Deskripsi Fisik

p. 68-80

Bahasa

English

ISBN/ISSN

-

Klasifikasi

NONE

Tipe Isi

-

Tipe Media

-

Tipe Pembawa

-

Edisi

-

Subjek

NEURAL NETWORKS
INFORMATION EXCHANGE
FEATURE EXTRACTION
REDUNDANCY
CONVOLUTIONAL NEURAL NETWORKS
ENCODING
KNOWLEDGE ENGINEERING
TRANSFORMERS
SELF-SUPERVISED LEARNING
BIOMEDICAL IMGAGING
MULTIMODAL SENSORS

Info Detail Spesifik

DOI: 10.1109/MCI.2022.3223487

Pernyataan Tanggungjawab

Ying Zhang, Rencan Nie, Jinde Cao, Chaozhen Ma

Versi lain/terkait

Lampiran Berkas

Komentar