Text

Robust Multimodal Representation Learning With Evolutionary Adversarial Attention Networks

Multimodal representation learning is beneficial for many multimedia-oriented applications, such as social image recognition and visual question answering. The different modalities of the same instance (e.g., a social image and its corresponding description) are usually correlational and complementary. Most existing approaches for multimodal representation learning are not effective to model the deep correlation between different modalities. Moreover, it is difficult for these approaches to deal with the noise within social images. In this article, we propose a deep learning-based approach named evolutionary adversarial attention networks (EAANs), which combines the attention mechanism with adversarial networks through evolutionary training, for robust multimodal representation learning. Specifically, a two-branch visual-textual attention model is proposed to correlate visual and textual content for joint representation. Then adversarial networks are employed to impose regularization upon the representation by matching its posterior distribution to the given priors. Finally, the attention model and adversarial networks are integrated into an evolutionary training framework for robust multimodal representation learning. Extensive experiments have been conducted on four real-world datasets, including PASCAL, MIR, CLEF, and NUS-WIDE. Substantial performance improvements on the tasks of image classification and tag recommendation demonstrate the superiority of the proposed approach.

Ketersediaan

Informasi Detail

Judul Seri

IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION; Vol.25 No.5 October 2021

No. Panggil

-

Penerbit

: .,

Deskripsi Fisik

p. 856-868

Bahasa

English

ISBN/ISSN

-

Klasifikasi

NONE

Tipe Isi

-

Tipe Media

-

Tipe Pembawa

-

Edisi

-

Subjek

MULTIMODAL
REPRESENTATION LEARNING
EVOLUTIONARY
ATTENTION MODEL
ADVERSARIAL NETWORKS

Info Detail Spesifik

DOI: 10.1109/TEVC.2021.3066285

Pernyataan Tanggungjawab

Feiran Huang, Alireza Jolfaei, Ali Kashif Bashir

Versi lain/terkait

Lampiran Berkas

Komentar