Text

VPN++: Rethinking Video-Pose Embeddings for Understanding Activities of Daily Living

Many attempts have been made towards combining RGB and 3D poses for the recognition of Activities of Daily Living (ADL). ADL may look very similar and often necessitate to model fine-grained details to distinguish them. Because the recent 3D ConvNets are too rigid to capture the subtle visual patterns across an action, this research direction is dominated by methods combining RGB and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the absence of appropriate sensors. This limits the usage of aforementioned approaches in real-world applications requiring low latency. Then, how to best take advantage of 3D Poses for recognizing ADL? To this end, we propose an extension of a pose driven attention mechanism: Video-Pose Network (VPN), exploring two distinct directions. One is to transfer the Pose knowledge into RGB through a feature-level distillation and the other towards mimicking pose driven attention through an attention-level distillation. Finally, these two approaches are integrated into a single model, we call VPN++ . It is worth noting that VPN++ exploits the pose embeddings at training via distillation but not at inference. We show that VPN++ is not only effective but also provides a high speed up and high resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the representative baselines on 4 public datasets. Code is available at https://github.com/srijandas07/vpnplusplus .

Ketersediaan

Informasi Detail

Judul Seri

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE; Vol.44 No.12 Part 2 December 2022

No. Panggil

-

Penerbit

: .,

Deskripsi Fisik

p. 9703-9717

Bahasa

English

ISBN/ISSN

-

Klasifikasi

NONE

Tipe Isi

-

Tipe Media

-

Tipe Pembawa

-

Edisi

-

Subjek

ATTENTION
EMBEDDING
POSE
TRIMMED VIDEOS
ACTIVITIES OF DAILY LIVING

Info Detail Spesifik

10.1109/TPAMI.2021.3127885

Pernyataan Tanggungjawab

Srijan Das, Rui Dai, Di Yang, Francois Bremond

Versi lain/terkait

Lampiran Berkas

Komentar