Radar and Video Multimodal Learning for Human Activity Classification

de Jong, R.J.; Heiligers, M.J.C.; de Wit, J.J.M.; Uysal, F.

doi:10.1109/RADAR41533.2019.171283

Radar and Video Multimodal Learning for Human Activity Classification

Title

Radar and Video Multimodal Learning for Human Activity Classification

Author

de Jong, R.J.
Heiligers, M.J.C.
de Wit, J.J.M.
Uysal, F.

Publication year

2019

Abstract

Camera systems are widely used for surveillance in the security and defense domains. The main advantages of camera systems are their high resolution, their ease of use, and the fact that optical imagery is easy to interpret for human operators. However, particularly when considering application in the defense domain, cameras have some disadvantages. In poor lighting conditions, dust or smoke the image quality degrades and, additionally, cameras cannot provide range information. These issues may be alleviated by exploiting the strongpoints of radar. Radar performance is largely preserved during nighttime, in varying weather conditions and in dust and smoke. Furthermore, radar provides range information of detected objects. Since their qualities appear to be complementary, can radar and camera systems learn from each other? In the current study, the potential of radar/video multimodal learning is assessed for the classification of human activity.

Subject

Human activity classification
Micro-Doppler
Video
Cameras
Dust
Network security
Object detection
Radar imaging
Security systems
Smoke
High resolution
Human operator
Lighting conditions
Multi-modal learning
Optical imagery
Radar performance
Range information
Radar