Anirudh Bindiganavale Harish1*, Peikun Guo1*, Bhargav Ghanekar1**, Diya Gupta1**, Akilesh Rajavenkatanarayan2, Manoj Kumar Sharma2, Maureen Elizabeth August2, Akane Sano1, Ashok Veeraraghavan1
1Rice University
2General Motors
*Equal Contribution, **Equal Contribution
Video Demonstrations
Abstract
Remote physiological sensing is an evolving area of research. As systems approach clinical precision, there is increasing focus on complex applications such as cognitive state estimation. Hence, there is a need for large datasets that facilitate research into complex downstream tasks such as remote cognitive load estimation. A first-of-its-kind, our paper introduces an open-source multimodal multi-vital sign dataset consisting of concurrent recordings from RGB, NIR (near-infrared), thermal, and RF (radio-frequency) sensors alongside contact-based physiological signals, such as pulse oximeter and chest bands, providing a benchmark for cognitive state assessment. By adopting a multimodal approach to remote health sensing, our dataset and its associated hardware system excel at modeling the complexities of cognitive load. Here, cognitive load is defined as the mental effort exerted during tasks such as reading, memorizing, and solving math problems. By using the NASA-TLX survey, we set personalized thresholds for defining high/low cognitive levels, enabling a more reliable benchmark. Our benchmarking scheme bridges the gap between existing remote sensing strategies and cognitive load estimation techniques by using vital signs (such as photoplethysmography (PPG) and respiratory waveforms) and physiological signals (blink waveforms) as an intermediary. Through this paper, we focus on replacing the need for intrusive contact-based physiological measurements with more user-friendly remote sensors. Our benchmarking demonstrates that multimodal fusion significantly improves remote vital sign estimation, with our fusion model achieving <3 BPM (beats per minute) error for vital sign estimation. For cognitive load classification, the combination of remote PPG, remote respiratory signals, and blink markers achieves 86.49% accuracy, approaching the performance of contact-based sensing (87.5%) and validating the feasibility of non-intrusive cognitive monitoring.
Dataset Access
Our multimodal physiological sensing dataset is available for research purposes. Due to the sensitive nature of physiological data, access requires signing a Data Use Agreement (DUA).
To request access to the dataset, please contact:
- Anirudh Bindiganavale Harish: anirudhbh@rice.edu
- Ashok Veeraraghavan: vashok@rice.edu
Please include your institutional affiliation and intended use case in your request.
Method Overview
Our approach combines multimodal remote sensing with contact-based physiological measurements to create a comprehensive benchmark for cognitive load assessment. The system integrates RGB, NIR, thermal, and RF sensors alongside traditional contact-based sensors.
Key Results
Vital Sign Estimation Performance
Multimodal fusion achieves <3 BPM error for heart rate estimation.
Vital Sign Estimation Performance
Multimodal fusion achieves <3 BPM error for respiratory rate estimation.
Cognitive Load Classification
86.49% accuracy using remote PPG, respiratory signals, and blink markers.
Citation
@inproceedings{
harish2025cogphys,
title={CogPhys: Assessing Cognitive Load via Multimodal Remote and Contact-based Physiological Sensing},
author={Anirudh Bindiganavale Harish and Peikun Guo and Bhargav Ghanekar and Diya Gupta and Akilesh Rajavenkatanarayan and MANOJ KUMAR SHARMA and Maureen Elizabeth August and Akane Sano and Ashok Veeraraghavan},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=VJEcCMx16R}
}