Embedded sensors in smart devices pose privacy risks, often unintentionally leaking user information. We investigate how combining an ambient light sensor with a device display can capture an image of touch interaction without a camera. By displaying a known video sequence, we use the light sensor to capture reflected light intensity variations partially blocked by the touching hand, formulating an inverse problem similar to single-pixel imaging. Due to sensors’ heavy quantization and low sensitivity, we propose an inversion algorithm involving an $\ell_p$-norm dequantizer and a deep denoiser as natural image priors, to reconstruct images from the screen’s perspective. We demonstrate touch interactions and eavesdropping hand gestures on an off-the-shelf Android tablet. Despite limitations in resolution and speed, we aim to raise awareness of potential security/privacy threats induced by the combination of passive and active components in smart devices, and promote the development of ways to mitigate them.
Initial Poster video at ICCP 2021.
Research Spotlight by MIT CSAIL in April 2022. Thanks Matt and Nate for neat production!
MIT News, Forbes, IEEE Spectrum, Ars Technica, and more (30+).
This work was supported in part by the DARPA REVEAL program (HR0011-16-C-0030) and MIT Stata Family Presidential Fellowship (Y.L.).
We thank L. Murmann, P. Sharma, A. Yedidia, V. K. Goyal, J. H. Shapiro, F. N. C. Wong, C. Saunders, S. P. Bangaru, T.-M. Li, X. Yuan, J. Suo, A. Kaspar, R. Spreitzer, F. Asim for discussions, and L. Anderson, T. Rubio for proofreading.
@article{Liu24PDI,
author = {Liu, Yang and Wornell, Gregory W. and Freeman, William T. and Durand, Fr{\'e}do},
title = {Imaging Privacy Threats from an Ambient Light Sensor},
journal = {Science Advances},
year = {2024},
month = {1},
volume = {10},
number = {2},
pages = {eadj3608},
url = {https://doi.org/10.1126/sciadv.adj3608},
publisher = {AAAS},
doi = {10.1126/sciadv.adj3608},
type = {Journal Article}
}