Multimodal AR that helps low-vision users cook independently
AR that shows you where to grab — published research helping people with low vision cook safely and independently.
CookAR is a head-mounted AR system that recognizes kitchen tools in real time and overlays affordance augmentations — visual cues that distinguish the part of a tool you grab from the part that can hurt you. It received the Best Paper Award (Belonging and Inclusion) at UIST 2024, ACM's flagship venue for user interface software and technology.
The system pairs a fine-tuned affordance segmentation model with a stereo-camera AR headset, generating real-time visual augmentations directly on the tools a user is reaching for. The team collected and annotated the first egocentric dataset of kitchen tool affordances to train it, then refined the augmentation designs through a lab study with ten low-vision participants.
I co-authored the research as part of a cross-institution team spanning the University of Washington, the University of Wisconsin–Madison, and UT Dallas.