USING FACIAL EXPRESSIONS FOR CUSTOM ACTIONS: DEVELOPMENT AND EVALUATION OF A HANDS-FREE INTERACTION METHOD

Abstract

This study explores a novel facial expression-based interaction method designed to provide an accessible and hands-free alternative for performing precision tasks. Traditional input devices, such as keyboards and mice, are often unsuitable for individuals with limited mobility or for hands-free environments. The proposed system leverages standard computer hardware and machine learning-based facial landmark detection to map customizable facial expressions to specific actions, making it a low-cost and adaptable solution. This study evaluates the usability, learnability, and potential applications of this interaction method through a task-based user study. Sixteen participants aged 19–34 completed a series of five trials, performing the same color and number-matching task on an interactive grid. This approach allowed the evaluation of the learning curve by analyzing how participants improved their skills and reduced task completion times with each subsequent trial. Participants also provided feedback on challenging facial expressions to identify usability challenges. The evaluation focused on task completion time, participant-reported challenging actions, and qualitative feedback to assess system usability, user adaptability, and potential applications. The results indicate a clear learning curve, with participants improving task completion times over repeated trials. Feedback highlighted the potential of this interaction method for assistive technologies, gamified facial exercises, and as a supplementary input tool, while also identifying challenges such as facial fatigue and action complexity. The findings demonstrate the system's promise as an accessible and adaptable alternative interaction method, with opportunities for future refinement and broader application.