User experience research plays a crucial role in software product development, focusing on user perceptions of the product and the emotions it invokes. However, many methods for measuring emotions still remain subjective and can lack sufficient accuracy and objectivity. We aim to address the subjectivity concern by proposing a multi-method user research approach, which could be applied in the context of interactions with software products and would be scalable and repeatable in remote user testing conditions.
We combine self-reporting, behavioral observation analysis, direct user speech, and AI-powered facial expression analysis. We evaluate our method in two case studies with 15 participants, analyzing the emotional responses of users interacting with a Utility App and an App Marketplace, utilizing the Customer Journey Map (CJM) Framework for deeper insights into emotional dynamics shifts.
The analysis results indicate that, although AI analysis of emotions has limitations, the overall methodology partially correlates with observer analysis. Both methodologies are more effective in reporting emotional downs, while self-reported data tends to show emotional shifts more boldly.