Emotional Decision Making Algorithm

Overview

This project investigates how emotional states influence decision-making by combining facial emotion recognition (FER) with the Iowa Gambling Task (IGT). By capturing real-time facial expressions and aligning them with decision points, the system reveals how subtle emotional shifts correlate with risk-taking behavior. The work positions emotion not as a byproduct of decision-making, but as an active and measurable driver of it.

Background

Decision-making under uncertainty is shaped by a complex interplay between cognition and emotion. While prior research has relied on physiological signals such as heart rate or skin conductivity, less attention has been given to facial expressions as a continuous and accessible source of emotional data. This project builds on that gap, exploring whether facial emotion recognition can provide meaningful insights into behavioral outcomes. It also engages with the broader question of whether awareness of one’s emotional state could enable more informed and intentional decisions.

Concept

The core idea is to externalize and quantify emotion at the exact moment of decision-making. By synchronizing facial expression data with behavioral choices, the project reframes decision-making as a temporally embedded process—where each choice is influenced by a dynamic emotional context. Rather than treating decisions as purely rational, the system reveals patterns where emotions such as sadness and anger correlate with poorer outcomes, while surprise is linked to more advantageous decisions.

The Project

This project investigates how emotional states influence decision-making by combining facial emotion recognition (FER) with the Iowa Gambling Task (IGT). By capturing real-time facial expressions and aligning them with decision points, the system reveals how subtle emotional shifts correlate with risk-taking behavior. The work positions emotion not as a byproduct of decision-making, but as an active and measurable driver of it.

Process

Participants completed a modified version of the Iowa Gambling Task under different emotional conditions while their facial expressions were recorded. Facial data was analyzed using FaceReader software, which classifies emotions and provides continuous measures such as valence and arousal. A custom Python pipeline was developed to synchronize timestamps between the behavioral data (IGT) and emotion data (FER), allowing each decision to be paired with corresponding emotional states. Statistical analysis, including logistic regression, was then used to identify relationships between specific emotions and decision outcomes. The results demonstrate that integrating FER with behavioral data can uncover meaningful patterns in how emotion shapes decision-making.