Development of Machine Learning Methods for Apparent First Impression Prediction: From single modality to causal methods

Abstract

First impressions play a crucial role in human interaction, shaping relationships and perceptions in both personal and professional settings. Accurately understanding and predicting these impressions can help reduce biases and promote fairer evaluations across areas like hiring, social networking, and interpersonal communication.
In computer science, scholars have aimed to improve first impression predictions using complex models, but these often lack explainability, making their decision-making processes unclear. Additionally, research in this area is limited by the lack of suitable datasets. Apparel, a key factor in shaping first impressions, has been largely ignored due to the absence of a database that effectively captures its impact on perception.
This study introduces a novel methodology that enhances model interpretability while maintaining strong predictive performance. A series of experiments were conducted using feature isolation and causal inference techniques to provide deeper insights into the decision-making process. The first phase employed a single-modal approach using Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to analyse static facial features and temporal emotion patterns, respectively. Focusing solely on visual data simplifies the data processing and model deployment while maintaining high predictive power. This approach achieved a top-level accuracy score of 0.9067, demonstrating its effectiveness. Beyond accuracy, the single-modal nature of our method provides significant advantages in deployability and interpretability, making it more practical for real-world applications.
The second phase investigates the causal impact of clothing attributes on first impressions using advanced causal inference techniques, including propensity score matching, instrumental variable analysis, and causal mediation analysis. The results demonstrate that specific apparel attributes significantly influence perceived personality traits, with formal clothing positively associated with conscientiousness and agreeableness, while casual attire is linked to openness and extraversion. These findings provide empirical evidence on the psychological impact of apparel, addressing a previously unexplored aspect of impression prediction.
To address the gap of dataset missing, the First Impression Static Appearance Dataset (FISAD) was developed, comprising 6,000 full-body images annotated with the Big Five personality traits. This dataset is the first large-scale resource dedicated to apparel and first impressions, bridging a significant research gap. Additionally, the Detect2Interact framework, a Large Language Model (LLM) based Visual Question Answering (VQA) system, was designed to streamline the dataset annotation process by automating labelling, effectively replacing time-consuming manual annotation.
These experiments collectively contribute to reducing model complexity and enhancing explainability, providing deeper insights into the factors that influence first impressions and paving the way for more accurate and interpretable AI models.

Date of Award	22 May 2025
Original language	English
Awarding Institution	Northumbria University
Supervisor	Honglei Li (Supervisor) & Wai Lok Woo (Supervisor)