Can ChatGPT Read Who You Are?

AI and psychology intersect to assess personality traits using ChatGPT. It shows competitive performance with a positive bias.

Major Takeaways

1. ChatGPT, a large language model-based chatbot, demonstrates competitive performance in inferring personality traits from short texts, outperforming human raters in several personality dimensions.

2. The study identifies a ‘positivity bias’ in ChatGPT’s assessments, as the chatbot tends to assign socially desirable scores across key personality dimensions.

3. ChatGPT’s performance in assessing personality traits is sensitive to the formulation of the prompt and the type of text, with different prompts impacting accuracy.

Introduction

Advancements in artificial intelligence, particularly in the analysis and generation of natural language, have allowed the development of intelligent assistants like ChatGPT, which can engage in coherent and contextually relevant conversations with users. While previous work has shown that personality traits can be reliably inferred from individual linguistic styles, the use of large language models in the domain of personality assessment through language analysis remains under-explored. This study aims to fill this gap by investigating ChatGPT’s abilities to infer personality characteristics from written text.

Related Work

The intersection between automatic natural language processing methods and psychology is an emerging field focusing on understanding and interpreting different aspects of human traits and behavior through language technologies. Previous research has established the link between individual linguistic patterns and personality traits, suggesting the potential of leveraging natural language processing (NLP) tools to automatically infer personality.

Method

The study analyzes data collected from a user study of 155 participants who wrote short texts in Czech and completed the Big Five Inventory (BFI) questionnaire to assess their personality traits. ChatGPT’s capabilities in inferring personality traits are evaluated by comparing its assessments with those of human raters and the participants’ self-assessments. Different prompts and types of text are used to understand the impact on ChatGPT’s performance in inferring personality traits.

Results

The study finds that ChatGPT’s assessments outperform human assessments according to most metrics in several personality dimensions, yet it also uncovers limitations in ChatGPT’s performance, such as a positivity bias, a dependency on the formulation of the prompt, and varying accuracy levels across different personality traits and text types.

Discussion

The study identifies a positivity bias in ChatGPT’s assessments and underscores the need for cautious and responsible use of AI in personal and psychological assessments, emphasizing ethical considerations related to privacy, consent, autonomy, and potential biases in automated personality analysis.

Acknowledgements

E.D. and N.O. are supported by the Valencian Government and Intel Corporation.

Critique

The study provides valuable insights into ChatGPT’s abilities in inferring personality traits from text. However, there might be limitations in generalizing findings to other language models or languages, and the results may be influenced by specific characteristics of the Czech language. Additionally, the study’s focus on ChatGPT’s performance and ethical implications could benefit from broader discussions about the implications for AI applications beyond personality assessment.

Appendix

Model	gpt-3.5-turbo-1106
Date Generated	2024-02-26
Abstract	http://arxiv.org/abs/2312.16070v1
HTML	https://browse.arxiv.org/html/2312.16070v1
Truncated	False
Word Count	10609