Abstract:
Researchers across disciplines are increasingly using Generative Artificial Intelligence (AI) for labeling text and images or as pseudo-respondents in surveys. But of which populations are Generative AI models most representative? We use an image classification task -- assessing crowd-sourced street view images of urban neighbourhoods -- to compare assessments generated by GPT-4o with those from a nationally representative survey, and a locally representative survey of city residents. While GPT-4o closely approximates the perceptions of a nationally representative survey, it poorly approximates the perceptions of those actually living in the city. Examination of perceptions of neighbourhood safety, wealth, and disorder reveals a clear bias in GPT-4o toward national averages over local perspectives. We also document gender and racial biases in GPT-4o's neighbourhood assessments compared to human respondents in both surveys. Tailoring GPT prompts to encourage alignment with subgroup perceptions generally does not reduce bias and, in some cases, exacerbates it. The results underscore the limitations of using Generative AI to study or inform decisions in local communities but also highlight its potential for approximating "average'' responses to certain types of questions. Finally, our study emphasises the importance of carefully considering the identity and representativeness of labelers — a principle that applies broadly, whether Generative AI tools are used or not.
About the speakers:
Ala Alrababah (Bocconi) is an Assistant Professor in the Department of Social and Political Sciences at Bocconi University. Formerly a Postdoctoral Fellow at the Immigration Policy Lab, ETH Zurich, he earned a PhD in political science from Stanford University in 2021. His research focuses on political violence, immigration and refugees, and authoritarian media, using a combination of fieldwork, experimental approaches, and computational methods.
Elias Dinas (EUI) holds the Swiss Chair in Federalism, Democracy and International Governance. He holds a PhD in Political Science from the European University Institute (2010) and his research interests include the dynamics of political socialization, the downstream effects of institutional interventions and the legacy of authoritarian rule on the ideological predispositions of citizens in new democracies. He has also a keen interest in research methodology. His work has been published, among others, in the American Political Science Review, the American Journal of Political Science, the Journal of Politics, and Political Analysis and mentioned in The Economist, the Atlantic and the New York Times.
Melissa Sands (LSE) is affiliated to the Department of Government at the London School of Economics in 2021. Previously she worked at the University of California, Merced as an Assistant Professor of Political Science. She received her PhD from the Harvard University Department of Government in 2017, and earned an MPA from the School of International and Public Affairs at Columbia University. She studies the consequences of context on political and civic behaviour. Her research has been published in the American Political Science Review, Nature, the Proceedings of the National Academy of Sciences, the Journal of Public Administration Research and Theory, and elsewhere.