In Generative AI We Trust: Can Chatbots Effectively Verify Political Information?

Kurzanzeige
dc.contributor.authorKuznetsova, Elizaveta
dc.contributor.authorMakhortykh, Mykola
dc.contributor.authorVziatysheva, Victoria
dc.contributor.authorStolze, Martha
dc.contributor.authorBaghumyan, Ani
dc.contributor.authorUrman, Aleksandra
dc.date.accessioned2025-01-13T12:32:32Z
dc.date.available2025-01-13T12:32:32Z
dc.date.issued2024
dc.description.abstractThis article presents a comparative analysis of the potential of two large language model (LLM)-based chatbots—ChatGPT and Bing Chat (recently rebranded to Microsoft Copilot)—to detect veracity of political information. We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ + -related debates. We compare how the chatbots respond in high- and low-resource languages by using prompts in English, Russian, and Ukrainian. Furthermore, we explore chatbots’ ability to evaluate statements according to political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts. We also systematically test how such evaluations are influenced by source attribution. The results show high potential of ChatGPT for the baseline veracity evaluation task, with 72% of the cases evaluated in accordance with the baseline on average across languages without pre-training. Bing Chat evaluated 67% of the cases in accordance with the baseline. We observe significant disparities in how chatbots evaluate prompts in high- and low-resource languages and how they adapt their evaluations to political communication concepts with ChatGPT providing more nuanced outputs than Bing Chat. These findings highlight the potential of LLM-based chatbots in tackling different forms of false information in online environments, but also point to the substantial variation in terms of how such potential is realized due to specific factors (e.g. language of the prompt or the topic).
dc.identifier.citationKuznetsova, E., Makhortykh, M., Vziatysheva, V., Stolze, M., Baghumyan, A., & Urman, A. (2024). In Generative AI We Trust: Can Chatbots Effectively Verify Political Information? Journal of Computational Social Science, 8(1), 15. https://doi.org/10.1007/s42001-024-00338-8
dc.identifier.doi10.1007/s42001-024-00338-8
dc.identifier.issn2432-2725
dc.identifier.urihttps://www.weizenbaum-library.de/handle/id/782
dc.language.isoeng
dc.rightsopen access
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectAI auditen
dc.subjectLLMsen
dc.subjectDisinformationen
dc.subjectMisinformationen
dc.subjectConspiracy theoryen
dc.titleIn Generative AI We Trust: Can Chatbots Effectively Verify Political Information?
dc.typeArticle
dc.type.statuspublishedVersion
dcmi.typeText
dcterms.bibliographicCitation.urlhttps://doi.org/10.1007/s42001-024-00338-8
local.researchgroupPlattform-Algorithmen und digitale Propaganda
local.researchtopicDigitale Märkte und Öffentlichkeiten auf Plattformen
Dateien
Originalbündel
Gerade angezeigt 1 - 1 von 1
Vorschaubild
Name:
Kuznetsova_ea_2024_In-Generative-AI-We-Trust.pdf
Größe:
2.02 MB
Format:
Adobe Portable Document Format
Beschreibung:
Herunterladen
Sammlungen
Digitale Märkte und Öffentlichkeiten auf Plattformen