JudgeGPT is a research project focused on the critical evaluation of news content generated by AI, specifically exploring the perception of authenticity in machine-generated vs. human-generated news. This early-stage work aims to gather insights on how individuals discern between real and artificial news content.
If you're interested in seeing this application in action and would like to participate in the evaluation of fake news, please visit our interactive survey at https://judgegpt.streamlit.app/.
The core of JudgeGPT lies in its interactive survey platform, built with Streamlit, which invites participants to read news fragments and assess whether they believe the content is generated by humans or machines. This process is important for understanding public perception but also for refining AI detection and generation methodologies in future developments. The project is integral to a larger research initiative, complementing our sister project RogueGPT, which focuses on the generation of (fake) news content.
The name JudgeGPT is thoughtfully chosen to reflect the core objective of this research project. The term "GPT" is employed in a pars pro toto manner, where it denotes not just the Generative Pre-trained Transformer models developed by OpenAI but extends to cover a broad spectrum of Large Language Models (LLMs). This choice signifies that while the project may initially focus on content generated by GPT models, it is inherently designed to evaluate news fragments produced by any advanced LLMs. The word "Judge" is used, as it directly relates to the action performed by participants within the project. Attendees are invited to judge the news fragments presented to them, determining their authenticity (real vs. fake) and origin (human-generated vs. machine-generated).
-
app.py: The main application script that powers the Streamlit web interface, facilitating the survey process, data collection, and interaction with a MongoDB database for result storage. -
requirements.txt: A simple file listing all necessary Python packages to ensure easy setup and deployment of the JudgeGPT application.
To participate in the development of JudgeGPT, follow these steps:
-
Clone the repository to your local machine.
-
Install the required dependencies listed in
requirements.txtusing pip:pip install -r requirements.txt
-
Launch the Streamlit application:
streamlit run app.py
Upon running the application, users are presented with a series of news fragments retrieved from a MongoDB database. Participants are asked to:
- Read each news fragment.
- Use sliders to rate their perception of the fragment's authenticity (real vs. fake) and source (human vs. machine).
- Submit their response, contributing to the research dataset.
This iterative process allows for the collection of valuable data on news authenticity perceptions, feeding into analytical studies aimed at improving AI news generation and detection frameworks.
JudgeGPT aims to provide a personalized user experience by automatically determining your language preference to tailor the survey content accordingly. However, should you wish to manually set your preferred language, you can easily change this in the app. Furthermore, it is possible to specify the user language through URL parameters. For instance, to set the language to German, you can use the URL https://judgegpt.streamlit.app/?language=de, or for French, https://judgegpt.streamlit.app/?language=fr.
Currently, JudgeGPT supports the following languages:
- English (
en) - German (
de) - French (
fr) - Spanish (
es)
As an early work in progress, JudgeGPT is continuously evolving, with updates and improvements being made regularly. The goal is to expand the scope of the survey, enhance the user interface, and deepen the analytical aspects of the project to provide richer insights into the dynamics of news authenticity in the age of AI.
Contributions to JudgeGPT are highly encouraged, whether in the form of code improvements, database enhancements, or analytical methodologies. If interested, please fork the repository and submit pull requests with your proposed changes.
While JudgeGPT has laid a foundational framework for evaluating perceptions of news authenticity, several exciting ideas remain on the horizon for implementation. These enhancements aim to deepen engagement, enrich the user experience, and provide more nuanced insights into the data collected:
-
Localization: Expanding the platform to support multiple languages and regional content, allowing for a more globally inclusive research scope. This would enable the collection of data across diverse linguistic and cultural contexts, offering a richer understanding of global perceptions of news authenticity.
-
Gamification: Introducing elements of gamification to encourage participation and make the evaluation process more engaging. This could include scoring systems, badges, or leaderboards to reward users for their contributions and accuracy in identifying fake vs. real news.
-
(Visualized) Results: Developing an interactive dashboard where participants can view real-time results and insights derived from the collective data. This visualization would not only make the project more transparent but also allow users to understand trends and patterns in news perception.
-
Personalized Feedback Mechanisms: Offering users personalized feedback on their performance, such as how often they correctly identify fake news or how their perceptions align with broader trends. This could further educate users on discerning news authenticity.
JudgeGPT is open-source and available under the GNU GPLv3 License. For more details, see the LICENSE file in the repository.
This project leverages pymongo for database interactions.
JudgeGPT is an independent research project and is not affiliated with, endorsed by, or in any way officially connected to OpenAI. The use of "GPT" within our project name is purely for descriptive purposes, indicating the use of generative pre-trained transformer models as a core technology in our research. Our project's explorations and findings are our own and do not reflect the views or positions of OpenAI or its collaborators. We are committed to responsible AI research and adhere to ethical guidelines in all aspects of our work, including the generation and analysis of content.