Skip to yearly menu bar Skip to main content


Poster

Assessing the Impact of ChatGPT in AI Conference Peer Reviews

Weixin Liang · Zachary Izzo · Yaohui Zhang · Haley Lepp · Hancheng Cao · Xuandong Zhao · Lingjiao Chen · Haotian Ye · Sheng Liu · Zhi Huang · Daniel McFarland · James Zou


Abstract:

There are many settings, where for accountability, transparency and other reasons, it is important to know how much of the text have been modified or written by AI. Academic peer review is one such setting. In this paper, we develop an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or written by a large language model (LLM). Unlike existing approaches which seek to classify individual pieces of text as human- or AI-generated, our maximum likelihood model considers the entire corpus of text and also leverages true human and AI generated reference texts, leading to greater accuracy and improved stability. We apply this approach to analyze reviews for AI conferences that took place after the release of ChatGPT---ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our estimator provides evidence suggesting that between 6.5\% and 16.9\% of the text in these reviews could have been substantially modified by LLMs. The estimated fraction is higher in reviews that have lower confidence, submitted close to the deadline, and are more likely to not respond to author rebuttals.

Live content is unavailable. Log in and register to view live content