Computational notebooks, such as Jupyter notebooks, are interactive computing
environments that are ubiquitous among data scientists to perform data wrangling and
analytic tasks. To measure the performance of AI pair programmers that automatically
synthesize programs for those tasks given natural language (NL) intents from users, we
build ARCADE, a benchmark of 1078 code generation problems using the pandas data
analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to …