We study the ability of pretrained large language models (LLM) to answer questions from
online question answering fora such as Stack Overflow. We consider question-answer pairs
where the main part of the answer consists of source code. On two benchmark datasets—
CoNaLa and a newly collected dataset based on Stack Overflow—we investigate how a
closed-book question answering system can be improved by fine-tuning the LLM for the
downstream task, prompt engineering, and data preprocessing. We use publicly available …