Evaluating AI Code Generation on Novel Frameworks Beyond Pretraining Data

Abstract

A study of large language models on code generation for novel frameworks. The APIs of these frameworks lie outside the common pretraining corpora. The study measures how well current models generalize to recent and specialized software ecosystems.

Record

Keywords