CodeFusion: A Pre-trained Diffusion Model for Code Generation

• DOI: 10.18653/v1/2023.emnlp-main.716
• Publication Date: 2023-12-10T16:58:19Z
• Authors: Mukul Singh; José Cambronero; Sumit Gulwani; Vu Le; Carina Negreanu; Gust Verbruggen
• Keywords: Disk formatting; Python; Code (set theory); Natural Language Generation

Imagine a developer who can only change their last line of code—how often would they have to start writing function from scratch before it is correct? Auto-regressive models for code generation natural language similar limitation: do not easily allow reconsidering earlier tokens generated. We introduce CodeFusion, pre-trained diffusion model that addresses this limitation by iteratively denoising complete program conditioned on the encoded language. evaluate CodeFusion task Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show (75M parameters) performs par with state-of-the-art auto-regressive systems (350M-175B in top-1 accuracy outperforms them top-3 top-5 due its better balance diversity versus quality.