Automatic Code Summarization via ChatGPT: How Far Are We?

• DOI: 10.48550/arxiv.2305.12865
• Publication Date: 2023-01-01
• Authors: Weisong Sun; Chunrong Fang; Yudu You; Yun Miao; Yi Liu; Yuekang Li; Gelei Deng; Shenghan Huang; Yuchen Chen; Quanjun Zhang; Hanwei Qian; Yang Liu; Zhenyu Chen
• Keywords: Python; Code (set theory)

To support software developers in understanding and maintaining programs, various automatic code summarization techniques have been proposed to generate a concise natural language comment for given snippet. Recently, the emergence of large models (LLMs) has led great boost performance processing tasks. Among them, ChatGPT is most popular one which attracted wide attention from engineering community. However, it still remains unclear how performs (automatic) summarization. Therefore, this paper, we focus on evaluating widely-used Python dataset called CSN-Python comparing with several state-of-the-art (SOTA) models. Specifically, first explore an appropriate prompt guide in-distribution comments. Then, use such ask comments all snippets test set. We adopt three metrics (including BLEU, METEOR, ROUGE-L) measure quality generated by SOTA NCS, CodeBERT, CodeT5). The experimental results show that terms BLEU ROUGE-L, ChatGPT's significantly worse than also present some cases discuss advantages disadvantages Based findings, outline open challenges opportunities ChatGPT-based