CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation

• DOI: 10.48550/arxiv.2401.06786
• Publication Date: 2024-01-01
• Authors: Yifei Xu; Yuning Chen; Xumiao Zhang; Xianshang Lin; Pan Hu; Yunfei Ma; Songwu Lu; Wan Du; Z. Morley Mao; Ennan Zhai; Dennis Cai
• Keywords: Benchmark (surveying); Benchmarking; Speedup; Code (set theory)

Among the thriving ecosystem of cloud computing and proliferation Large Language Model (LLM)-based code generation tools, there is a lack benchmarking for in cloud-native applications. In response to this need, we present CloudEval-YAML, practical benchmark configuration generation. CloudEval-YAML tackles diversity challenge by focusing on YAML, de facto standard numerous tools. We develop with practicality mind: dataset consists hand-written problems unit tests targeting scenarios. further enhanced meet needs rephrasing questions concise, abbreviated, bilingual manner. The 1011 that take more than 1200 human hours complete. To improve during evaluation, build scalable evaluation platform achieves 20 times speedup over single machine. best our knowledge, first an in-depth 12 LLMs, leading deeper understanding as well effective methods task performance reduce cost.