Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM

• DOI: 10.48550/arxiv.2402.00097
• Publication Date: 2024-01-31
• Authors: Gabriel Ryan; Siddhartha Jain; Mingyue Shang; Shiqi Wang; Xiaofei Ma; Murali Ramanathan; Baishakhi Ray
• Keywords: Regression testing; Code (set theory)

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software (SBST) methods often struggle with complex units, achieving suboptimal test coverage. Recent work using large language models (LLMs) for generation have focused on improving quality through optimizing the context and correcting errors model outputs, but use fixed prompting strategies that prompt to generate tests without additional guidance. As result LLM-generated suites still suffer from low In this paper, we present SymPrompt, code-aware strategy LLMs generation. SymPrompt's approach is based recent demonstrates can solve more logical problems when prompted reason about problem multi-step fashion. We apply methodology by deconstructing testsuite process into multi-stage sequence, each of which driven specific aligned execution paths method under test, exposing relevant type dependency focal model. Our enables pretrained complete cases any training. implement SymPrompt TreeSitter parsing framework evaluate benchmark challenging open source Python projects. enhances correct generations factor 5 bolsters relative coverage 26% CodeGen2. Notably, applied GPT-4, symbolic path prompts improve over 2x compared baseline strategies.