Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

• DOI: 10.1145/3654777.3676345
• Publication Date: 2024-10-11T14:50:36Z
• Authors: Majeed Kazemitabaar; Jack Williams; Ian Drosos; Tovi Grossman; Austin Zachary He...; Carina Negreanu; Advait Sarkar
• Keywords: FOS: Computer and information sciences; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Science - Human-Computer Interaction; Human-Computer Interaction (cs.HC)

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle challenging task of data analysis programming, which requires expertise in processing, and statistics. However, our formative study (n=15) uncovered serious challenges verifying AI-generated results steering AI (i.e., guiding system produce desired output). We developed two contrasting approaches address these challenges. The first (Stepwise) decomposes problem into step-by-step subgoals with pairs editable assumptions code until completion, while second (Phasewise) entire three editable, logical phases: structured input/output assumptions, execution plan, code. A controlled, within-subjects experiment (n=18) compared systems against a conversational baseline. Users reported significantly greater control Stepwise Phasewise systems, found intervention, correction, verification easier, suggest design guidelines trade-offs for AI-assisted tools.