ChatGPT as your Personal Data Scientist

• DOI: 10.48550/arxiv.2305.13657
• Publication Date: 2023-01-01
• Authors: Md. Mahadi Hassan; Alex Knipper; Shubhra Kanti Kar...
• Keywords: Interface (matter)

The rise of big data has amplified the need for efficient, user-friendly automated machine learning (AutoML) tools. However, intricacy understanding domain-specific and defining prediction tasks necessitates human intervention making process time-consuming while preventing full automation. Instead, envision an intelligent agent capable assisting users in conducting AutoML through intuitive, natural conversations without requiring in-depth knowledge underlying (ML) processes. This agent's key challenge is to accurately comprehend user's goals and, consequently, formulate precise ML tasks, adjust sets model parameters accordingly, articulate results effectively. In this paper, we take a pioneering step towards ambitious goal by introducing ChatGPT-based conversational data-science framework act as "personal scientist". Precisely, utilize Large Language Models (ChatGPT) build interface between models (Scikit-Learn), which turn, allows us approach problem with realistic solution. Our pivots around four dialogue states: Data Visualization, Task Formulation, Prediction Engineering, Result Summary Recommendation. Each state marks unique conversation phase, impacting overall user-system interaction. Multiple LLM instances, serving "micro-agents", ensure cohesive flow, granting granular control over conversation's progression. summary, developed end-to-end system that not only proves viability novel concept science but also underscores potency LLMs solving complex tasks. Interestingly, its development spotlighted several critical weaknesses current highlighted substantial opportunities improvement.