Business Utility of Large Language Models as Exploratory Data Analysis Agents
Rafa{\l} {\L}ab\k{e}dzki, Patryk Miziu{\l}a, Hubert Rutkowski, Szymon Betlewski, Cezary Depta, Szymon Janowski, Jaros{\l}aw Kochanowicz, Jan Kanty Milczek
Why It Matters
What makes this one worth your time
Understanding the limitations and capabilities of LLMs in EDA can guide businesses in adopting AI tools for data-driven decision-making.
This study assesses the reliability of LLMs for exploratory data analysis in business applications.
Summary
The paper evaluates the effectiveness of large language models as exploratory data analysis agents in business contexts, focusing on their reliability and repeatability in identifying supplier-product combinations linked to low quality and sales loss.
Key contributions
- Introduction of a Business utility metric for assessing EDA agents.
- Evaluation of fifteen model-variant configurations across multiple experimental conditions.
- Identification of key factors affecting the reliability of LLMs in EDA tasks.
Notable insights
- The proposed Business utility metric combines quality and repeatability into a single operational measure, which could be a valuable tool for evaluating AI performance.
- The study highlights that average performance metrics can be misleading without considering variability and repeatability.
Possible limitations
- The abstract does not address potential biases in the agent-based supply chain simulation or the generalizability of the findings to other business contexts.
Abstract
arXiv:2606.00051v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used in analytical workflows, but their suitability as exploratory data analysis (EDA) agents in business settings remains uncertain. In practice, a deployable EDA agent must provide not only useful average performance but also sufficient repeatability to support trust in its outputs. We evaluate this requirement in a controlled, business-relevant benchmark built on an agent-based supply chain simulation. The task is to identify supplier-product combinations responsible for low quality and downstream sales loss by reasoning from indirect operational traces rather than from explicit labels. Fifteen model-variant configurations from eight model families were evaluated under four experimental conditions that varied data representation, prompt clarity, and signal strength, with five trajectories per condition. Outputs were scored against deterministic ground truth using the Jaccard index and assessed through a framework that combines mean score (ms), coefficient of variation (CV), exploratory cross-condition significance tests, and Business utility, a risk-adjusted metric that we propose to summarise quality and repeatability in a single operational measure. The results show that most configurations are not reliable enough for autonomous EDA use, even when their average scores appear acceptable. GPT-5.4 with extra-high reasoning effort achieved the strongest overall profile, with an experiment-averaged ms of 0.8748 and an experiment-averaged Business utility of 0.6952, while the next-best configurations lost substantially more utility after variability discounting. Our findings suggest that evaluation of EDA agents should treat average quality, repeatability, and condition sensitivity as complementary dimensions of operational trustworthiness.