Draft:DABstep Benchmarks

DABstep (Data Agent Benchmark for Multi-step Reasoning) is a benchmark for evaluating artificial intelligence agents on complex, multi-step data analysis tasks. It was developed jointly by Adyen and Hugging Face and first published in 2025.

Background

Existing benchmarks for data analysis AI evaluated models on isolated code generation tasks or synthetic question-answering problems. DABstep was designed to address limitations in these earlier evaluations by grounding tasks in actual operational workloads from a financial analytics environment.

Design and structure

DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Tasks are derived directly from operational workloads at Adyen and reflect the complex, iterative problem-solving scenarios faced by professional data analysts.

The benchmark integrates both structured data, including CSV and JSON files representing transaction telemetry and business metadata, and unstructured documentation such as Markdown files defining domain-specific formulas and rules. Tasks require technical proficiency in data manipulation, including filtering, aggregation, and joins, as well as the ability to extract and apply domain-specific rules from documentation.

Tasks are divided into two difficulty splits. The easy split covers tasks resolvable by reasoning over a single file or data source. The hard split requires multi-step reasoning across multiple files and source documents simultaneously and accounts for the majority of the benchmark's tasks.

DABstep uses factoid-style evaluation in which each task output maps to a binary outcome — correct or incorrect — enabling objective scoring at scale without requiring human interpretation. Unlike benchmarks such as SWE-bench or MLE-bench, DABstep is designed for low-barrier usage; generating answers requires only access to a code execution environment, and participants can submit answers directly to a leaderboard for automatic evaluation. Results

Results from the benchmark's original evaluation revealed a substantial performance gap between AI agent capability and real-world data analysis requirements. Even the best agent at the time of publication achieved only 14.55% accuracy on the hard tasks, while performance on the easy split was considerably higher, with the top model reaching 76.39% accuracy.

Subsequent systems improved substantially on the original baseline results. DS-STAR, a role-decomposed data science agent developed at Google, raised accuracy on DABstep to 45.2% and secured the top rank on the public leaderboard as of September 2025.

Publication and availability

The benchmark paper was submitted to the NeurIPS 2025 Datasets and Benchmarks Track. The benchmark is released with a public leaderboard on Hugging Face Spaces and an open dataset at huggingface.co/datasets/adyen/dabstep.

References

References Egg, Alex; Iglesias, Martin; Kingma, Friso; Mora, Andreu; Von Werra, Leandro; Wolf, Thomas (2025). "DABstep: Data Agent Benchmark for Multi-step Reasoning". arXiv:2506.23719. Submitted to NeurIPS 2025 Datasets and Benchmarks Track. Adyen Tech (February 2025). "Data Agent Benchmark for Multi-step Reasoning (DABstep)". Medium / Adyen Tech Blog. Hugging Face (February 2025). "DABstep: Data Agent Benchmark for Multi-step Reasoning". Hugging Face Blog. Google Research (2025). "DS-STAR: A state-of-the-art versatile data science agent". Google Research Blog. DABstep Leaderboard. Hugging Face Spaces. huggingface.co/spaces/adyen/DABstep. DABstep Dataset Card. Hugging Face Datasets. huggingface.co/datasets/adyen/dabstep.