We need data engineering benchmarks for LLMs

Tools like Copilot and GPT-based copilots promise to reduce the repetitive burden of data engineering tasks, suggest code, and even debug complex pipelines. But how do we measure whether they’re actually good at this? Frankly, the industry is lagging behind when it comes to evaluation methods. While SWE-bench offers a framework for software engineering, data engineering is just left out—no tailored benchmarks, no precise way to gauge their effectiveness. It’s time to change that.

Why SWE-Bench Falls Short SWE-bench evaluates LLMs on real-world software engineering tasks by using GitHub issue–pull request pairs from popular repositories.

Success is measured by the quality, reliability, and scalability of data workflows, not just correctness of code.

Data engineers solve complex systems-level problems involving constant data motion, quality maintenance, and continuous evolution. Treating these disciplines as equivalent is doing data engineering a disservice.

DE deals with raw, messy, and large-scale data that must be cleaned, transformed, and made accessible.

i.e. handling data quality, schema evolution, and compliance, not just code correctness.

DE focuses on automating and orchestrating multi-step workflows, which requires understanding task dependencies, scheduling, and retries.

Edge-cases! DE must handle schema drift, missing values, malformed records, and outliers, which are rarely part of SWE workflows.

DE is less about writing application logic and more about making data usable, accessible, and reliable. Data engineers don’t just write code. They manage workflows that operate across multiple systems, handle changing requirements, and scale with data growth. A DE benchmark needs to reflect these realities.

One might argue that text-to-SQL benchmarks are a step toward evaluating LLMs for data engineering tasks. While useful, text-to-SQL falls far short of the needs of DE benchmarks for several reasons:

Text-to-SQL only handles querying structured data. Data engineering is so much more: it involves transforming, orchestrating, and making sense of chaotic, mixed-format data.

Lack of Pipeline Context. DE isn’t more than single queries. it’s about creating end-to-end workflows that deliver business value.

Handling real-world problems like schema drift or malformed records

Data engineers work with Spark, Airflow, dbt—not just databases. Generating a SQL query is child’s play compared to orchestrating a complex Spark job across petabytes of data.

Reliability, scalability, and optimization are make-or-break factors for DE. These are entirely missed by simple SQL benchmarks.

In short, text-to-SQL is just one piece of the DE puzzle. Evaluating DE copilots requires a broader, pipeline-focused benchmark.

A DE-bench would simulate real-world DE workflows, evaluating LLMs on their ability to solve practical, pipeline-oriented problems. Here’s how it might look:

Dataset Sources:

Use public datasets (e.g., NYC Taxi data, Kaggle datasets).

Generate synthetic data to simulate edge cases (e.g., missing values, schema drift).

Data Ingestion: Load raw data from APIs or files into a database or data lake.

Data Transformation: Normalize data formats, remove duplicates, and compute aggregates.

Pipeline Orchestration: Create an Airflow DAG for a multi-step ETL pipeline.

Schema Management: Migrate data between schemas safely.

A DE-bench would provide a structured, objective framework for assessing LLMs on real-world DE tasks, ensuring that these tools are reliable, efficient, and robust. It’s time we hold DE copilots to the same high standards as their SWE counterparts.