Agents in Data engineering

Databricks: agents building declarative Spark pipelines used ~34x less compute

Databricks says agents now create over 70% of its declarative (SDP) pipelines. In 528 builds on 22 tasks with Opus 4.8, procedural runs used 596 executor-seconds vs 17 for declarative, with 51% less code.

Why it matters: Declarative builds cost about 2.3x more in tokens but catch structural errors before a cluster runs. Databricks says logic errors still need tests and review.