A 528-build benchmark shows that declarative pipeline authoring catches structural faults before execution, cutting cluster ...

Spark Declarative Pipelines provides an easier way to define and execute data pipelines for both batch and streaming ETL workloads across any Apache Spark-supported data source, including cloud ...

Today, at its annual Data + AI Summit, Databricks announced that it is open-sourcing its core declarative ETL framework as Apache Spark Declarative Pipelines, making it available to the entire Apache ...

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...

The name of Databricks' annual conference has gone from "Spark Summit" to "Spark + AI Summit" and now to "Data + AI Summit." The evolution of the event name tracks Databricks' own transition from the ...