Lakeflow Spark Declarative Pipelines

Topic Overview

Lakeflow Spark Declarative Pipelines (formerly known as Delta Live Tables) is a declarative framework for building reliable data pipelines on Databricks. Instead of writing imperative Spark code where you specify every step, you declare your desired end state—your target tables and their definitions—and the system handles the rest. This includes orchestration, dependency management, error handling, and data quality validation.

The declarative approach means you focus on what data you want, not how to compute it. Lakeflow Spark Declarative Pipelines automatically figures out the optimal execution order, handles retries, and ensures your data quality rules are enforced. This is especially powerful for building medallion architecture pipelines (Bronze, Silver, Gold) where you have many interdependent transformations.

Pipelines are defined in notebooks using SQL or Python and then deployed as pipeline objects in Databricks. You run them in either Development mode (for iteration) or Production mode (for reliable scheduled execution). The exam expects you to understand how to design pipelines, choose between Streaming Tables and Materialized Views, apply data quality expectations, and troubleshoot pipeline failures.

Key Concepts

What are Lakeflow Spark Declarative Pipelines?
Streaming Tables vs Materialized Views
Expectations (data quality constraints)
Triggered vs Continuous mode
Development vs Production mode
Pipeline dependencies and DAG resolution

Code Examples

Creating a Streaming Table with Auto Loader

CREATE STREAMING TABLE bronze_orders
COMMENT "Raw orders from cloud storage"
AS
SELECT *
FROM cloud_files(
  "s3://my-bucket/orders/",
  "csv",
  map(
    "cloudFiles.format", "csv",
    "header", "true"
  )
)

Creating a Materialized View with Aggregation

CREATE MATERIALIZED VIEW silver_orders_daily
COMMENT "Daily order summary"
AS
SELECT
  DATE(order_date) AS order_day,
  COUNT(*) AS total_orders,
  SUM(order_amount) AS total_revenue,
  AVG(order_amount) AS avg_order_amount
FROM bronze_orders
WHERE order_date IS NOT NULL
GROUP BY DATE(order_date)

Expectations / Data Quality Constraints

CREATE STREAMING TABLE silver_customers
CONSTRAINT valid_email EXPECT (email LIKE '%@%.%') ON VIOLATION DROP
CONSTRAINT positive_age EXPECT (age > 0) ON VIOLATION FAIL
CONSTRAINT not_null_id EXPECT (customer_id IS NOT NULL) ON VIOLATION WARN
AS
SELECT *
FROM bronze_customers