Built a metadata-driven Azure lakehouse that incrementally ingests Spotify-style SQL data with ADF, processes new Parquet files with Databricks Auto Loader, and publishes SCD-managed Delta facts and dimensions.
spotify azure pyspark data-engineering change-data-capture azure-sql-database azure-data-lake azure-data-factory star-schema metadata-driven azure-databricks delta-lake scd-type-2 data-lakehouse unity-catalog medallion-architecture databricks-asset-bundles databricks-autoloder lakeflow incremental-loading
-
Updated
Aug 14, 2026 - Python