ETL pipeline using pyspark (Spark - Python)
-
Updated
Apr 4, 2020 - CSS
ETL pipeline using pyspark (Spark - Python)
Notes/blogs/tutorials/talks around basics & better optimzed usage of Apache Spark
An experimental investigation into Apache Spark's Catalyst optimizer, exploring when it optimizes effectively, what defeats it, and when optimization itself adds overhead. Through six controlled experiments, we analyzed predicate pushdown, join reordering, rule criticality, and algebraic equivalence, using PySpark in Google Colab.
🐍 Simplify Python development with reusable code snippets, tools, and examples to enhance productivity and streamline your projects.
🐍 Explore key Python concepts and enhance your coding skills with organized notes for easy reference and learning.
Add a description, image, and links to the catalyst-optimizer topic page so that developers can more easily learn about it.
To associate your repository with the catalyst-optimizer topic, visit your repo's landing page and select "manage topics."