snowpipe

Learn how to auto-ingest streaming data into Snowflake using Snowpipe.

CLI script for interacting with Snowpipe REST APIs

Realtime data pipeline using Kafka + Spark + AWS S3 (Terraform) + Snowflake

The goal is to build an example of a simple data collection pipeline which collects data from multiple customers and uploads the data to Snowflake could look like. The OpenMeteo Api acts as the "customer system" in this case.

❄️ Stream real-time data into Snowflake with Amazon Kinesis Firehose

Real-time ETL (Extract, Transform, Load) data pipeline to process insurance claims data with Snowflake, Apache Airflow, AWS S3, EC2, python pandas and creating a real time data visualization dashboard using Tableau.

Get the streaming data from the S3 bucket with SQS queue. Load into Snowflake with Snowpipe and modify the data with Snowflake task

Real-time fintech analytics on Snowflake: Snowpipe/Tasks ingestion, Snowpark medallion, Cortex AI (forecasting, anomaly detection, fraud LLM, RAG, Text2SQL, agent), and a Streamlit dashboard.

Production-style real-time data ingestion pipeline built with AWS and Snowflake, featuring pre-stream validation, fault isolation, and automated Snowpipe ingestion.

Production-grade Snowflake ELT platform - dbt, AWS S3, Snowpipe, GitHub Actions

Cloud data lake: GCS ingestion into Snowflake via Snowpipe + Streamlit insights app

Designed a cloud-to-cloud migration pipeline through migrating data with event-driven architecture and automated data loads. Implemented data quality checks at each stage, ensuring data consistency and reliability.

Real-time financial data pipeline using AWS, Snowflake, Terraform, and dbt.

End-to-end real-time data warehouse project using Snowflake, AWS, Apache NiFi, Snowpipe, Streams, and Tasks.

Spotify ETL pipeline on AWS

Real-time streaming pipeline: Kinesis -> Lambda -> Firehose -> S3 -> Snowpipe -> Snowflake with Streams & Tasks