Projects

Event-Driven Lakehouse

A local-first, backend-agnostic medallion lakehouse with queue-driven ingestion. Reliably ingests irregularly-arriving files with exactly-once processing and schema enforcement across raw → aggregate → curated layers. A queue layer (local watcher or Azure Storage Queue) feeds N async workers with in-flight deduplication, while a single-writer commit buffer prevents Delta Lake write contention and enables near-linear worker scaling. YAML-defined schemas mean zero-Python table definitions, and it runs without a JVM or Spark — querying via DuckDB locally or scaling to ADLS Gen2 in the cloud.