Overview
Company: Transfr
My role: Event-model development, cost investigation, incremental design, and deployment planning
Stack: BigQuery, dbt, Snowplow, Airflow
The problem
On a five-minute schedule, small inefficiencies repeat hundreds of times a day. I investigated the Snowplow processing path to understand which data each run scanned and how incremental windows affected both cost and deduplication.
Investigation and design
I replaced vendor normalization packages with a custom incremental, partitioned base event model and repointed the downstream staging models to it. I moved deduplication before context unnesting to make event selection deterministic.
I compared MERGE behavior with and without a target predicate, then added explicit target-window filtering to the incremental design.
I measured arrival latency across tens of millions of events and used the observed lag to configure a source lookback and a wider target window.
The correctness constraint
Source and target windows must be designed together. If the source includes a previously processed event but the target predicate excludes its existing row, a MERGE can insert a duplicate. The wider target window was therefore part of the correctness design as well as the performance strategy.
Delivery
I shipped the event-model remodel and incremental tuning, including partitioning, target predicates, and latency-informed processing windows.
I also documented deployment sequencing, including pausing the near-real-time DAG for a full refresh, and a breaking column rename for downstream consumers.
What this project shows
I use measurements to investigate performance, account for the correctness implications of optimizations, and plan how changes reach production safely.