Overview
Company: Transfr
My role: Infrastructure provisioning, CDC configuration, orchestration integration, and pipeline validation
Stack: Terraform, BigQuery, Cloud Composer 3, Cloud Build, Secret Manager, IAM, Estuary, Aurora PostgreSQL
The problem
The team needed a third environment where dbt, Airflow, and the reporting API could be validated against QA data. The GCP project was nearly empty, so making it usable required work across infrastructure, permissions, ingestion, and deployment.
What I built
I provisioned landing datasets, a staging bucket, service accounts, Composer on a Shared VPC, and Cloud Build integration. I brought API enablement under Terraform and configured access through IAM and Secret Manager. Failure alerts covered DAGs and builds.
I connected multiple QA databases through Estuary CDC and integrated the new environment into the dbt targets, Airflow environment detection, and CI configuration.
The engineering decisions
Make access explicit. I diagnosed permissions independently across cloud IAM, database SELECT grants, logical-replication publications, and source rediscovery. A working connection alone did not establish that every intended table would replicate.
Review infrastructure changes before applying. I used Terraform moved blocks for state renames and reviewed saved plans for unintended destroys. The environment was delivered through applies with zero destroys, with no drift across development, production, and test at final verification.
Validate the complete build. Once infrastructure and ingestion were available, I checked model selection parity, investigated build failures, created missing views, and full-refreshed incremental facts where needed. I also identified a test snapshot configured to write to production and excluded it.
Results
The test environment came online with CDC from multiple databases and an initial load of roughly 8 million rows. Infrastructure, deployment configuration, and monitoring were managed as code, and pipeline validation extended beyond provisioning to actual warehouse dependencies and builds.
What this project shows
I can carry platform work across service boundaries and troubleshoot the gaps between an infrastructure plan, a successful deployment, and a functioning data pipeline.