Data Engineering
40× faster survey ETL at million-record scale
North America-scale travel surveys generated millions of trip records, but the legacy ETL pipeline took hours per run, blocked analyst iteration, and duplicated manual validation steps across studies.
Approach
- Profiled the existing pipeline and identified row-by-row database writes and redundant validation as primary bottlenecks.
- Rebuilt ingestion with SQLAlchemy bulk operations, async I/O for network-bound steps, and set-based validation in PostgreSQL.
- Introduced staging tables and composite indexes aligned to join keys used in spatial assignment and weighting.
- Added row-count gates and golden-file regression checks so speed improvements never traded away correctness.
Results
- Pipeline runtime reduced by up to 40× on representative study volumes.
- Analysts could re-run ETL multiple times per day during methodology reviews.
- Standardized patterns reused across subsequent large mobility studies.
- Python
- SQLAlchemy
- PostgreSQL
- PostGIS
- Pandas