A multi-source data pipeline, built to be cloned
A public-data pipeline across thirty-plus state sources — connector per source, validation against ground truth, and a readiness gate that decides which sources are trustworthy enough to use. Then cloned to a second market in weeks.
The challenge
Public data lives in dozens of incompatible places — Socrata APIs, flat CSV dumps, XML, and ASP.NET forms that actively fight automation — and every source drifts over time. Assembling it into something trustworthy and current is mostly grunt work, which is exactly where automation earns its keep. The harder problem isn't collecting the data. It's knowing when to trust it.
The build
A connector per source type across thirty-plus state jurisdictions, with a single source of truth for what's actually usable from each. A validation layer re-checks output against live ground truth so nothing ships unverified, and a change-detection engine flags what's new since the last run. Where sites block automation outright, a real-browser egress path handles it. Sources that throttle get a circuit breaker that rests them and escalates the backoff, because losing an endpoint costs a whole jurisdiction rather than a single request — and the health report names every resting source, so one being quietly avoided is never invisible.
Knowing which data you can trust
The part I'd build again first: a readiness gate that decides, per source, whether the data is good enough to act on. A source qualifies only after enough clean runs to arm its own volume guard, a passing data-quality scan, and a freshness check calibrated to how slowly that particular source actually publishes. The first run of that gate cleared four of eighteen sources — and that was the correct answer, not a disappointing one. Most pipelines will happily serve you stale or half-broken data with total confidence. This one refuses, and tells you which test it failed.
Reusing the engine
The real return showed up on the second build. A different market, a different public data source, a different buyer — and the collection, normalization, change-detection, validation, and delivery layers all transferred as-is. What had taken months the first time stood up in weeks, because the engine had already been paid for. That's the argument for building this kind of thing properly once: the first one is a project, and every one after it is a configuration.
The result
Thirty-plus jurisdictions assembled and kept current with no manual touch, a test suite in the hundreds guarding the parts most likely to drift silently, and a reusable pattern for any 'pull data from everywhere, keep it clean, act on what changed' problem. Reliability is the whole point: a pipeline that fails silently is worse than no pipeline, so this one checks its own work and says so when it can't.