StackDX ingests public oil & gas data — wells, production, permits — from across the United States (and Canada), and the pipeline doing that work was a legacy, hand-rolled, Python "EtlJob" stack that was hard to test, harder to extend, and made it difficult to tell whether a failure was a logic bug or an architecture one. Touching legacy code that was serving live data to customers but was built on top of a shaky foundation was a known challenge that shaped many of the decisions and workflows that followed.
I made the case to replace the stack outright rather than keep patching it, and rebuilt it Dagster-native: every table an asset with a schema contract, all transformations on Polars or DuckDB where appropriate, and the whole thing deployed as Terraform-managed AWS infrastructure on RDS, ECS and S3. The primary goal here was to leverage Dagster's python-native and refined approach to orchestration while clearly separating out the logic required within our data jobs.
In addition to the architecture and orchestration rebuild I also implemented a rigorous, defined and enforced data pipeline structure: source -> raw -> landing -> clean -> regrain -> coalesce -> mart -> exposure. Brand new mart data schemas were defined from scratch to match a more normalized data model for oil and gas wells, while the outgoing exposures were kept consistent with what the API endpoints expected.
- A production Dagster platform spanning 15 US states and 945 transformation assets, distilling a 3.8 TB source layer of 1.03M regulator files into a 10.3 GB exposure layer that serves customer endpoints — 127M well production records, 172M lease production records, and 4.1M wellheaders.
- Targeted Polars and DuckDB rewrites on the critical datasets, with incremental materializations and coalesced fact/mart layers that isolate the expensive work and cut end-to-end processing time on the worst offenders.
- Those layer boundaries are contracts rather than naming conventions — each stage has operations it is forbidden from performing.
landingrenames and casts and does nothing else, no joins are allowed beforecoalesce, grain only changes inregrain, andmartis where Dataframely validates the final schema. That answers the old stack's worst property directly: the layer a failure surfaces in now tells you which of the two it is. Reusablecoalesce_sourcesandexposure_transformscarry the last two hops, so downstream consumers keep the legacy schemas they were built against and a new state extends the pattern instead of needing bespoke work. - CI/CD on GitHub Actions enforcing ruff, pyright, and schema-contract tests against asset outputs, so a staging or production release stopped being something anyone had to think about.
- A Claude Code toolchain built for this project specifically — a
parquet-inspectorsub-agent, an auto-format hook, and task skills for pipeline triage and schema work. I profiled my own agent conversation history to find where the time was actually going; parquet inspection came out well ahead of everything else, so I promoted it into a first-class tool. The slash commands that carried the conversion itself were retired once it landed.
The legacy stack is retired now. New states onboard through framework patterns instead of copy-paste, all 945 assets follow standardized conventions, and the documentation is refined enough that additional engineers pick the platform up at pace without me walking them through it.
This is the data platform engineering pillar end to end — a modern platform architected and shipped by one lead engineer while the thing it replaced was still running. The AI-native toolchain running through it is what made that pace possible.