StackDX ingests public oil & gas data — wells, production, permits — from across the United States (and Canada), and the pipeline doing that work was a legacy, hand-rolled, Python "EtlJob" stack that was hard to test, harder to extend, and made it difficult to tell whether a failure was a logic bug or an architecture one. Touching legacy code that was serving live data to customers but was built on top of a shaky foundation was a known challenge that shaped many of the decisions and workflows that followed.
I made the case to replace the stack outright rather than keep patching it, and rebuilt it Dagster-native: every table an asset with a schema contract, all transformations on Polars or DuckDB where appropriate, and the whole thing deployed as Terraform-managed AWS infrastructure on RDS, ECS and S3. The primary goal here was to leverage Dagster's python-native and refined approach to orchestration while clearly separating out the logic required within our data jobs.
In addition to the architecture and orchestration rebuild I also implemented a rigorous, defined and enforced data pipeline structure: source -> raw -> landing -> clean -> regrain -> coalesce -> mart -> exposure. Brand new mart data schemas were defined from scratch to match a more normalized data model for oil and gas wells, while the outgoing exposures were kept consistent with what the API endpoints expected.
- A production Dagster platform spanning 15 US states and 945 transformation assets. It distills a 3.8 TB source layer of 1.03M regulator files into a 10.3 GB exposure layer. That layer serves customer endpoints and carries 127M well production records, 172M lease production records and 4.1M wellheaders.
- Targeted Polars and DuckDB rewrites on the critical datasets. Incremental materializations and coalesced fact/mart layers isolate the expensive work. This cut end-to-end processing time on the worst offenders.
- Those layer boundaries are contracts, and each stage has operations it is forbidden from performing.
landingrenames and casts and does nothing else. No joins are allowed beforecoalesce. Grain only changes inregrain.martis where Dataframely validates the final schema. This answers the old stack's worst property directly. The layer a failure surfaces in now tells you which kind of failure it is. Reusablecoalesce_sourcesandexposure_transformscarry the last two hops. Downstream consumers keep the legacy schemas they were built against, and a new state extends the pattern without bespoke work. - CI/CD on GitHub Actions enforces ruff, pyright and schema-contract tests against asset outputs. A staging or production release stopped being something anyone had to think about.
- A Claude Code toolchain built for this project. It has an auto-format hook and task skills for pipeline triage, exposure coverage and schema work. The slash commands that carried the conversion itself were retired once it landed.
Since moving into the product seat in May 2026 I've kept building on it. Mississippi went in as a sixteenth state, with production back to 1980 and permits from 1997, and there's a new well-detail data model where every exposure row carries its api10/api12/api14 keys and production is served at the grain the regulator filed it, rolled up and never down. Those exposures load into Redshift by state-partitioned COPY, the layer behind the new US details pane, which I prototyped along with the forecasting and type curve feature. The piece I'm most pleased with is the forecasting app's decline engine - a deterministic bounded Arps fit per well and phase, ported from my own TypeScript prototype into the Dagster pipeline and run monthly, with EUR and fit diagnostics landing in the same schema a user-saved forecast will use.
The legacy stack is retired now. New states onboard through framework patterns instead of copy-paste, all 945 assets follow standardized conventions, and the documentation is refined enough that additional engineers pick the platform up at pace without me walking them through it.
This is the data platform engineering pillar end to end — a modern platform architected and shipped by one lead engineer while the thing it replaced was still running. The AI-native toolchain running through it is what made that pace possible.