Work · 02Signal

A weekly ingestion pipeline over public publishers that move the ground every release.

A field-services company wanted to know, every week, where building permits had just been issued across its service area — ranked, so a field team could act on it at the start of the week rather than read it. The data is public. Several municipal publishers release it, each in a different format and on a different cadence, through content-management systems whose document identifiers rotate on every release. A link that resolved last week returns nothing this week.

Client: A field-services company. Not named.
Layer: 2 of 4. Signal.
Source: Public municipal permit data.
Measured: 2026-09-11, from the build records.

The substitute a buyer will reach for first is a permit-data aggregator. Whether one covers a given set of publishers, on what refresh schedule, in what schema and with what ranking is a question answered on the day, per publisher, before any build is proposed. If the aggregator's list is good enough, that is the recommendation; this pipeline is for the case where it is not.

01The constraint

What was actually hard.

Never the parsing. Each format needs its own extractor, and that is a day's work. The hard part is twofold. The publishers change the ground under the pipeline on every release, so nothing can be hardcoded and every run has to rediscover its own inputs. And the output has to be an operating list — a route — not a data dump: a field team spends real days on it, so precision beats volume.

02The mechanism

How it works.

Document links are resolved off each publisher's listing page on every run. Nothing is hardcoded. If a listing page moves, the run fails loudly at that source rather than silently reading a stale document.
One extractor per source format. A malformed PDF fails that source's branch, not the run; every other publisher still lands.
Dedupe by permit number across a rolling twelve-month table with an archive boundary. A backdated record arriving late must not resurrect an archived one, so the check reads both tables.
A source log per run — which publisher, which document, how many rows — so a coverage gap is visible the week it opens, not the quarter after.
Ranking to street and ZIP clusters with job-type flags and a recency weight. The page the team reads is the ranked cluster list, cut to four columns for legibility.
Qualifying addresses pushed into the CRM through the bridge in 01, tagged with a dedicated lead status so the field team can filter to them.
Acceptance was specified as three passes before the live enable: a live end-to-end run; a dead document identifier plus a malformed PDF; and dedupe verified across the archive boundary with a backdated record — followed by a CRM dry run against a hand-checked week before anything was written live. Those three passes are reusable verbatim as a buyer's acceptance clause.

The highlighted element is the only one that makes the pipeline survivable: every run re-reads each publisher's listing page to find that release's document, so a rotated identifier is a normal Tuesday rather than an outage. Everything downstream of it is ordinary, which is the point.

03The figures

What it measured.

Each figure carries the date it was taken. A number without a date is not on this page.

Row counts, publisher counts and jurisdictions are deliberately absent: together they size the publishing bodies closely enough to name them, which is a fact about someone else's coverage area rather than a fact about this build.
Figure	Value	Measured
Cadence	weekly, one fixed slot	2026-09-11
Build size	56 nodes · 22 code nodes	2026-09-11
Acceptance passes specified before the live enable	3, plus a CRM dry run	2026-09-11
Rolling window the dedupe reads across	12 months, live table plus archive	2026-09-11

One honest limitation: a source closed mid-build, and that is the normal case.

Partway through, a jurisdiction being evaluated for inclusion stopped serving permit documents to automated clients altogether. Its data became obtainable only through a manual request process with its own clock, not a scheduled workflow. It was handled as a one-off list and left out of the pipeline. Coverage is a function of each publisher's posture, not of the code, and a buyer running a pipeline over public sources should assume at least one source will close during the life of the build.

Two conditions stated before the build, because they decide whether to build at all.

The output is residential addresses: no sample rows, exports or screenshots of it appear anywhere, and the client's use is bounded to its own field outreach. A buyer whose plan is to resell permit-derived lists to third parties is a data-broker question for counsel before it is a build, and is declined.

04Doing it again

What it would take to do again.

The reusable parts are the listing-page resolver, the dedupe-with-archive pattern, the source log, the ranking frame and the three acceptance passes. The client-specific parts are the extractors — one per publisher format — the job-type flags and the cluster geography. A new publisher is a new extractor and a source-log row; nothing else changes.

A fit

A public source that resists automation, and a team that acts on the output on a schedule.

Not a fit

A source with a stable API or a bulk export. A scheduled download does that.

Up against a constraint like this one?

Thirty minutes, no prep, no pitch. Bring the thing that is slow; if it is not a fit, I say so and point you somewhere useful.

Book an intro call