Work · 02 Permit signal · Signal
A weekly pipeline over public data that moves every time it is published.
A field-services company wanted a weekly list of the building permits just issued across the area it serves. It wanted the list ranked, so the field team could work from it at the start of the week instead of reading it. The data is public. Several city offices publish it, each in a different file format and on a different schedule. Each office posts through a website that gives every new document a fresh address. A link that worked last week returns nothing this week.
- Client
- A field-services company. Not named.
- Layer
- 2 of 4. Signal.
- Source
- Public municipal permit data.
- Measured
- 2026-09-11, from the build records.
- Published
The cheaper answer a buyer finds first is a company that sells permit data. Does one of them cover these particular offices? How often does it refresh? How good is its ranking? I answer those on the day, office by office, before I propose a build. If the data seller covers them well enough, that is my recommendation. This pipeline is for the case where it does not.
01 The constraint
What was actually hard.
Not the reading of the files. Each format needs its own reader, and that is a day of work. Two other things are hard. First, the addresses change every time an office publishes, so no address can be written into the code. Every run has to find its own files again. Second, the result has to be a working list, close to a driving route, not a pile of data. A field team spends real days on it, so a short accurate list beats a long one.
02 The mechanism
How it works.
- Every run finds the current document link on the office's own index page. No address is written into the code. If an index page moves, the run stops at that office and says so. It does not quietly read last week's file.
- One reader per file format. A broken PDF stops that one office and nothing else. Every other office still comes through.
- Duplicates are caught by permit number, across twelve months of records and the older archive. A record that arrives late with an old date must not bring an archived record back to life, so the check reads both tables.
- Every run writes its own log: which office, which document, how many rows. A gap in coverage shows up the week it starts, not three months later.
- Results are grouped by street and ZIP code, marked by job type, and weighted so recent permits rise. The page the team reads is that grouped list, cut to four columns so it can be read quickly.
- Qualifying addresses go into the CRM through the bridge in build 01. They carry their own lead status, so the field team can filter to them.
- Three tests were written before the pipeline was switched on. The first was one full live run. The second used a dead document link and a broken PDF. The third checked for duplicates using a late record with an old date. After those, the pipeline ran against a hand-checked week without writing anything to the CRM. A buyer can lift those three tests word for word into their own contract. The tests were specified. This page does not claim which of them were run.
City offices, one per file format, feed a step that finds the current document link on the office's own index page on every run. One reader per format turns the files into one record shape with the same fields for every office. A duplicate check by permit number reads the last twelve months and the archive, sending duplicates to a run log and new records to the table, which moves older rows into the archive. Ranking by street and ZIP code, job type and recency produces the weekly page for the field team and sends qualifying addresses into the CRM through build 01.
03 The figures
What it measured.
Each figure carries the date it was taken. A number without a date is not on this page.
| What was measured | Value | Measured |
|---|---|---|
| Cadence | weekly, one fixed slot | 2026-09-11 |
| Build size | 56 steps · 22 of them hand-written code | 2026-09-11 |
| Tests written before the pipeline was switched on | 3, plus one practice run | 2026-09-11 |
| Period the duplicate check reads across | 12 months, plus the archive | 2026-09-11 |
Partway through, one office we were looking at stopped serving permit documents to software altogether. Its data could then be had only by filing a request by hand and waiting. That is not something a scheduled job can do. We took that office's records once, by hand, and left it out of the pipeline. How much ground the pipeline covers depends on what each office decides to allow, not on the code. A buyer building on public sources should expect at least one source to close during the life of the build.
The result is a list of home addresses. No sample rows, no exports and no screenshots of it appear anywhere, and the client may use it only for its own field visits. Some buyers plan to sell permit-based lists on to other companies. That is a question for a lawyer before it is a question for me. I decline the work.
04 Doing it again
What it would take to do again.
Five parts carry over to the next client. The step that finds the current link. The duplicate check that reads the archive too. The run log. The ranking method. The three tests. The parts built for each client are the readers, one per file format, the job-type markings and the local geography. Adding a new office means one new reader and one new row in the log. Nothing else changes.
A fit
A public source that is hard for software to read, and a team that works from the result on a set day.
Not a fit
A source that offers a steady feed or a full download. A scheduled download does that job.
Up against a constraint like this one?
Thirty minutes, no prep, no pitch. Bring the thing that is slow; if it is not a fit, I say so and point you somewhere useful.