case study // 2026-09-10 // automation + data QA
A nightly lead-generation pipeline wrote status: ok into its own report while
delivering rows that had nothing to do with the order, and a second job celebrated success after
scraping zero records. Everything looked green for five days. Here is what was wrong, how I
found it, and the gate I built so a delivery has to prove itself.
Two scheduled jobs fed a productised lead-generation service: one watched an inbox directory for
new orders and fulfilled them on a 15-minute tick, the other processed a pending queue at 02:00.
Both wrote a manifest.json summarising the delivery, and both reported that summary
back to a dashboard.
The manifest was written by the same code that produced the delivery. That is the whole problem: it is a self-report, and nothing independently checked it.
Three separate faults had piled up, and they had nothing to do with each other — which is why the first fix attempt (a rebuild of the job) didn't help.
fault 1 — the data was wrong
An order for pest control in New York was delivered as orthodontists, an escape room and a moving company. One of its eight rows was on-target. An order for moving companies received the same buildings back. The cause wasn't the scraper's parsing: the search URL carried a state segment, the source returned a redirect, and the scrape quietly fell back to a generic city listing. Two live probes settled it:
GET /search/ny/new-york/pest-control → 301, 0 bytes (what the orders used) GET /search/new-york/pest-control → 200, 16 pest-control companies
fault 2 — success was unfalsifiable
The nightly job ran a pipeline that exits 0 after scraping zero records and writes no file at all. The return code could not distinguish "delivered 500 leads" from "delivered nothing", so the order was filed as completed either way. A job that cannot fail is not a job, it's a rumour.
fault 3 — the failures were invisible
One job delivered to a local log nobody read, and another had been pinned to a model that no longer existed. Between them they failed 16 times in nine days with no notification anywhere, because there was no failure alert to receive — only a delivery channel.
Not by reading the code first, but by asking the durable run history what had actually happened. The job table said "ok" in places, the execution records said otherwise, and one delivery's own rows contradicted its manifest:
FAIL delivery verification — order fvr-20260831-cronlive01 FAIL NICHE_MOSTLY_WRONG: only 1/8 rows match the ordered niche delivered {'dental-care': 1, 'entertainment-industry': 1, 'moving-and-storage': 1}
An independent gate that reads the order, reads the delivery, and refuses to take the manifest's word for it. It fails on 16 conditions — wrong or partially-wrong niche, a manifest that lies about its own row count, masked phone numbers, fabricated sample data reaching a real order, duplicates, uncontactable rows, a missing deliverable, a format mismatch, an ordered column that isn't there. It warns on the softer signals, like emails shipped without provenance.
It runs inside the fulfilment path before an order can be marked done: a failed delivery is moved aside, the job exits non-zero, and a daily sentinel reports anything that has gone quiet. The deliverable is now a decision, not a press release.
The same run that used to pass with junk now fails loudly, and the fixed source delivers what was ordered:
before 1/8 rows on-target manifest: status ok after 8/8 rows on-target gate: PASS on a real scrape tests 11/11 cases plus 4 verified failure branches
The two real deliveries on disk are still there and still fail the gate — which is the correct answer for a historical artefact that was wrong.
Every unattended pipeline eventually reports success for something it didn't do. The question is whether anything checks. If your delivery marks itself green, nobody will notice the day the rows stop matching the order — and the person who finds out will be your customer.
I build these checks for lead data, scraping pipelines and scheduled jobs, and I publish the tool this case study is about:
github.com/Jrudani21/delivery-gate ↗ — MIT licensed, standard library only, with the eleven test cases and two runnable samples.
The decision behind that build is written up too — a source-checked note on what makes a project worth showing, every claim quoted verbatim from primary sources and verified against the fetched page text: research/building-good-projects.md ↗