A mid-market distributor's accounts-payable dashboard was overstating the amount due by an eight-figure sum — and the sync feeding it reported green on every run. One overnight full-system review found the mechanism, purged the corruption with a self-healing upsert, and made the entire failure class impossible to repeat silently.
A mid-market distributor ran its supplier payments through a Keycloak-authenticated React SPA sitting over a no-code data layer, kept in sync with the company's cloud accounting system by an automation workflow. The platform looked finished. The numbers on it did not add up — panel totals disagreed with each other, disagreed with the accounting system, and the headline amount due was overstated by an eight-figure sum. Finance staff had quietly stopped believing the screen and gone back to cross-checking spreadsheets, which meant the platform was now generating work instead of removing it.
The dangerous part was the shape of the failure. Nothing was down. No error had ever fired. The sync workflow showed a perfect record of green runs while it corrupted the invoice table on every single cycle. Each sync requested page one of the accounting API with a duplicated pagination parameter — the workflow appended its own page counter alongside a static page=1, the API honored the first one, and the same page of invoices came back up to eight times per run. The upsert step then bulk-inserted every copy with no within-run deduplication.
The result compounded silently for months: the invoice table grew to 4,063 rows of which only 1,338 were real invoices — 67% of the table was duplicates — and every duplicate row inflated the amount-due totals a little further. A monitoring stack that only watches whether jobs succeed can never catch this class of failure. The job was succeeding. It was the data that was failing.
The review method was deliberate: in a single overnight full-system pass, Del Rey traced every number the dashboard displayed back through the data layer to the accounting system — treating the accounting API as the read-only source of truth and refusing to accept any figure that could not be reproduced from it.
The reconciliation surfaced the whole chain in one night. The invoice table held 4,063 rows against 1,338 real invoices in the accounting system. Grouping by invoice identifier showed the duplicates were exact copies stamped by successive sync runs — which pointed straight at the fetch step, not at user behavior or the accounting data itself. Reading the workflow's HTTP nodes exposed the smoking gun: a static page=1 query parameter sitting alongside the paginator's own same-name parameter. The API honored the first, so "fetch all pages" degenerated into "fetch page one, up to eight times."
The same pass caught what a narrower bug hunt would have missed. A display-priority rule on payment folios was inverted — a short internal folio was outranking the full invoice reference everywhere it appeared, so staff saw truncated identifiers on documents that demanded exact ones. Role checks that looked enforced in the UI were not enforced behind it. And buried in the workflow history was a previous incident: a stale credential had silently stopped payment notifications for roughly 21 hours, and nothing had alerted anyone — the same monitoring blind spot as the sync, wearing a different disguise.
One review, one method — trust nothing the screen says until the source of truth reproduces it — and the platform's three separate problems turned out to be one problem: failures that don't announce themselves.
Deleting 2,725 rows from a production financial table is not a script you run and hope. The fix was engineered as an idempotent, self-healing upsert with a hard discipline around every mutation — prove the plan before executing it, and prove convergence after.
The cloud accounting system was treated as an untouchable source of truth for the entire operation. Every call against it was a GET; every mutation happened only in the platform's own data layer. Whatever went wrong downstream, the client's books could not be damaged by the repair — a constraint set before the first command ran, not negotiated afterward.
The complete invoice table — all 4,063 rows, duplicates included — was exported before a single row was touched. If the dedup logic had a flaw, the rollback was one restore away. Backups of prior state preceded every destructive step in the engagement, not just the headline purge.
The rebuilt sync runs its full dedup pass in a reporting mode before it is allowed to delete: group rows by invoice identifier, keep the newest row per invoice (the one matching current accounting-system state), and emit the surplus it would remove. The dry run reported a surplus of exactly 2,725 rows — no deletion yet, just an auditable plan.
Before going live, the arithmetic had to close: 4,063 rows in the table, 1,338 real invoices in the accounting system, surplus 2,725 — an exact match, to the row. Per-entity totals were reconciled against the deduped baseline to the cent, both before and after. A dry run that had reported any other number would have stopped the operation cold.
Only after the dry run matched the source of truth did the purge run live: 2,725 surplus rows removed with zero errors, leaving 1,338 rows and zero duplicate groups. The root cause was fixed in the same release — the duplicated pagination parameter eliminated, real pagination-to-the-end wired in, and within-run deduplication added so the upsert is idempotent by construction.
The proof of an idempotent, self-healing sync is its next cycle. It ran on schedule against the cleaned table and reported: surplus 0, deleted 0, 0 errors. And because the dedup pass now runs inside every sync, the pipeline doesn't just avoid re-corrupting the table — if duplicates ever appear again by any path, the next cycle detects and heals them, and surfaces the anomaly instead of swallowing it.
Cleaning the data was half the job. The other half was making sure the platform can never again fail quietly — no workflow without a live error handler, no day without a health signal, no number on screen that the source of truth can't reproduce.
The rebuilt pipeline carries its own repair logic: every cycle deduplicates against the invoice identifier, keeps the row matching current accounting-system state, and reports any surplus it heals. The discipline that fixed the corruption — dry-run first, verify the exact count against the source of truth, delete under backup, then prove convergence — is now the pipeline's permanent behavior, not a one-time rescue script. The evidence chain reads: dry-run surplus 2,725 exact → live delete 2,725 with 0 errors → next cycle surplus 0. The table went from 67% duplicates to zero duplicate groups, and the eight-figure overstatement disappeared with them.
The review uncovered an earlier stale-token outage that had silently stopped payment notifications for roughly 21 hours — nothing alerted, because the workflows' error paths pointed nowhere useful. All 17 automation workflows were re-pointed onto a live error handler, so a failed run now produces an immediate, human-visible alert instead of a quiet gap.
Error handlers catch loud failures; the digest catches quiet ones. A daily health report now summarizes sync outcomes, row counts, and anomaly signals — so a failure mode that produces no error but wrong data, the exact class that caused this incident, surfaces within a day instead of compounding for months.
An inverted priority rule had the short internal folio outranking the full invoice reference across the platform. The precedence was flipped everywhere it appears, so finance staff see the complete, unambiguous document reference — recorded as a client-approved decision, not a silent change.
Role checks that existed only in the interface were hardened into server-side enforcement: writes to the data layer now require an elevated role on a verified token, fail-closed, with a proper read-only tier for operational staff. Authorization the UI merely implied is now something the backend guarantees.
I rescue production data systems — tracing every displayed figure back to its source of truth, fixing the mechanism instead of the symptom, and proving convergence before I call it done.
Let's Work Together