Thirteen iterations: what a 53-million-candidate bug taught me

A tool of mine once generated 52,793,600 candidate locations when it was asked for 200. The run had been going for over an hour with no end in sight, and the log finally told me why: the candidate generator was placing the requested count inside every one of the road network's 263,968 segments. Multiply that out against the origins and the OD matrix was headed toward roughly 13.7 billion pairs.

Here's the uncomfortable part: the code looked fine. It had been reviewed. The logic read sensibly. Nothing about it said "this will try to boil the ocean." The defect was only visible at runtime, on real data — which is why the first lesson from this build became a personal working principle: run it, don't just read it. I now budget for iterative execution on representative data as a first-class part of any build, not a final checkbox.

The fix itself came from an unglamorous place: the vendor's documentation. Reading the OD Cost Matrix docs carefully revealed that the solver snaps every input location to the nearest network edge on load — meaning my entire "pre-place candidates on roads" step was unnecessary. Generate exactly N points inside the boundary, let the solver do the snapping, and the matrix collapses by five orders of magnitude. Runtime went from over an hour to minutes. Second lesson: verify against primary sources. The docs everyone skips are where several of my best fixes have come from — including one more from the same read: the docs confirmed that setting a cutoff silently drops unreached destinations with no flag at all. That's not a crash; that's a plausible-but-wrong answer feeding a real decision. I removed the cutoff entirely.

It took thirteen iterations to take that tool from "works on the demo" to "trusted in production" — reproducible seeds, network-snapped outputs that never land in water, a clean-copy step that flattens whatever joins and group layers enterprise data throws at it, and normalization for ArcGIS's charming habit of passing blank parameters as the literal string #. None of those hardening steps were in the original plan. All of them came from running the tool against messy reality and refusing to let a quiet failure slide.

The AI part people ask about

I built this tool in partnership with AI, and this project is a good illustration of what that partnership actually looks like — because the AI didn't find the bug. Running the tool found the bug. What AI did was compress everything around that moment: drafting the refactors, restructuring the candidate generator once I'd diagnosed the explosion, and turning each lesson into hardened code far faster than I could type it.

The division of labor matters. I owned the direction: what to test, which log line smelled wrong, when a "working" answer wasn't trustworthy, which documentation claim needed verifying against a real run. The AI multiplied my output inside that direction. That's the difference between directing AI and merely prompting it — the judgment about what's true and what matters stayed with me, and everything mechanical accelerated. Thirteen iterations in a timeframe that would otherwise have allowed four or five.

If you take one thing from this note: the most dangerous failures in data tooling are the quiet ones — the plausible number, the silently truncated matrix, the result that looks fine and is wrong. Design against those first. The loud crashes take care of themselves.

Full case study: Network Center of Gravity Tool →