[{"data":1,"prerenderedAt":421},["ShallowReactive",2],{"monument:profile":3,"\u002Fblog\u002Ffive-dollar-root-cause-analysis":4,"\u002Fblog\u002Ffive-dollar-root-cause-analysis-surround":410},null,{"id":5,"title":6,"authors":7,"badge":13,"body":15,"date":399,"description":400,"extension":401,"image":402,"meta":403,"navigation":404,"path":405,"seo":406,"stem":408,"__hash__":409},"posts\u002F3.blog\u002F01.five-dollar-root-cause-analysis.md","The $5 Root Cause Analysis",[8],{"name":9,"to":10,"avatar":11},"Kyle Johnson","https:\u002F\u002Fwww.kylejohnson.ai",{"src":12},"\u002Fimg\u002Fkyle_headshot.jpg",{"label":14},"Case Study",{"type":16,"value":17,"toc":383},"minimark",[18,29,40,43,48,51,91,102,106,109,112,115,119,122,125,129,132,164,167,170,174,177,182,189,192,195,207,210,214,217,237,240,244,251,318,321,324,327,331,334,338,341,352,355,375],[19,20,21],"p",{},[22,23],"img",{"alt":24,"className":25,"src":27,"width":28},"Ink-line illustration of a ruined classical column, broken partway up its shaft, with fallen column drums and carved fragments scattered beside it between two trees",[26],"rounded-lg","\u002Fimg\u002Fclipart\u002Fclassical-column-ruin-wide.webp",1200,[19,30,31,32,39],{},"On Tuesday I exported a 33-second clip from ",[33,34,38],"a",{"href":35,"rel":36},"https:\u002F\u002Fcamerashy.app",[37],"nofollow","Camera Shy"," and the captions were garbage. Single words sat on screen for six and seven seconds while the voice moved on without them. Every gate we run had passed that same morning: typecheck, lint, two Playwright passes, a 7-for-7 production smoke test. I told Claude Code the captions were wrong, gave it nothing else, and went to do something else.",[19,41,42],{},"When I came back it had done the best product data science I've watched all year. I spent six years doing product data science at Meta, so I don't say that casually. The technical detail is at the bottom of this post for anyone who wants it, and it's worth the read. But the thing I keep coming back to is the receipt. The whole investigation, from metric design through postmortem, cost about five dollars in API calls.",[44,45,47],"h2",{"id":46},"what-five-dollars-bought","What five dollars bought",[19,49,50],{},"Here is the work, described the way I'd describe it if a team had done it.",[52,53,54,62,68,74,80],"ul",{},[55,56,57,61],"li",{},[58,59,60],"strong",{},"A data scientist's metric design."," Caption offset measured at each phrase's entrance, a 400ms pass threshold derived from the pipeline's own mechanics, a match-rate metric that fails the run on its own, and a drift slope that tells \"one wrong addition\" apart from \"the clock runs at the wrong rate.\"",[55,63,64,67],{},[58,65,66],{},"An engineer's instrument."," An OCR-plus-transcription harness that measures the exported video from the outside, built on the principle that you cannot validate an exporter against its own output.",[55,69,70,73],{},[58,71,72],{},"A controlled experiment."," Four arms on the same clip. One root cause found, two suspects cleared, one tempting shallow fix ruled out.",[55,75,76,79],{},[58,77,78],{},"A calibration pass."," It caught a 600ms systematic bias in its own matcher and removed it before trusting a single number.",[55,81,82,85,86,90],{},[58,83,84],{},"QA's fixtures and the postmortem."," A golden regression fixture, the failing production numbers preserved as a negative control, a ",[87,88,89],"code",{},"check:caption-sync"," command, and three rules written into the repo's agent docs so no future session relearns the lesson.",[92,93,96],"callout",{"color":94,"icon":95},"primary","i-lucide-receipt",[19,97,98,101],{},[58,99,100],{},"The receipt."," One agent, one afternoon, roughly $5 in API calls. The bug shipped in the morning, I caught it after lunch, and the measured, guarded fix was back in production by 7:39 that evening, most of it unattended.",[44,103,105],{"id":104},"how-this-used-to-get-staffed","How this used to get staffed",[19,107,108],{},"I know how this loop runs at a big company because I've been on the teams that ran it. A data scientist writes the metric definitions. An engineer builds the extraction harness. QA writes the fixtures. Someone drafts the postmortem. Two weeks go by and a dozen people attend the review. Any one piece of what happened here would have made a solid entry in a promo packet, and in 2022 the whole sequence would have headlined a half-year review.",[19,110,111],{},"It took a dozen people because the pieces lived in different heads, and every handoff leaks context. The DS who sets the threshold has never read the export code, so the threshold is a round number. The engineer who builds the harness wasn't in the metric review, so the harness measures something slightly different from what was defined. The QA lead inherits both and writes fixtures for the version they understood. Each seam costs a meeting and a little truth.",[19,113,114],{},"Here, one process held all of it at once: the pipeline code, the product, the bug report, the metric rationale, and the experiment. That is why the 400ms threshold has a derivation (250ms of caption hold, plus or minus 125ms of sampling error at four frames per second, about 100ms of Whisper boundary slop) instead of a guess. That is why the experiment tested the shallow fix before the write-up rejected it. Data science, engineering, and product judgment in one brain, with all the context, is what made the work good. The five dollars is what makes it hard to think about.",[44,116,118],{"id":117},"the-part-i-nearly-missed","The part I nearly missed",[19,120,121],{},"What actually landed in front of me was a small diff and a summary: delete one parameter, add a guard, add a check command. If I had only read the diff I would have said \"great\" and merged it. The metric design, the four arms, the debiasing pass, all of it lived in sub-agent transcripts I only saw because I happened to be watching them scroll by.",[19,123,124],{},"Which means I have almost certainly done this before. Handed an agent a problem, received a correct fix, and never noticed the caliber of the analysis underneath it. The work is now routinely better than the attention we pay to it. That's a strange position for a person who used to be paid specifically to notice.",[44,126,128],{"id":127},"what-i-think-this-means","What I think this means",[19,130,131],{},"Some inferences, held loosely.",[133,134,135,143,150,157],"card-group",{},[136,137,140],"card",{"icon":138,"title":139},"i-lucide-building-2","If you're at an enterprise",[19,141,142],{},"The two-week, twelve-person RCA loop is now a background task. What stays expensive sits on either side of it: someone with enough context to ask the right question, and someone with enough taste to check the answer and the authority to act on it. Headcount planning that assumes analysis is the bottleneck is planning for 2022.",[136,144,147],{"icon":145,"title":146},"i-lucide-store","If you run a small business",[19,148,149],{},"You can now afford a data scientist, rented one question at a time, at a quality that used to take a FAANG team half a year. The constraint moves from \"can we afford to look\" to \"do we know what to look at.\"",[136,151,154],{"icon":152,"title":153},"i-lucide-rocket","If you're a small software startup",[19,155,156],{},"\"We'll instrument it later\" is dead. A measurement harness with a golden fixture and a negative control cost less than the coffee I drank while it was built. Every \"we think it works\" can be \"we measured it\" at the moment the code ships.",[136,158,161],{"icon":159,"title":160},"i-lucide-flask-conical","If you're a data scientist",[19,162,163],{},"Analysis divorced from the codebase is now the weak position. The agent did this well because it had the repo, the pipeline, the product, and the bug in one context. The DS who can read and run the code, and who spends their hours on framing and verification instead of on the harness, is the one still needed.",[19,165,166],{},"One prediction. Within a couple of years, \"we ran a root cause analysis\" at most companies will mean an agent ran it and a human read the write-up. The write-up becomes the deliverable, and the skill that matters is reading it well: knowing that a low match rate is a louder alarm than a big offset, that a measuring stick needs calibrating before you read it, that a fix without a negative control is a hope. Everyone knows agents write code fast. The thing I keep updating on is the judgment.",[168,169],"hr",{},[44,171,173],{"id":172},"the-nuts-and-bolts","The nuts and bolts",[19,175,176],{},"For the curious. This is what actually happened, in the order it happened.",[178,179,181],"h3",{"id":180},"it-refused-to-trust-the-codes-own-opinion","It refused to trust the code's own opinion",[92,183,186],{"color":184,"icon":185},"neutral","i-lucide-quote",[19,187,188],{},"\"You cannot validate an exporter against its own output.\"",[19,190,191],{},"That sentence is in the first file of the evaluation harness the agent wrote, and the whole fix hangs on it. If the caption pipeline is broken, a fresh export produces a file the code believes is correct, and checking the file against that belief passes. That is exactly how a defect that burned wrong words into finished videos sailed through every green check we had.",[19,193,194],{},"So instead of reading the code and guessing, it built an instrument that measures the export from the outside:",[196,197,198,201,204],"ol",{},[55,199,200],{},"Pull the audio out of the finished video and transcribe it with word-level timestamps, treating the export's own audio as ground truth.",[55,202,203],{},"Sample the frames, crop to the caption band, and run OCR on the text actually drawn on screen.",[55,205,206],{},"Match each on-screen phrase to the run of spoken words it was made of.",[19,208,209],{},"The transcript says when the words were spoken. The frames say when they were shown. The gap between those two clocks is the entire question.",[178,211,213],{"id":212},"the-metrics","The metrics",[19,215,216],{},"Each one came with a stated rationale.",[52,218,219,225,231],{},[55,220,221,224],{},[58,222,223],{},"Median offset."," The gap between when a phrase is spoken and when it first appears on screen, taken at each phrase's entrance, with a 400ms pass threshold. Measured at the entrance because, in the agent's words, \"a phrase that arrives on the right word is forgiven a long exit, and one that arrives late is wrong no matter when it leaves.\" The 400ms is derived from the pipeline: 250ms of caption hold, plus or minus 125ms of sampling error at four frames per second, and about 100ms of Whisper boundary slop.",[55,226,227,230],{},[58,228,229],{},"Match rate."," The share of on-screen phrases found anywhere in the clip's own audio. It fails the run on its own, because a caption that matches nothing was never spoken in that clip at all.",[55,232,233,236],{},[58,234,235],{},"Drift."," A least-squares slope of offset against on-screen time. A constant offset means one wrong addition. An offset that grows means the caption clock runs at the wrong rate. Those are different lines of code.",[19,238,239],{},"I have sat through metric-review meetings with senior data scientists that produced weaker definitions than these.",[178,241,243],{"id":242},"four-arms-one-root-cause","Four arms, one root cause",[19,245,246,247,250],{},"The suspect was a vocabulary hint. We had been passing the session transcript to Whisper's ",[87,248,249],{},"prompt"," parameter so that names and jargon would transcribe correctly. The agent tested it the way you'd want an experiment run: same clip, same audio, same gateway, two runs per arm.",[252,253,254,270],"table",{},[255,256,257],"thead",{},[258,259,260,264,267],"tr",{},[261,262,263],"th",{},"Arm",[261,265,266],{},"Result",[261,268,269],{},"Verdict",[271,272,273,285,296,307],"tbody",{},[258,274,275,279,282],{},[276,277,278],"td",{},"Prompt off",[276,280,281],{},"88 segments, longest word 1.8s",[276,283,284],{},"Baseline",[258,286,287,290,293],{},[276,288,289],{},"Prompt on",[276,291,292],{},"428 segments, longest word 6.44s",[276,294,295],{},"Root cause",[258,297,298,301,304],{},[276,299,300],{},"Prompt on, untrimmed",[276,302,303],{},"619 segments",[276,305,306],{},"Bounding the prompt won't fix it",[258,308,309,312,315],{},[276,310,311],{},"Audio cross-correlation",[276,313,314],{},"0.46s lag",[276,316,317],{},"Audio path cleared",[19,319,320],{},"Whisper had been transcribing the prompt text itself, folding it into the output as fabricated words. And 88 divided by 428 is 0.21, which reproduced the one-fifth-speed caption clock I'd seen in the export, to two decimal places.",[19,322,323],{},"The third arm ran the same clip untrimmed, with the prompt text fully inside the audio window, and still got 619 segments. That killed the tempting shallow fix of bounding the prompt to the trim window. The write-up's conclusion: \"The option is unsafe in general. It is deleted, not bounded.\" The fourth arm cross-correlated the extracted audio against the source and measured a lag of 0.46 seconds, matching the 450ms of padding we intended, which cleared the audio path entirely.",[19,325,326],{},"Four arms, one root cause, two exonerated suspects.",[178,328,330],{"id":329},"it-debiased-its-own-instrument","It debiased its own instrument",[19,332,333],{},"This is the part that got me. Measured against a known-good export, the first version of the phrase matcher reported a median offset of 600ms. The agent noticed its matcher was dating each phrase from a window padded with words the phrase didn't actually contain, trimmed the window, and re-measured: 0ms median offset at an identical 90 percent match rate. It found and removed a systematic bias in its own measurement tool before trusting a single number from it. Plenty of human teams run dashboards for years without ever asking that question.",[178,335,337],{"id":336},"what-shipped","What shipped",[19,339,340],{},"The production fix was small:",[52,342,343,349],{},[55,344,345,346,348],{},"Delete the ",[87,347,249],{}," parameter.",[55,350,351],{},"Add a guard that rejects any transcription with a word longer than two seconds, words running past the end of the submitted audio, or fewer than one word per five seconds, and falls back to the older caption path when it trips.",[19,353,354],{},"The measurement work is what made a deletion that small trustworthy. The durable part is everything around it:",[52,356,357,363,366,369],{},[55,358,359,360,362],{},"A ",[87,361,89],{}," command any future session can run.",[55,364,365],{},"A golden fixture with a hand-written script as ground truth, passing at 90 percent match and 0.00 seconds median offset.",[55,367,368],{},"The failing production numbers preserved as a negative control: 10 percent match, 15.36 seconds median offset.",[55,370,371,372,374],{},"Three rules in the repo's agent docs, including \"Whisper transcribes ",[87,373,249],{},"; never pass it,\" with the 88-versus-428 numbers attached.",[19,376,377,378,382],{},"Camera Shy is in early access at ",[33,379,381],{"href":35,"rel":380},[37],"camerashy.app",". The captions arrive on time now, and I can prove it to within 400 milliseconds.",{"title":384,"searchDepth":385,"depth":385,"links":386},"",2,[387,388,389,390,391],{"id":46,"depth":385,"text":47},{"id":104,"depth":385,"text":105},{"id":117,"depth":385,"text":118},{"id":127,"depth":385,"text":128},{"id":172,"depth":385,"text":173,"children":392},[393,395,396,397,398],{"id":180,"depth":394,"text":181},3,{"id":212,"depth":394,"text":213},{"id":242,"depth":394,"text":243},{"id":329,"depth":394,"text":330},{"id":336,"depth":394,"text":337},"2026-08-14","An agent root-caused a caption bug in Camera Shy with a measurement harness, a four-arm experiment, and a debiased instrument. Total spend: about five dollars. The bug is the small story. The bill is the big one.","md",{"src":27},{},true,"\u002Fblog\u002Ffive-dollar-root-cause-analysis",{"description":407,"title":6},"World-class product data science, engineering, and product judgment in one brain with full context now costs about $5. A Camera Shy caption bug, an AI agent's OCR measurement harness and controlled Whisper experiment, and what that price means for enterprises, small businesses, startups, and data scientists.","3.blog\u002F01.five-dollar-root-cause-analysis","8WmQzE6x6g2igrPH6sm8YaGLoymcBOyJ288qXZbw4cU",[411,416],{"title":412,"path":413,"stem":414,"description":415,"children":-1},"A Hypervisor for Products","\u002Fblog\u002Fa-hypervisor-for-products","3.blog\u002F00.a-hypervisor-for-products","One control plane now watches ten Monument Labs products, keeps a falsifiable ledger for each, and proposes its own fixes through a gate it cannot talk its way past. Notes on running a portfolio on about ten minutes of attention a day.",{"title":417,"path":418,"stem":419,"description":420,"children":-1},"Seven More Tips for Vibe Coders","\u002Fblog\u002Fseven-more-tips-for-vibe-coders","3.blog\u002F02.seven-more-tips-for-vibe-coders","Part two: the details that separate a professional product from an obviously vibe-coded one. Link cards, finished metadata, social login, selection control, and layouts that behave.",1788295449782]