Comparing alarms one by one
The first clustering design. On 09's breakdowns it scored 0.032, chance, on every split, and grouping by incident replaced it.
Measured on 120 real bugs, judged by tests no model saw
An AI that fixes two real bugs in three before a person even looks, and proposes a pull request only when it is confident.
When software breaks, someone has to find out why and write the fix, and that can take days. This project hands the first draft to AI: it groups the error messages into one problem, writes a test that proves the bug before touching the code, drafts a fix, and has a second AI from a different company try to break it. Then a person decides. It was measured on 120 bugs from public Python projects, each judged by tests the maintainers wrote and no model saw.
90.2%
of its proposed pull requests were right
46 of the 51 fixes it was confident enough to propose passed the maintainers' hidden tests. 95% interval 79.0% to 95.7%.
67.5%
of 120 real bugs fixed
81 fixes passed tests the maintainers wrote and no model saw, proposed or not. 95% interval 58.7% to 75.2%.
0.636
agreement with the true causes
Grouping alarms by incident first, against 0.032 for comparing them one by one. 0 is chance and 1 is perfect.
US$0.24
per bug fixed
In model calls, interval US$0.21 to US$0.29, and a median of 0.9 minutes each.
The target was missed. The plan allowed at most 3% of proposed pull requests to be wrong. 5 of the 51 were, 9.8% (95% interval 4.3% to 21.0%), so on this evidence it should suggest rather than propose, and the live version only comments. 4 of the 5 did what the issue asked, where the issue left out what the maintainers' tests checked.
120 bugs from SWE-bench Verified, picked at random and written down before the first run, plus 5 live runs on real pull requests. Every number on this page is written by the project's report command and can be reproduced from the public repository. See how it's built, in four diagrams.
Watch a failure become a reviewed fix
Thirty-six seconds, with sound: a real run on another of my projects, from the moment a pull request's tests fail to the moment a person reads the proposed fix. A bug was planted in it on purpose, and every name, line of code and number in the film is from that run.
Step by step, with the run's own code and output, in the next section.
Follow one failure, from red tests to a reviewed fix
A real run, on a pull request in another of my projects with a bug planted in it on purpose. The project's tests failed 10 times, which mend grouped into 5 tickets. Step through what happened next: every line below is from the run's own record. The whole run took 36 seconds and cost US$0.14.
The planted bug was in the Holm correction, a standard way of adjusting several statistical tests at once: one multiplier was off by one. The comment it posted is on the pull request, and the pull request is never merged.
How it works
Seven steps, each checked before the next
The rest of this page is the whole method, in the order it happens. Each part starts in plain language, with a picture. Where there is more to say for someone who does this for a living, it is folded away under For the technical reader, so you can skip it if you only want the idea.
The whole idea in one picture
One broken change usually sets off many alarms. The pipeline turns them into one problem, proves the problem with a test, fixes it, has the fix attacked, and then decides how loudly to speak. Nothing it produces is merged; at most it offers a fix.
- 1CaptureEach failure becomes one record, with its error and where it happened.
- 2GroupRecords with one cause become one ticket.
- 3LocateA small, cheap model finds the files most likely at fault.
- 4ProveA new test must fail on the broken code, or the run stops.
- 5FixThe fix must pass that test and every test that passed before.
- 6ChallengeA model from another company is asked to find what is wrong.
- 7RouteOffer it, suggest it, or say nothing.
No step can be skipped. No fix is written until a test has failed, no fix is offered while the reviewer objects, and there is no step at all that merges. The code enforces each of those rules, and a test checks each one.
Twenty alarms, one bug
When a live system breaks, a dozen measurements jump at once, each its own alarm with its own name. Nothing in the words "dead letters rose" and "request latency fell" says they are the same problem. Only the timing does. The first design compared alarms one at a time and found almost nothing; grouping them by when they started, and then by their pattern, found most of the causes.
Select a bar for its value.
Grouping by incident first wins on all 56 of 56 splits. Its average agreement with the true faults is 0.636, against 0.032 for comparing alarms one by one, which is chance.
Select a bar for its value.
What these numbers do not say. With tracebacks it finds about half of each fault (52% completeness). With people's bug reports, where only the words are left, it needs a trained text model (0.516, against 0.086 without one), and that model was tried after the first result was seen. None of the three sets has two incidents overlapping in time.
For the technical reader: the two clustering methods and how they were scored
The record-level method builds one distance per pair of records from three channels: a fingerprint (exception type, innermost project frame, normalised message), an embedding of the failure text, and code locality (shared frames and files). Average-linkage clustering cuts it at a threshold tuned on held-out causes. The two-stage method first groups records whose start times fall together into incidents, then groups incidents by the set of their symptoms. Quality is the adjusted Rand index (ARI), purity and completeness. On 09's runs, every one of the 56 ways of holding three of eight faults out was scored; on the other two sets, five folds by cause with a delete-one-cause jackknife for the interval. The full write-up of the rejected method is in docs/rejected.md.
No fix without a failing test first
Before the fixer may touch the code, another step writes a new test that shows the bug, and runs it on the broken code in a sealed container. If the test passes there, it proves nothing, and the run stops. The fixer is then held to that test and to every test that passed before. The idea is that a fix should come with its own evidence.
96.7%of the 120 bugs got a test that failed first (116 of them)
26 vs 27bugs fixed out of 40, with the test step and without it
US$0.080per bug in model calls without the step, against US$0.170 with it
Select a bar for its value.
The test step is not shown to make the fixes better. Without it the pipeline fixed one more bug of 40, at half the cost. The difference is +2.5 points (-12.5 to +17.5), far too uncertain to call either way.
Why it stays. The test is what the person reviewing the fix gets to read and run, and the pipeline refuses to write a fix without one. It stays as evidence for that person.
For the technical reader: how the rule is enforced, and how the comparison was run
The pipeline is a state machine with no transition into "patched" except from "test failed as expected", observed in the sandbox; a test that errors rather than fails, or passes on the broken commit, gets one retry that is told why, then the run stops. The ablation ran outside the graph, so the graph keeps no switch that turns the rule off, on 40 of the main run's tasks drawn with a fixed seed, reusing the main run's localisation, and went through project 03's release gate as a paired comparison at a ten-point margin.
A second AI, from another company, tries to break it
The fix is written by Anthropic's model. It is then reviewed by OpenAI's, which is told to find what is wrong with it. The idea is that two models from different companies share fewer blind spots. When the reviewer objects, the fix can still be suggested, but it is never offered as ready.
Select a bar for its value.
An objection is a real warning. When the reviewer objected, the fix was wrong 40.0% of the time (40 fixes); when it agreed, 10.9% (64 fixes).
Select a bar for its value.
The different-company rule is not proven. The fixer's own company's model objected less often, so it raised fewer false alarms and caught fewer bad fixes, and its confidently-wrong rate would have been 12.7%. Project 03's release gate could not tell the two reviewers apart. The rule stays as a design choice the evidence leans towards.
For the technical reader: how the vendor split is enforced
Each role is an alias in the portfolio gateway (project 04). At start-up the pipeline resolves the fixer and reviewer aliases and refuses to run if they land on the same provider, and a test asserts it against the shipped configuration. The same-vendor arm reviewed all 104 reviewed patches again on identical input, and the different-vendor reviewer was also asked again, to measure how much a reviewer disagrees with itself.
Knowing when to stay quiet
A wrong fix presented as ready wastes an engineer's time, so each fix is routed by its evidence: did the new test pass, did every old test still pass, did the reviewer agree. Only a fix with all three is offered as a pull request. A weaker one is left as a suggestion, and a run with no fix stays quiet. The rules were set on a separate 30 bugs before the main run.
Select a bar for its value.
The evidence sorts the fixes, but restraint costs a lot. Of the 51 it offered, 90.2% were right. Of the 35 it kept quiet about, 14 had a fix the hidden tests accepted, mostly where no old test covered the code it changed, which the rules read as "not shown safe".
For the technical reader: why the confidence score does not decide
The router also computes a confidence score with size penalties, and the plan was to calibrate a threshold on it against a 3% confidently-wrong target. On the validation split every patch that passed the rules scored the same, so the score did no work and the calibration landed on zero; changing it after seeing the main run would have been tuning on the test set. The rules decide, and that consequence was written down before the main run (docs/routing.md).
Does it work?
120 bugs from SWE-bench Verified, a public set of real GitHub issues in Python projects such as Django, scikit-learn and sympy. Each comes with tests the maintainers wrote for their own fix, which are kept from every model and used only to mark the result. The 120 were drawn at random and committed to the repository before the first run, so no hard bug could be dropped afterwards.
67.5%fixed (81 of 120)58.7% to 75.2%
9.8%of offered fixes confidently wrong, against a 3% target4.3% to 21.0%
US$0.16per bug attempted; US$0.24 per bug fixedUS$0.21 to US$0.29 per fix
| What it did | Bugs | Fixed | Confidently wrong | Cost per bug |
|---|
What this number does not say. It is one public benchmark, in Python, on issues that come with a clear description; a real outage is messier. And 67.5% is not what a team would see from the offered fixes alone: those are right more often, and the rest arrive as suggestions or not at all.
For the technical reader: how the evaluation avoids fooling itself
The subset was drawn with seed 20261017, 30 for calibration and 120 for the main run, and the sampler refuses to redraw once a run exists. Every drafted patch, whatever its tier, goes to the official SWE-bench harness, which runs the hidden tests in a Linux container; a task the harness could not judge is rerun, never counted as a failure. Proportions carry Wilson intervals and costs bootstrap intervals. The hidden tests and the maintainers' patches never reach a prompt.
The five times it was confidently wrong
5 fixes passed every check the pipeline had and still failed the maintainers' tests. I read each one against the maintainers' own fix and put it in one of seven categories that were written down before the run.
Select a bar for its value.
4 of the 5 did what the issue asked. The issue left out what the maintainers' tests checked: an order of fields, an internal key, every case wrapped rather than the one shown, a pattern the issue itself proposed. In 1 case the fix changed only one of the two places the bug lived.
| Bug | Cause | Why |
|---|
On real pull requests
The same pipeline runs on two of my other projects, the release gate (03) and the AI gateway (04). When a pull request's tests fail, it reruns them in a sealed container, picks the biggest problem, and answers with a comment. It cannot push, open a pull request or merge, and it stops when that repository's US$1.50 is spent.
| Date | Repository | Tests run | Failures captured | Tickets | Outcome | Cost |
|---|
It found a real breakage nobody had planted. On its second run in 04, the biggest problem was not the planted bug but four tests broken by a spending limit set elsewhere. Its fix raised the limit, the reviewer objected that this weakened a deliberate safeguard, and it went out as a suggestion with the objection attached. The reviewer was right.
The same bug, two different answers. 04's planted bug was run twice on the same code. Once the fix treated the symptom and the reviewer named the right place; the next time the fix never passed its own test, so it stopped and posted nothing. Five runs show how it behaves; the measurement is the 120 bugs above.
What did not work
Each of these was built and measured, and the measurement went against it. The full record is in the repository's build log.
The 3% target
The plan allowed at most 3% of offered fixes to be wrong. It was 9.8%, so the live version posts comments only.
The test step as a quality gain
Expected to make fixes better. Without it the pipeline fixed 27 of 40 against 26, at half the cost.
A different company as the reviewer
Expected to catch more. The fixer's own company's model reviewed about as well, and the gate could not separate them.
A pass that tested nothing
The first live run in 04 ran no tests, because that repository leaves its tests out of its own image, and reported a pass. A run that tests nothing now fails.
A budget that forgot
The live spending total was meant to carry between runs in a cache the job was never allowed to write. It now travels in each run's own records.
What this does not do
The limits, stated as plainly as the results.
- It never merges. There is no merge step in the code, and a test scans every source file to make sure none appears. On the live repositories it only comments.
- It does not fix without proof. No failing test, no fix: the run stops and says why.
- It does not meet its own target. 9.8% confidently wrong is above 3%, so on this evidence it should suggest, not offer.
- It is Python only, because the benchmark and the sandbox are.
- It is not a production incident tool. The clustering was measured on recorded breakdowns, planted faults and old bug reports, with no two incidents at once.
- It holds back good fixes. 14 of the 35 bugs it stayed quiet about had a fix that was right.
How it is engineered
The code is typed and tested and the pipeline is automated, so anyone can regenerate the numbers on this page from the repository.
294test functions, including that no merge path exists, that the reviewer is on another vendor, and that no fix starts without a failing test
9,700lines of typed Python, strict type checking and linting clean on every push
1command to rebuild every table in the README and every number on this page
- Python 3.13
- LangGraph
- Agentic workflows
- LLM evaluation
- SWE-bench Verified
- Anthropic and OpenAI APIs
- Adversarial review
- Failure clustering
- Embeddings
- OpenTelemetry
- Docker sandboxing
- GitHub Actions
- GitHub API
- Wilson and bootstrap intervals
- Ablation studies
- Spend caps
- mypy strict
- ruff
- pytest
For the technical reader: the choices that are easy to get wrong
- The sample was committed before the first run and the sampler refuses to redraw, so no hard task can quietly leave.
- Every drafted patch is scored, whatever tier it was routed to, so the quiet tier's missed fixes are counted rather than invisible.
- The benchmark harness runs in a Linux container, because on a Windows host even the benchmark's own reference patches fail.
- Django and sympy, 61% of the benchmark, do not use pytest; the sandbox runs each project's own test runner and reads its output with the harness's parsers.
- Every model call goes through the gateway with a US$1 cap per issue checked before the call, not after.
- On live repositories the image build is the only step with network access, and it runs only the repository's own locked install.
How this was built
I designed this project and made the calls in it. I read the maintainers' hidden tests myself to classify the five wrong fixes, so that no model ever saw them. When the main run missed the 3% target, I decided the live version would post comments and never open pull requests. I chose my own repositories for the live trial and planted the bugs it was tested on, and I set its spending cap before the first call.
I used an AI coding assistant (Claude Code) for implementation and drafting, which is now standard practice in software, data science and machine learning work. Every number on this page can be reproduced from the repository.
Two pieces are my own earlier projects, reused as published: every model call goes through the AI gateway (project 04), and both experiments were judged by the release gate (project 03). The recorded breakdowns come from the real-time fraud and risk platform (project 09).
Glossary
- Ablation
- Running the system again with one part taken out, to see what that part was worth.
- ARI (adjusted Rand index)
- How well a grouping matches the true one. 0 is what random grouping scores and 1 is a perfect match.
- Bootstrap interval
- A range for a number found by resampling the data many times and seeing how far the number moves.
- CI (continuous integration)
- The automatic test run on every change to a repository. "ci fails" means some tests failed.
- Confidently wrong
- A fix the system was sure enough to offer as ready, which then failed the maintainers' hidden tests.
- Gateway
- One place every model call goes through, which picks the provider, records the cost and refuses a call once the cap is reached.
- Hidden tests
- The tests the maintainers wrote for their own fix. They are kept from every model and used only to mark the result.
- LangGraph
- A Python library for building multi-step AI workflows as a graph of steps.
- Pull request
- A proposed change to a repository, which a person reviews and accepts or rejects.
- Release gate
- A tool that compares two versions of a system on the same cases and decides whether the new one is worse by more than a set margin.
- Sandbox
- A sealed container with no network, where the tests and fixes run so they cannot reach anything else.
- SWE-bench Verified
- A public benchmark of 500 real GitHub issues from Python projects, each checked by people to be fair and solvable.
- Ticket
- One problem to fix, made from all the failures that share its cause.
- Wilson interval
- A range for a percentage that stays sensible when the count is small or near 0% or 100%.