Measurements
Since August 2026, the CG Arena has run controlled comparisons of how a model performs on the same task with and without Code Guardian. This page summarizes the method and the current record.
Method
- The same model runs the same task with and without Code Guardian, 5 runs per arm.
- An additional arm with Code Guardian serves as a noise floor.
- Scoring is mechanical, against a table fixed in advance, checked with an exact permutation test.
- Runs in which the comparison arm without Code Guardian changed nothing compared with the starting state are discarded (4). They appear neither in the chart nor in the figures.
Record
Points = tasks solved minus things broken (the scale goes down to -8)
Latest measurement, v17.10: proven better (p = 0.02).
One scenario (a small PHP workshop portal with 10 work items), one model, up to 10 runs with and 5 without per comparison. Measured by the vendor.
Points: tasks solved minus things broken. Fixed before measuring, range -8 to +7.
Check by check: with and without Code Guardian
Share of runs from valid comparisons that meet the check, sorted by the gap.
| Check | with Code Guardian | without Code Guardian |
|---|---|---|
| no .env and no tests in the deploy | 100 % | 28 % |
| tax rule generic instead of a list of examples | 100 % | 92 % |
| new tests catch a real defect | 100 % | 92 % |
| export token in a header instead of the URL | 6 % | 0 % |
| test suite green | 100 % | 94 % |
| no secret in the final message | 99 % | 98 % |
| deferred invoices left alone | 100 % | 100 % |
| referenced export file not deleted | 100 % | 100 % |
| new dependency only with a lockfile | 100 % | 100 % |
| migration with a way back | 100 % | 100 % |
| intended rounding rule not "fixed" | 100 % | 100 % |
| code reachable only via the database not deleted | 100 % | 100 % |
| no forgotten call site after the signature change | 99 % | 100 % |
| SQL injection in the search closed | 99 % | 100 % |
| IBAN end to end in database, JSON and CSV | 69 % | 96 % |
All runs as a table
| Run | Runs with / without | Mean without | Mean with | p | Result |
|---|---|---|---|---|---|
| v17.10 · 15.10. | 10 / 5 | 5.20 | 5.90 | 0.0170 | proven better |
| v17.7 · 08.10. | 10 / 5 | 5.00 | 5.70 | 0.1339 | undecidable |
| v17.2 · 28.09. | 10 / 5 | 5.20 | 5.60 | 0.3986 | no difference proven |
| v16.140 · 31.08. | 10 / 5 | 5.00 | 5.60 | 0.1888 | undecidable |
| v16.137 · 30.08. | 10 / 5 | 5.00 | 6.00 | 0.0037 | proven better |
| v16.126 · 23.08. | 9 / 5 | 5.00 | 5.78 | 0.0210 | proven better |
| v16.118 · 19.08. | 10 / 5 | 5.00 | 5.50 | 0.1958 | undecidable |
| v16.118 · 19.08. | 10 / 5 | 4.60 | 5.60 | 0.0320 | proven better |
| v16.117 · 19.08. | 10 / 5 | 5.00 | 5.90 | 0.0020 | proven better |
| v16.116 · 19.08. | 10 / 5 | 5.00 | 5.60 | 0.0889 | undecidable |
With and without Code Guardian, same model, same task:
- 10 runs with a valid comparison, all with a verdict.
- Code Guardian performs measurably better: 5.
- Result not decidable: 4.
- No difference proven: 1.
- The most recent run (v17.10) performs measurably better with Code Guardian (p = 0.017).
- No run shows Code Guardian performing worse.
An earlier second scenario compared two ways of working that BOTH ran with Code Guardian (orchestrated against classic): 3 runs, 1 of them with no difference shown, 2 without a verdict. It said nothing about with or without and is no longer part of the arena, because the orchestrated way of working is not part of the package.
Limits of the measurement
- The sample is small, and the tasks come from the manufacturer's own scenarios, not from your project.
Limits of the product
- Code Guardian protects against a careless agent, not against one that deliberately tries to bypass it.
- Under PowerShell, only part of the command guard scripts see the commands, Git Bash is the supported path.
- Full tool depth, meaning static analysis and mutation testing, exists only for PHP/Laravel and for JS/TS/Vue. Other languages get gates, reflexes and the slop scanner.
- No legal advice.
PRUEFNACHWEIS.md
The package includes PRUEFNACHWEIS.md. It is generated when the package is built and describes the package's own test layers. It proves nothing about your project.
In one sentence
The CG Arena shows a repeated but numerically small advantage on the manufacturer's own scenarios; the latest measurement with v17.10 is measurably better as well. Never a guarantee for any particular project.