← writing

A benchmark can measure its own metadata

May 20, 2026 · 6 min · AI evaluations · revised evaluation postmortem

I built BRIDGE-bench to ask a reasonable question: can a tool-using language model find the vulnerability that was actually exploited in real DeFi contracts?

Then I found a more important problem in the benchmark itself.

The files contained provenance headers so a human could see where each example came from. The loader sent the entire file to the model. That meant comments such as exploit names, incident descriptions, and in one case the exact ground-truth label quietly became part of the evaluation prompt.

A benchmark that was supposed to measure code analysis was also measuring the model’s ability to recognize the dataset’s metadata.

The audit

I stopped looking at aggregate F1 and printed the exact strings the model received.

Across the locked 24-contract corpus:

  • 13 / 24 prompts contained severe author-injected leakage: the header named or described the exploit mechanism, or stated a label.
  • 7 / 24 contained moderate incident metadata.
  • after stripping author-added comments, the audit found 0 severe / 0 moderate leakage;
  • after additionally neutralizing protocol identity, all 24 examples passed the audit’s CLEAN condition.

Those are zero-cost dataset checks. They do not require another model call.

The original Opus run reported roughly 35% semantic F1 and 54% recall on the expanded 24-contract benchmark. Those measurements happened. What changed is what they are allowed to mean. They are now treated as contaminated upper bounds, not clean vulnerability-detection performance.

Euler made the problem impossible to hand-wave away

The Euler example’s author-added header described donateToReserves, explained the missing solvency check, and included the label missing_solvency_check. The model returned that label and the scorer counted it as a true positive.

But the vulnerable donateToReserves function was not present in the committed source bundle used by the benchmark. After comment stripping, the symbol disappeared entirely.

That one example establishes a hard lower bound on the methodological problem: at least one scored detection could not have been inferred from the vulnerable function in the supplied code, because the function was absent. The answer was available in the annotation.

It does not prove every finding was copied from metadata. Many prompts contained real code plus leaky prose, and a model may still need to reason about the code after recognizing the exploit family. The point is that the aggregate score mixed mechanisms we had not separated.

The original number was answering two questions at once

The intended question was:

Given contract source, can the model identify the exploited vulnerability?

The accidental second question was:

Given source plus provenance that may name or describe the exploit, can the model recover the expected label?

Once those inputs are mixed, a single F1 number cannot tell us how much comes from code reasoning, incident recognition, or memorized associations with famous protocol names.

This is a common evaluation failure mode. Public benchmarks are often assembled from bug reports, CVEs, postmortems, GitHub issues, filenames, commit messages, or annotated examples. Human-facing provenance can be useful for auditability while still being toxic as model input.

The sanitizer is a measurement tool, not a cosmetic cleanup

The benchmark now defines three prompt conditions:

  1. raw — reproduce the historical prompt;
  2. stripped — remove author-injected provenance comments while preserving the program;
  3. anon — additionally neutralize protocol identity.

The important part is verifying that sanitization does not accidentally change the code being evaluated.

The zero-cost preflight checks that stripping preserves the code-token stream and that anonymization keeps structural fingerprints stable. That validation caught a real sanitizer bug: an early address-neutralizing expression could rewrite the first 40 hex characters of 64-character bytes32 constants. If that corruption had reached a model run, a lower sanitized score could have been blamed on “removing leakage” when the sanitizer had actually changed the program.

That is the same measurement lesson twice: the control itself has to be controlled.

The experiment I am not pretending to have run

The clean causal experiment is paired remeasurement on the same 24 contracts:

raw → stripped → anon

with contract-level deltas and the same scoring stack. The repository now has a locked runner, sanitizer invariants, population checks, and post-run analysis for that experiment.

I am intentionally not running the paid model calls in the current writing program. Therefore I do not claim:

  • the sanitized F1;
  • the size of Δ(raw → stripped);
  • the size of Δ(stripped → anon);
  • a causal percentage of the historical score attributable to leakage.

A missing result is preferable to a fabricated one.

The strongest counterargument

A leaky header does not necessarily make the model’s output trivial. If a comment says “oracle manipulation,” the model may still have to find the relevant price path and explain why the code is vulnerable.

Agreed. That is why I am not subtracting 13 examples from the score or saying the model had zero capability.

The dataset audit establishes two narrower facts:

  1. the confound is systematic, not a one-off;
  2. the Euler example demonstrates a scored label that the committed vulnerable source could not support on its own.

The magnitude of score inflation remains unresolved without paired inference.

What I changed about how I build benchmarks

I now do the cheapest adversarial check before trusting a headline metric:

Print the exact prompt. Grep it for the label. Then grep it for a prose description of the answer.

After that:

  • separate provenance from model-visible input;
  • treat filenames and protocol identity as potential side channels;
  • make the sanitized population explicit rather than assuming every JSON row is a valid example;
  • verify transformations with invariants before using them as controls;
  • keep historical contaminated runs, but label them as historical rather than silently replacing them.

The benchmark became more useful when it became less flattering.

What this establishes

BRIDGE-bench’s original model score is not clean evidence of vulnerability-detection capability. Severe author-injected leakage existed in 13 of 24 prompts, and the benchmark now has a validated zero-cost path for stripping that information while preserving the program structure.

What this does not establish

It does not establish sanitized model performance, memorization rate, or the causal size of the contamination effect. Those require the paired inference experiment I have chosen not to run here.

Primary evidence

Evidence classes: the 13/24 audit, stripped/anon cleanliness checks, and Euler source-absence test are observed repository audits. The conclusion that the historical score mixes code analysis with metadata recognition is derived from those observations. The causal size of the effect is unknown.