TraceFuse: Beating a 2024 ICSE Method and Claude Opus 4.6 With One Cheap Model
TraceFuse combines a 2024 traceability method with one small-model call, outperforming a frontier LLM at 15x lower cost.
Summarize this article with:
TL;DR
TraceFuse is the requirement-to-test traceability method behind KomAInu. It fuses three signals into one score:
- Two TRIAD variants, used as lexical priors.
- A single backward pass with Haiku 4.5 — 21 calls total, one per test file.
- Geometric fusion, a "soft AND" that only scores a pair highly when every signal agrees.
| Method | AP | MAP | Source |
|---|---|---|---|
| TraceFuse | 48.60% | 63.30% | This work |
| TRIAD +b+o+i | 44.73% | 62.03% | ICSE 2024 |
| Claude Opus 4.6 (text only) | 39.66% | 56.00% | Baseline |
| Claude Opus 4.6 (text + code) | 39.22% | 53.75% | Baseline |
TraceFuse beats the ICSE 2024 TRIAD baseline by +3.87 AP and beats Claude Opus 4.6 by +8.9 to +9.4 AP — while running on Haiku, priced around $1 per million tokens against roughly $15 per million for Opus. That is close to a 15x cost difference for a better score.
This is my own engineering benchmark on a single, established research dataset, not a peer-reviewed result.
TraceFuse is not a research exercise — it runs in production as the traceability engine behind KomAInu, mapping SRS requirements to the tests that verify them.
See the traceability matrix in KomAInu →The Research Question
Frontier models are good at reading a requirement and a test and judging whether one verifies the other. So the question worth answering honestly was: do I need the frontier model, or does a small model plus a smarter method win?
TRIAD, published at ICSE 2024, is the incumbent classical baseline for this problem. It scores 44.73% AP and 62.03% MAP using weighted word statistics and source-code bridges — biterms and transitive links through the code that sits between a requirement and the test that exercises it.
To keep the comparison honest, every method in this benchmark receives the same artifacts, complete, with zero truncation. No method gets an information advantage over another — the only variable is the method itself.
The LibEST Dataset
- 52 requirements
- 21 C test files
- 14 source code files, used as an intermediate bridge
- 352 oracle requirement-to-test links
- 1,092 possible requirement/test pairs in the search space
Two metrics are reported throughout: AP (Average Precision) over one global ranked list of all 1,092 pairs, and MAP (Mean Average Precision), which averages a per-requirement AP across all 52 requirements. AP rewards a method that ranks true links near the top overall; MAP rewards consistency across every individual requirement, not just the easy ones.
TraceFuse Architecture
Signal 1 — Dual TRIAD prior
Rather than picking one TRIAD configuration, TraceFuse combines TRIAD's best-AP variant (b_o_i) with its best-MAP variant (o_i) as a two-vote ensemble. This inherits TRIAD's structural understanding of the intermediate source-code artifacts without committing to a single tuning.
Signal 2 — Backward pass
- Single direction: test → requirements, not requirements → test.
- Haiku 4.5, run at medium reasoning effort.
- 21 calls total — one per test file, not one per requirement pair.
- Per-test normalization to correct for calibration drift between calls.
Signal 3 — Geometric fusion
score(req, test) =
(0.15 + TRIAD_b_o_i)^1.0 ·
(0.15 + TRIAD_o_i)^1.0 ·
(0.15 + Haiku_backward)^0.5Multiplying the signals together, instead of averaging them, creates a "soft AND": every signal needs a reasonable confidence for the product to be high. One loud signal can no longer override the other two — the three components have to agree.
Cross-validation
Fusion weights were tuned with 5-fold cross-validation: split the 52 requirements into 5 groups, tune the fusion weights on 4 groups, then measure AP/MAP on the held-out group. Repeat for every group and average the result.
Globally, TraceFuse scores 48.60% AP / 63.30% MAP. Averaged across the 5 held-out folds, it scores 60.09% AP / 63.46% MAP — consistent with the global numbers, which is the point of cross-validating in the first place.
What Didn't Survive Cross-Validation
Two ideas looked promising in early experiments and were cut from the final method. The first was extended-thinking mode with an 8,192-token reasoning budget. The second was a forward pass — requirements → tests, in addition to the backward pass.
On a full-grid search over the whole dataset, adding these produced a +1.7 AP gain. But under 5-fold cross-validation, tuning only on the held-out folds, that gain nearly disappeared: AP moved by roughly +0.05 and MAP dropped. That is the textbook signature of overfitting to a fully-visible evaluation set — a configuration that looks better only because it was tuned against the exact data it is being scored on.
Both features were removed from the final method. The version reported above is the one that holds up out-of-fold.
How the Frontier Model Compares
Claude Opus 4.6 was tested on the identical inputs, with no method-specific advantage, in two configurations:
| Input | AP | MAP | Cost |
|---|---|---|---|
| Requirements + tests only | 39.66% | 56.00% | $9.62 |
| Requirements + tests + source code | 39.22% | 53.75% | $13.87 |
- Opus underperformed the 2024 TRIAD baseline by 5 to 8.9 MAP points.
- Adding the bridge source code made Opus worse, not better.
- TRIAD and TraceFuse exploit that source code structurally, through biterms and transitive links. A general-purpose LLM just sees it as extra unstructured context to wade through.
TraceFuse (48.60 / 63.30) > TRIAD (44.73 / 62.03) ≫ Opus 4.6 (39.66 / 56.00)
Limitations
- Single dataset — every number above is measured on LibEST. Generalization to other codebases is untested.
- Not peer-reviewed — this is an engineering combination of a published method and a model call, not an independently reviewed research result.
- Dependent on TRIAD — TraceFuse fuses on top of TRIAD's IR prior. It is not a standalone LLM technique.
- One frontier model — Opus 4.6 is the only frontier model tested here; other frontier models are untested.
Takeaway
A cheap model, asked one well-chosen question, cross-validated against overfitting, and fused with a structured prior, beats throwing a frontier model at the same, complete data.
The frontier model's underperformance here comes from a methodology mismatch, not from information scarcity — it had every artifact TraceFuse had. The practical recommendation: test the fusion before you upgrade the model.
Frequently Asked Questions
What is TraceFuse?
TraceFuse is the requirement-to-test traceability method behind KomAInu. It scores every requirement/test pair by fusing three signals: two TRIAD variants from ICSE 2024 used as lexical priors, and one backward pass with Haiku 4.5 that reads each test file and names the requirements it verifies. The signals are multiplied, not averaged, so a pair only scores high when all three agree.
How accurate is AI requirements-to-test traceability?
On the LibEST dataset (52 requirements, 21 C test files, 352 true links), TraceFuse scores 48.60% AP and 63.30% MAP. Under 5-fold cross-validation it scores 60.09% AP and 63.46% MAP on held-out requirements. This is an engineering benchmark on one dataset, not a peer-reviewed result.
How does TraceFuse compare to TRIAD and Claude Opus 4.6?
TraceFuse beats the ICSE 2024 TRIAD baseline by +3.87 AP (48.60% vs 44.73%) and beats Claude Opus 4.6 by +8.9 to +9.4 AP. It runs on Haiku, priced around $1 per million tokens against roughly $15 for Opus: close to a 15x cost difference for a better score.
Why does a frontier LLM score lower on traceability?
Claude Opus 4.6 received the same complete artifacts as every other method. Adding the source code made it worse (39.22% AP vs 39.66%), because a general-purpose model reads that code as extra unstructured context. TRIAD and TraceFuse use the same code structurally, through biterms and transitive links.
What are the limits of this benchmark?
Every number is measured on one dataset, LibEST, so generalization to other codebases is untested. The result is not peer-reviewed. TraceFuse depends on TRIAD's prior and is not a standalone LLM technique. Opus 4.6 is the only frontier model tested.
Reproduce the Benchmark
Base commit 524929e. Requirements: Java 11+ with Maven for TRIAD, Python 3.12 with the standard library only for the rest.
# 1. TRIAD reference scores (Maven + Java)
mvn compile && java -cp target/classes TriadRunner
# 2. Claude Opus 4.6 baseline
python openrouter_req_test_baseline.py
# 3. TraceFuse finalization (no extra API calls)
python tracefuse_v2_finalize.py
# 4. 5-fold cross-validation
python tracefuse_cv.pySource data referenced in this write-up:
- RESULTATS_BENCHMARK.md
- result/tracefuse_req_test/tracefuse_v2_dualprior_bwdonly/summary.json
- result/llm_req_test_file_baseline/bedrock/.../summary.json
- result/triad_req_test_export/JSD/summary.csv