TraceFuse: I Beat a 2024 ICSE Method and Claude Opus 4.6 With One Cheap Model

TL;DR

TraceFuse is the requirement-to-test traceability method behind KomAInu. It fuses three signals into one score:

• Two TRIAD variants, used as lexical priors.
• A single backward pass with Haiku 4.5 — 21 calls total, one per test file.
• Geometric fusion, a "soft AND" that only scores a pair highly when every signal agrees.

TraceFuse beats the ICSE 2024 TRIAD baseline by +3.87 AP and beats Claude Opus 4.6 by +8.9 to +9.4 AP — while running on Haiku, priced around $1 per million tokens against roughly $15 per million for Opus. That is close to a 15x cost difference for a better score.

This is my own engineering benchmark on a single, established research dataset, not a peer-reviewed result.

TraceFuse is not a research exercise — it runs in production as the traceability engine behind KomAInu, mapping SRS requirements to the tests that verify them.

The Research Question

Frontier models are good at reading a requirement and a test and judging whether one verifies the other. So the question worth answering honestly was: do I need the frontier model, or does a small model plus a smarter method win?

TRIAD, published at ICSE 2024, is the incumbent classical baseline for this problem. It scores 44.73% AP and 62.03% MAP using weighted word statistics and source-code bridges — biterms and transitive links through the code that sits between a requirement and the test that exercises it.

To keep the comparison honest, every method in this benchmark receives the same artifacts, complete, with zero truncation. No method gets an information advantage over another — the only variable is the method itself.

The LibEST Dataset

• 52 requirements
• 21 C test files
• 14 source code files, used as an intermediate bridge
• 352 oracle requirement-to-test links
• 1,092 possible requirement/test pairs in the search space

Two metrics are reported throughout: AP (Average Precision) over one global ranked list of all 1,092 pairs, and MAP (Mean Average Precision), which averages a per-requirement AP across all 52 requirements. AP rewards a method that ranks true links near the top overall; MAP rewards consistency across every individual requirement, not just the easy ones.

TraceFuse Architecture

Signal 1 — Dual TRIAD prior

Rather than picking one TRIAD configuration, TraceFuse combines TRIAD's best-AP variant (b_o_i) with its best-MAP variant (o_i) as a two-vote ensemble. This inherits TRIAD's structural understanding of the intermediate source-code artifacts without committing to a single tuning.

Signal 2 — Backward pass

• Single direction: test → requirements, not requirements → test.
• Haiku 4.5, run at medium reasoning effort.
• 21 calls total — one per test file, not one per requirement pair.
• Per-test normalization to correct for calibration drift between calls.

Signal 3 — Geometric fusion

score(req, test) =
  (0.15 + TRIAD_b_o_i)^1.0 ·
  (0.15 + TRIAD_o_i)^1.0 ·
  (0.15 + Haiku_backward)^0.5

Multiplying the signals together, instead of averaging them, creates a "soft AND": every signal needs a reasonable confidence for the product to be high. One loud signal can no longer override the other two — the three components have to agree.

Cross-validation

Fusion weights were tuned with 5-fold cross-validation: split the 52 requirements into 5 groups, tune the fusion weights on 4 groups, then measure AP/MAP on the held-out group. Repeat for every group and average the result.

Globally, TraceFuse scores 48.60% AP / 63.30% MAP. Averaged across the 5 held-out folds, it scores 60.09% AP / 63.46% MAP — consistent with the global numbers, which is the point of cross-validating in the first place.

What Didn't Survive Cross-Validation

Two ideas looked promising in early experiments and were cut from the final method. The first was extended-thinking mode with an 8,192-token reasoning budget. The second was a forward pass — requirements → tests, in addition to the backward pass.

On a full-grid search over the whole dataset, adding these produced a +1.7 AP gain. But under 5-fold cross-validation, tuning only on the held-out folds, that gain nearly disappeared: AP moved by roughly +0.05 and MAP dropped. That is the textbook signature of overfitting to a fully-visible evaluation set — a configuration that looks better only because it was tuned against the exact data it is being scored on.

Both features were removed from the final method. The version reported above is the one that holds up out-of-fold.

How the Frontier Model Compares

Claude Opus 4.6 was tested on the identical inputs, with no method-specific advantage, in two configurations:

• Opus underperformed the 2024 TRIAD baseline by 5 to 8.9 MAP points.
• Adding the bridge source code made Opus worse, not better.
• TRIAD and TraceFuse exploit that source code structurally, through biterms and transitive links. A general-purpose LLM just sees it as extra unstructured context to wade through.

TraceFuse (48.60 / 63.30) > TRIAD (44.73 / 62.03) ≫ Opus 4.6 (39.66 / 56.00)

Limitations

• Single dataset — every number above is measured on LibEST. Generalization to other codebases is untested.
• Not peer-reviewed — this is an engineering combination of a published method and a model call, not an independently reviewed research result.
• Dependent on TRIAD — TraceFuse fuses on top of TRIAD's IR prior. It is not a standalone LLM technique.
• One frontier model — Opus 4.6 is the only frontier model tested here; other frontier models are untested.

Takeaway

A cheap model, asked one well-chosen question, cross-validated against overfitting, and fused with a structured prior, beats throwing a frontier model at the same, complete data.

The frontier model's underperformance here comes from a methodology mismatch, not from information scarcity — it had every artifact TraceFuse had. The practical recommendation: test the fusion before you upgrade the model.

Reproduce the Benchmark

Base commit 524929e. Requirements: Java 11+ with Maven for TRIAD, Python 3.12 with the standard library only for the rest.

# 1. TRIAD reference scores (Maven + Java)
mvn compile && java -cp target/classes TriadRunner

# 2. Claude Opus 4.6 baseline
python openrouter_req_test_baseline.py

# 3. TraceFuse finalization (no extra API calls)
python tracefuse_v2_finalize.py

# 4. 5-fold cross-validation
python tracefuse_cv.py

Source data referenced in this write-up:

• RESULTATS_BENCHMARK.md
• result/tracefuse_req_test/tracefuse_v2_dualprior_bwdonly/summary.json
• result/llm_req_test_file_baseline/bedrock/.../summary.json
• result/triad_req_test_export/JSD/summary.csv