Calibrating Semantic Renames Instead of Trusting Confidence

“The model is 95% confident” sounds like a reason to apply a rename automatically.

It is not.

Model confidence is produced by the model’s own internal scoring behavior. It does not tell us that 95 out of 100 similarly scored names are semantically useful, safe for that binding class, or free of protected-surface violations.

Automation needs calibration against outcomes.

Exact recovery and useful naming are different metrics

Sometimes the original identifier survives in a source map or debug artifact. In those cases, exact-name recovery can be measured.

Often it does not survive. Several names may be equally appropriate for the recovered role. A variable could reasonably become activeRoute, currentRoute, or selectedRoute.

The primary question is then semantic usefulness: does the proposed name help an engineer understand the binding’s role without asserting more than the evidence supports?

A calibration corpus should distinguish:

• exact historical match where recoverable;
• accepted semantic synonym;
• useful but overly specific name;
• wrong role;
• unsafe contract mutation;
• appropriate abstention;
• unnecessary abstention.

Evidence sources can be correlated

Suppose a runtime observation and a generated report both derive from the same string label. Counting them as two independent votes creates false confidence.

Evidence fusion needs correlation roots. Signals that share one origin contribute as one family, while genuinely independent sources can strengthen the result.

Useful evidence may include source maps, runtime object roles, call behavior, property access, consumer context, framework metadata, scene relationships, and neighboring recovered symbols. Their value depends on independence and surface class, not just count.

Risk differs by binding class

A wrong private local name is annoying. A wrong exported name can break consumers. A mutation that touches a protected route or storage key can break behavior.

Calibration should therefore report separate risk for local and externally visible bindings. A single global threshold hides the exact place where mistakes are expensive.

The most important metrics include:

• semantic-usefulness accuracy;
• false-confident wrong-rename rate;
• false-abstention rate;
• protected-surface violation rate;
• calibration error;
• coverage at each accepted risk level.

Any protected-surface violation should block automatic rollout, even if aggregate accuracy looks strong.

Rollout is a sequence, not a switch

A measured rename system can advance through explicit stages:

1. score in shadow mode while applying nothing;
2. collect adjudicated outcomes;
3. show advice and queue every change for review;
4. permit bounded local-binding automation;
5. pilot externally visible names under separate gates;
6. maintain rolling safety evidence.

If the evidence window becomes stale, too small, or unsafe, the system should fall back to review. A configured number is not a substitute for a current calibration artifact.

Abstention is useful output

A rename engine that changes every short identifier will look productive in a demo. It will also invent meaning.

A useful abstention explains why it stopped:

• insufficient independent evidence;
• close competing candidates;
• protected or exported surface;
• vendor-owned region;
• stale plan;
• unsupported syntax;
• calibration unavailable for this class.

That explanation directs the next evidence-gathering step and preserves trust.

KodeBack uses AI where semantic judgment is genuinely needed, but the system—not the model—owns mutation authority. Calibration is how a confidence score becomes an auditable risk decision, and how “I think this name is right” becomes either a bounded action or an honest refusal.