Evaluating ADACL Performance: Metrics & Analysis

Welcome to Lesson 41 of the SNAP ADS Learning Hub! Module 7 covered the technical machinery of ADACL — architecture, initialization, features, scoring, production optimisation. Module 8 turns to the question: how do we know it works? That sounds simple but it isn’t, because ADACL operates in a regime where the conventional answers (accuracy, F1 score, AUROC) are either uncomputable, uninformative, or actively misleading.

This lesson covers the evaluation methodology that gates every ADACL model release. The methodology is opinionated — it deliberately ignores some metrics that machine-learning textbooks treat as standard and emphasises others that production operators care about. The reasoning behind each choice is worth understanding before you ship anything based on the framework.

A useful framing: evaluating a smoke detector is not the same problem as evaluating a thermometer. A smoke detector’s accuracy is determined by how many real fires it catches and how many burnt-toast incidents it ignores. A thermometer’s accuracy is determined by how close its reading is to the true temperature at every moment. ADACL is more thermometer than smoke detector — its value comes from the calibrated continuous score, not from a binary hit/miss decision. Evaluation has to follow that grain.

What Makes ADACL Evaluation Hard

Three structural challenges complicate evaluation:

1. Ground truth is scarce and biased

Real labelled anomalies on quantum hardware are precious. The handful we have are by definition the events that operators noticed and investigated — the easy cases. The anomalies we missed, or that resolved themselves before anyone looked, are not in the dataset. So our “true positive” set is small and skewed toward obvious cases. Evaluating only against this set systematically over-estimates accuracy.

2. The output is continuous, but most metrics assume binary

Precision, recall, F1, accuracy — all assume a binary decision. To use them with a continuous score, you have to pick a threshold first. Pick a different threshold and you get a different number. None of the conventional metrics give you a single answer; they give you a curve, and reducing the curve to a number means choosing one point on it.

3. The score’s value matters, not just its rank

A binary classifier needs only to put anomalies above non-anomalies in score order. ADACL needs to put a 0.7-anomaly at 0.7, not at 0.95 or 0.4. Metrics like AUROC are insensitive to that — they only care about ranking. Calibration error is the metric that catches scale errors that AUROC misses.

The Five Metrics That Matter

ADACL’s evaluation suite reports five primary metrics. Each captures a different aspect of model behaviour, and they are not redundant.

1. Calibration error

The mean absolute difference between the model’s predicted score and the operator-validated target for examples in the validation set:

calibration_error = mean(|score_predicted − score_operator_target|)

This is the single most important metric. A model with low calibration error puts a 0.7 example at 0.7. A model with high calibration error may still rank examples correctly (and thus look good on AUROC) but its scores cannot be used for fine-grained policy decisions. Calibration error directly measures the thing ADACL exists to deliver.

The target value for “calibration error” depends on how granular the policy layer needs to be. For a system with policy thresholds every 0.1, calibration error should be under 0.05. For a system with policy thresholds every 0.05, calibration error needs to be under 0.025.

2. Precision @ k

For the top k highest-scoring alerts in a held-out period (typically a week of operation), what fraction were operator-confirmed as real anomalies?

precision_at_k = (operator-confirmed alerts in top k) / k

This is the metric operators care about most. Tier-1 ops engineers triage a fixed budget of alerts per shift; they want the model’s top alerts to be the most informative. Precision@k captures this directly. Target: above 0.7 for k=10, above 0.5 for k=50.

3. Time-to-detect

For confirmed real anomalies, the elapsed time from the underlying anomaly’s onset (as best can be estimated post-hoc) to ADACL’s first alert. Lower is better; faster detection means more lead time for mitigation.

time_to_detect = first_alert_time − estimated_onset_time

This is the metric that distinguishes ADACL from naive anomaly detectors. A detector that catches anomalies only after they’ve reached catastrophic magnitudes is useless; ADACL’s continuous score and physics-informed features should catch drift earlier.

4. False-positive rate per shift

For a typical 8-hour operator shift, how many ADACL alerts are dismissed as false positives by operators? Lower is better, with a floor — zero is suspicious (probably means the model is too conservative and is missing real events).

fp_per_shift = (operator-dismissed alerts in 8h) / 1

Target: below 3 per shift on a typical workload. Anything higher creates alarm fatigue (Lesson 30).

5. Calibration drift over time

How much does the calibration error change week-over-week? Stable models have flat calibration-drift curves; degrading models have rising ones.

calibration_drift = week_T_calibration_error − week_T-1_calibration_error

This is the early-warning indicator for “the model needs retraining.” Target: drift below 0.005 per week under nominal conditions.

Metrics ADACL Evaluation Deliberately De-emphasises

For completeness, three commonly-reported metrics that ADACL evaluation reports but does not gate on:

• Raw accuracy / F1 score. Both require binarising the output, which discards the calibration signal that’s the framework’s main contribution. Useful as a coarse sanity check; useless as a primary metric.
• AUROC. Ranking-only, insensitive to scale. A model with perfect AUROC and terrible calibration is not useful in production. Tracked for completeness but not gated on.
• Loss on the validation set. The loss combines MSE and hinge terms in proportions chosen for training, not for human evaluation. A model can have lower validation loss but worse calibration error if the hinge term over-emphasises the tail. Training metric, not evaluation metric.

Handling the Ground-Truth Gap

The hardest part of ADACL evaluation is that the labelled-positive set is small and biased. ADACL’s methodology has three patches:

1. Augmented validation set

A held-out slice of the same physics-informed augmented examples used in training serves as a synthetic validation set. Augmented examples have known continuous deviation labels, so calibration error can be computed exactly. The catch: the augmentation distribution is what it is — a model that’s well-calibrated on augmented examples is well-calibrated on real examples only to the extent the augmentation distribution matches reality. Augmented metrics are necessary but not sufficient.

2. Operator gold set

200-500 carefully-curated examples that operators have triaged into “definitely real anomaly,” “definitely normal,” and “marginal.” This set is small but high-quality; it captures the operator-aligned definition of “anomalous” that the production model needs to match. Calibration error and precision@k are computed against this set.

3. Counterfactual replay

For past production alerts where operators eventually identified the underlying cause, the methodology replays the historical feature stream through the new model and checks whether it would have detected the same event with the same lead time. This catches regressions in time-to-detect even when no formal labelled positive exists for the event.

The three together approximate a real validation set well enough to gate model releases.

Evaluation Cadence

Production ADACL evaluation runs at three cadences:

• Per-PR (every change to model code or training data): full augmented validation pass + gold-set calibration error + counterfactual replay on the last quarter of incidents. ~30 minutes. Gates merge to main.
• Per-release (every model retrain): all of the above + extended counterfactual replay across the last year + ML-on-call review of the explanation reports for the top 20 historical incidents.
• Weekly (operational): calibration error + precision@k + false-positive rate on the rolling operator-feedback window. Tracked in the drift dashboards (Lesson 40).

Per-PR is automated. Per-release requires a human sign-off. Weekly is automated but its outputs are reviewed by the ML on-call.

Common Failure Modes Evaluation Catches

Three regressions evaluation routinely surfaces:

• Calibration regression with stable ranking. AUROC unchanged, calibration error up. Usually a training-loss-weighting bug. Caught by the calibration-error gate.
• Time-to-detect regression on slow drifts. Precision@k unchanged on the obvious-anomaly subset, but counterfactual replay shows the new model detects slow drifts later. Caught by the per-release counterfactual replay.
• False-positive surge on a specific operating mode. Aggregated FP rate looks fine, but operator feedback stratified by workload shows one mode produces 80 % of the false positives. Caught by stratified weekly review.

Key Takeaways

• Understanding the fundamental concepts: ADACL evaluation gates on five metrics — calibration error, precision@k, time-to-detect, false-positive rate, and calibration drift over time. The traditional binary metrics (accuracy, F1, AUROC) are tracked but de-emphasised because they discard the calibration signal that’s the framework’s main contribution.
• Practical applications in quantum computing: Ground truth is scarce; the evaluation methodology patches the gap with augmented validation, an operator-curated gold set, and counterfactual replay on historical incidents. The three together are sufficient for production gating.
• Connection to the broader SNAP ADS framework: The evaluation methodology is what allows the rest of the framework — augmentation, continuous labelling, calibration-refresh cycles — to be shipped safely. Without disciplined evaluation, each of those moving parts can drift independently and degrade the system invisibly.

What’s Next?

Lesson 42 looks at one specific claim the evaluation methodology is designed to verify: that ADACL “breaks the binary confidence ceiling.” We will see what that phrase concretely means, how it’s measured, and the empirical evidence that the framework delivers the gain its name promises.

---

Ready to continue? Use the navigation buttons below to move to the next lesson or return to the module overview.