Consistency & Reproducibility

Welcome to Lesson 44 of the SNAP ADS Learning Hub! Module 8 has so far made the case that ADACL delivers measurable gains over binary and traditional baselines. This final lesson of the module asks the harder question: are those gains durable? A model that scores 0.92 calibration today and 0.65 next month is not delivering the gains it advertised; it’s delivering a snapshot. Production trust comes from outputs that are consistent across reruns, reproducible across versions, and auditable across time.

This lesson is the MLOps half of ADACL’s evaluation story. The framework’s accuracy gains (Lesson 42), evaluation methodology (Lesson 41), and method comparisons (Lesson 43) all assume the underlying numbers are stable. Stability isn’t automatic — it has to be engineered. Here is how ADACL engineers it.

A useful framing: a high-precision watch is impressive on day one of being calibrated. The thing that makes it a useful instrument is keeping that precision across temperature changes, mechanical shocks, and a year of operation. ADACL’s consistency and reproducibility discipline is the equivalent of a watch’s compensation mechanisms — the ungreased pieces that determine whether the impressive-on-paper precision actually survives contact with reality.

Three Faces of Reproducibility

When we say ADACL is “reproducible,” we mean three distinct properties that are independently engineered:

1. Run-to-run determinism. The same inputs, the same model version, the same calibration → the same score, bit-identical.
2. Version-to-version traceability. When a model is retrained or recalibrated, why the score changed for a given input is recoverable from logged data.
3. Long-horizon stability. The calibration error metric stays roughly constant week-over-week under nominal operation; degradation triggers an alert before it becomes visible in operator workflow.

Each property is gated by a different engineering practice. Let’s walk them.

Property 1: Run-to-Run Determinism

A model that produces score 0.73 for an input on one run and 0.74 on the next is not auditable — when an operator asks “why did this alert fire?”, the answer cannot be “it depends on the GPU’s mood today.”

ADACL’s run-to-run determinism is engineered through:

Deterministic inference

• Random seeds are pinned in every stochastic operation. The CNN forward pass is deterministic by construction; what isn’t deterministic by default (e.g., some CUDA reduction operations) is replaced with deterministic alternatives.
• Float precision is held constant; no mixed-precision auto-cast at inference time.
• The post-hoc calibration transformation is a pure mathematical function (isotonic regression or Platt scaling) with no internal randomness.

Feature-pipeline determinism

The 394-feature pipeline (Lesson 37) is deterministic given the same input window. The harder part is making sure the input window itself is deterministic — sliding windows have boundary conditions, time synchronisation depends on system clocks, sensor reads can have microsecond-level jitter. ADACL’s feature pipeline pins window boundaries to absolute timestamps, not to wall-clock-at-compute, so the same archival input produces the same features regardless of when the replay happens.

Quantisation reproducibility

The int8 quantised production model is bit-identical to its calibration baseline. Quantisation is computed offline once per model version and shipped as a frozen artifact; production never re-quantises on the fly.

The combined effect: running the same input through the same model version produces the same score, every time. Auditors can verify this by replaying historical alerts and confirming the original scores.

Property 2: Version-to-Version Traceability

Models retrain. Calibrations refresh. The question that needs an answer when a model changes is: “this input now scores 0.62 where it previously scored 0.78 — why?”

ADACL’s traceability is built through:

Model versioning with full provenance

Every model release carries a manifest:

• Training data digest (hash of the labelled set + augmentation seed).
• Code version (git SHA of the training pipeline at training time).
• Hyperparameter snapshot.
• Validation report (Lesson 41) snapshot.
• Calibration baseline used.

Any of these changing produces a new model version. Two runs with identical manifests are guaranteed to produce identical weights.

Calibration versioning

Calibration is versioned independently of the model. Calibration history is a time-series of post-hoc transformations, each tied to the operator-feedback window it was fit against. An alert’s full context is (model_version, calibration_version, input_features, timestamp) — the four-tuple that lets the alert be reproduced exactly.

Score-diff explanations

For inputs that score differently across model versions, ADACL’s reporting tools surface a score-diff explanation: which features contributed to the difference, decomposed by family. This is the operator-facing artifact when someone asks “why did the new model rate this differently?” — it points at concrete features rather than at “the new model is different.”

Property 3: Long-Horizon Stability

Calibration drifts. Feature distributions shift. Sensors degrade. Without engineered counter-measures, the framework’s metrics would erode invisibly until the operational quality collapsed. ADACL’s stability discipline:

Drift dashboards

Three real-time tracking surfaces (covered in Lesson 40): score-distribution drift, calibration-error drift, feature-distribution drift. Each gets its own alert threshold and on-call escalation.

Held-out gold set

A 200-500-example curated set of operator-validated examples (Lesson 41) is scored weekly. If accuracy on the gold set degrades, the model is rolled back to the previous version. The gold set is never used for training or calibration — it’s a perpetual external reference.

Calibration freshness loop

Calibration is auto-refreshed weekly against rolling operator feedback (Lesson 40). The refresh is additive — it adjusts the post-hoc transformation, never the underlying model — so refresh cycles don’t introduce model-level instability.

Shadow-mode trial of new versions

New model versions are run in shadow mode for at least one week before being promoted to production. During shadow mode, the new model scores every input alongside the production model; divergence patterns are reviewed before the new model is allowed to fire actual alerts.

Rollback discipline

The last three model versions are always retained in deployable form. Any operational regression triggers immediate rollback to the previous version while investigation proceeds. Rollback is automated; investigation is not.

What Each Property Catches

The three properties guard against different failure modes:

• Determinism fails when an operator can’t reproduce an alert from logged inputs. Indicates pipeline non-determinism (random seeds, mixed precision, clock-dependent feature boundaries).
• Traceability fails when the team can’t explain why a model version produces different scores from the previous one. Indicates missing provenance metadata, especially around training-data digests and calibration baselines.
• Stability fails when calibration error trends upward week-over-week with no obvious cause. Indicates either real distribution drift (which needs retraining) or model degradation (which needs rollback).

Mature ADACL deployments report on all three in their weekly operational review. Looking at any one alone misses the others.

Reproducibility Beyond the Numbers

The reproducibility discipline isn’t only about score values; it extends to two other operational artifacts:

Explanation reports

For every alert, the explanation report (top contributing features, augmentation-class attribution if relevant) is computed and stored at alert time. Re-deriving the explanation later from the same input must produce the same report — auditors check this on a sample of historical incidents each quarter.

Operator decisions

The operator decision on an alert (confirm/dismiss/escalate/mitigate) is captured with full input context. Re-presenting the same alert with the same context to the same operator should produce the same decision (modulo human variability). This is checked informally via “blind replay” exercises during quarterly operations reviews.

Both artifacts are part of the audit trail that makes ADACL trustworthy at the regulatory level (financial-grade quantum hardware, medical-device monitoring) where reproducibility isn’t a nice-to-have.

Common Pitfalls

Three failure modes seen in early ADACL deployments — flagged here for new operators to design around:

• Calibration version slipping. The model version is pinned in the audit trail, but the calibration version isn’t. New deployments must include (model_version, calibration_version) together; either one floating is a defect.
• Wall-clock-dependent features. A feature that uses now() somewhere in its computation produces different values on different replay runs. Audit-replay must use the archival timestamp, not the wall clock, throughout the pipeline.
• Float-precision drift. Different hardware accelerators (different GPU models, mixed-precision libraries, even kernel versions) can produce slightly different floats. The int8 quantised model defends against this; running float32 in production does not.

The discipline to avoid each of these comes from running through the audit-replay exercise once per quarter at minimum. The first time you do it, you find bugs. After you’ve fixed them, the audit is fast.

Key Takeaways

• Understanding the fundamental concepts: Reproducibility in ADACL is three properties — run-to-run determinism, version-to-version traceability, and long-horizon stability. Each is engineered through specific practices (pinned seeds, model + calibration versioning, drift dashboards, gold set, shadow trials, rollback discipline).
• Practical applications in quantum computing: Audit-replay of historical alerts is the gold-standard test for reproducibility. Mature deployments run the audit quarterly; it’s the discipline that makes the framework’s accuracy gains durable rather than transient.
• Connection to the broader SNAP ADS framework: Module 8 closes here. The framework’s accuracy claims (Lesson 42), evaluation methodology (Lesson 41), and method comparisons (Lesson 43) rest on the reproducibility infrastructure described in this lesson. Without it, the gains are real but fragile.

What’s Next?

Lesson 45 covers ethical AI and responsible deployment — the considerations that go beyond accuracy and reproducibility into the question of should this be deployed here, in this context, with these controls? It is the final lesson of Module 8 and the bridge to Module 9’s deployment-focused content.

---

Ready to continue? Use the navigation buttons below to move to the next lesson or return to the module overview.