Optimizing ADACL for Production

Welcome to Lesson 40 of the SNAP ADS Learning Hub! Module 7 has worked through ADACL’s technical machinery — architecture (Lesson 35), initialization (Lesson 36), features (Lessons 37–38), scoring (Lesson 39). This final lesson of the module covers what changes when you take that machinery out of a training notebook and put it into a live production environment monitoring real quantum hardware around the clock.

Production optimisation is a different skill from research-grade implementation. The model that gives the best validation score is not necessarily the model that survives a year of operational use. The pipeline that runs cleanly on a single workstation is not necessarily the pipeline that copes with three nines of uptime requirements, mid-life retraining, and operator handoffs. This lesson covers the gap.

A useful analogy: a research-grade ADACL prototype is the airplane that flies for the first time. The production system is the airplane after a thousand operational hours, where the seals have aged, the avionics have been retrofitted twice, and three different maintenance crews have rotated through. Many of the most important engineering decisions only become visible in that later state. This lesson is the tour of those decisions.

What “Production” Actually Means for ADACL

Concrete production requirements that shape every optimisation:

• Continuous inference at the rate the system produces features — typically one score per qubit per minute, 24/7.
• End-to-end latency under 30 seconds from feature computation to score publication.
• Calibration freshness maintained automatically; no human in the loop for routine drift.
• Operational footprint that fits within the existing monitoring infrastructure’s compute budget (typically a single mid-range GPU or a small CPU cluster).
• Alert quality that doesn’t degrade as the underlying system drifts.
• Auditability — every alert must be reproducible from logged inputs.

These six dimensions drive most of the optimisation effort. Let’s walk them in turn.

Model Optimisation: Quantisation and Pruning

ADACL’s research model is float32 with ~250k parameters (Lesson 35). In production, two transformations:

Post-training quantisation to int8

The model is converted to int8 weights with a calibration step on a representative held-out set. Accuracy loss is typically below 0.5 % calibration error on the operator-validated set — well below the model’s headroom against natural fluctuation. The win: a 4× memory reduction and ~2-3× inference speedup on CPU-only deployments, which matter when the production budget doesn’t include a dedicated GPU.

Structured pruning

A small fraction of the convolutional channels (~10 %) consistently shows near-zero contribution to the output across the validation set. Pruning these channels recovers another 10-15 % of inference time. The pruned model is re-evaluated against the full validation suite before shipping.

The combined int8 + pruned model is what ships. It is not what’s used during retraining — retraining always uses the float32 baseline and re-quantises at the end, because quantisation noise interferes with the gradient signal during training.

Feature-Compute Scheduling

The 394 features (Lesson 37) span a wide range of compute costs. Cheap moment-based features take microseconds; randomised-benchmarking features take seconds. Computing all of them every minute is wasteful. ADACL’s production scheduler runs them on three tiers:

• Tier 1 — always on: the ~80 cheapest features (moments, basic quantiles, single-shot readout contrast). Computed every minute.
• Tier 2 — periodic: the ~200 medium-cost features (autocorrelations, frequency-domain transforms, density-matrix-derived quantities). Computed every 5 minutes; the value carries forward to the in-between minute slots.
• Tier 3 — triggered: the ~100 expensive features (full state tomography, randomised benchmarking, gate-tomography sequences). Computed every 30 minutes baseline, or triggered immediately when tier-1 features cross a coarse threshold.

The triggered-recompute pattern is the operational equivalent of “if the cheap sensor says something interesting, take the expensive measurement.” It keeps the compute budget low under nominal operation while still surfacing the high-fidelity features when they matter.

Calibration Freshness

A model whose scores were calibrated against last quarter’s operator judgments will gradually drift out of alignment with this quarter’s operators. ADACL’s production calibration is auto-refreshing:

• Every operator action (confirm/dismiss/escalate on an alert) is captured with full input context.
• A rolling window of recent operator decisions (typically last 90 days) is held as the “current calibration set.”
• Once a week, the post-hoc calibration transformation is re-fit against the rolling set and shipped without retraining the underlying CNN.
• If the calibration transformation has drifted more than a threshold from the previous one, an alert fires to the ML on-call: probable distribution shift, may warrant a full retrain.

This separation — fast-cycle calibration refresh, slow-cycle model retraining — keeps the system responsive to operational drift without forcing constant retrains.

Operational Footprint

ADACL in production fits within a single small node:

• Memory — ~50 MB for the quantised model + ~500 MB for the feature-cache rolling buffer.
• CPU — one core dedicated to inference; 4-8 cores for feature computation depending on the system’s complexity.
• GPU — optional. With a GPU, inference batches across multiple qubits run in parallel. Without, qubits are processed serially with a typical 100ms inference time each.
• Network — minimal. Features stream in from the quantum-hardware control plane; scores stream out to the alerting layer. Both are sub-kilobit/s.
• Disk — ~10 GB/day of logged feature snapshots for reproducibility; rotated weekly to a cheaper archive tier.

The footprint was deliberate. ADACL was sized to fit alongside existing quantum-hardware control software, not to require a separate ML infrastructure stack.

Alert Quality Maintenance

A production anomaly detector that ships with 80 % precision on day 1 and 40 % precision after six months has effectively failed. ADACL has three safeguards against quality drift:

1. Drift dashboards

Real-time tracking of three quantities: model score distribution, calibration error against the rolling operator set, and feature distribution against the training baseline. Each gets a dedicated dashboard tile; sustained departure from baseline fires an ML on-call alert.

2. Shadow scoring

A second copy of the latest trained-but-not-shipped model runs in shadow mode on every input. Its scores aren’t surfaced to operators, but they’re compared to the production model’s scores. Sustained divergence is a signal that the production model is degrading or the shadow has improved enough to warrant promotion.

3. Held-out gold set

A small set (~200 examples) of “definitely real anomaly” and “definitely normal” examples — carefully curated and never used for training or calibration. The production model is scored against this set weekly. If accuracy on the gold set drops, the model is rolled back.

These three measures catch different drift modes: drift dashboards catch general shift; shadow scoring catches model-vs-data mismatch; the gold set catches catastrophic regressions.

Auditability and Reproducibility

Every alert ADACL fires must be reproducible. The runbook for “tell me why ADACL alerted at 14:32 UTC last Tuesday” is:

1. Pull the feature snapshot for the alert timestamp from archive.
2. Pull the model version and calibration version active at that time.
3. Re-run the inference. The score must match within the int8 quantisation tolerance.
4. Pull the explanation report (top contributing features) from archive — also stored at alert time.
5. If any of the above is missing or mismatched, the alert is treated as un-auditable and an incident is filed.

This level of bookkeeping is non-negotiable in regulated environments (financial-grade quantum-hardware deployments, medical applications) and useful everywhere else. The storage cost is modest; the trust dividend is large.

On-Call Runbook Highlights

For ML on-call engineers operating ADACL, the most common runbook entries:

• Score distribution shift > 2σ on the dashboard → check feature distribution dashboard; if also shifted, distribution drift; trigger retrain. If feature distribution is fine, calibration drift; trigger weekly recalibration ahead of schedule.
• Gold-set accuracy drop > 5 % → immediate rollback to previous model version. Investigate root cause on the rolled-back model; do not roll forward until cause is found.
• Inference latency > 30s P99 → check feature scheduler; tier-3 features may be running on the wrong schedule. Check GPU saturation if applicable.
• Operator-flagged false-positive surge → pull the recent operator feedback. If consistent across multiple operators, the calibration is mis-tuned; trigger recalibration. If isolated, may be operator alarm fatigue.
• Operator-flagged false-negative (missed real anomaly) → highest-severity ticket. The missed example goes into the labelled positive corpus immediately and triggers an out-of-cycle retrain.

These runbook entries are the difference between an ML system that operations teams trust and one they don’t.

When to Retrain vs Recalibrate

A subtle judgment call that comes up often:

• Recalibrate when the model’s scores have drifted from operator judgment but the underlying feature distributions are stable. Cheap, frequent (weekly).
• Retrain when the feature distributions themselves have shifted, or when accuracy on the gold set has degraded. Expensive, infrequent (monthly or as needed).

Most drift is in the calibration layer. Most catastrophic regressions are in the model. The two are addressed differently; conflating them produces unnecessary churn.

Key Takeaways

• Understanding the fundamental concepts: Production ADACL is a quantised + pruned + tiered-scheduled + auto-recalibrated version of the research artifact. Six dimensions of optimisation — latency, footprint, calibration freshness, alert quality, auditability, operator workflow — each drive their own engineering decisions.
• Practical applications in quantum computing: The full pipeline fits on a small node, runs in ~30 seconds end-to-end, and maintains alert quality through drift dashboards, shadow scoring, and a held-out gold set. Operator feedback loops keep calibration current without forcing constant model retrains.
• Connection to the broader SNAP ADS framework: Module 7 has built the technical machinery; this lesson is the bridge to Module 8 (performance & validation in production) and Module 9 (deployment across domains). The production-optimisation patterns here travel beyond quantum hardware to any high-stakes drift-monitoring application.

What’s Next?

Module 8 opens with ADACL’s performance and validation methodology — how we know the framework works, what metrics matter, and how validation is done in a regime where ground truth is scarce. Lesson 41 walks through the evaluation suite that gates every model release.

---

Ready to continue? Use the navigation buttons below to move to the next lesson or return to the module overview.