ADACL Neural Network Architecture

Welcome to Lesson 35 of the SNAP ADS Learning Hub! Module 6 set up what ADACL does and why it makes each architectural commitment. Module 7 zooms into the technical machinery that actually makes it work. We open with the neural-network architecture itself — the specific layer sequence, the way the 394-feature input is shaped, and the design choices that let a relatively small model deliver calibrated continuous anomaly scores in the small-label regime quantum hardware lives in.

The architecture is not the heroic ingredient in ADACL. The augmentation pipeline (Lesson 32), the continuous deviation labels (Lesson 33), and the physics-informed features (Lesson 37) carry most of the weight. But the architecture has to be matched to those ingredients, and a wrong choice here can defeat the rest of the framework. This lesson walks through why ADACL settled on a CNN regressor, what its specific layer sequence looks like, and where the regularisation lives.

A good analogy: the architecture is the chassis of a race car. The chassis itself doesn’t win races — the engine, the tyres, and the driver do — but the wrong chassis will undo all three. ADACL’s CNN is a chassis tuned for the specific load profile of physics-informed continuous regression on a 394-feature input.

Input Tensor Shape

Before the architecture, the input. The 394 engineered features (Lesson 37 unpacks the full list) are organised into a tensor with deliberate structure:

Input shape: (batch, time_windows, features_per_window, channels)
            = (B, T, F, C)

• B — minibatch size, typically 32 or 64.
• T — number of rolling time windows in the example. Default is 16, covering ~16 minutes of recent system history at the default windowing.
• F — features per window. The 394 features grouped by family, padded to a power of two (typically 512) for tensor-core friendliness.
• C — channels, typically 1 (single observed device) but extensible to multi-device deployments.

The shape matters because it determines the kernel layout in the convolutional layers. The (T, F) plane has both temporal locality (adjacent time windows are correlated) AND family locality (adjacent feature positions belong to the same family — statistical, quantum-specific, frequency-domain, or physics-constraint). A 2D CNN kernel exploits both.

The Layer Sequence

ADACL’s architecture is intentionally shallow by 2025 standards — depth is not the bottleneck here; calibration is. The default configuration:

Block 1 — Temporal-feature convolution

Conv2D(channels=32, kernel=(3,5), stride=1, padding="same")
LayerNorm
ReLU
Conv2D(channels=32, kernel=(3,5), stride=1, padding="same")
LayerNorm
ReLU
MaxPool2D(pool=(1,2))

The first convolutional block reads short temporal windows (3 rows, ~3 minutes) across local feature neighbourhoods (5 columns). LayerNorm rather than BatchNorm because batch sizes in the small-label regime are too noisy for BatchNorm’s running statistics to stabilise. MaxPool on the feature axis halves the dimensionality before block 2.

Block 2 — Cross-family convolution

Conv2D(channels=64, kernel=(3,5), stride=1, padding="same")
LayerNorm
ReLU
Conv2D(channels=64, kernel=(3,5), stride=1, padding="same")
LayerNorm
ReLU
MaxPool2D(pool=(2,2))

Block 2 doubles the channel count and pools on both axes. After two pools the spatial dimensions are (T/2, F/4) — i.e. 8 × 128 by default. The expanded channel count lets the network represent richer combinations of local features detected in block 1.

Block 3 — Global context

Conv2D(channels=128, kernel=(3,3), stride=1, padding="same")
LayerNorm
ReLU
GlobalAveragePooling2D
Dropout(p=0.3)

Block 3 deepens the receptive field with a smaller 3×3 kernel and then collapses the spatial dimensions entirely via global average pooling. After this the tensor shape is (B, 128) — a 128-dim summary of the example. Dropout at 0.3 provides the regularisation that, combined with augmentation and weight decay, keeps the model from overfitting the small real-positive set.

Head — Regression output

Dense(64) + ReLU
Dropout(p=0.2)
Dense(1) + Sigmoid

A small two-layer regression head emits the final score in [0, 1]. The sigmoid is the only nonlinearity that’s explicitly capped; the rest of the network uses ReLU (in the convolutions) which the next lesson on Xavier/He initialization (Lesson 36) addresses.

Total parameter count: approximately 250k. That’s small by modern standards — for comparison, a ResNet-50 has 25M parameters, 100× more. ADACL’s small footprint is intentional; in the small-label regime, smaller is better.

Why This Shape, Not Deeper / Bigger / Transformer

Three alternatives that were tested and rejected during ADACL’s development:

Deeper CNNs

ResNet- and DenseNet-style architectures with 50+ layers were evaluated. With 394 features and ~10k effective examples (real + augmented per epoch), they overfit badly even with aggressive regularisation. The shallow three-block design hits the sweet spot — enough capacity to model the feature interactions, not so much that it memorises the augmented examples.

Fully-connected networks

A 4-layer MLP on the flattened 394-feature vector was evaluated. It loses the temporal-locality prior that the CNN exploits and ends up needing 10× more parameters to match the CNN’s accuracy. In the labelled-data regime, this is fatal.

Transformer encoders

Transformer architectures on the (T, F) input were evaluated. They modestly outperformed the CNN on validation accuracy when given 10× more data — i.e. they would be the right choice in a label-rich regime. In the small-label quantum regime, the CNN wins on sample efficiency and operational simplicity.

The right architecture depends on the data regime. ADACL’s regime makes the CNN the winner.

Activation Choices

A small but important decision: ADACL uses ReLU in the convolutional and dense layers but sigmoid only at the output. Two reasons sigmoid is chosen for the final layer rather than, say, a clipped linear:

• Bounded output. The continuous deviation labels live in [0, 1]. A sigmoid output is naturally bounded there without requiring a post-hoc clip.
• Smoothness near the boundaries. A sigmoid is smooth at 0 and 1, so gradient updates don’t go to zero near the boundaries the way they would for a clipped linear. This matters for the hinge term in the loss (Lesson 33), which actively pushes high-anomaly predictions near 1.

ReLU in the hidden layers is the default choice; combined with He initialization (Lesson 36) it produces stable training.

Regularisation Stack

ADACL’s regularisation has four layers, each addressing a different overfitting mode:

1. Data augmentation (Lesson 32). The largest source of regularisation by far — 100×-1000× synthetic examples per real example.
2. LayerNorm + Dropout. LayerNorm stabilises activations; Dropout (0.3 after block 3, 0.2 in the head) randomly drops units to prevent co-adaptation.
3. Weight decay at 1e-4 in AdamW. A mild L2 prior on weights, sufficient given the augmentation does the heavy lifting.
4. Early stopping. Validation loss is monitored each epoch; training halts after 5 consecutive epochs without improvement. Typical final training length is 20-30 epochs.

The four layers work together, not in isolation. Removing any one produces measurable overfitting; the full stack delivers stable convergence.

What Operators Can Tune

For deployment engineers extending ADACL to new operating contexts, the levers most likely to need adjustment:

• Window count T. Default 16; increase to 32 if drift signatures span longer time scales, decrease to 8 for faster early warning.
• Augmentation multiplier. Default 100 augmented examples per real positive; increase for very-small-label deployments.
• Dropout rate in block 3. Default 0.3; raise to 0.5 if validation loss diverges from training loss early.
• Score head depth. Default 2 layers; raise to 3 if the system has strong feature interactions that the convolutions don’t fully capture.

Everything else (kernel sizes, channel counts, activation choices) is well-tuned for the default deployment and rarely needs touching.

Key Takeaways

• Understanding the fundamental concepts: ADACL uses a three-block CNN regressor with ~250k parameters, deliberately shallow to fit the small-label regime. The input is a 4D tensor exploiting both temporal locality (across windows) and family locality (across feature groups).
• Practical applications in quantum computing: Compared to deeper CNNs, MLPs, or transformers, this architecture wins on sample efficiency, calibration stability, and operational simplicity — the three qualities that matter most for production quantum-hardware monitoring.
• Connection to the broader SNAP ADS framework: The architecture is the chassis, not the engine. Augmentation, continuous labels, and physics-informed features are what carry the framework; the architecture’s job is to fit those ingredients without getting in their way.

What’s Next?

Lesson 36 covers the initialization of the architecture we just described — Xavier and He initialization, why naive small-random weights fail in this network, and the specific weight-initialisation choices that keep the regressor training stably from epoch 1.

---

Ready to continue? Use the navigation buttons below to move to the next lesson or return to the module overview.