Every number in a model is a compromise.

Number formats: range versus precision

A checkpoint is a dictionary of named tensors — each one a dtype, a shape and a flat run of bytes (.safetensors is literally a JSON header of offsets plus one byte buffer; .pt is a zip of a pickle plus raw storages). Size on disk is parameters × bytes per element. Quantization changes the bytes-per-element term. Before changing it, you need to know what each format can and cannot hold.

Floating point in one sentence

A float is (−1)^sign × 1.mantissa × 2^{exponent − bias}. Exponent bits buy range (how big and how small). Mantissa bits buy precision (how finely spaced the values are near any given magnitude). Spacing is relative: between 1 and 2, FP16 has steps of 2⁻¹⁰; between 1024 and 2048 its steps are 1. That relative spacing is why floats handle wide dynamic ranges gracefully and integers don't.

Subnormals, saturation and special values

When the exponent field is all zeros, IEEE formats switch to subnormals: the implicit leading 1 becomes 0, giving a gradual slide to zero instead of a cliff. FP16 subnormals reach down to about 6e−8. At the other end, IEEE formats overflow to inf. The OCP FP8 E4M3 variant deliberately has no infinity — it reclaims those bit patterns for extra finite values and saturates at ±448, keeping a single NaN encoding. Kernels typically clamp before casting so a large activation becomes 448, not garbage.

Microscaling: how 4-bit becomes usable

FP4 alone covers 0.5 to 6. Real tensors span orders of magnitude. The fix is block scaling baked into the format:

Smaller blocks and a finer-grained scale (NVFP4's FP8 scale vs MX's power-of-two) cost a little metadata and buy accuracy. This is per-group quantization, chapter 3, pushed into silicon.

The quantization equation

Integers have no exponent. To store real numbers in INT8 you pick a scale s (the real-valued step between neighbouring integers) and optionally a zero-point z (the integer that represents real zero):

Symmetric vs asymmetric

Weights are near-symmetric around zero → symmetric, per-channel. Activations after ReLU/GELU are skewed → asymmetric is tempting, but many accelerators prefer symmetric activations too because the zero-point cross-term costs an extra correction. Symmetric INT8 usually uses −127…127, not −128, so the grid stays symmetric.

The two kinds of error

Every range choice trades rounding error against clipping error:

• Rounding error — inside the range, each value moves by at most s/2. Treating it as uniform noise, its variance is s²/12. Bigger range → bigger s → more rounding noise everywhere.
• Clipping error — values outside the range get slammed to the edge. Error grows with how far outside they were.

Absmax never clips but lets one outlier inflate s for everybody. Tighter ranges clip a few values to keep the grid fine for the many. Rule of thumb: each extra bit halves s and buys about 6 dB of signal-to-quantization-noise ratio (SQNR).

Things to try. Weights at 8 bits: absmax is fine. Turn on outliers: SQNR collapses under absmax, and percentile/MSE rescue it by clipping. GELU activations: switch to asymmetric and watch SQNR jump — symmetric wastes the negative half. Drop to 2 bits: every choice is bad, which is why 2-bit needs the heavy machinery in chapter 5.

What the hardware actually computes

For a linear layer y = W x with symmetric per-channel weights (scale s_w per output row) and a per-tensor activation scale s_x, zero-point z_x:

Then the output is requantized for the next layer: q_y = round( y / s_y ) + z_y. In practice s_ws_x/s_y is folded into one fixed-point multiplier plus a bit shift, and the bias is pre-quantized to INT32 at scale s_ws_x. That's why it matters that per-channel weight scales are "free": they factor out of the sum over k. A per-channel activation scale would sit inside the sum and break the integer GEMM — keep that in mind for SmoothQuant.

Two very different mechanisms behind the labels

Granularity: how many scales

One scale for a whole tensor is cheap and fragile. More scales track local ranges better at the cost of metadata and kernel complexity.

Per-tensor lets the loudest channel set the step for all eight; quiet channels round to almost nothing. Per-channel fixes the across-channel problem. Per-group also fixes within-channel spikes, which is why 4-bit schemes all go per-group.

Calibration: choosing ranges for activations

Weights are known offline — you can inspect every value. Activations only exist when data flows. Calibration runs a few hundred representative inputs, records each activation tensor's statistics, and picks a range.

Range-selection methods

How the KL/entropy calibrator works

Build a histogram of |activation| (TensorRT uses 2048 bins). For each candidate threshold i (starting around bin 128): take bins 0…i as reference distribution P, fold the clipped tail mass into the last bin, re-bin P into 128 levels (the INT8 positive half) to get Q, expand Q back, and compute KL(P‖Q). Choose the threshold with minimum KL. Intuition: preserve the shape of the distribution the next layer sees, not every extreme value.

Static vs dynamic

Static: activation scales frozen from calibration. Zero runtime cost; wrong if deployment data drifts. Dynamic: scale computed per tensor/token at runtime (e.g. per-token absmax). More accurate for LLM activations, costs a reduction pass per tensor.

What makes a good calibration set

• Representative, not large. 128–512 samples often suffice; covering the input modes matters more than count.
• Deduplicated. Consecutive robot frames are near-identical; effective sample size is roughly N divided by the correlation length.
• Held-out. Evaluate on data you didn't calibrate on.
• Seeded. Unseeded shuffles near a threshold make exports flaky between runs.
• Covers every axis the model conditions on. For diffusion that includes timesteps.

PTQ algorithms: from rounding to rotating

Post-training quantization (PTQ) quantizes a finished model with a little calibration data and no full retraining. Every algorithm below attacks one of three problems: rounding is locally greedy, errors compound across weights, or activations have outliers.

The baseline: round-to-nearest (RTN)

Quantize each weight independently to its nearest level. Works at 8 bits for almost everything. At 4 bits and below it degrades, because the best choice for each weight alone isn't the best choice for the layer's output.

AdaRound — learn which way to round

Nagel et al., 2020. Key observation: what matters is the layer output Wx, not each weight. Rounding some weights "the wrong way" can let errors cancel. AdaRound keeps the scale fixed and learns, for each weight, a binary choice: floor or ceil.

Rows shaded red are weights where the output-optimal choice rounds away from nearest. AdaRound approximates this search with gradient descent, because 2ⁿ is impossible for real layers.

Descendants. BRECQ (2021) reconstructs whole blocks (e.g. a residual block) instead of single layers, using second-order (Fisher) weighting — enables 2–4 bit. QDrop (2022) randomly drops activation quantization during reconstruction so weights become robust to activation noise.

GPTQ — compensate error as you go

Frantar et al., 2022, built on Optimal Brain Quantization (OBQ). Quantize one weight column at a time; after each, update the not-yet-quantized weights to absorb the error, using the inverse Hessian of the layer's reconstruction loss.

Watch the unquantized weights drift after each step — that drift is error compensation. Compare the final output error against RTN. GPTQ is greedy, so on some layers it gains little; the demo picks layers where the effect is visible. GPTQ quantizes 175B-parameter models to 3–4 bits in a few GPU hours; it's the default weight-only method for LLMs alongside AWQ.

AWQ — protect the salient weights

Lin et al., 2023 (activation-aware weight quantization). Observation: about 1% of weight channels matter disproportionately, and you find them by looking at activation magnitude, not weight magnitude. Instead of keeping them in FP16 (mixed precision is hardware-unfriendly), scale them up before quantizing:

SmoothQuant — move the difficulty from activations to weights

Xiao et al., 2022. Enables true W8A8 on LLMs. Activations have a few channels with huge magnitudes; weights are well-behaved. Since per-channel activation scales can't be used inside an integer GEMM (chapter 2), SmoothQuant applies a mathematically equivalent per-channel rescale:

Start with smoothing off: two activation channels dominate, and per-tensor INT8 leaves the typical channel about ten levels. Turn it on and sweep α. At α = 0 the weights become perfectly flat and all the difficulty lands on activations; at α = 1 it all lands on the weights. Around α ≈ 0.5 both operands keep a healthy share of levels — that balance is the whole method.

LLM.int8() — just don't quantize the outliers

Dettmers et al., 2022. Vector-wise INT8, plus mixed-precision decomposition: feature dimensions whose activations exceed a threshold (magnitude 6) are pulled out and multiplied in FP16; the remaining ~99.9% run in INT8. Accurate, but the split GEMM is slower than pure INT8 — it's a memory-saving method more than a speed method.

Rotations — QuaRot and SpinQuant

If outliers live in a few channels, rotate the space so every channel gets a bit of every outlier. For any orthogonal matrix Q:

QuaRot (2024) uses Hadamard rotations to reach end-to-end W4A4 with 4-bit KV cache. SpinQuant (2024) learns the rotations (optimized on the Stiefel manifold) and beats random ones. Same idea underlies QuIP / QuIP# for 2-bit weights.

The rest of the zoo

Quantization-aware training

When PTQ loses too much, let the model learn to live with rounding. QAT inserts fake-quantization nodes — quantize then immediately dequantize, all in floating point — so the forward pass sees exactly the error it will see at deployment.

• LSQ (learned step size, 2020): the scale s is a trainable parameter with a gradient derived through the rounding, scaled by 1/√(N·Q_p) for stable training.
• PACT (2018): learns the activation clipping value α — clipping becomes a trainable trade-off instead of a calibration heuristic.
• Recipe: start from the trained FP model, PTQ-initialize scales, fine-tune a few percent of the original schedule with a lower learning rate, freeze BatchNorm statistics late.

When to use it: 4-bit activations, small models with little redundancy (mobile CNNs, compact ViTs), or when PTQ misses the accuracy gate. Cost: a training pipeline, training data, and GPU time. For LLM-scale models people prefer PTQ plus a light fine-tune (e.g. QLoRA-style) over full QAT.

Choosing a recipe

The profile follows from two questions: what bounds this workload, and what the hardware executes natively.

The workflow that doesn't lie to you

1. Fix the reference first. Compare exported graph vs framework at FP32/FP16 before quantizing anything. If export isn't faithful, quantization results are meaningless.
2. Profile the bound. Memory or compute, per component.
3. Sensitivity analysis. Quantize one layer (or block) at a time, measure the drop. Rank layers.
4. Keep the usual suspects high: first and last layers, LayerNorm/RMSNorm, softmax, attention score matmuls, anything feeding a residual stream with outliers.
5. Calibrate on representative, held-out data covering every conditioning axis.
6. Gate on the task metric — closed-loop success for a policy, not just cosine or MSE on actions.

Where time goes: the roofline

Every kernel is limited either by how fast the chip computes or by how fast it can be fed. The roofline model makes that a single plot.

Batch-1 decode is the canonical memory-bound case. A matrix-vector product reads each FP16 weight (2 bytes) once and does 2 FLOPs with it (multiply + add): AI ≈ 1 FLOP/byte. Modern accelerators have ridge points in the hundreds. The chip idles more than 99% of its math. That's why weight-only INT4 speeds decode up almost 4×: same FLOPs, a quarter of the bytes, AI ≈ 4.

Set M = 1 (one token) then M = 512 (a prefill or a ViT with 512 patches) and watch the point cross the ridge. Same weights, same layer, different bound — which is why one model can need two quantization strategies.

Kernel fusion: stop writing to memory

Unfused, every op reads its inputs from DRAM and writes its output back. For cheap elementwise ops that round-trip costs far more than the math. Fusion keeps intermediates in registers or shared memory (LDS on AMD) and writes once.

Kinds of fusion

• Vertical (producer–consumer): conv/GEMM + bias + activation, LayerNorm + residual add. The epilogue of a GEMM kernel applies elementwise ops before the single write.
• Horizontal: independent ops with shared input launched as one kernel — Q, K, V projections become one larger GEMM.
• Reduction fusion: softmax, LayerNorm, RMSNorm as one pass with warp/wavefront shuffles instead of several kernels.
• Quantization fusion: dequantize inside the GEMM (weight-only kernels), requantize in the epilogue, fold Q/DQ into neighbours.

FlashAttention — fusion that changed the field

Naive attention materializes the N×N score matrix in DRAM: write S = QKᵀ, read it for softmax, write P, read it for PV. FlashAttention tiles Q, K and V into on-chip blocks and never materializes S. The trick that makes tiling legal is the online softmax:

It's IO-aware: same FLOPs (slightly more, due to recomputation in the backward pass), far fewer bytes. On AMD this lives in Composable Kernel and AITER.

CUDA and HIP graphs: kill the launch tax

Every kernel launch costs CPU time in the driver — a few microseconds — and the CPU has to stay ahead of the GPU. A batch-1 VLA step is hundreds of small kernels; a four-step flow-matching head multiplies that. When kernels are shorter than launch overhead, the GPU starves. On an embedded CPU also running ROS, launch timing jitters.

A graph records the sequence once (stream capture), instantiates it, and replays it with a single launch.

The rules that bite

• Static shapes. A captured graph bakes in sizes; new shapes need re-capture (or one graph per shape bucket).
• Static addresses. Copy new inputs into pre-allocated buffers; never allocate inside capture.
• No host synchronization or host-side work inside capture. In TensorRT terms: set tensor addresses and input shapes before capture, capture only enqueueV3.
• Warm up first. Run once eagerly so lazy initialization happens outside capture.
• It only removes launch cost. A compute- or bandwidth-bound kernel isn't faster in a graph.

Pruning and sparsity

Remove weights (or whole structures) that contribute little. The catch: zeros only save time if hardware can skip them.

Scoring what to cut

• Magnitude: drop smallest |w|. Simple, surprisingly strong with fine-tuning.
• Movement pruning: drop weights moving toward zero during fine-tuning — better for transfer learning.
• SparseGPT (2023): one-shot, GPTQ-style — prune with the inverse Hessian and update remaining weights to compensate. 50–60% unstructured on large LLMs with small loss, no retraining.
• Wanda (2023): score = |W_ij| · ‖X_j‖₂, prune per output row, no weight update. Activation-aware, like AWQ is for quantization.
• Structured: head importance, channel L1 norms, layer dropping (LLM-Pruner, depth pruning) — then fine-tune or distill to recover.

The 2:4 workflow

Train dense → prune each group of 4 to its 2 largest magnitudes → fine-tune to recover → export. Storage: half the values plus 2-bit indices per kept value. In TensorRT, build with --sparsity=enable and it uses sparse tactics only where they are actually faster.

Token merging

ToMe (2022) merges similar tokens in a ViT via bipartite soft matching — r tokens per layer, training-free, often ~2× throughput with a small accuracy drop. For a VLA with multiple cameras the vision tokens dominate the LLM's sequence, so token reduction is often a bigger win than any kernel.

Distillation: a small model that thinks like a big one

Train a student to match a teacher's outputs, not just hard labels. The teacher's full distribution carries "dark knowledge" — which wrong answers are nearly right.

Variants worth knowing

• Feature / hint distillation: match intermediate representations (FitNets), attention maps, or hidden states through a small projection. Useful when student and teacher widths differ.
• Sequence-level / on-policy distillation: student generates, teacher scores or corrects — avoids exposure bias.
• Regression outputs (actions): no logits, so match actions directly (MSE/L1) or match the teacher's predicted velocity field.
• Step distillation for diffusion: progressive distillation trains a student to do two teacher steps in one, halving repeatedly (Salimans & Ho, 2022). Consistency models learn to map any point on the ODE trajectory straight to its endpoint — one or two steps. Rectified flow / reflow straightens trajectories so few Euler steps suffice. For robot policies, Consistency Policy applies this to diffusion policies.

TensorRT, end to end

TensorRT is an ahead-of-time compiler plus a runtime. You hand it a graph (almost always ONNX); it spends minutes searching for the fastest implementation of that graph on this exact GPU, and serializes the result as an engine (plan file). At runtime you deserialize and execute. Build is slow and hardware-specific; run is fast.

The objects

What happens during the build

Precision: weakly typed vs strongly typed

Weakly typed (the classic mode): you set flags like FP16, INT8, FP8; TensorRT treats them as permissions and may pick any allowed precision per layer if it's faster. Great for speed, hard to reason about accuracy, and you may need layer precision constraints to pin sensitive layers.

Strongly typed (NetworkDefinitionCreationFlag::kSTRONGLY_TYPED, trtexec --stronglyTyped): tensor types come from the graph — ONNX dtypes and explicit Q/DQ nodes are authoritative. No precision autotuning. This is the modern path for FP8/NVFP4 graphs produced by ModelOpt: what you exported is what runs.

Implicit vs explicit quantization

Rule of thumb for Q/DQ placement: quantize the inputs of GEMM/conv (weights and activations), don't sprinkle Q/DQ on elementwise ops, and keep residual adds in higher precision unless the fusion pattern supports it.

Dynamic shapes

Mark dynamic dims as −1 in ONNX, give every dynamic input a min / opt / max in an optimization profile. TensorRT tunes kernels for opt but guarantees correctness across the range. At runtime call setInputShape before enqueue. Wide ranges cost performance; several profiles (e.g. short and long sequences) can help. Data-dependent sequence lengths — padded VLM tokens — are the usual reason a transformer engine needs a dynamic profile, or a fixed padded budget.

Plugins

For ops TensorRT can't express or fuses badly: implement IPluginV3 (TensorRT 10; older code uses IPluginV2DynamicExt) with shape inference, supported formats/types, enqueue with your CUDA kernel, and serialization; register a creator; load the .so at build and run time. The ONNX node's op type must match the plugin name. Plugins are opaque to the optimizer — fusion stops at their boundary, so they're a last resort.

Build knobs that matter

• Workspace / memory pool — scratch memory tactics may use. Too small silently excludes fast tactics.
• Builder optimization level (0–5, default 3) — higher searches more tactics, builds slower.
• Timing cache — persist tactic timings so rebuilds are fast and more reproducible.
• Tactic sources — cuBLAS, cuBLASLt, cuDNN, internal kernels; can be restricted.
• Sparsity — allow 2:4 sparse tactics.
• DLA (Jetson Orin has two DLA cores) — offload supported layers with GPU fallback; each GPU↔DLA transition costs a reformat and sync.
• Hardware / version compatibility — engines are tied to GPU architecture and TensorRT version by default; compatibility modes relax this at some performance cost.

trtexec cheat sheet

Debugging accuracy: Polygraphy

# Compare TensorRT against ONNX Runtime on the same inputs
polygraphy run model.onnx --trt --onnxrt --atol 1e-3 --rtol 1e-3

# Mark every intermediate as an output to find the first diverging layer
polygraphy run model.onnx --trt --onnxrt \
    --trt-outputs mark all --onnx-outputs mark all

# Search for which layers must stay in higher precision
polygraphy debug precision model.onnx --fp16 --check ...

# Clean the graph before building
polygraphy surgeon sanitize model.onnx --fold-constants -o clean.onnx

Runtime patterns

engine  = runtime.deserialize_cuda_engine(plan_bytes)
ctx     = engine.create_execution_context()
for name in input_names:
    ctx.set_input_shape(name, shape)          # dynamic dims, before enqueue
for name, ptr in buffers.items():
    ctx.set_tensor_address(name, ptr)         # pre-allocated, persistent
ctx.execute_async_v3(stream)                  # enqueueV3; capture this in a CUDA graph

• Allocate once, reuse forever; pinned host memory for H2D copies (or none at all on unified-memory Jetsons).
• One execution context per concurrent stream.
• Don't synchronize after every engine in a multi-engine pipeline — chain them on one stream.
• Pin the whole stack (JetPack, CUDA, TensorRT, ModelOpt, opset) and record versions next to every benchmark.

Pitfalls that show up in practice

The AMD mirror

Every concept above has an AMD counterpart. Know the mapping and most NVIDIA experience transfers.

Drill: cards and quiz

Click a card to flip it. Answer out loud before flipping — the point is fluency, not recognition.

Quiz