For edge AI engineers coming from TensorRT · checked against Qualcomm docs, Oct 2026

One SoC,
six places to run a tensor

A Snapdragon or Dragonwing chip gives you a CPU, a GPU, an NPU, a DSP and a micro-power NPU. Qualcomm's software stack is the set of tools that decides which one runs your model and how well. This guide walks the whole path, from a PyTorch checkpoint to a tuned inference loop on the NPU.

train → ONNX → AIMET (quantize) → qairt-converter (DLC) → context binary (per SoC) → QNN runtime → HTP / GPU / CPU / LPAI

Module 01

The map: every name, decoded

Half the difficulty is naming. Qualcomm renamed most products at least once, and docs use old and new names side by side. Click any block.

Rosetta stone: TensorRT → Qualcomm

You know (Jetson)	Qualcomm equivalent	What differs
ONNX parser	`qairt-converter`	Output is a DLC file, not an engine. Layout is converted to NHWC.
Engine / `.plan`	Context binary (`.bin`)	Tied to QAIRT version and SoC / HTP arch. Same pinning discipline as JetPack + TensorRT.
Builder optimisation	Graph finalize / offline prepare	Run on x86 host for HTP. On-device prepare exists but is slow.
INT8 calibrator, Q/DQ	`qairt-quantizer`, AIMET encodings, QDQ ONNX	On the NPU quantization is the default path, not an optimisation.
`trtexec`	`qnn-net-run`, `qnn-throughput-net-run`	Reads raw tensor files from an input list.
Plugins	Op packages (QNN), UDO (SNPE)	One implementation per backend. HTP ones need the Hexagon SDK.
DLA	HTP (the NPU)	Same mental model: fast, fixed-point first, fallbacks are expensive.
CUDA zero-copy	rpcmem / DMA-BUF shared buffers	Registered with `QnnMem_register`.
Polygraphy	`qnn-accuracy-debugger`	Layer-wise compare against the framework reference.
`tegrastats`	`sysMonApp`, Snapdragon Profiler	DSP load is reported separately from CPU load.

I can explain QAIRT vs QNN vs SNPE vs AI Hub

Module 02

The silicon: six backends

A QNN backend is one shared library that drives one piece of hardware. Click a block on the floorplan.

ISP · modem · memory

FastRPC: the CPU ↔ DSP call bridge every NPU job crosses

NPU, HTP and DSP are not three chips. "NPU" is the marketing name. "HTP" (Hexagon Tensor Processor) is the QNN backend name for it. Physically it lives in the compute DSP subsystem (cDSP / NSP). In SNPE you still select it with the runtime named DSP. The QNN DSP backend is a different thing: older Hexagon cores without the tensor accelerator.

Inside the Hexagon NPU

Scalar cores

Control flow, glue, ops with no vector kernel. Slow path.

HVX (vector)

1024-bit SIMD. Runs many element-wise and non-conv ops. You reserve HVX threads per graph (default 4).

HMX (tensor)

Matrix engine for conv and matmul. This is where the TOPS come from. Works on fixed-point and, on newer chips, FP16.

VTCM is a small, fast on-chip memory the tensor engine works from. Bigger VTCM per graph means fewer tile shuffles, but it is shared between all graphs on that core. Default is 4 MB.

HTP generations

Arch	Typical chips	What changed
`v68`	Snapdragon 888, QCS6490 (RB3 Gen 2)	First HTP. Quantized only.
`v69`	8 Gen 1	FP16 arrives on select SoCs.
`v73`	8 Gen 2, QCS8550, X Elite, QCS9100 / SA8775P	Weight sharing across graphs. Workhorse for LLMs.
`v75`	8 Gen 3	Generational speed-up.
`v79`	8 Elite	Generational speed-up.
`v81`	8 Elite Gen 5, X2 Elite	BF16 on Windows, concurrent resource sharing.

Chip-to-arch mapping is from memory of public material. Confirm yours in the SDK's supported-devices page before you set dsp_arch.

I know which library drives which block

Module 03

Precision & quantization

Precision decides the backend more than anything else. A float model cannot use most of the NPU. A badly quantized model uses it and gives wrong answers.

Backend	FP32	FP16	INT8	INT16 act.	4-bit weights	Notes
CPU						FP32 and quantized 8-bit from the same library.
GPU						Modes: FP32, FP16, hybrid (FP16 tensors, FP32 accumulators).
HTP						INT8/INT16 on every SoC. FP16 on select SoCs. BF16 on Windows v81+. FP32 graphs run as FP16 when relaxed precision is on.
DSP						8-bit fixed point.
LPAI						Quantized only. Float models are rejected.
HTA						16-bit activations only with 8-bit weights.

supported conditional not supported

Reading the notation

w8a8: 8-bit weights, 8-bit activations. Fastest, least robust.
w8a16: the default for transformers and anything with wide activation ranges.
w4a16: LLM weights. Needs block-wise tricks (LPBQ) to hold accuracy.
ExecuTorch writes it the other way round: 16a8w.
Weights are usually symmetric and per-channel. Activations are asymmetric and per-tensor.

AIMET, in order of effort

Calibrate: QuantizationSimModel + compute_encodings on a few hundred samples.
Free fixes: batch-norm folding, cross-layer equalization.
Better rounding: AdaRound, Sequential MSE, BN re-estimation.
Find the bad layer: QuantAnalyzer, then mixed precision (manual or automatic) to lift only those layers to 16-bit.
Last resort: QAT. For LLMs: LPBQ, OmniQuant, SpinQuant.
Export gives ONNX + an .encodings JSON. Feed that to the converter with --quantization_overrides.

Three ways to quantize, pick one. AIMET (most control), qairt-quantizer on a DLC (quickest, fewer algorithms), or an AI Hub quantize job (AIMET-ONNX run for you in the cloud). Do not stack them.

I can pick a bit-width scheme for a model

Module 04

Op support explorer

QNN defines well over a hundred ops. Each backend implements a subset, and each op has its own rank, datatype and parameter limits. Search an op, or filter to see what a backend lacks.

listed in Qualcomm's table or backend supplement expected in current SDKs, not checked here not supported? not checked

How far to trust this table. Rows up to ElementWiseNeg come from Qualcomm's supported-operations table, and the whole GPU and DSP columns come from their op supplements (docs published Aug 2026). I could not load the rest of the HTP, LPAI and HTA columns, so those show hollow dots or "?". Before you design around an op, open SupportedOps and the HTP supplement for your SDK version.

Why a "supported" op still falls back

Rank. Most HTP ops stop at 4D or 5D. A 6D reshape in a window-attention block breaks the partition.
Datatype. int64 indices, bool masks and float64 are common ONNX leftovers. Truncate or cast them before conversion.
Dynamic shapes. Backends want static dims. The GPU backend states it plainly: variable input and output dimensions are not supported.
Parameter limits. Example from the ORT QNN EP: Gather needs positive indices, LpNormalization only p = 2, MatMul on HTP only for specific uint8/uint16 input pairs.
Precision column. An op can exist for INT8 and not for FP16, or the other way round.

One fallback op in the middle of a graph costs two FastRPC crossings plus a dequantize and a quantize. Three small NPU islands are usually slower than running the whole thing on the GPU.

I know how to check an op before committing to a backend

Module 05

Which backend, and why

Answer four questions. The recommendation updates as you click.

Why some models go to the NPU and others to the GPU

NPU wins when

The model is quantized and the graph is static.
Conv and matmul dominate, so HMX does the work.
You care about energy per inference or sustained thermals.
You can build the context binary offline.

GPU wins when

The model is float and quantization hurts accuracy (depth, flow, some diffusion parts, regression heads).
The op set is exotic or shapes change.
The data is already on the GPU (camera, render, OpenCL pipeline).
The NPU is busy with a higher-priority model.

Workload	Usual home	Reason
Detector / segmenter (YOLO, DeepLab)	HTP w8a8 / w8a16	Conv-heavy, quantizes well.
ViT, DETR, VLA vision tower	HTP w8a16	LayerNorm and softmax need 16-bit activations.
LLM / VLM decoder	HTP w4a16 via Genie	Split into several context binaries, KV cache handled by Genie.
Monocular depth, super-resolution in float	GPU FP16	Accuracy-sensitive, no calibration needed.
NMS, tracker, pre/post-processing	CPU	Branchy, dynamic sizes, tiny compute.
Keyword spotting, tactile or IMU classifier	LPAI	Runs while the application processor sleeps.
Small MLP distilled from a random forest	CPU or LPAI	RPC overhead exceeds the compute on HTP. LPAI if it must be always-on.

I can defend a backend choice in a design review

Module 06

End to end: checkpoint to device

Two lanes reach the same artifacts. The SDK lane runs on your x86 Linux box and in CI. The AI Hub lane runs in Qualcomm's cloud on real phones and boards.

Changed in Sept 2026: AI Hub Workbench no longer compiles straight to a QNN context binary. Compile to qnn_dlc, then run a link job to get the binary.

The three files you will ship

DLC

Graph + weights + encodings. Portable across SoCs and backends. Prepared on device at load, which is slow on HTP.

Context binary

Already compiled for one HTP arch and one QAIRT version. Loads fast. Rebuild it when either changes.

Wrapper model

*_ctx.onnx for ONNX Runtime or .pte for ExecuTorch. A thin shell that points at, or embeds, the context binary.

I have run every step once

Module 07

Runtime: load, run, tune

Your process never touches the NPU directly. It talks to a stub library on the CPU, which sends FastRPC calls to a skeleton library inside the DSP. Everything below follows from that.

The QNN C API call flow

What you can and cannot control

Backend	Knob	Where	Effect
HTP	Performance profile	`perf_profile`; power config via `QnnDevice_getInfrastructure`	Sets DCVS, voltage corners and sleep latency. `burst` is fastest, `power_saver` family trades latency for energy.
	RPC control latency, polling time	`rpc_control_latency`, `rpc_polling_time` (µs)	Keeps the CPU side awake and polling. Cuts wake-up jitter on sporadic inference.
	HVX threads	`hvx_threads` per graph	How many vector threads the graph reserves. Default 4. Lower it to leave room for a second model.
	VTCM size	`vtcm_mb` per graph	Fast-memory budget. Default 4 MB.
	Finalize optimisation	`O` = 1, 2, 3	3 gives the fastest graph and the longest prepare. Use it for offline builds.
	Context priority	low / normal / normal_high / high	Which graph wins when several share the NPU.
	Device and core	`device_id`, `core_id`, `core_type` (NSP or HPASS)	Picks the NPU on chips with several (automotive SA-series, HTP MCP).
	Process domain	`pd_session` signed / unsigned	Unsigned works for any app. Signed needs a signed DSP image from the OEM and unlocks more resources.
	Async execution	`QnnGraph_executeAsync`, queue depth	Pipeline several frames without blocking the caller.
GPU	Precision mode	graph custom config	FP32, FP16 or hybrid.
	Performance hint	context custom config	HIGH (default) pushes clocks up, LOW saves power.
	Kernel cache	`QNN_GPU_KERNEL_REPO` on disk	Skips OpenCL kernel compilation on the next start.
CPU	Cores and threads	OS affinity on the calling thread (`sched_setaffinity`, `taskset`)	The QNN CPU backend documents no thread-count option. Higher layers do: ORT intra-op threads, LiteRT `number_of_threads`, Genie `n-threads` and `cpu-mask`.
CPU	Op debug callback	graph custom config	Returns intermediate outputs as the CPU backend executes.
LPAI	Clock and bandwidth	`fps`, `ftrt_ratio`, `enpu_clock_scale`	Sizes eNPU clocks to your frame rate.
LPAI	Core, affinity, memory	`core_selection` bitmask, `affinity` soft/hard, `mem_type` ddr/tcm	Real core pinning. TCM only in aDSP direct mode.

About "assigning cores" on the NPU. You do not pin NPU work to cores the way you pin CPU threads. You choose a device and core on multi-NPU chips, reserve HVX threads and VTCM per graph, and set priority. The Hexagon RTOS schedules the rest.

Config builder

Set the knobs, copy the files. The same settings are shown for qnn-net-run and for the ONNX Runtime QNN execution provider.

Key names follow the HTP backend-extension schema as I know it. Validate against the example JSON in your SDK's docs/QNN/general/htp folder.

Talking to a model that is already loaded

Read its I/O. QnnSystemContext_getBinaryInfo on the binary gives graph names, tensor names, shapes, datatypes and quantization scale/offset. On the CLI: qnn-context-binary-utility dumps the same as JSON.
Feed it without copies. Allocate with rpcmem_alloc (DMA-BUF), register with QnnMem_register, set the tensor memType to memhandle. The NPU reads the camera buffer in place.
Skip CPU quantization. Keep I/O tensors in their native uint8/uint16 type and fill them yourself with q = round(x / scale) - offset.
Switch behaviour live. Several graphs can sit in one context and share weights (up to 64 per context, v73+). Pick one per call. LoRA adapters can be applied to a loaded context. ORT exposes ep.dynamic.workload_type to drop priority at runtime.
See inside. No mid-graph tensor access on HTP at runtime. Use --profiling_level detailed for per-op time, optrace for a timeline, and debug builds that mark every layer as an output for value dumps.
Watch the DSP. adb shell sysMonApp getLoad-style queries and Snapdragon Profiler show cDSP utilisation and clocks. CPU tools will not show NPU load.

I can write a load-and-execute loop and tune it

Module 08

Debugging & the traps everyone hits

When it will not load

Version skew. Context binary, libQnnHtp.so, the stub and the libQnnHtpV73Skel.so on the DSP must come from the same QAIRT release.
Skel not found. Set ADSP_LIBRARY_PATH to the folder holding the skel. On Android declare libcdsprpc.so with uses-native-library.
Wrong arch. A v75 binary will not run on a v73 chip.
Too big. One HTP session has a limited address space. LLMs are split into several binaries for this reason.

When it is slow

Fallback islands. Check which compute unit each layer landed on before reading any latency number.
On-device prepare. Loading a DLC on HTP compiles it on the phone. Ship a context binary.
Cold NPU. First inference after idle pays wake-up. Use burst plus RPC polling, or accept it.
Layout transposes. NCHW models gain transposes at I/O. Feed NHWC.
CPU quant/dequant. Float I/O on a quantized graph converts every frame on the CPU.

When it is wrong

Run the quantized DLC on the CPU backend first. If it is already wrong, it is quantization, not the NPU.
Compare layer by layer with qnn-accuracy-debugger.
Usual suspects: softmax, LayerNorm, sigmoid tails, add of two very different ranges. Lift them to 16-bit.
FP16 on HTP overflows above ~65k. Watch un-normalised activations.

Reproducibility checklist

Pin QAIRT version, HTP arch and soc_model in artifact metadata.
Store DLC and encodings next to the binary so you can rebuild.
Build binaries in Docker on x86 Linux, never on a dev laptop.
Record device firmware: the DSP image changes behaviour too.
Use the QNN Saver backend to capture a failing API sequence and replay it.

I have a debug order I follow

Module 09

Check yourself

Eight questions. Each one is a mistake people make in their first month on the stack.

0 / 8

Answer to score.

A four-week path

Week	Do	You are done when
1	AI Hub Workbench: compile, profile and run inference on a MobileNet and a YOLO for your target chip.	You can read a per-layer profile and say which layers left the NPU.
2	Same model through the QAIRT CLI on your own machine. Quantize with AIMET, w8a8 then w8a16.	Your local context binary matches AI Hub latency within a few percent.
3	Write a C++ loader with the QNN API: context from binary, shared buffers, perf config.	Zero-copy camera frames in, p99 latency stable.
4	A hard model: a ViT or a small VLA block. Fix fallbacks, add mixed precision, wire it into CI.	Rebuild from checkpoint to binary is one pipeline run with pinned versions.

Quiz done