This repo documents how to get llama.cpp running with the **Qualcomm Hexagon CDSP v68** (NPU/DSP) backend on a **Radxa Dragon Q6A** board (SA8775P).
## Overview
The Q6A has a Qualcomm QCS6490 SoC with a Hexagon CDSP v68 that can accelerate matrix operations in llama.cpp via FastRPC. The key insight from weeks of debugging: **let libcdsprpc handle `FASTRPC_IOCTL_INIT_CREATE` internally** — do NOT attempt it manually. Use the system's `libcdsprpc.so`, not the SDK's cross-compiled version.
## Prerequisites
### Build Machine (x86_64)
- Ubuntu 24.04 (or similar with cross-compilation packages)
The error occurs because **the SDK's cross-compiled `libcdsprpc.so` does NOT handle `FASTRPC_IOCTL_INIT_CREATE` internally** for unsigned PDs. The Q6A system `/usr/lib/libcdsprpc.so.1` does. The fix: compile and link natively on the Q6A (or link against the system library).
Attempting `FASTRPC_IOCTL_INIT_CREATE` via ioctl on `/dev/fastrpc-cdsp-secure` always returns EINVAL because the kernel expects the struct to be set up by libcdsprpc's internal state machine. The correct approach:
```c
/* ONLY these two calls are needed — libcdsprpc handles INIT_CREATE */
1.**`CMAKE_SYSROOT` breaks the linker** — Ubuntu's `gcc-aarch64-linux-gnu` packages install linker scripts with absolute paths (e.g., `/usr/aarch64-linux-gnu/lib/libm.so` contains `GROUP( /usr/aarch64-linux-gnu/lib/libm.so.6 ... )`). When `--sysroot` is set to `/usr/aarch64-linux-gnu`, these absolute paths double up to `/usr/aarch64-linux-gnu/usr/aarch64-linux-gnu/lib/...` and fail. Solution: let the compiler use its built-in sysroot.
2.**`rpcmem_init` is optional** when linked against `libcdsprpc.so`. The SDK's cross-compiled `libcdsprpc` only exports `rpcmem_alloc`/`free` but not `rpcmem_init`/`deinit`. The Q6A system libcdsprpc has them all.
## Why the NPU Isn't Accelerating Inference
After extensive testing, the Hexagon backend loads and initializes successfully but **never actually offloads any computation**. Every test shows:
The NPU reports 2048 MiB but uses 0 MiB for model, context, and compute. Source code analysis reveals three reasons:
### 1. `offload_op` callback is NULL
The backend device struct registers `/* .offload_op = */ NULL`, so the scheduler never proactively moves tensors to the NPU. Even when `-ngl N` is specified, without this callback no layers get claimed.
It never calls `rpcmem_alloc2()` or checks kernel ION/DMA-BUF heap sizes. The 2GB is a rough placeholder. On QCS6490 with 11GB system RAM, the CDSP carveout is typically 1-4 GB depending on firmware config, but the code never checks.
### 3. Q4_K_M is not a supported quantization for HTP
The Hexagon HTP kernels only support these quantization types:
-`GGML_TYPE_Q4_0`
-`GGML_TYPE_Q8_0`
-`GGML_TYPE_IQ4_NL`
-`GGML_TYPE_MXFP4`
**Q4_K_M is NOT in this list.** Every MUL_MAT operation with Q4_K_M weights fails the `supports_op` type check, regardless of buffer placement. This means even if you fix the buffer allocation path, the 7B Q4_K_M model still won't offload.
### Summary of Issues
| Issue | Root cause |
|-------|-----------|
| 0 MiB used for model/context/compute | `offload_op = NULL` in device registration |
| 2048 MiB cap | Hardcoded constant, not a FastRPC/ION query |
| Q4_K_M tensors don't offload | Q4_K_M not in HTP supported type list |
| Ops always rejected by supports_op | Chicken-and-egg: tensors never in Hexagon buffers |
## Performance Benchmarks
All tests on Radxa Dragon Q6A (QCS6490, 11GB RAM, 8x ARM Cortex cores, Ubuntu 24.04).
### 1B Model (Llama 3.2, Q4_K_M)
| Metric | CPU-only | With Hexagon backend |
|--------|----------|---------------------|
| Prompt processing | 32.3 t/s | 32.0 t/s |
| Generation | 4.5 t/s | 4.5 t/s |
No difference — CPU handles the 1B model natively.
### 7B Model (DeepSeek R1 Distill Qwen, Q4_K_M)
Need `-c 2048` or smaller to fit in 11GB RAM (model alone is 4460 MiB, KV cache at default 128K context adds 7GB).
| Context | Test | Prompt t/s | Gen t/s | Model | Context | Compute | Total |
**Every NPU vs CPU comparison is identical.** The Hexagon backend never offloads any tensors, so all computation runs on CPU in both cases.
Memory at 64K context (8.5 GiB total) approaches the 11 GiB ceiling but fits with swap support.
### What would need to change to get actual NPU offload
1. Implement `offload_op` callback in `ggml_backend_hexagon_device_registration`
2. Wire the repack buffer type for weight tensors so quantized weights land in Hexagon-accessible memory
3. Query actual rpcmem capacity instead of hardcoding 2GB
4. Use a Q4_0 or Q8_0 quantized model (Q4_K_M not supported by HTP kernels)
## Known Issues
- **Minimal stub library** (`htp_minimal_impl.c`) fails to load on the DSP with error `0x80000442` — the full `libggml-htp-v68.so` (generated by the cmake build from `ggml-hexagon/main.c`) works correctly.
- **DSP library is rebuilt every time** the cmake build runs.
- The `htp_iface.idl` declares `dst` as `in sequence<uint8>` (input-only) but it's an output buffer. Should be `rout`.
