A spec sheet says 40 TOPS. Your OCR model still takes 400 ms and warms the back of the tablet.

Both numbers are true. The gap between them is the whole story of on-device AI — and it is not a marketing problem, it is an engineering one.

This article explains what the silicon can do, what the software layer actually lets you use, and which workloads waste the hardware. If you are specifying an Android handheld, tablet or kiosk for computer vision or speech, this is the part that decides whether the project ships on time.

What “on-device AI” actually means

On-device AI means the model runs on the same processor that drives the screen. No round trip to a server. That single architectural choice changes five things that matter in the field:

For a phone that summarises messages, these are conveniences. For a production line that inspects 2,000 labels an hour, they are the difference between a working system and a support ticket.

Four compute units, four jobs

Every modern Android SoC contains the same four kinds of silicon. Vendors bundle them under names like “AI Engine” or “AI performance”, which is why the marketing reads the same across very different chips.

The important consequence is not which unit is “fastest”. It is that a real model is a mixture, and the runtime decides which unit executes which part.

When the NPU cannot execute an operator, it hands that subgraph back to the CPU. This is called fallback. It does not raise an error, it does not log a warning by default, and the app keeps working — just slowly. Measure the assignment per layer before you believe any speed claim, including your own.

Why TOPS does not compare

TOPS means trillions of operations per second. Three things make the number incomparable across vendors.

1. Precision changes the number by multiples. The same NPU might quote 40 TOPS at INT8, 10 TOPS at FP16, and refuse FP32 entirely. “100 TOPS” without a precision is not a specification.

2. Quantisation legally buys multipliers. Current flagship NPUs support INT4, INT2 and even 1.58-bit formats. Fewer bits per operation means more operations per second from the same circuit. That is real acceleration, not a trick — but it means two vendors can quote very different TOPS for hardware of similar size.

3. Most models never reach peak. Peak throughput needs large, contiguous matrices. Real models are full of transposes, reshapes, normalisation and non-maximum suppression. Those either run on the CPU or leave the NPU waiting on memory.

A detail worth noticing

At its September 2025 summit, Qualcomm announced the Snapdragon 8 Elite Gen 5 and did not publish an absolute TOPS figure for the phone chip. It published relative numbers instead: the Hexagon NPU is 37% faster than the previous generation with 16% better performance per watt.

In the same keynote, the laptop chip — Snapdragon X2 Elite — was given a clean 80 TOPS.

That is a deliberate choice, and it tells you something useful: when absolute numbers stop flattering, vendors switch to percentages. For anyone specifying hardware, the practical rule is simple. Ignore the headline. Measure milliseconds and milliwatts on your own model.

What the current silicon looks like

Qualcomm: from Hexagon 690 to fused AI

The Hexagon line has been the AI block in Snapdragon for years. Hexagon 690, for example, shipped in the Snapdragon 855 generation around 2019 and was quoted at roughly 7 TOPS. Successive generations raised that by an order of magnitude, and the 8-series moved to a fused architecture where scalar, vector and tensor work share one pipeline.

The current flagship generation is worth reading closely, because the architecture tells you what it was designed for:

The INT2 support is the tell. Two-bit quantisation is how you fit a usable language model into phone memory without draining the battery. An NPU that supports INT2 is designed to run local LLMs, not only to sharpen photographs.

The industrial line matters more than the phone line

Phone chips get the headlines, but the silicon in a warehouse handheld is a different product family. Qualcomm’s Dragonwing line targets exactly that, and the Q-6690 is the most interesting part to appear in years:

Two things here are more consequential than the TOPS figure.

RFID moved into the SoC. That removes a separate module from the bill of materials, shrinks the board and removes a failure point.

Compute became a software tier. Four feature packs let one board ship as three products at three price points, with compute, multimedia and camera capability unlocked by licence rather than by redesign. Hardware is designed once; capability is sold in layers.

Six TOPS sounds small next to a flagship phone. For a rugged handheld it is not: 6 TOPS of INT8 is comfortable headroom for a 6 MB OCR model, a wake-word model and a defect classifier running on the same device. Industrial buyers are not shopping for peak throughput — they are shopping for eight years of supply and a device that does not throttle at hour three.

MediaTek: efficiency as the design goal

MediaTek’s Dimensity 9500 (September 2025) takes a different route. Its ninth-generation NPU 990 with Generative AI Engine 2.0 is quoted against the previous generation:

The architecture detail worth knowing is compute-in-memory. In a conventional design, weights move between memory and the compute array, and that movement often costs more energy than the arithmetic. Doing multiply-accumulate inside the memory array removes the trip.

The goal is not a better benchmark. It is a model that can stay loaded and running without the user noticing the battery. That distinction decides whether always-on wake words and continuous vision are practical.

In January 2026 MediaTek added the Dimensity 9500s and 8500, pushing this class of NPU into mid-range price bands. Industrial handhelds, tablets and kiosks are usually the first devices to benefit, because they buy mature silicon one tier below flagship.

The software stack decides more than the chip

This is the part most teams underestimate. A capable NPU with an outdated runtime is a slow CPU with extra steps.

NNAPI has been deprecated

For years the standard advice for Android was one line: add the NNAPI delegate and the hardware takes over. That path is now closed.

Google’s own documentation states that the Neural Networks API (NNAPI) is deprecated. The reasoning is straightforward: after NNAPI shipped in Android 8.1, on-device machine learning moved to transformers and diffusion models, and a system API that only changes with Android releases could not keep up. Google also warns that on many future devices NNAPI is expected to resolve to a CPU backend — meaning the “NPU enabled” flag would compile, run, and accelerate nothing.

The recommended paths are now:

1. LiteRT — the renamed TensorFlow Lite, shipped as an independently updatable runtime.
2. AICore with Gemini Nano — a foundation model already present on the device.
3. ML Kit GenAI APIs — turnkey generative features for apps.

If you are starting a project in 2026, NNAPI is a fallback, not a strategy.

LiteRT is the unified NPU entry point

LiteRT exposes one interface across NPU vendors, so you do not integrate a different compiler for each chip family:

Two compilation modes, and the choice visibly affects the user’s first launch:

Large models — LLMs, segmentation networks — should be compiled ahead of time. Small models, including most OCR and classifier graphs, are usually fine on device.

Deployment is handled by AI Packs: compiled artifacts are packaged and delivered through Google Play’s per-device delivery, so a user only downloads the build for their own SoC. NPU acceleration requires API level 31 or higher.

Gemini Nano: the model you do not have to train

The most underrated route is to use the model the platform already ships.

Gemini Nano runs on more than 140 million devices. On Tensor G5 (Pixel 10 series, TSMC 3 nm) the TPU delivers up to 60% more compute, and the latest Nano runs 2.6x faster and 2x more efficiently in features such as screenshot search and recorder transcription. On a Pixel 10 Pro, the current model processes text at roughly 940 tokens per second with image encoding around 0.6 s.

The next generation, Gemini Nano 4, is built on the Gemma 4 family and splits into Fast and Full variants; the Fast variant is quoted as up to 4x faster with 60% less battery on the TPU.

You reach it through ML Kit GenAI APIs — summarisation, proofreading, rewriting, image description — without converting a model, without checking operator support, and without shipping weights in your APK.

Android 17: from operating system to “intelligence system”

Android 17 (June 2026) reframes the platform. The mechanism most relevant to device builders is AppFunctions with Android MCP: an app exposes its capabilities as callable tools, and on-device agents can discover and execute them, with direct access to the app’s local state.

The compute implication is direct. An agent that runs locally is the only way to get low latency and keep data on the device. That turns “always-on, low-power inference” from a nice extra into a baseline requirement.

Android 17 also introduces a hybrid inference API for routing between local and cloud models, and A2UI for agents that render interface elements. The split is consistent: frequent, private, small work stays on the device; long-context and heavy reasoning goes to a server.

What you can actually run

OCR: the most mature workload, with one specific trap

OCR suits on-device inference unusually well. The input is one image, the output is a few dozen characters, the model is small, it is called constantly, and the content is often sensitive — serial numbers, batch codes, documents.

It also has a structural problem on NPUs:

This is why teams enable NPU acceleration and see no gain. Detection falls back to the CPU because the input size varies; recognition is accelerated but its compute is small enough that data movement eats the benefit.

The reliable pattern is hybrid inference:

1
2
3
4

frame capture
 └─► detection (CPU, fixed size) → text boxes
      └─► per-box crop → recognition (NPU, 3×48×320 static) → text
           └─► regex field extraction → output

Vision: beyond reading, to deciding

In industrial settings the value is rarely recognition — it is judgement, and judgement has a deadline.

There is also a hardware trend worth tracking: ISPs are being designed with AI in the loop. Tensor G5 introduced a new ISP, and MediaTek’s Imagiq 1190 does RAW-domain pre-processing. Low light, glare and motion blur — historically the hardest part of any vision project — are being partly absorbed by silicon.

Speech: the workload with the highest frequency

Speech is more demanding than OCR because it is continuous.

In an intercom, the split that works is: noise suppression, wake word and voice activity detection on the device; the cloud only forwards and synthesises what is necessary. That combination is how you get both low latency and low running cost.

Three integration routes

A. LiteRT — the recommended default

1
2
3
4
5
6
7
8
9
10

// LiteRT CompiledModel with NPU acceleration (API 31+)
// Refer to the official docs for the current API surface
val options = CompiledModel.Options(Accelerator.NPU)
val model = CompiledModel.create(
    context.assets,
    "ocr_rec.tflite",
    options,
    env
)
// Reuse input and output buffers across calls

Multi-vendor, officially supported, with AOT compilation and per-device delivery through Play. The cost is that models must be converted to .tflite, and custom operators need implementing.

B. ONNX Runtime — the shortest path from PyTorch

1
2
3
4
5
6
7

// Qualcomm: QNN execution provider, direct to Hexagon
val qnn = OrtSession.SessionOptions()
qnn.addQnn(mapOf("backend_path" to "libQnnHtp.so"))

// Other devices: NNAPI as a transitional fallback only (deprecated)
val nnapi = OrtSession.SessionOptions()
nnapi.addNnapi()

No framework change if your models are already ONNX. The cost is that QNN bundles vendor .so files, so the APK grows, and the NNAPI route is on its way out.

C. Vendor SDK — the performance ceiling

Qualcomm QNN/QAIRT, MediaTek NeuroPilot, Samsung ENN. This extracts the most from the hardware at the cost of lock-in and one build pipeline per vendor.

Where it breaks

Five limits account for most failed pilots.

The fifth one is the one that kills industrial deployments. A production line does not forgive a device that benchmarks well and then slows down after half an hour.

Why industrial devices get there first

Consumer on-device AI sells convenience: photo editing, summaries, translation. Industrial deployments pay for different reasons.

Add chips like the Q-6690 — eight years of Android upgrades, compute unlockable by software tier — and the product logic changes. A board can ship today as a basic barcode terminal and be upgraded later to run OCR, defect detection and voice commands. The device is no longer fixed at the factory.

This is also why we build on standard ONNX and standard inference runtimes. A model that runs on today’s CPU and accelerates on tomorrow’s NPU, without a rewrite, is worth more than a model that is 20% faster on one vendor’s silicon and locked to it.

Two bets worth making

1. Hybrid routing becomes a platform API. Android already ships the routing primitives. Developers will stop hand-writing the “local or cloud” decision.

2. Low-bit quantisation brings LLMs to mid-range silicon. MediaTek supports BitNet 1.58-bit in hardware; Qualcomm supports INT2. Model compression and hardware formats are converging, and a 3B-parameter model on a mid-range chip will stop being news.

The quieter consequence: once assistants listen and watch continuously, the metric that matters shifts from peak TOPS to milliwatts while resident. Compute-in-memory, low-power NPUs and event-driven pipelines will matter more than another round of benchmark gains.

One page

💬 Questions or a custom requirement?

This global site is brand new, and our online tools are currently in Chinese. Tell us what you need and we’ll set up a fast, tailored solution for you — an English version is coming in the next few weeks.

📧 sales@happylab.me · 🌐 global.happylab.me

Specifications and platform behaviour reflect public information as of October 2026 and change with firmware, drivers and OS versions. Validate on your own hardware before committing to production.