Engineering · Arm Create · Mobile AI
Zygote: the agent that lives on a $150 phone
How Zygote — a fully on-device agent harness for the Arm Create: AI Optimization Challenge — squeezed a 1.2B model into a Samsung Galaxy M17 with 2.2 GB of free RAM, and cut time-to-first-token by 34×.
No NPU, no GPU, and 2.2 GB to hold it all
The challenge: run an AI agent entirely on-device — private, offline, responsive — on Arm-powered phones. Our device is a Samsung Galaxy M17 (SM-M176B): an Exynos 1330 with 2× Cortex-A78, 6× Cortex-A55, no NPU, and 8 GB of RAM — of which the OS leaves about 2.2 GB free.
LLM decode is memory-bandwidth-bound: every token reads the entire weight matrix. For a 664 MB Q4_0 model that is ~9.2 GB/s of sustained reads at 14 tokens/s — half the theoretical LPDDR4X bandwidth. The constraint set is unforgiving: no 8B-class model fits (a Q4 8B-A1B is 4.6 GB on disk), so the realistic ceiling was Liquid's LFM2.5 family.
After benchmarking all three tiers on the phone, we committed to the 1.2B-Instruct Q4_0as the single model and dropped the 2.6B — 15.1 t/s prefill and 1.49 t/s decode meant minute-long waits, and its tool-call reliability didn't justify it. The 230M stayed on as the fast router tier — 251 t/s prefill means ~2 ms intent classification — and as the speculative-decoding draft when enabled (section 04).
Every number here was measured with the benchmark instrument on the actual phone — prefill, decode, per-turn TTFT, RAM available. No emulator results, no vendor claims.
| Model · Q4_0 | Prefill | Decode |
|---|---|---|
| LFM2.5-230M | 251 t/s | 51.1 t/s |
| LFM2.5-1.2B-Instruct | 67 t/s | 14.7 t/s |
| LFM2.5-2.6B | 15.1 t/s | 1.49 t/s |
$ adb shell am instrument -w \
-e class com.example.llama.ZygoteBenchmarkTest \
com.example.llama.aichat.test/androidx.test.runner.AndroidJUnitRunner
230M Q4_0 | prefill 251 t/s | decode 51.1 t/s
1.2B Q4_0 | prefill 67 t/s | decode 14.7 t/sWhy Q4_0 beats Q4_K_M here — KleidiAI
Arm's KleidiAI microkernels are the multiplier. Our llama.cpp build uses its Q4_0 DotProd kernels on the A78s — the chip has Armv8.2 dotprod, so SME2 and i8mm paths are out. That single choice was worth 2.06× prefill throughput over the generic Q4_K_M kernels. Which is why the quantization story is short: Q4_0 + KleidiAI, measured, done.
Killing TTFT, one measured win at a time
Time to first token, cold session — four wins on one curve
01
Tool relevance filtering
59.6 s → 13.4 s
Only always-on plus intent-matched tools stay; the prompt sheds ~1,700 tokens.
02
Slim tool schemas
part of the same cut
Types and required lists kept, parameter descriptions dropped.
03
Context 8192 → 4096
13.4 s → 4.4 s
KV cache halved; decode catches up with the standalone benchmark.
04
Native KV prefix cache
4.4 s → 1.7 s
Only the new suffix is decoded, every turn after the first.
On a phone, TTFT is the experience. A 60-second wait feels broken; four seconds feels like a native app. Each of the four wins above was measured in isolation on the same session type before the next one was attempted.
1 · Tool relevance filtering.The tool list is the largest single prompt cost. Liquid's docs say it plainly: only include tools relevant to the request. We inject the always-on tools plus intent-matched phone tools, cutting the prompt from ~2,000 tokens of schemas to a few hundred. Prefill is linear in prompt length — this was the 59.6 s → 13.4 s jump.
2 · Slim tool schemas. Every parameter carried a description. For a phone model, x, y, text are self-describing. We kept types and required lists, dropped descriptions.
3 · Context 8192 → 4096. The KV cache is allocated up front; at 8192 it consumed over a gigabyte and pressured decode — the 1.2B measured 7.5 tok/s in-app against 14.2 in the benchmark. Halving the context fixed the gap: 13.2 tok/s, TTFT 4.4 s.
4 · Native KV prefix caching.The engine re-prefilled the whole conversation every turn. We implemented llama-server-style prompt caching in the C++ engine: cache the last prompt's tokens, find the common prefix, decode only the suffix. Turn 2+ TTFT collapsed to ~1.7 s, and the cache grows with the conversation.
Or: why LFM2.5 is special
Prompt caching sounds simple until you hit a Gated DeltaNet. LFM2.5's hybrid architecture keeps a recurrent state alongside the attention KV cache — and trimming it requires rollback snapshots, which llama.cpp gates behind an architecture allowlist that doesn't include LFM2.5. The first attempt silently no-op'd, and every turn after the first failed with a batch-allocation error.
Two fixes landed in our local llama.cpp build: enable the recurrent rollback window (n_rs_seq = 256, sized to the generation cap so memory stays bounded), and a fallback that resets the cache when a trim exceeds the window. The result is a prefix cache that is correct on the first try and fast on every turn.
Same session · four turns
Pushing tokens/sec on a shared memory bus
Decode is bandwidth-bound, so the levers are: read the weights as few times as possible, and don't waste the A78s. The KV cache uses Q8_0 quantized keys and values to cut attention traffic, and threads are split by phase: 4 for decode, 8 for prefill.
The split came from a regression the benchmark instrument caught. Moving to 8 decode threads looked reasonable on paper — more cores, more parallelism — but decode is a memory-bandwidth-bound GEMV: 7 active threads contending for the same DRAM bus measured 4.07 tok/s, a 71% collapse from 14.2. Prefill is the opposite: a compute-bound GEMM that scaled to 67 tok/s at 8 threads vs 45.7 at 4. The benchmark made the mistake visible; the split fixed both maxima at once. The lesson: on a shared-memory phone, thread count is not monotonic.
The next step was speculative decoding: the 230M drafts tokens, the 1.2B verifies them in one batched pass. We wired the full DSpark-style loop into the native engine — draft context mirroring the target KV, in-flight token bookkeeping, hybrid-cache rollback for partial trims. The measured result was an honest negative: the 230M is a different model family (a data-extraction model, not a chat model), so it agrees with the 1.2B only 20–39% of the time, and the per-round overhead made decode slower (3.7–5.3 tok/s vs 14 single-model). It stays in the codebase behind a flag, with the numbers documented — a measured negative result beats a claimed one.
Baseline
7.5
11 tools · 8192 ctx
4096 ctx
13.2
+76%
Warm
15.6
+18%final
Decode throughput journey · tokens/sec
What it adds up to
| Metric | Baseline | Final | Δ |
|---|---|---|---|
| Time to first token (turn 1) | 59.6 s | 4.4 s | 13.5× |
| Time to first token (turn 2+) | 59.6 s | 1.7 s | 34× |
| Prefill throughput (pp128) | 45.7 tok/s | 67 tok/s | 1.47× |
| Decode throughput (tg64) | 7.5 tok/s | 14.7 tok/s | 1.96× |
| KV cache memory | ~1 GB @ 8192 | ~250 MB @ 4096 | 4× less |
| Prompt tokens (tools) | ~2,000 | ~300 | 6.5× less |
| Model footprint | — | 664 MB | fits 2.2 GB free |
More important than any single number: everything runs inside one Android app process on a $150 phone, fully offline, with per-session telemetry measured from real runs — tokens/sec, TTFT, RAM available, battery. The agent loops, calls tools (todo, shell, phone control), and streams its thinking live through a tiny embedded HTTP server and a hyper-minimal PWA.
The full framework — Kotlin harness, native llama.cpp integration, PWA, benchmark instrument, and this optimization log — is open source on GitHub. Known issues are tracked there too, including the session-API quirks and message-rendering edge cases we hit along the way.
Read the source