Benchmarks
Measured, with the conditions attached.
On-chip timings are from an ESP32-S3 (QFN56 rev0.2), 240 MHz, 16 MB flash, 8 MB PSRAM. Laptop timings name the runtime.
Time per decision on an ESP32-S3
| Model | Size | Time per decision |
|---|---|---|
| sparrow-pico-rover-v1.1 | ~2 KB | 9.3 µs |
| sparrow-nano-general-v1.1 | 2.2 MB | 3.8 ms |
| sparrow-nano-robot-v1.2 | 1.11 MB | 3.5 ms |
Time per decision on an ESP32-S3
Log scale: pico is in µs, nano in ms. Every bar carries its unit.
First build to today
On the ESP32-S3: int8 SIMD on the vector unit (ESP-NN kernel), weights in PSRAM / internal RAM.
First on-chip build vs today, same board. Today's nano models also use a smaller hidden layer. Each row has its own scale. Board: ESP32-S3 (QFN56 rev0.2), 240 MHz, 16 MB flash, 8 MB PSRAM.
Show as table
| Model | Time per decision on ESP32-S3, first build to today |
|---|---|
| sparrow-pico-rover-v1.1 | 42.9 µs → 9.3 µs |
| sparrow-nano-general-v1.1 | 9.6 ms → 3.8 ms |
| sparrow-nano-robot-v1.2 | 73.5 ms → 3.5 ms |
Speed on a laptop
| Model | Time per question |
|---|---|
| sparrow-pico-rover-v1.1 | 0.18 µs |
| sparrow-nano-general-v1.1 | 0.06 ms (C) |
| sparrow-nano-robot-v1.2 | 0.08 ms (C) |
| sparrow-micro-general-v1.1 | 0.22 ms (C) |
| sparrow-tiny-general-v1.0 | 23 ms (numpy) |
| sparrow-small-general-v1.0 | 64 ms (numpy) |
Time per question on a laptop
Log scale. Values span 0.18 µs to 64 ms.
Against the public 2B
Navigation from sensor numbers
8 / 10
sparrow-nano-robot-v1.2 (1.11 MB)
3 / 10
public 2B Strands Decider
navigation calls from sensor numbers right with a 1.11 MB nano-robot; the public 2-billion-parameter decider gets 3 / 10.
Robot decisions (RobotBench)
36/55
sparrow-nano-robot-v1.2
40/55
public 2B Strands Decider
The 2B is broader on skills; the 1.11 MB nano is better at navigation from sensor numbers. Details in the RobotBench table below.
pico vs its spec's rules
94.0%
sparrow-pico-rover-v1.1
94.2%
its 2B teacher
94.0% vs the spec's rules (its 2B teacher: 94.2%).
General text
sparrow-nano-general-v1.1: JevBench 122/231, MASSIVE intent 0.588.
Yes/no judgement on JevBench is near chance at every tier.
RobotBench
Robot decisions over sensor readings and commands. Released model IDs only, plus the two Strands Decider references.
Right answers out of 55
Linear scale, 0 to 55.
| Model | Right | skill | nav | check | recover | urgency |
|---|---|---|---|---|---|---|
| Strands Decider 4B (teacher) | 46/55 | 14/15 | 6/10 | 10/12 | 10/10 | 6/8 |
| Strands Decider 2B (public) | 40/55 | 15/15 | 3/10 | 10/12 | 10/10 | 2/8 |
| sparrow-nano-robot-v1.2 | 36/55 | 10/15 | 8/10 | 10/12 | 6/10 | 2/8 |
| sparrow-nano-general-v1.1 | 32/55 | 10/15 | 4/10 | 9/12 | 7/10 | 2/8 |
| sparrow-micro-general-v1.1 | 30/55 | 10/15 | 5/10 | 7/12 | 6/10 | 2/8 |
| sparrow-small-general-v1.0 | 24/55 | 7/15 | 2/10 | 7/12 | 5/10 | 3/8 |
| sparrow-tiny-general-v1.0 | 20/55 | 7/15 | 2/10 | 6/12 | 3/10 | 2/8 |
55 items: each item moves the score by one. Small differences between models are within run-to-run noise.
MASSIVE routing
This is where Sparrow is weakest: open-ended language it never trained on.
Domain routing accuracy
Linear scale, 0 to 1.
| Model | Domain (18 options) | Intent within domain |
|---|---|---|
| Strands Decider 2B (public) | 0.86 | 0.871 |
| sparrow-small-general-v1.0 | 0.673 | 0.599 |
| sparrow-micro-general-v1.1 | 0.635 | 0.542 |
| sparrow-nano-robot-v1.2 | 0.518 | 0.522 |
| sparrow-nano-general-v1.1 | 0.5 | 0.588 |
| sparrow-tiny-general-v1.0 | 0.368 | 0.592 |
JevBench
| Model | Right (of 231) |
|---|---|
| sparrow-tiny-general-v1.0 | 123/231 |
| sparrow-nano-general-v1.1 | 122/231 |
| sparrow-small-general-v1.0 | 122/231 |
| sparrow-micro-general-v1.1 | 117/231 |
| sparrow-nano-robot-v1.2 | 115/231 |
Read with care. These are local proxy scores on the public tasks. Yes/no judgement on JevBench is near chance at every tier, so use a full decider or an LLM for yes/no judgement calls.
Same answers on the chip as on the host
On the chip
The C runtime matches the Python runtime to within 1e-5 on the host; on the ESP32-S3, int8 builds agree with the host on 150-155 of 155 decisions.
| Model | Same answer as host |
|---|---|
| sparrow-nano-general-v1.1 | 152/155 |
| sparrow-nano-robot-v1.2 | 150/155 |
python c/tests/parity.py MODEL| Model | Cases | Argmax agree | Max prob diff |
|---|---|---|---|
| sparrow-nano-robot-v1.2 | 286 | 286/286 | 2e-06 |
| sparrow-nano-general-v1.1 | 286 | 286/286 | 1e-06 |
| sparrow-micro-general-v1.1 | 586 | 586/586 | 3e-06 |
Methodology
- RobotBench v0: 55 robot decisions written by hand, separately from the robot training generator.
- MASSIVE: Amazon MASSIVE 1.1 en-US test, 600 utterances, never trained on. Evaluation only (CC BY 4.0).
- JevBench: public tasks, local v1.5 proxy, not leaderboard scores. Evaluation only (MIT).
- Laptop timings: local runs, one CPU core, per question; the runtime (C or numpy) is named next to each value.
- ESP32-S3 timings: measured on the board: ESP32-S3 (QFN56 rev0.2), 240 MHz, 16 MB flash, 8 MB PSRAM.
- Tables are generated by
bench/aggregate.pyfrombench/results/. Model IDs are shown exactly as run.
Where it falls short
Open-ended language it never trained on: well behind the 2B (MASSIVE domain routing 0.37-0.67 across released Sparrow models vs 0.86).
Yes/no judgement on JevBench is near chance at every tier.
pico answers only the questions it was compiled for.
RobotBench was written by hand, separately from the robot training generator. MASSIVE was never trained on, and it is where the gap to the 2B shows.
Run it on your hardware and tell us what you measure.
Share your resultsEvaluation data: Amazon MASSIVE (CC BY 4.0, evaluation only) · JevBench (MIT, evaluation only).