# An $8 chip just ran a 28.9M-parameter LLM. What else can it do?

URL: https://interfaze.ai/blog/an-8-chip-just-ran-a-28-9m-parameter-llm-what-else-can-it-do

A developer named Slava S. (slvDev) got a 28.9-million-parameter language model running on an ESP32-S3, a microcontroller that costs about $8. It runs fully offline at roughly 9.5 tokens per second and draws current in the tens-of-milliamps range.

## The numbers

| Spec              | Value                                                 |
| ----------------- | ----------------------------------------------------- |
| **Parameters**    | 28.9M stored (25M in a flash lookup table)            |
| **Chip**          | ESP32-S3, ~$8, 512 KB SRAM / 8 MB PSRAM / 16 MB flash |
| **Speed**         | ~9.5 tok/s end to end                                 |
| **Model size**    | 14.9 MB at 4-bit quantization                         |
| **Connectivity**  | None, everything runs on the device                   |
| **Training data** | Microsoft TinyStories (synthetic short stories)       |
| **Repository**    | [slvDev/esp32-ai](https://github.com/slvDev/esp32-ai) |

## How it fits

The ESP32-S3 has 512 KB of fast SRAM. A 28.9M-parameter model should not fit.

It does because most of those parameters are not used for computation. They sit in a giant embedding table that gets read, not multiplied against.

Google's **Per-Layer Embeddings** (PLE), the technique behind the Gemma 3n and Gemma 4 model families, lets you keep that table somewhere slow and just pull the rows you need. slvDev put the 25-million-row table in flash and memory-mapped it.

Each token reads about six rows, roughly 450 bytes. The dense "thinking" core (~559K parameters) stays in fast SRAM where it belongs.

<br />

```mermaid
flowchart LR
    SRAM["SRAM (512 KB)\n~559K dense core\n273 KB at 4-bit"] --> COMPUTE["Token computation\nevery token"]
    PSRAM["PSRAM (8 MB)\n3.1M output head\nKV cache + buffers"] --> COMPUTE
    FLASH["Flash (16 MB)\n25M PLE table\n~12 MB"] -->|"~6 rows / token\n~450 bytes"| COMPUTE
    COMPUTE --> TOKEN["Next token\n~9.5 tok/s"]
```

The lookup table is basically free to access. It sits in flash and gets sampled a few rows at a time, while the expensive math only touches the ~559K core in SRAM. You get the quality of 28.9M stored parameters while paying, on the hot path, for about half a million.

## What people are building

The previous record for an LLM on this class of chip was about 260K parameters, so this is roughly 100x more. The original GPT-1 had 117M parameters. This model holds about a quarter of that on a chip you can buy for the price of two coffees.

Fair reaction. But the model's own author is pretty blunt about what it cannot do, and that matters more than the headline number.

## What the model cannot do

slvDev's [`RESULTS.md`](https://github.com/slvDev/esp32-ai/blob/main/RESULTS.md) says it plainly: quote the 28.9M figure as "parameters resident via a memory-hierarchy split," never as a capability multiple. The model was trained on Microsoft's [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset, short synthetic stories simple enough that a small model can learn to write them coherently.

It will not answer questions, follow instructions, write code, or recall facts. The dense core doing the actual reasoning is still ~559K parameters, and the flash trick does nothing to change that ceiling. It makes coherent short stories better, not the model smarter.

That is the catch with running a general-purpose LLM on a microcontroller. The parameter count looks good on paper, but the intelligence bottleneck is the dense core, and 512 KB of SRAM puts a hard cap on how big that core can be.

## What actually works on-device today

esp32-ai is a cool technical demo. But a different class of model is already shipping in production on the same chip: small, specialized models built for exactly one job.

| Model                 | Task                        | Size                          | Latency on ESP32-S3  | Notes                                                             |
| --------------------- | --------------------------- | ----------------------------- | -------------------- | ----------------------------------------------------------------- |
| microWakeWord         | Wake word detection         | ~50-60 KB INT8 TFLite         | Real-time, streaming | Custom wake words, used in Home Assistant / ESPHome               |
| ESP-SR WakeNet        | Wake word detection         | Proprietary, optimized for S3 | Real-time            | Espressif's own engine, supports "Alexa", custom phrases          |
| ESP-SR MultiNet       | Offline command recognition | Proprietary                   | Real-time            | Up to 200 speech commands (ESP32-S3), no cloud                    |
| TFLM person detection | Person yes/no (96x96 RGB)   | 250 KB INT8                   | 54 ms (ESP-NN)       | TensorFlow Lite Micro with Espressif ESP-NN optimizations         |
| ESPDet-Pico           | Object detection            | 0.36M params                  | >7 FPS at 224x224    | Espressif's YOLOv11-based, single-class competitive with YOLOv11n |
| VAD (edge, INT8)      | Voice activity detection    | ~11 KB INT8                   | 4 ms                 | Binary speech/silence on 30 ms windows                            |
| Keyword spotting      | 8-word recognition          | ~31 KB INT8                   | 12 ms                | MFCC input, simple CNN                                            |

These run in single-digit milliseconds and do their one job well. A 50 KB wake word model will beat a 28.9M LLM at wake word detection every time, because it was built for that and nothing else.

Which raises the more useful question: what should run locally, and what should get sent to a bigger model?

## The routing pattern: local classify, selective escalation

For a real device, you want a pipeline. Small specialist models do what they are good at on-chip.

When something falls outside their scope, a lightweight classifier decides: handle locally or escalate? If it escalates, the device ships the raw capture to a gateway that calls a full model like Interfaze.

<br />

```mermaid
flowchart TD
    INPUT["Sensor input\naudio / image / text"] --> LOCAL["On-device specialist\nwake word, VAD, person detect"]
    LOCAL -->|"Handled locally"| DONE["Local action\nrelay, LED, display"]
    LOCAL -->|"Needs reasoning"| CLASSIFY["Lightweight classifier\nlocal vs escalate"]
    CLASSIFY -->|"Local"| DONE
    CLASSIFY -->|"Escalate"| GATEWAY["Wi-Fi/BLE gateway"]
    GATEWAY --> INTERFAZE["Interfaze API\nOCR, STT, object detection,\nstructured output"]
    INTERFAZE --> RESPONSE["Structured result\nback to device"]
```

The classifier does not need to be big. model2vec's `potion-base-8M` (7.5M params, ~8 MB on disk) classifies intents at roughly 92% the accuracy of `all-MiniLM-L6-v2` (22.7M params, ~80 MB) while running tens of thousands of sentences per second on CPU.

On a Raspberry Pi or a beefier MCU with Wi-Fi, that classifier runs in microseconds. On the ESP32 itself, you often do not even need an embedding model: MultiNet's fixed command set or a simple decision tree handles routing for most cases.

If the task maps to a local specialist, run it locally. Otherwise, ship the raw image, audio clip, or sensor reading to the cloud.

## Escalation in practice: ESP32 to Interfaze

The ESP32 grabs the data (an image, an audio clip) and sends it to a gateway over Wi-Fi or BLE. The gateway forwards it to Interfaze and gets back structured JSON plus metadata: bounding boxes, confidence scores, transcription chunks.

**ESP32-side: capture and forward**

```cpp
#include <WiFi.h>
#include <HTTPClient.h>

void escalateToGateway(const uint8_t* imageData, size_t imageLen) {
    HTTPClient http;
    http.begin("http://192.168.1.100:3000/escalate");
    http.addHeader("Content-Type", "application/octet-stream");
    int code = http.POST(const_cast<uint8_t*>(imageData), imageLen);
    if (code == 200) {
        String response = http.getString();
        // Parse structured JSON response, act on it locally
        handleEscalationResult(response);
    }
    http.end();
}
```

**Gateway-side: call Interfaze for object detection**

The gateway receives the raw image and sends it to Interfaze. This example detects objects on a construction site image using structured output.

**Interfaze SDK · typescript**

```typescript
import { Interfaze, responseFormat } from "interfaze";
import { z } from "zod";

const interfaze = new Interfaze({
	apiKey: "<INTERFAZE_API_KEY>",
});

const DetectionSchema = z.object({
	objects: z.array(
		z.object({
			name: z.string().describe("describe the object in the image"),
			top_left_x: z.number(),
			top_left_y: z.number(),
			bottom_right_x: z.number(),
			bottom_right_y: z.number(),
		})
	),
});

const response = await interfaze.chat.completions.create({
	messages: [
		{
			role: "user",
			content: [
				{ type: "text", text: "Detect all objects and equipment on this site" },
				{
					type: "image_url",
					image_url: {
						url: "https://r2public.jigsawstack.com/interfaze/examples/construction.png",
					},
				},
			],
		},
	],
	response_format: responseFormat(z.toJSONSchema(DetectionSchema), "detection_schema"),
});

const result = JSON.parse(response.choices[0]?.message.content ?? "{}");
console.log("Detected objects:", result.objects);

console.log("Bounding boxes + confidence:", response.precontext?.[0]?.result);
```

**Vercel AI SDK · typescript**

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateText, Output } from "ai";
import { z } from "zod";

const interfaze = createOpenAI({
	apiKey: "<INTERFAZE_API_KEY>",
	baseURL: "https://api.interfaze.ai/v1",
});

const DetectionSchema = z.object({
	objects: z.array(
		z.object({
			name: z.string().describe("describe the object in the image"),
			top_left_x: z.number(),
			top_left_y: z.number(),
			bottom_right_x: z.number(),
			bottom_right_y: z.number(),
		})
	),
});

const { output, response } = await generateText({
	model: interfaze.chat("interfaze-beta"),
	output: Output.object({ schema: DetectionSchema }),
	messages: [
		{
			role: "user",
			content: [
				{ type: "text", text: "Detect all objects and equipment on this site" },
				{
					type: "image",
					mediaType: "image/png",
					image: "https://r2public.jigsawstack.com/interfaze/examples/construction.png",
				},
			],
		},
	],
});

console.log("Detected objects:", output?.objects);

//@ts-expect-error precontext is not typed
const precontext = response.body?.precontext;
console.log("Bounding boxes + confidence:", precontext?.[0]?.result);
```

**LangChain SDK · typescript**

```typescript
import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";

const interfaze = new ChatOpenAI({
	configuration: {
		baseURL: "https://api.interfaze.ai/v1",
	},
	apiKey: "<INTERFAZE_API_KEY>",
	model: "interfaze-beta",
});

const DetectionSchema = z.object({
	objects: z.array(
		z.object({
			name: z.string().describe("describe the object in the image"),
			top_left_x: z.number(),
			top_left_y: z.number(),
			bottom_right_x: z.number(),
			bottom_right_y: z.number(),
		})
	),
});

const structuredModel = interfaze.withStructuredOutput(DetectionSchema);

const response = await structuredModel.invoke([
	{
		role: "user",
		content: [
			{ type: "text", text: "Detect all objects and equipment on this site" },
			{
				type: "image_url",
				image_url: {
					url: "https://r2public.jigsawstack.com/interfaze/examples/construction.png",
				},
			},
		],
	},
]);

console.log("Detected objects:", response.objects);
```

**Interfaze SDK · python**

```python
from interfaze import Interfaze
from pydantic import BaseModel, Field
from typing import List

interfaze = Interfaze(
	api_key="<INTERFAZE_API_KEY>",
)

class DetectedObject(BaseModel):
    name: str = Field(..., description="describe the object in the image")
    top_left_x: float
    top_left_y: float
    bottom_right_x: float
    bottom_right_y: float

class DetectionSchema(BaseModel):
    objects: List[DetectedObject]

response = interfaze.chat.completions.parse(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Detect all objects and equipment on this site"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://r2public.jigsawstack.com/interfaze/examples/construction.png"
                    },
                },
            ],
        }
    ],
    response_format=DetectionSchema,
)

print("Detected objects:", response.choices[0].message.parsed)

print("Bounding boxes + confidence:", response.precontext[0].result if response.precontext else None)
```

**LangChain SDK · python**

```python
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from pydantic import BaseModel, Field
from typing import List

interfaze = ChatOpenAI(
	base_url="https://api.interfaze.ai/v1",
	api_key="<INTERFAZE_API_KEY>",
	model="interfaze-beta",
)

class DetectedObject(BaseModel):
    name: str = Field(..., description="describe the object in the image")
    top_left_x: float
    top_left_y: float
    bottom_right_x: float
    bottom_right_y: float

class DetectionSchema(BaseModel):
    objects: List[DetectedObject]

structured_llm = interfaze.with_structured_output(DetectionSchema)

response = structured_llm.invoke([
    HumanMessage(
        content=[
            {"type": "text", "text": "Detect all objects and equipment on this site"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://r2public.jigsawstack.com/interfaze/examples/construction.png"
                },
            },
        ]
    )
])

print("Detected objects:", response.objects)
```

Interfaze puts the structured JSON in `content` and the raw CNN-grade metadata (bounding boxes, confidence scores) in `precontext`. No second parsing step. The [object detection docs](https://interfaze.ai/docs/vision/object-detection) have the full response shape.

Same idea works for audio. The ESP32 runs VAD, records the speech segment, and the gateway sends it to Interfaze for transcription. See the [speech-to-text docs](https://interfaze.ai/docs/audio/speech-to-text).

## Use cases: what stays local, what escalates

| Use case                 | On-device (ESP32)                                              | Escalate to Interfaze                                                |
| ------------------------ | -------------------------------------------------------------- | -------------------------------------------------------------------- |
| **Retail shelf audit**   | Person detection triggers camera capture                       | Full object detection + OCR on shelf labels for stock counts         |
| **Wildlife camera trap** | Motion/person detection filters false triggers                 | Species classification on real detections, structured metadata       |
| **Field inspection**     | Wake word activates recording, VAD segments speech             | Transcription + structured extraction of defect reports              |
| **Industrial anomaly**   | Vibration/sensor anomaly detection (8-14 KB model)             | Image analysis of flagged equipment, damage classification           |
| **Offline voice kiosk**  | Wake word + command recognition (MultiNet, up to 200 commands) | Complex queries forwarded to full model for natural language answers |
| **Agricultural monitor** | Soil sensor anomaly detection, local threshold alerts          | Periodic image analysis of crop health, pest identification          |

Same pattern every time: cheap filtering on the device, expensive analysis in the cloud, and only when the local model says something is worth looking at.

## The cost and latency tradeoff

| Approach                                         | Hardware cost          | Per-inference cost          | Latency                                   | Privacy               | Capability                                          |
| ------------------------------------------------ | ---------------------- | --------------------------- | ----------------------------------------- | --------------------- | --------------------------------------------------- |
| **Everything cloud**                             | ~$0 (any Wi-Fi device) | API cost per call           | Network round-trip + inference            | Data leaves device    | Full model reasoning                                |
| **Everything on-chip**                           | ~$8 (ESP32-S3)         | $0                          | 4-95 ms depending on model                | Complete              | Limited to small specialist models                  |
| **Hybrid (local filter + Interfaze escalation)** | ~$8 (ESP32-S3)         | API cost only on escalation | Local: ms, escalated: network + inference | Most data stays local | Full reasoning when needed, fast local for the rest |

The hybrid row is where the economics get interesting. Most sensor events (empty frames, silence, normal readings) never leave the device, so you only pay API costs on the events that actually need a big model.

## Where esp32-ai fits in this picture

slvDev's project is a proof of concept, not a production architecture. What it proves is that Per-Layer Embeddings can map onto a microcontroller's memory hierarchy: you can park 25M parameters in flash and read them cheaply.

But a TinyStories model with a 559K dense core is not something you would ship. In a real product, the on-device slot goes to a purpose-built specialist, and anything that needs reasoning, structured output, or multimodal understanding goes to a real model over the network.

Cramming bigger LLMs onto smaller chips is a fun stunt. The useful version is simpler: tiny specialist models that stay on and stay local, paired with a capable API for the moments you actually need intelligence.

## Takeaway

- **esp32-ai** proves Per-Layer Embeddings work on an $8 ESP32-S3 by mapping 25M parameters into flash. The dense core is still ~559K parameters, and the model only writes short stories.
- **Specialized edge models** (wake word, VAD, person detection, command recognition) already ship on the same chip, run in single-digit milliseconds, and are the practical choice for production.
- **Local specialist + classifier + cloud escalation** keeps most data on-device and only burns API cost on the events the local layer flags as worth analyzing.
- [Interfaze](https://interfaze.ai/docs) fits the escalation slot: structured output, bounding boxes, confidence scores, OCR, STT, and object detection in one call.

## Links

- [slvDev/esp32-ai repository](https://github.com/slvDev/esp32-ai)
- [TinyStories paper (arXiv:2305.07759)](https://arxiv.org/abs/2305.07759)
- [microWakeWord project](https://microwakeword.com/)
- [Espressif ESP-SR (WakeNet, MultiNet, VADNet)](https://github.com/espressif/esp-sr)
- [Interfaze object detection docs](https://interfaze.ai/docs/vision/object-detection)
- [Interfaze speech-to-text docs](https://interfaze.ai/docs/audio/speech-to-text)
- [Interfaze structured output docs](https://interfaze.ai/docs/structured-output)
- [Interfaze run-task mode](https://interfaze.ai/docs/run-tasks)