perf: per-layer tempfile round trip and autocommit SQLite dominate MLX inference CPU #12

New issue

Open

opened 2026-10-14 00:58:28 +00:00 by peterswimm · 0 comments

peterswimm commented

2026-10-14 00:58:28 +00:00

Owner

Measured from Toilville/toilville_rituals while profiling ritual_engine for #355 there (harness: toilville_rituals PR #548, tool/profiling/). The setup was v0.1.6 (3fa4564) under ritual_engine's mlx_smoke test: one 64-token greedy Qwen3-0.6B-4bit run on an M5 Max, release build, Time Profiler at 1 ms.

What the profile shows

78% of the process's CPU samples have a forge_kv_cache frame on the stack. The heaviest parts:

Share of samples	Where
42.9%	`sqlite3_step`
26.3%	`ForgeKvSession::enqueue_layer_kv`
14.7%	`unixSync` (SQLite fsync)
14.1%	`mlx::core::save_safetensors`
1.5%	Model load (`load_qwen3`), for comparison

Charged to leaf frames, about three-quarters of the samples are kernel file syscalls: open, pwrite, fsync, unlink and unlinkat.

Why

Qwen3-0.6B has 28 layers, so a 64-token run enqueues about 1,800 layer deltas. Each delta goes through three steps.

On the generating thread, encode_kv_delta (mlx_cache.rs) creates a tempfile, save_safetensors into it, reads it back with std::fs::read, and deletes it. That is 4–5 syscalls plus a file create and unlink per layer per token. enqueue_layer_kv is documented as non-blocking, but this happens before the channel send.
The worker, in allocate_block (store.rs), writes the bytes again to the tensors dir with std::fs::write.
The worker then makes two autocommit inserts, insert_block_row and record_event. The connection uses SQLite's default rollback journal, and nothing sets journal_mode or synchronous. So each insert creates, syncs and deletes a -journal file: about 3,600 synced transactions per 64-token run.

Suggested direction

None of this has been tried yet; each item is a starting point, not a measured fix.

Serialise the delta in memory (mlx's safetensors writer can target a buffer, or write the header and raw as_slice bytes directly). This takes the tempfile round trip off the generating thread.
Open the store with PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;.
Batch each token's 28 layer jobs (or the whole run) into one transaction in the worker.
Re-measure with the same harness, and report tokens per second before and after.

Measured from Toilville/toilville_rituals while profiling `ritual_engine` for #355 there (harness: toilville_rituals PR #548, `tool/profiling/`). The setup was v0.1.6 (`3fa4564`) under `ritual_engine`'s `mlx_smoke` test: one 64-token greedy Qwen3-0.6B-4bit run on an M5 Max, release build, Time Profiler at 1 ms. ## What the profile shows 78% of the process's CPU samples have a `forge_kv_cache` frame on the stack. The heaviest parts: | Share of samples | Where | |---|---| | 42.9% | `sqlite3_step` | | 26.3% | `ForgeKvSession::enqueue_layer_kv` | | 14.7% | `unixSync` (SQLite fsync) | | 14.1% | `mlx::core::save_safetensors` | | 1.5% | Model load (`load_qwen3`), for comparison | Charged to leaf frames, about three-quarters of the samples are kernel file syscalls: `open`, `pwrite`, `fsync`, `unlink` and `unlinkat`. ## Why Qwen3-0.6B has 28 layers, so a 64-token run enqueues about 1,800 layer deltas. Each delta goes through three steps. 1. **On the generating thread**, `encode_kv_delta` (`mlx_cache.rs`) creates a tempfile, `save_safetensors` into it, reads it back with `std::fs::read`, and deletes it. That is 4–5 syscalls plus a file create and unlink per layer per token. `enqueue_layer_kv` is documented as non-blocking, but this happens before the channel send. 2. **The worker**, in `allocate_block` (`store.rs`), writes the bytes again to the tensors dir with `std::fs::write`. 3. **The worker then makes two autocommit inserts**, `insert_block_row` and `record_event`. The connection uses SQLite's default rollback journal, and nothing sets `journal_mode` or `synchronous`. So each insert creates, syncs and deletes a `-journal` file: about 3,600 synced transactions per 64-token run. ## Suggested direction None of this has been tried yet; each item is a starting point, not a measured fix. - [ ] Serialise the delta in memory (mlx's safetensors writer can target a buffer, or write the header and raw `as_slice` bytes directly). This takes the tempfile round trip off the generating thread. - [ ] Open the store with `PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;`. - [ ] Batch each token's 28 layer jobs (or the whole run) into one transaction in the worker. - [ ] Re-measure with the same harness, and report tokens per second before and after.