WASTE — Weight-Aware Streaming Tensor Engine
Kimi K3 — 2.78 trillion parameters — running on a consumer laptop.
$ waste run ~/models/k3.waste ‘What is the capital of Italy?’ waste: no –budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB) The capital of Italy is **Rome**. [16 tokens, 25.78 s, 0.62 tok/s | experts 9038 hit / 14514 miss = 38%]
WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.
Its current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.45 – 0.62 tokens per second. This is not a distilled, pruned, or reduced variant.
WASTE was written for that one model and that one constraint: K3 does not fit in the RAM of current mainstream consumer systems. It is 1.42 TB as published and 982 GB after conversion. But a mixture of experts activates about 4% of itself per token, so almost all of that weight is idle at any instant — and idle weight does not need to be in memory, it needs to be reachable in time. WASTE keeps it on disk in a layout where one expert costs exactly one read, streams what each token actually needs, and spends every remaining byte of RAM on the part that repeats.
Where this stands
The engine is correct: every layer is validated against a PyTorch reference, the final logits agree to 3.6e-06, and the vision tower matches its own oracle to 2.3e-06. It is also slow — half a token per second, twenty-six seconds for the sentence above.
Both of those matter, and the second one should not be read as a disclaimer. We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine: we found none for trillion-scale NVMe streaming, and the best-documented 671B-class recipes assume a server with a terabyte of DDR5. That is a report of what our search turned up rather than a survey — this repository carries no bibliography and no comparison table, so read it as an invitation to send a counter-example, not as a result. The interesting part is not the speed, it is that the whole thing is in the reachable range on a single consumer machine — and that from here the question is engineering rather than feasibility.
Where the levers were is not where they are. The two that looked biggest — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused: this family’s router has no tail to demote, and a cache the machine will not leave resident cannot be bought at any price. What paid instead was never about which bytes to read but when. Overlapping the expert reads with the arithmetic is worth ~1.6x; starting the next layer’s reads on its own router’s guess, one residual early, takes the hit rate from 14% to 38% at no extra bytes at all.
Both of those are exact — the cache statistics and the logits are unchanged — which is the property that makes them shippable rather than tuning. docs/EFFICIENCY.md is the account of how each lever was priced, including the three that were built before being measured and the two that were then taken back out.
What that opens up, concretely: a frontier-scale model that answers with no network, no per-token invoice, and nothing leaving the machine — which is the difference between “you may not send that data to an API” and “run it here”. The format and the engine are not K3-specific in any deep way; K3 is simply the hardest case that exists today, and a model that streams at 2.78T streams comfortably at 48B.
Every number in this document was measured on the commit it is published with, and the ones that were wrong are recorded as wrong in docs/LEARNED.md rather than quietly corrected.
Why the name
Every token answered by a cloud service is paid for twice: once on the invoice, and once in the electricity of a datacenter running a model that would fit — barely, awkwardly, but genuinely — on hardware already sitting on a desk. WASTE means to be the first concrete step toward ending that waste of tokens. The acronym came second.
What you need
Sizes here are powers of two, the way df and the engine both report them: the container is 982 GiB, which a disk vendor would call 1.05 TB.
The RAM floor is what the engine refuses to start below, and it is almost entirely the 27.28 GB resident trunk. Useful throughput starts higher: on a 64 GB machine the engine gives itself a 46 GB budget, of which 17.56 GB is expert cache, and that is the top of the measured curve. A 32 GB machine can technically open the model and will page badly; treat 64 GB as the real requirement.
Storage speed is not a detail. A token reads 17 GB of experts. On the internal SSD that is 12.78 GB/s and the model streams; over a USB enclosure it is 0.94 GB/s and the same token takes thirteen seconds. Convert onto internal NVMe, and use the external disk for the download only.
If a terabyte is not available, the same engine and the same format run Kimi-Linear-48B-A3B-Instruct from a 19 GB container with a 1.87 GB floor, at 10.7 tok/s. That is the good path for trying WASTE out before committing a disk to K3.
What it is
Self-contained. One libwaste.a, one waste binary, nothing at run time beyond libc and pthreads.
Zero dependencies. No BLAS, no ONNX, no Python in the inference path, nothing to install. The Python under tools/ converts models and validates the engine; it never runs alongside it.
Fully embeddable. Twenty-six public functions in src/waste.h: open a model under a RAM ceiling, generate, save the session, close. The CLI is a client of that API and touches nothing private — if the CLI can do it, so can an embedding host.
waste_cfg cfg; waste_cfg_init(&cfg); cfg.ram_budget_bytes = 46ULL << 30; /* a hard ceiling, not a hint; 0 sizes it to this machine */
waste_ctx *ctx; if (waste_open(“/path/to/k3.waste”, &cfg, &ctx) != WASTE_OK) return 1; waste_generate(ctx, ids, n, ¶ms, on_token, user); waste_close(ctx);
The path is the container directory the converter wrote — no ~ expansion here, that is the shell’s job.
How it works
Placement decides the speed
A model is converted once into a .waste container: a JSON manifest, a resident trunk, and one expert bank per layer. Each expert record is 4 KiB-aligned with its gate, up and down matrices adjacent, so routing to an expert costs exactly one pread — not three, not a seek per matrix. The arithmetic was never the bottleneck.
Reads bypass the page cache (F_NOCACHE on macOS, O_DIRECT on Linux, FILE_FLAG_NO_BUFFERING on Windows). That is deliberate: with a container smaller than RAM the kernel would cache everything, and the hit rates measured that way are a fiction that does not survive contact with a 982 GB model.
Every record’s header is checked on the way in — right magic, the expert the index asked for, offsets that fit — so a bank that has been truncated or spliced stops the generation and names the record instead of answering from the wrong bytes. That costs nothing measurable. The record also carries a crc32 over its payload, and checking that is –verify, off by default: it is a pass over every record on every cache miss, about 5% on Kimi-Linear and 1% on K3. Worth it for a container you copied or downloaded and have not read since; not worth it on every token of one you converted yourself. See docs/FORMAT.md.
Reading ahead, and reading before the router has spoken
A layer knows all sixteen of its expert ids the moment its router runs, so the reads go out on their own threads and the arithmetic consumes them as they land instead of blocking on each. That is worth ~1.6x on K3, and the cache statistics are identical to the digit with it on and off — the engine does the same work, it just stops waiting for it.
The next layer’s ids are not known: its router eats a hidden state that does not exist yet. But the router does exist, and it is resident. So at the end of each layer, once its own reads are consumed and the disk is about to go idle through the next layer’s attention, the engine runs layer L+1′s router on layer L’s hidden state and starts fetching the six experts it names. One residual early, that guess is right 92% of the time at rank 1 and 81% over the first six.
It is exact by construction: the real router still decides, and the guess only decides when bytes move. The demand hit rate goes from 14% to 38% and the total bytes read do not change — those records were going to be read anyway. WASTE_LOOKAHEAD=0 turns it off.
The same trick in the prefill path was built and removed. A decode layer claims 16 cache slots so the speculative records survive; a chunk layer claims about 550, evicts them before use, and reads them twice — 6.9% more bytes for no time saved. docs/LEARNED.md §34 – 36.
Three bits per expert weight
Experts are stored as residual vector quantization — three stages of 256-entry codebooks over 8-dimensional vectors, 3.00 bits per weight — and the matrix is never materialized. For each token the engine builds a table of partial dot products, one per codebook entry per vector position, after which every expert row is three table reads and two adds.
The trunk stays at 4 and 8 bits. The model was trained with quantization-aware training on the experts only, so it has no trained tolerance for a squeezed trunk: a 3-bit trunk was built and measured, the cache prediction held, the throughput did not, and the output collapsed.
The cache floor is one token’s working set
The most predictive number in this project. K3 touches 16 experts in each of 92 layers per token: 17.0 GB. Below that, an expert cached for one token is evicted before the next token asks for it, and the hit rate is not low — it is zero.
What crossing it buys has changed, though, and the table below is the first one to show it. Going from a 0% hit rate to 17% is worth about 8% of throughput now — 0.50 to 0.54 — because read-ahead already hides most of the I/O the cache would have saved. The sharp bend in this curve is no longer the climb above the floor; it is the collapse above 46 GB, where the engine stops fitting in the machine.
The hit-rate column predates the router lookahead, which roughly doubles it at every budget — 14% to 38% at 46 GB. It does not move the decode column’s shape, because what collapses the 52 and 58 GB rows is the machine running out of memory and not the cache missing.
Ranges, not measurements, and the width is the finding. Every run behind a row reports cache statistics identical to the digit — the engine does the same work each time — so what varies is the machine, not the engine.
The two rows that fit are tight. 32 and 46 GB reproduce to within a few percent, because the engine’s whole footprint fits with room to spare and nothing has to be taken from anything.
52 GB has no value. Two runs of the default configuration gave 0.04 and 0.15; three more with the trunk wired gave 0.46, 0.19 and 0.03 — seven-fold in the column above and fifteen-fold across both configurations, against 3652 hit / 8124 miss every single time. That budget sits exactly where the engine’s footprint either does or does not fit beside whatever else the machine is holding, and which side it lands on is decided before the process starts. A row that spans 15x is not a slow row; it is a row whose mean would invite a comparison there is nothing to compare.
58 GB is uniformly bad and reproducibly so.
Order still matters, and more than the table shows. Re-run after the 52 and 58 GB rows have driven the machine into paging, 46 GB collapses — 0.02 tok/s in one such run — while again reporting identical counts. Sweep upward, one budget per quiet machine, and treat anything measured after a paging row as void.
Read-ahead made the rows that fit faster and left the others where they were, so the step is larger than when this was first measured: 46 GB went 0.32 to 0.54. Wiring the resident trunk with WASTE_MLOCK=trunk does not move it either — 32 and 46 GB are unchanged, 58 GB stays hopeless, and 52 GB has no value to change. docs/LEARNED.md §30 – 33.
Everything in the memory design exists to get above that line, which is why the engine works to free RAM rather than to save it.
And there is a ceiling on the other side, closer than it looks. Read that table twice: the hit rate climbs all the way down. At 58 GB on a 64 GB machine the cache serves 39% of experts from RAM and the engine is twenty times slower than at 46 GB, where it serves 17%. The engine is inside its budget; the machine is not, so the OS pages out the expert cache, and a “hit” becomes a page fault instead of the disk read the engine was managing.
So the usable window is narrow. It opens at ~46 GB, where the cache finally clears one token’s working set, and it has already closed by 52 — on an otherwise idle machine, with 49 GB free before the run. It is also sharp enough to move under a change that looks unrelated: taking 1.11 GB of embedding table off the resident set fed straight into the cache at a fixed budget, and on the build of the day that was enough to push 58 GB from 0.32 tok/s to 0.04.
So the default does not fill the machine. Expert cache is only worth anything in whole multiples of that working set, and the remainder above a multiple buys a few points of hit rate while pushing the machine towards paging. When it picks a budget for itself the engine steps down a whole working set at a time and takes the largest that fits under seven eighths of RAM: K3 asks for floor + 3× — 80.63 GB — and gets floor + 1× on this laptop, a 46 GB budget and a 17.56 GB cache. That is the top of the curve above, reached with no flag. A 128 GB machine still gets the full 3×.
An earlier version took every byte up to the cap instead, which put a 27 GB cache on this machine — between two budgets measured at 0.11 and 0.04 tok/s. The real lesson is that a cache you do not control is not a cache, and the corollary is that an engine should stop asking for memory before the OS starts taking it back.
Linear attention, and an absorbed KV cache
K3′s attention is a 3:1 hybrid: Kimi Delta Attention, which carries a fixed-size recurrent state instead of a growing KV cache, and gated multi-head latent attention. The MLA layers cache the 512-wide latent rather than expanded per-head keys and values, with kv_b_proj absorbed into the query and the output:
q_nope · (W_kb c) == (W_kbᵀ q_nope) · c Σ_s a_s (W_vb c_s) == W_vb (Σ_s a_s c_s)
Identical logits to 1.2e-05, and 53× less cache: 11.25 GB becomes 0.21 GB at 4K context. It is also what makes long context possible at all — the expanded layout wants 360 GB at 128K tokens, the latent one 7.2.
Performance and memory
MacBook Pro M5 Pro, 64 GB, container on the internal SSD. Every figure was measured on the commit it is published with.
Kimi K3 — 2.78T parameters, 982 GB container
The floor is almost entirely the resident trunk. Useful throughput starts above ~46 GB, where the expert cache finally clears one token’s working set, and is gone again by 52, where the machine starts paging. Below the first line extra RAM buys nothing; above the second it costs, badly. The window is one budget wide on this machine.
The tower is not what an image costs. Encoding 1024 patches takes 15.7 s; the 256 positions it produces then go through the 92 MoE layers like any other token, which is the other 731 s. An image is priced as text of the same length, so the patch budget in vision.json is a real dial: halving the grid halves the prompt.
Kimi-Linear — 48B parameters, 19 GB container
The same engine and the same format, on a model that fits comfortably. This is what WASTE looks like when it is not fighting.
Where the time goes
Decode on K3, 17.32 GB of cache and still cold — 6.7% hit over ten steps, which is the state a fresh prompt starts in:
Reproduce with WASTE_PROFILE=1 WASTE_LOOKAHEAD=0 WASTE_CACHE_MB=17735 ./test_forward MODEL 1008,10484,318,15383,387 out.bin 5. The lookahead is off there on purpose: this is the cold-cache shape, and with it on the hit rate is 38% and the I/O share correspondingly lower. The I/O share also falls as the cache warms, so a long session sits under this either way; the ranking does not change.
The I/O already runs near the hardware limit — 17.0 GB per token at ~9.9 GB/s against the SSD’s measured 12.78 — so it only gets cheaper by happening less often. For a long time that read as which means cache, which means RAM, and it was half right: the other half is when it happens. Overlapping the reads with the arithmetic and starting the next layer’s on its own router’s guess between them cost no RAM at all. What follows is the memory half of that story.
Getting started
git clone https://github.com/sqliteai/waste && cd waste make # libwaste.a, waste, libwastevq make check # 23 pass, 11 skip on a fresh clone
No configure step and no dependency resolution. make check needs no model: it builds a small synthetic container and runs the engine against it. The eleven skips are the checks that need something a clone does not carry — the PyTorch oracle, the round-trip against the source shards, anything driving the CLI with text, since the synthetic container carries no tokenizer, and the K3 checks, which want the container and the release on disk. With both containers present the suite is 36 checks.
Converting Kimi K3
Conversion is the one step that needs Python, and it happens once. The source is moonshotai/Kimi-K3 exactly as published — 96 safetensors shards, 1.42 TB, nothing patched:
# 1. preflight: reachable? how big? does it fit? tools/fetch_weights.sh –dest /Volumes/staging/k3 –dry-run
# 2. download — resumable, safe to kill, safe to re-run tools/fetch_weights.sh –dest /Volumes/staging/k3
# 3. convert into a container uv run –with torch –with safetensors python tools/convert.py \ –src /Volumes/staging/k3 \ –out ~/models/k3.waste –jobs 3
That produces the 982 GB container every number above was measured on. It takes about 4.7 hours with three processes on the M5 Pro (23.7 with the pure-torch encoder — see docs/K3.md), and wants ~1.0 TB free on the target volume. The converter is resumable too: a layer whose bank is already written is skipped, so an interrupted run costs only the layer it was in the middle of.
The download is the part that goes wrong. A 1.42 TB pull over hours will hit dropped connections, CDN 5xx and at least one interrupted run, so every shard resumes mid-file rather than restarting, retries with exponential backoff and jitter, and counts as done only when its size matches Content-Length — recorded in a state file, so a re-run skips finished shards without even a HEAD request. –check re-verifies everything on disk against the remote and downloads nothing (96 shards in 34 s). –repo points it at another model, HF_TOKEN at a gated one. macOS and Linux.
Give –dest a staging disk rather than the volume that will hold the container. The shards are read once, by the converter; the container is read continuously, at every token. On this machine the external enclosure measures 0.94 GB/s against the internal NVMe’s 12.78 — see docs/GATES.md, Gate H — which is the difference between a model that streams and one that stalls.
tools/pipeline.sh chains the whole thing unattended — download, convert, round-trip the container against the source weights, generate, then diff the logits against the PyTorch oracle — and leaves a report next to the container. The same converter handles the other member of the family, Kimi-Linear-48B-A3B-Instruct, into the 19 GB container of the second benchmark; –src is the only thing that changes.
Pre-converted containers are on their way to huggingface.co/sqliteai, at which point this whole section becomes a download and the Python is only needed for models we have not published.
Running it
The container is the directory the converter wrote, so give it that path — ~/models/k3.waste throughout this README:
waste run ~/models/k3.waste “The capital of France is” -n 32 waste chat ~/models/k3.waste # multi-turn, state kept waste eval ~/models/k3.waste “2 + 2 =” –top-k 5 # next-token distribution waste plan ~/models/k3.waste –budget 46G # what fits, what does not echo “prompt” | waste run ~/models/k3.waste # stdin works too
-n is a cap, not a requirement: without it generation stops at the container’s end-of-sequence token or at 128 tokens, whichever comes first. The examples pass it because 128 tokens of K3 is six minutes.
–budget is optional, and leaving it out is the right default rather than a fallback: the engine takes the container’s recommendation, steps it down a whole token working set at a time until it fits under seven eighths of physical RAM, and never goes below the floor — a budget you set explicitly under the floor is refused rather than swapped into. It then says on stderr what it landed on, so the same command on two machines is not silently two different runs:
waste: no –budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)
–verify checks each expert record’s crc32 as it comes off the disk. It is off by default, and that is a throughput decision rather than a claim that containers do not rot: it is a pass over every record on every cache miss, about 5% on Kimi-Linear and about 1% on K3, where the read dominates. Turn it on once for a container you copied, downloaded, or left on a disk you do not trust, and for anything whose wrong answers would be believed; leave it off for one you converted yourself and have been reading since. WASTE_VERIFY=1 in the environment does the same thing, and the server takes –verify as well. Any of them turns it on; none of them turns it off.
What is checked either way: a short read, and a record header that does not describe the expert the bank index asked for. Those are O(1), they cost nothing measurable, and they are what keeps a damaged offset out of the arithmetic — –verify only adds the pass over the payload.
waste –help lists all nine commands. –json makes eval, tokenize, plan, info and bench machine-readable.
Serving it
serve/ is an OpenAI-compatible HTTP server — the second client of the public API, alongside the CLI, reaching the same engine through ctypes rather than keeping a copy of the model code in Python:
make libwaste.dylib # or libwaste.so on Linux python3 -m serve ~/models/k3.waste –port 8000
curl localhost:8000/v1/chat/completions \ -H ‘Content-Type: application/json’ \ -d ‘{“model”:“k3″,“messages”:[{“role”:“user”,“content”:“Why is the sky blue?“}]}’
/v1/chat/completions (streaming and not), /v1/completions, /v1/models, /health. It carries the whole of K3′s prompt format, not the four-string subset a container’s chat.json can hold: tool definitions and tool results, typed call arguments, JSON response schemas, tool_choice, the think channel and thinking_effort, and images — plus the parser that reads the reply back into reasoning, answer and tool_calls. Stdlib only.
The prompt renderer is a port of encoding_k3.py from the release, and the test suite checks it against that file segment for segment on a corpus of 38 conversations whenever the weights directory is on disk. docs/SERVE.md is the reference.
Images
K3 is multimodal — a 401M ViT, 27 layers, patch 14 — and so is the engine. –image attaches a picture; repeat it for several: