Molt

The three bad options

On a phone or a laptop the model shares memory with everything else. When the camera app launches, the OS wants memory back now. An inference process conventionally has three choices, and all three are bad:

• Hold the big model. The process is reclaimed. The answer is lost mid-sentence.
• Always run the small model. You survive — and pay for it on every request, including the overwhelming majority where there was no pressure at all.
• Switch models and restart the request. You survive, and the user watches a multi-second stall while the prompt is re-read.

Molt adds a fourth: change the model between two tokens, and take the attention state with you.

Why per-request model selection cannot fix this

Every conventional elastic-serving stack picks a model per request. That is useless here, and the reason is structural rather than an implementation detail: a long answer takes tens of seconds, an app launch takes one, so the spike lands inside the answer and the next request boundary is far away in the future. Between the two, the process is holding a footprint the OS has already decided it cannot have.

The ladder

Three rungs of one model family, sharing a tokenizer. Two of the three transplant routes are structurally different problems, which is the point of this particular ladder:

The pressure trace drops the budget to 2,002 MiB — well below what tier0 needs (5,889 MiB) — and it does so in two steps, so every rung of the ladder is exercised in a single run. The spike is timed to land mid-answer on purpose.

Results

The trade-off, stated plainly

Condition C re-reads the prompt on the new model, so its handoff divergence is the irreducible floor: two different models simply disagree by that much about the next token. Molt does not reach that floor.

So the honest summary is a purchase, not a free lunch: a 4.0× cheaper switch costs 5.7× more distribution discontinuity at the seam than a full re-prefill would. Most of that gap is the value projection, which is the weakest map in the system.

What a switch costs

Carrying the cache versus re-reading the prompt, both rungs kept warm so the comparison is about the KV work rather than about model loading, which both strategies pay identically.

Ablations

Each mechanism removed on its own, same trace, same prompts. Reported as measured, including where a mechanism did not pay for itself — an ablation table that only ever confirms the design is not an ablation table.

How the cache crosses

HuggingFace caches keys after RoPE. Two rungs with different head_dim rotate by different angles, so no position-independent matrix can map one cache onto the other — the required map would depend on each token's absolute position. Molt therefore sandwiches the learned matrix between an un-rotation at the source's angles and a re-rotation at the destination's. That one detail is what makes a cross-size transplant well-posed at all.

Three mechanisms, fitted offline by closed-form ridge regression on a few thousand tokens of generic text:

1. A learned linear projection per destination layer, inside the RoPE sandwich, plus a depth remap.
2. A diagonal scale re-alignment when the rungs share weights and differ only in precision.
3. Selective top-k recompute: the destination's last few layers are recomputed natively from a projected boundary hidden state, rather than projected. Verified bit-exact against a full forward pass.

Held-out fit residuals (relative RMS, on windows the map never saw):

The value map is the weak one, and most of the residual post-migration quality gap lives there.

Three tenants, one budget, nobody killed

A foreground conversation and two background batch jobs share one moving budget. The scheduler has four levers, applied synchronously, before anyone takes a step, in order of increasing harm: defer admission, demote a rung, park the weights while keeping the cache, and finally shed the oldest context. Terminating is not on the list.

Max overshoot is max over time of (usage − budget), sampled after enforcement at every scheduling round. Negative means the budget was never exceeded.

It runs as a service

The prototype ships a streaming server whose elasticity is visible on the wire: every token event names the rung that produced it, and a switch arrives as its own event rather than as a surprise. Below is a real session — not a mock-up — with the budget cut to 2651 MiB eight tokens into the answer.

[start on tier0 · 17 prompt tokens · TTFT 3122 ms]
 An operating system reclaims memory from background
[budget cut to 2651 MiB]
 processes
⇆ tier0 → tier1  (molt, 260 ms, 79% FLOPs saved, pressure 0.81)
 to free up resources and improve system performance. When a process is
 no longer actively using memory, the operating system can reclaim that
 memory for other processes or applications
[done: 40 tokens · rungs used tier0, tier1 · 1 migration costing 260 ms · killed=False]

The sentence “…reclaims memory from background processes to free up resources…” is written by two different models. The word before the switch and the word after it are separated by a 260 ms migration and nothing else — no re-prefill, no restart, no dropped connection.

What this does not show

Stated plainly, because a prototype that hides its edges is worth less than one that marks them.

• The INT8 rung buys memory, not speed. bitsandbytes is CUDA-only, so the portable fallback dequantises on use: prefill is slightly faster than fp32, single-token decode is roughly 4× slower. A tuned kernel removes this; the memory result stands either way.
• The value projection is weak compared with the key projection, and up-shifts (expanding a cache) are harder than down-shifts. Per-head or low-rank non-linear maps are the obvious next step.
• Cross-family migration is not supported. Every rung must share a tokenizer; the loader refuses otherwise.
• The budget is simulated by default for reproducibility. A real allocating child process is available and gives the same qualitative result, noisier.
• Server-side cache sharing and weight streaming from flash are deliberately out of scope: this is about on-device multi-tenancy and mid-generation migration.
• Three metrics were discarded during the work for measuring nothing — step-to-step JSD (saturated at ln 2 on real text), judge perplexity alone (rewards degenerate repetition), and a single-needle recall task (every condition scored 1.00, including the ones that were killed). The replacements are what the table reports.

Molt · elastic on-device inference · research prototype
Every number on this page is read directly from the benchmark's own output; the page is generated, not transcribed.
Ladder qwen2.5-1.5b-ladder · cpu · trace spike_mid_answer