Deltafin: Local MoE Execution and SSD-Streaming of Frontier Models

Deltafin: Local MoE Execution and SSD-Streaming of Frontier Models

The Deltafin project represents a major technical breakthrough in the local-first, privacy-focused execution of massive Mixture-of-Experts (MoE) models. By streaming expert weights directly from consumer-grade solid-state drives (SSDs) on Apple Silicon, Deltafin enables the execution of unpruned frontier-class models without requiring multi-million dollar data center infrastructure.

1. The Local Memory Constraint for Frontier Models

Frontier-grade models like Moonshot's Kimi K3 are incredibly capable but carry an immense computational footprint. Kimi K3 is a 2.8-trillion-parameter MoE model with approximately 1.45 TB of expert weights.

• Infrastructure Requirements: Running K3 in a traditional datacenter typically requires a cluster of 16 high-end nodes with roughly 4.8 TB of aggregate VRAM (costing upwards of $2 million).
• The Compromise of Quantization: Most existing attempts to run massive models locally rely on aggressive quantization (e.g., compressing expert weights down to 3 bits). While this allows the model to fit into system memory, it degrades model quality and reasoning capability in unpredictable ways.

2. Deltafin's Core Technology: SSD Expert Streaming

Deltafin bypasses the memory bottleneck entirely through a unique Rust-based architecture that streams untouched expert weights on-demand from disk.

• Zero Expert Pruning: Deltafin preserves every single expert byte exactly as Moonshot shipped it (at MXFP4 precision). The model's 16 routed experts decide every token, ensuring identical output quality to the cloud-hosted frontier model.
• SSD Streaming Architecture: Rather than loading the entire 1.45 TB model into RAM, Deltafin streams individual experts (stored as 17.5 MB files) from disk on a per-token basis using direct file reads (pread with F_NOCACHE).
• Hardware Setup: The M5 Max MacBook Pro setup utilizes 128 GB of unified memory for the resident attention trunk (50 GB) and streams expert weights from a striped array of four high-speed SSDs (utilizing three Thunderbolt 5 enclosures plus the internal SSD).
• Evidence: The Deltafin repository states: "Deltafin is a single native binary that runs full Kimi K3. Nothing pruned. Nothing skipped. K3 decides every token... If Deltafin helps make frontier models usable on a $15,000 home setup, instead of a $2,000,000 infrastructure... we believe that is worthwhile progress" (https://github.com/argonautlabsai/deltafin).

3. Performance Metrics and Bottlenecks

While SSD expert streaming makes running a 2.8T model locally possible, it introduces unique performance trade-offs:

• 1 Token Per Second: Once running, Deltafin achieves a steady throughput of 1.00 token/second over a 512-token completion (or 1.13 tokens/second over 128 tokens) on an M5 Max MacBook Pro.
• The Prefill Penalty: The biggest current bottleneck is the "Time to First Token" (TTFT). For a 512-token prompt, the prefill phase takes approximately 6.3 minutes and requires reading 9 TB of data from disk (a 6.2x read-amplification of the 1.45 TB model). This occurs because the prefill engine currently reads each layer's experts once per 64-row pass.
• Drive Scaling Ladder: Throughput scales linearly with the number of SSD devices connected. A single-drive layout achieves only 57% of the four-drive decode rate, while two drives achieve 78%, and three drives reach 92%.
• Evidence: The project's author Argonautlabs explains: "1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens... Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo..." (https://news.ycombinator.com/item?id=49616265).

4. Practical Use Cases for Local Research Orchestrators

At 1 token/second and several minutes of prefill, Deltafin is entirely impractical for interactive chat. However, it represents a highly valuable execution path for unattended, scheduled research jobs:

• Absolute Data Privacy: For highly sensitive financial, legal, or intellectual property reviews, corporate data never leaves the local machine.¹²
• Asynchronous Processing: Tasks like day-end, week-end, or month-end financial reconciliation and report drafting can be scheduled as overnight jobs. The orchestrator can run the local model unattended, compiling the final report to be delivered the next morning.
• Evidence: Argonautlabs explains: "Not useful for chat, agreed... It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through the numbers, flag what doesn't reconcile, draft the report — and there the two things that matter are that the model is good enough to trust with the judgement... and that the data never leaves the machine" (https://news.ycombinator.com/item?id=49616520).

For platforms like Market Map & Positioning: Where Does Hey, Lefty Fit?, supporting local-first execution paths via Deltafin integration provides an unmatched value proposition for enterprise clients who refuse to send proprietary data to cloud APIs.