- What: A memory-optimization technique called "streaming experts" for Mixture-of-Experts (MoE) models.
- Hardware: Successfully tested on Apple M2/M4 Max MacBooks and even iPhones.
- Models: Supports massive models including Qwen3.5-397B and the 1-trillion-parameter Kimi K2.5.
- Performance: Achievement of up to 1.7 tokens per second for 1T-parameter models on consumer-grade hardware.
A burgeoning technical breakthrough known as "streaming experts" is reportedly allowing independent developers to run massive artificial intelligence models on consumer hardware that lacks the RAM traditionally required for such tasks. By streaming necessary weights from a device's SSD for each token processed, researchers have demonstrated the ability to run the 1-trillion-parameter Kimi K2.5 model on high-end MacBooks and the 397-billion-parameter Qwen3.5 model on an iPhone.
The 'Streaming Experts' Mechanical Shift
The technique centers on the architecture of Mixture-of-Experts (MoE) models. Unlike traditional dense models where every parameter is activated for every calculation, MoE models utilize only a fraction of their total parameters—referred to as "experts"—for any given token.
Traditionally, even the inactive weights of an MoE model must reside in the system's Random Access Memory (RAM) or Video RAM (VRAM) to ensure high-speed access. However, as reported by technologist Simon Willison and researcher Dan Woods, the "streaming experts" trick bypasses this requirement. Instead of storing the entire model in memory, the system fetches only the specific expert weights needed for a particular token directly from the Solid State Drive (SSD) in real-time.
This approach significantly lowers the barrier to entry for running "frontier-class" models. While SSDs are considerably slower than RAM, the experiment suggests that the trade-off in speed is acceptable for users who prioritize running massive models locally over sheer inference velocity.
From Qwen to Kimi: Scaling Down the Hardware
The rapid evolution of this technique has been documented through a series of experimental milestones over the past week. Developer Dan Woods recently demonstrated Qwen3.5-397B-A17B—a model with 397 billion parameters—running on hardware equipped with only 48GB of RAM. Under normal circumstances, a model of that scale would require several hundred gigabytes of VRAM to function.
The scale of these achievements escalated quickly. According to social media reports from user @seikixtc, the technique was used to run Kimi K2.5, a colossal 1-trillion-parameter model with 32 billion active weights, on an M2 Max MacBook Pro with 96GB of RAM. This was followed by an update from developer Daniel Isaac, who reportedly achieved speeds of approximately 1.7 tokens per second while running Kimi K2.5 on a 128GB M4 Max MacBook.
Perhaps most surprising was the claim from developer @anemll, who demonstrated the 397-billion-parameter Qwen3.5 model running on an iPhone. While the performance was limited to 0.6 tokens per second, the feat represents a significant milestone in mobile AI capabilities, proving that ultra-large models are no longer strictly tethered to massive data centers.
Impact on Developers and the AI Industry
The implications of "streaming experts" are profound for the local LLM (Large Language Model) movement. For the first time ever, individual developers and researchers can experiment with trillion-parameter models without investing in tens of thousands of dollars worth of enterprise-grade GPU clusters.
"This technique has legs," Willison noted, highlighting that tinkerers are currently engaged in "autoresearch loops" to find further optimizations. By shifting the bottleneck from RAM capacity to SSD read speeds, the industry may see a renewed focus on high-speed storage performance for AI tasks.
For users, this means:
- Local Privacy: The ability to run massive, highly capable models on personal devices without sending data to the cloud.
- Cost Efficiency: Massive reduction in the "hardware tax" required to test state-of-the-art models.
- Democratization: Smaller labs and independent researchers can now benchmark and probe models that were previously the exclusive domain of companies like OpenAI or Google.
The punchy reality for the industry is clear: this changes how developers will view hardware limitations, moving the conversation from "how much RAM do I need?" to "how fast is my storage?"
What’s Next for Local Inference
The current results are largely experimental and unverified by official peer-reviewed studies, relying on social media reports and open-source repositories. However, the momentum behind these "autoresearch" efforts suggests that further speed gains are likely.
As Gen 5 SSDs and faster unified memory architectures become standard, the "tokens per second" gap between SSD-streaming and RAM-resident models is expected to narrow. Developers are already looking toward specialized software kernels and more efficient "flash-attention" implementations to squeeze more performance out of the SSD-to-processor pipeline.
If these optimizations continue, the next generation of mobile and desktop operating systems could potentially integrate "streaming expert" layers to provide ultra-intelligent local assistants that were previously thought to be years away from local feasibility.