Create SwiftUI app or simple Go wrapper

Title

Build a Blazing-Fast On-Device Voice Agent with RunAnywhere’s MetalRT and RCLI

Why this matters for builders

RunAnywhere’s MetalRT is a custom Metal inference engine that lets you run LLMs, speech-to-text, and text-to-speech entirely on Apple Silicon with zero framework overhead and no cloud round-trips. It ships the fastest end-to-end voice pipeline on the Mac today (RCLI), beating llama.cpp, MLX, and Ollama across every modality while keeping everything local. This removes the last real excuse for shipping sluggish “AI features” on Mac — latency compounding in chained pipelines is now solvable in <600 ms end-to-end.

When to use it

• You want a fully on-device voice assistant that feels instantaneous
• You are building Mac-native tools, productivity agents, or accessibility apps
• You need sub-200 ms STT + fast LLM decode + sub-200 ms TTS in one binary
• You want to avoid API keys, network, and privacy concerns
• You are targeting M1–M4 Macs and want maximum performance without writing your own shaders

The full process

1. Define the goal

Decide what your voice agent should actually do. Good first scope:

• Push-to-talk or hotword activation
• Local STT → LLM reasoning → TTS spoken response
• Optional local RAG over your notes/docs
• 3–5 custom macOS actions (open app, search Spotlight, control volume, etc.)

Write a one-paragraph spec:

“I want a lightweight menu-bar agent that listens on Cmd+Shift+Space, transcribes speech with MetalRT, asks a local 4B Qwen3 model for an answer (with optional RAG over ~/Notes), then speaks the response using the fastest TTS. All processing stays on-device. Show per-stage latency in a small HUD. Ship as a single Go or Swift binary.”

2. Shape the spec & prompt your coding assistant

Use this starter prompt with Cursor, Claude, or Grok:

You are an expert Mac developer. Build a minimal on-device voice agent using RunAnywhere's rcli and MetalRT.

Requirements:
- Use the official rcli binary (brew install or install.sh)
- Spawn rcli in non-interactive mode or use its library bindings if available
- Record audio from microphone on hotkey (use AVFoundation or SwiftUI + HotKey package)
- Pipe audio to rcli STT
- Take text, optionally augment with local RAG (simple TF-IDF or embed with MetalRT)
- Send to LLM via rcli or direct MetalRT API
- Take LLM output and feed to TTS
- Show live latency numbers for STT/LLM/TTS
- Graceful fallback to llama.cpp if MetalRT is not installed
- Target macOS 14+, universal binary

Output a clean project structure with main.swift or main.go and clear instructions.

3. Scaffold the project

mkdir voice-agent && cd voice-agent
brew tap RunanywhereAI/rcli https://github.com/RunanywhereAI/RCLI.git
brew install rcli
rcli setup   # ~1 GB models, do once

swift package init --type executable

Or use the official install script:

curl -fsSL https://raw.githubusercontent.com/RunanywhereAI/RCLI/main/install.sh | bash

Create a thin wrapper that calls rcli as a subprocess with JSON mode (recommended for reliability). The RCLI repo already contains a fast concurrent pipeline with lock-free ring buffers — study main.go or the Swift parts and reuse the threading model.

4. Implement core loop

Focus on three concurrent threads as described in the RCLI architecture:

• Audio capture thread → ring buffer
• STT worker → MetalRT speech model
• LLM + TTS worker with double-buffered audio output

Key snippet (Go-style pseudocode you can ask your AI to expand):

// Simplified from RCLI pattern
type Pipeline struct {
    stt  *MetalRTSTT
    llm  *MetalRTLLM
    tts  *MetalRTTTS
    audioRing *LockFreeRing
}

func (p *Pipeline) ProcessVoice(ctx context.Context) {
    audio := captureUntilSilence()
    text, sttLatency := p.stt.Transcribe(audio)           // ~101 ms for 70s audio
    enriched := p.maybeAddRAG(text)
    response, llmLatency := p.llm.Generate(enriched)      // 186 tok/s on Qwen3-4B
    audioOut, ttsLatency := p.tts.Synthesize(response)    // 178 ms

    showHUD(fmt.Sprintf("STT:%dms LLM:%dms TTS:%dms", sttLatency, llmLatency, ttsLatency))
    play(audioOut)
}

Prompt your coding assistant to turn this into real Swift or Go using the exact MetalRT flags from the RCLI repo.

5. Validate performance

Run benchmarks on your target hardware:

# Test individual stages
rcli benchmark stt
rcli benchmark llm --model qwen3-4b
rcli benchmark tts

Measure full pipeline latency. Target: <650 ms from last word spoken to first word heard.

Use the built-in TUI latency readouts from RCLI during development. Add --latency flag if available or parse the stdout.

Test with real voice: varied accents, background noise, long utterances. Record 10 sample interactions and log timings.

6. Ship it safely

• Bundle the required ~1 GB models or provide a first-run rcli setup step
• Add a fallback path: if MetalRT fails, call llama.cpp server or Ollama
• Sign the binary (codesign) and notarize for distribution
• Provide both a CLI daemon and a tiny SwiftUI menu-bar app
• License as MIT like RCLI
• Document exact hardware requirements (M1 or newer recommended)

Create a one-command installer that runs brew install rcli, downloads your wrapper, and adds it to Login Items.

Pitfalls and guardrails

What if MetalRT is not installed?

Always detect with rcli doctor or by checking for the MetalRT dylib. Fall back gracefully to llama.cpp with a visible warning and one-click install command.

What if latency feels bad on older M1?

Use smaller models (Qwen3-0.6B or LFM2.5-1.2B) for the first release. Profile with Instruments — the biggest wins come from pre-allocating all GPU memory at startup.

What if the user has no microphone permission?

Request AVAudioSession permission on first launch and show a clear onboarding screen. Do not silently fail.

What if RAG is too slow?

Start with simple keyword + TF-IDF over a small folder. Only add embedding search once you have the base voice loop under 500 ms. The RCLI repo already ships a ~4 ms RAG implementation over 5K chunks — copy that pattern.

What if I want to expose this as an API to other apps?

Use XPC or a local Unix socket. Keep the MetalRT engine in a privileged helper so you can share the loaded models across processes.

What to do next

• Add 10 useful macOS voice actions (the RCLI already has 38 — extend the list)
• Implement context memory between conversations
• Add hot-swap model UI so users can toggle between 0.6B and 4B models
• Measure battery impact and add a “low-power” mode that uses smaller models
• Open-source your wrapper and send a PR to the RCLI examples folder
• Write a short blog post showing your end-to-end latency numbers

Sources

• Original Show HN: https://news.ycombinator.com/item?id=47326101
• RCLI GitHub (MIT): https://github.com/RunanywhereAI/RCLI
• MetalRT LLM benchmarks: https://www.runanywhere.ai/blog/metalrt-fastest-llm-decode-e...
• MetalRT Speech benchmarks: https://www.runanywhere.ai/blog/metalrt-speech-fastest-stt-t...
• Voice pipeline details: https://www.runanywhere.ai/blog/fastvoice-on-device-voice-ai...
• Demo video: https://www.youtube.com/watch?v=eTYwkgNoaKg

(Word count: 982)

This guide gives you a repeatable, AI-assisted process to turn RunAnywhere’s breakthrough into your own shippable on-device voice product in a weekend.