AMD Ryzen AI NPU Driver Guide with Lemonade Stack

Title:
How to Run Local LLMs on AMD Ryzen AI NPU Under Linux Using Lemonade + FastFlowLM

Why this matters for builders
Lemonade’s FastFlowLM is an NPU-first runtime that lets you run small LLMs exclusively on the Ryzen AI Neural Processing Unit under Linux, freeing your GPU and CPU for other work while keeping power consumption and latency low.

Until now, AMD’s Ryzen AI NPUs had almost no usable Linux story for LLMs. With the recent open-source Lemonade server release and FastFlowLM’s Ubuntu/Arch support, builders can finally ship efficient, on-device inference pipelines on modern Ryzen AI laptops and mini-PCs without touching proprietary Windows drivers.

When to use it

You own a Ryzen AI 300-series (Strix Point) or Ryzen 8000/9000 AI laptop
You want to run 1–7B parameter models at usable speed while the discrete GPU stays idle
You need low idle power for always-on local agents or edge tools
You prefer open-source runtimes over closed AMD ROCm or Windows-only Ryzen AI SDK
You are building Linux-native AI desktop apps, CLI tools, or local RAG assistants

The full process

1. Define the goal (30 minutes)

Write a one-paragraph spec before touching any code.

Example goal:
“Build a local chat CLI that loads Phi-3.5-mini (ONNX, NPU-optimized) onto the Ryzen AI NPU via FastFlowLM on Ubuntu 24.04, streams tokens at >25 t/s, keeps GPU/CPU usage under 10 %, and exposes a simple HTTP endpoint for future Electron frontend use.”

Success metrics:

Model runs 100 % on NPU (confirmed with ryzenai tools)
Power draw < 15 W during inference
First token latency < 400 ms
Works on clean Ubuntu 24.04 install

2. Shape the spec & prompt your coding assistant

Give your AI coding tool (Cursor, Claude, Windsurf, etc.) a precise system prompt.

Starter prompt you can copy-paste:

You are an expert Linux systems + ML deployment engineer.

Target: Ubuntu 24.04 on AMD Ryzen AI 300 laptop.
Goal: Create a FastFlowLM + Lemonade server setup that runs Phi-3.5-mini-instruct-onnx-ryzenai-npu exclusively on the NPU.

Requirements:
- Use the official Lemonade FastFlowLM Linux guide
- Install all dependencies with apt and pip in a reproducible way
- Provide a simple Python CLI that accepts --prompt and streams output
- Add a FastAPI endpoint at /generate for future use
- Include a health-check that verifies the model is actually running on NPU (use ryzenai tools or npu-top)
- Write clear README with exact hardware requirements and expected tokens/sec
- Add error handling for missing NPU driver

Output structure:
1. setup.sh (idempotent)
2. requirements.txt
3. main.py (CLI + FastAPI)
4. Dockerfile (optional but nice)
5. README.md

3. Scaffold the project

Run these commands to create the skeleton:

mkdir ryzen-ai-llm-linux && cd ryzen-ai-llm-linux
git init
mkdir -p src tests
touch README.md setup.sh requirements.txt src/main.py

4. Implement with FastFlowLM + Lemonade

Follow the official Lemonade Linux guide (linked in sources) but here is the practical flow builders actually use in 2025:

setup.sh (key parts)

#!/bin/bash
set -e

echo "Installing system dependencies..."
sudo apt update
sudo apt install -y python3-pip python3-venv git build-essential

# Usually involves the Ryzen AI Software 1.7+ Linux package

python3 -m venv venv
source venv/bin/activate

pip install -r requirements.txt

# Clone Lemonade FastFlowLM
git clone https://github.com/lemonade-server/fastflowlm.git
cd fastflowlm && pip install -e .
cd ..

echo "Setup complete. Run: source venv/bin/activate && python src/main.py"

requirements.txt

fastapi==0.115.*
uvicorn==0.32.*
python-dotenv
pydantic
# FastFlowLM will pull its own ONNX and RyzenAI bindings

src/main.py (minimal working example)

import os
from fastapi import FastAPI
from pydantic import BaseModel
import subprocess

app = FastAPI()

class Prompt(BaseModel):
    text: str
    max_tokens: int = 256

def is_npu_active() -> bool:
    try:
        out = subprocess.check_output(["npu-top", "--summary"], text=True, timeout=3)
        return "NPU" in out and "util" in out.lower()
    except:
        return False

@app.post("/generate")
async def generate(prompt: Prompt):
    if not is_npu_active():
        return {"error": "NPU not detected or model not loaded on NPU"}
    
    # FastFlowLM inference call – simplified
    result = subprocess.run([
        "fastflowlm-run",
        "--model", "Phi-3.5-mini-instruct-onnx-ryzenai-npu",
        "--prompt", prompt.text,
        "--max-tokens", str(prompt.max_tokens)
    ], capture_output=True, text=True)
    
    return {"response": result.stdout.strip(), "npu": True}

# CLI fallback
if __name__ == "__main__":
    import sys
    if len(sys.argv) > 1:
        print("Loading model on NPU...")
        os.system(f"fastflowlm-run --model Phi-3.5-mini-instruct-onnx-ryzenai-npu --prompt '{sys.argv[1]}'")

5. Validate

Run this checklist:

ls /dev/accel shows the NPU device
npu-top reports the model is using the NPU during generation
Measure power with powerstat or AMD’s tools (< 15 W sustained)
Benchmark tokens/sec with a 200-token prompt (target > 25 t/s for Phi-3.5-mini)
Confirm GPU usage stays near 0 % with nvidia-smi or radeontop
Test the FastAPI endpoint with curl

Quick validation command:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"text": "Explain why NPUs are great for local LLMs", "max_tokens": 150}'

6. Ship it safely

Push to a public GitHub repo with clear hardware requirements in README
Add a hardware.md listing exact Ryzen AI SKUs that work today
Provide a one-click setup.sh that fails fast with helpful messages
Consider packaging as a Snap or Flatpak later for non-technical users
Document that this is still early-stage Linux support — expect driver updates

Pitfalls and guardrails

### What if the NPU is not detected?
Check that you installed the latest Ryzen AI Linux driver from AMD (v1.7+ at time of writing). Run ryzenai-diagnostics or lsmod | grep npu. Reboot after driver install.

### What if performance is terrible?
Make sure you are using the NPU-optimized ONNX model (Phi-3.5-mini-instruct-onnx-ryzenai-npu) and not a generic GGUF or PyTorch model. FastFlowLM only accelerates specific ONNX variants.

### What if I only see CPU usage?
The model was loaded to CPU. Force NPU path with the dedicated --device npu flag in FastFlowLM or use the Lemonade server wrapper.

### What if the model is too big?
Stick to 3–4B class models for now. 7B+ models may require INT4 quantization specifically tuned for the NPU’s 50 TOPS.

What to do next

Add streaming response support to the FastAPI endpoint
Build a tiny Electron or Tauri frontend that talks to localhost:8000
Create a RAG pipeline using the NPU for the embedding step and CPU/GPU for retrieval
Benchmark against llama.cpp on CPU and ROCm on GPU — publish numbers
Monitor Lemonade and AMD’s GitHub for new supported models (Mistral, Gemma-2, etc.)

Sources

Original Phoronix article: https://www.phoronix.com/news/AMD-Ryzen-AI-NPUs-Linux-LLMs
Lemonade FastFlowLM Linux guide: https://lemonade-server.ai/flm_npu_linux.html
AMD Ryzen AI Software Linux LLM docs: https://ryzenai.docs.amd.com/en/latest/llm_linux.html
Community discussion: https://www.reddit.com/r/artificial/comments/1rr1en0/amd_ryzen_ai_npus_are_finally_useful_under_linux/
r/LocalLLaMA threads on Ryzen AI NPU usage

(Word count: 942)

AMD NPU driver & Ryzen AI stack (follow latest from AMD docs)