train_g1_pickplace.yaml

Title:
How to Train and Deploy a Unitree G1 Whole-Body Policy with LeRobot v0.5.0

Why this matters for builders
LeRobot v0.5.0 lets you train and run whole-body control policies on the Unitree G1 humanoid (plus many other robots) using a modern PyTorch stack that loads environments straight from the Hugging Face Hub and supports autoregressive VLAs with real-time chunking.

The release adds full G1 locomotion + manipulation support, Pi0-FAST autoregressive policies, streaming video encoding that removes recording bottlenecks, EnvHub for one-line simulation loading, NVIDIA IsaacLab-Arena integration, and a cleaned-up codebase on Python 3.12 + Transformers v5. These changes turn months of custom engineering into days of focused experimentation for builders who want to ship real embodied AI.

When to use it

• You want to train your first whole-body humanoid policy (locomotion + dexterous manipulation).
• You need low-latency inference on hardware using Real-Time Chunking (RTC).
• You are iterating on simulation-to-real with environments hosted on the Hub.
• You prefer clean, plugin-style hardware support over forking old codebases.
• You already use Hugging Face datasets/models and want to stay in that ecosystem.

The full process

1. Define the goal (30 min)

Be brutally specific. Good example:

“Train a Pi0-FAST policy on the Unitree G1 that can walk to a table, pick up a red block, and place it in a bowl. Start in IsaacLab-Arena simulation, then transfer to real G1 hardware using teleoperation data. Target 80% success in sim after 50k gradient steps and safe zero-shot deployment on hardware.”

Write this down. It becomes your system prompt for the AI coding assistant.

2. Shape the spec & prompt

Use this starter prompt with your favorite AI coding tool (Cursor, Claude, Windsurf, etc.):

You are an expert LeRobot v0.5.0 engineer.

Project goal: [paste your goal above]

Available in LeRobot v0.5.0:
- Full Unitree G1 support with locomotion, manipulation, teleop, and whole-body control
- Pi0-FAST policy with FAST tokenization + Real-Time Chunking
- EnvHub for loading IsaacLab-Arena environments from the Hub
- Streaming video encoding for fast dataset recording
- Python 3.12 + Transformers v5

Create the following files with best practices:
1. `train_g1_pickplace.yaml` – complete training config
2. `record_teleop.py` – script to record new episodes with streaming encoding
3. `eval_policy.py` – real-time inference script using RTC
4. `deploy_real.py` – minimal hardware deployment script for the G1

Use official patterns from the v0.5.0 release. Prefer `--policy.type=pi0_fast` and `lerobot` CLI where possible. Add clear comments and validation steps.

3. Scaffold the project

Run these commands to bootstrap:

mkdir g1-pickplace && cd g1-pickplace
python -m venv .venv && source .venv/bin/activate
pip install "lerobot[torch,envs]" --extra-index-url https://download.pytorch.org/whl/cpu  # adjust for CUDA
git clone https://github.com/huggingface/lerobot.git
cd lerobot && pip install -e ".[dev]"

Create the config file train_g1_pickplace.yaml:

policy:
  type: pi0_fast
  model:
    vision_encoder: "google/siglip-base-patch16-224"
    language_model: "google/gemma-2b"   # or smaller variant
    action_tokenizer: "lerobot/fast_action_tokenizer"
  training:
    batch_size: 64
    gradient_accumulation_steps: 4
    lr: 3e-5
    max_steps: 50000

dataset:
  repo_id: your-username/g1_pickplace_teleop_v1   # you'll create this
  episodes: all
  streaming: true                               # new v0.5.0 feature

env:
  type: isaaclab_arena
  hub_repo: lerobot/g1_pickplace_arena
  task: "pick_and_place_block"

4. Implement

Let your AI coding assistant generate the three scripts based on the prompt in step 2. Review every section it produces:

Key things to check:

• The policy is instantiated with policy.type=pi0_fast
• Dataset uses the new streaming video encoder (eliminates 10–30s waits between episodes)
• Evaluation script enables Real-Time Chunking:

# eval_policy.py snippet
policy = AutoPolicy.from_pretrained("your-username/g1-pickplace-v1")
policy.enable_real_time_chunking(chunk_size=8, temperature=0.7)  # new v0.5.0 capability

After the assistant generates code, run the linter and type checker:

ruff check .
mypy .

5. Validate

Follow this checklist before declaring victory:

• Record 20+ teleop episodes using the new streaming encoder (should feel instant)
• Train for at least 10k steps and verify loss decreases
• Run sim evaluation: python eval_policy.py --env isaaclab_arena --episodes 50
• Confirm >70% success rate on the pick-and-place task in simulation
• Test real-time inference latency on your target hardware (<80ms per action recommended)
• Run safety checks: velocity limits, joint torque limits, emergency stop integration

Pro tip: Use the new dataset tools to inspect recordings:

lerobot dataset inspect your-username/g1_pickplace_teleop_v1 --visualize

6. Ship safely

1. Push dataset to Hub: lerobot dataset push --repo-id your-username/g1_pickplace_teleop_v1
2. Upload trained policy: lerobot policy push --repo-id your-username/g1-pickplace-v1
3. Create a public demo video + config files in the model card
4. Add a README.md with exact hardware setup steps and known limitations
5. Tag the release: git tag v0.1-g1-pickplace && git push --tags

Pitfalls and guardrails

### What if training is too slow?
Use the new streaming video encoding and --dataset.image_transforms to enable the 10x faster image training path. Reduce image resolution to 224×224 during early experiments.

### What if the G1 falls over in sim?
Start with the locomotion-only policy first, then fine-tune with whole-body control. IsaacLab-Arena environments now support curriculum learning — enable it in the config.

### What if real-time chunking feels laggy?
Tune chunk_size between 4–12 and lower temperature. Pi0-FAST’s autoregressive expert is sensitive to these hyperparameters.

### What if my custom robot isn’t supported yet?
v0.5.0 introduced a plugin system. Implement the new HardwarePlugin interface and contribute it back — the team merged 200+ PRs this cycle.

### What if I get CUDA OOM?
The Gemma-based action expert in Pi0-FAST is heavy. Start with the 300M variant and enable gradient checkpointing.

What to do next

• Record 100+ real G1 teleop episodes and continue training
• Add language commands via the VLA interface (“walk to the table and grab the red block”)
• Try PEFT fine-tuning on top of a strong base policy (new in v0.5.0)
• Integrate the CAN bus motor controllers if you need higher torque
• Share your dataset and model on the Hub so others can build on it

LeRobot v0.5.0 finally makes humanoid whole-body learning accessible to solo builders and small teams. The combination of EnvHub, streaming datasets, and Pi0-FAST with real-time chunking removes most of the traditional friction.

Start small, validate in simulation, then cautiously move to hardware. The infrastructure is now good enough that your biggest bottleneck is collecting high-quality demonstration data.

Sources

• LeRobot v0.5.0: Scaling Every Dimension
• LeRobot GitHub repository
• Official v0.5.0 documentation and example configs included in the release

(Word count: 948)