PhyAgentOS — Plugins & Drivers

Plugin Discovery Chain

External plugins don't activate automatically. The main repository needs to know what driver you provide, where the entry class is, and where the profile is — all declared through a manifest file.

1

Manifest

PhyAgentOS_plugin.toml declares driver name, module, class, and profile path

2

Deploy

Deploy script clones/copies the plugin to the local plugin directory

3

Register

hal/plugins.py parses the manifest and writes to the local registry

4

Load

When not found in the built-in driver table, dynamically import from external registry

5

Launch

Watchdog starts with --driver <name> just like a built-in driver

The main repo only sees the BaseDriver interface — it doesn't care which SDK the plugin uses internally.

ReKep Plugin Installation

ReKep is currently the most complete external plugin reference. Install online or debug locally.

Online Install

python scripts/deploy_rekep_real_plugin.py \
  --repo-url https://github.com/baiyu858/PhyAgentOS-rekep-real-plugin.git

Local Debug

python scripts/deploy_rekep_real_plugin.py \
  --repo-url ../PhyAgentOS-rekep-real-plugin

# With solver dependencies
python scripts/deploy_rekep_real_plugin.py \
  --repo-url https://github.com/baiyu858/PhyAgentOS-rekep-real-plugin.git \
  --with-solver

# Register only, skip dependency install
python scripts/deploy_rekep_real_plugin.py \
  --repo-url ../PhyAgentOS-rekep-real-plugin \
  --no-install-deps

Invoking via Watchdog

This is the closest to real-world usage. The Agent generates actions, and the Watchdog monitors ACTION.md and invokes the plugin to execute.

# Terminal 1
paos onboard
python hal/hal_watchdog.py --driver rekep_real --workspace ~/.PhyAgentOS/workspace

# Terminal 2
paos agent

Recommended for demonstrating the complete "perception → decision → execution" closed loop.

Direct ACTION.md Debugging

Want to bypass the Agent and directly verify whether a specific action can be handled by the plugin? Manually editing ACTION.md is the fastest approach.

Plugin-Native Action

{
  "action_type": "real_execute",
  "parameters": {
    "instruction": "pick up the chili pepper and place it in the plate",
    "execute_motion": true
  },
  "status": "pending"
}

Generic High-Level Action

{
  "action_type": "pick_up",
  "parameters": {
    "target": "red_apple"
  },
  "status": "pending"
}

ReKep supports both protocols. The former uses plugin-specific capabilities, while the latter is converted to a natural-language instruction before execution.

Direct Runtime Debugging

The most common need during development: bypass the Agent and Markdown first, and independently verify whether the runtime itself has issues.

# Preflight
python runtime/dobot_bridge.py preflight --pretty

# Dry-run
python runtime/dobot_bridge.py execute \
  --instruction "pick up the red block and place it on the tray" \
  --pretty

# Real execution
python runtime/dobot_bridge.py execute \
  --instruction "pick up the red block and place it on the tray" \
  --execute_motion \
  --pretty

# Switch robot family
python runtime/dobot_bridge.py preflight \
  --robot_family cellbot \
  --robot_driver your_driver

Runtime-level debugging is best for isolating issues with third-party SDKs, bridge scripts, or real-robot adapters, because it narrows the failure scope to the minimum.

Built-in Driver Reference

Complete reference for all drivers shipped with the PhyAgentOS repository.

Driver Name	Robot	Type	Status	Notes
`simulation`	Built-in Simulator	Sim	verified	Default simulator with a simple tabletop scene. Supports move_to, pick_up, target_navigation, strategy.
`go2_edu`	Unitree Go2	Quadruped	partial	Quadruped locomotion and navigation. Supports target_navigation using onboard sensors.
`franka`	Franka Research 3	Arm	partial	7-DOF robotic arm. Supports Cartesian-space motion and joint-space control.
`piper`	AgileX PIPER	Arm	partial	Lightweight 6-DOF arm. Primarily used with ReKep for grasping tasks.
`dobot`	Dobot Nova 2	Arm	partial	Collaborative robot arm. Deployed via the ReKep real-robot plugin bridge.
`xlerobot`	XLeRobot	Dual-Arm	partial	Dual-arm mobile manipulation platform. Supports coordinated bimanual tasks.
`pipergo2_manipulation`	PIPER + Go2	Composite	partial	Mobile manipulation: PIPER arm mounted on Go2 quadruped. Combines locomotion and grasping.

Driver Config Pass-through

Pass a JSON config file directly to the driver constructor using the --driver-config flag.

# Pass driver-specific configuration
python hal/hal_watchdog.py --driver franka --driver-config examples/franka_config.json

The JSON object is forwarded directly to the driver's constructor. Each driver defines its own config schema — refer to the driver's documentation for available keys. Common fields include ip, control_mode, gripper_type, and sensor_config.

Example config: {"ip": "192.168.1.100", "control_mode": "joint_position", "gripper_type": "parallel_jaw"}

Choosing Between Plugins vs Built-in Drivers

When to create a separate plugin repository vs contributing directly to the built-in driver table.

Scenario	Use Plugin Repo	Use Built-in
Heavy SDK dependencies (ROS, ReKep solver, CUDA)	Plugins isolate dependency bloat	Keep main repo lightweight
Real robot with proprietary SDK	Plugins can have their own license	Main repo stays MIT-only
Independent versioning needed	Plugin repos version independently	Versions tied to PhyAgentOS releases
Small fix to simulation	Overhead of separate repo not worth it	PR directly to hal/drivers/
Improving existing built-in driver	Plugin would duplicate effort	PR directly to the driver file
General utility (logging, math, I/O)	Generally not plugin-shaped	Add to hal/utils/ or PhyAgentOS/utils/