Markdown File Formats & Examples

Where Files Live

In Single mode, all files are in one directory. In Fleet mode, shared files go global, and robot-private files go in their respective workspaces.

ENVIRONMENT.md · Source of Truth

This is the "single source of truth" for the system's current environment state. Scene graph, robot poses, and navigation state all live here. Watchdog and perception modules update it; Agent and Critic read it.

{
  "schema_version": "PhyAgentOS.environment.v1",
  "updated_at": "2026-03-17T10:20:30Z",
  "scene_graph": {
    "nodes": [
      {
        "id": "obj_red_apple",
        "class": "apple",
        "object_key": "red_apple",
        "center": {"x": 0.05, "y": 0.05, "z": 0.75},
        "size": {"x": 0.06, "y": 0.06, "z": 0.06},
        "confidence": 0.96,
        "frame": "map",
        "track_id": "track_apple_001",
        "last_seen_at": "2026-03-17T10:20:29Z"
      }
    ],
    "edges": [
      {"source": "obj_red_apple", "relation": "ON", "target": "furniture_table", "confidence": 0.97}
    ]
  },
  "robots": {
    "go2_edu_001": {
      "connection_state": {
        "status": "connected",
        "transport": "ssh",
        "host": "192.168.1.23",
        "port": 22,
        "last_heartbeat": "2026-03-17T10:20:30Z"
      },
      "robot_pose": {
        "frame": "map",
        "x": 1.23, "y": -0.45, "z": 0.0, "yaw": 1.57,
        "stamp": "2026-03-17T10:20:30Z"
      },
      "nav_state": {
        "mode": "navigating",
        "status": "running",
        "goal_id": "nav_goal_001"
      }
    }
  },
  "objects": {
    "red_apple": {
      "type": "fruit",
      "color": "red",
      "location": "table"
    }
  }
}

EMBODIED.md · Capability Profile

This is not real-time state — it's a capability manual for the Agent. Watchdog copies it from hal/profiles/*.md at startup.

# EMBODIED.md Template

## Identity
- **Name**: Unitree Go2 EDU
- **Type**: quadruped
- **Driver/Profile**: go2_edu

## Sensors
- **Camera**: RGBD
- **LiDAR**: yes
- **Odometry**: yes

## Supported Actions
| Action | Parameters | Description |
|--------|-----------|-------------|
| `move_to` | `x, y` | Navigate to a target pose |
| `target_navigation` | `target_label, timeout_s` | Move toward a visual target |

## Physical Constraints
- **Max speed**: 1.0 m/s
- **Safety distance**: 0.4 m
- **Collision policy**: stop when obstacle too close

The trick to writing this file: don't stuff technical details — let the Agent quickly know "what is this, what can it do, where are the boundaries."

ACTION.md · Action Queue

The direct dispatch file for the execution pipeline. The Agent writes actions here, and the Watchdog reads them and hands them to the Driver.

{
  "schema_version": "PhyAgentOS.action_queue.v1",
  "actions": [
    {
      "id": "2f18c6a912ab",
      "action_type": "target_navigation",
      "parameters": {
        "robot_id": "go2_edu_001",
        "target_label": "apple",
        "timeout_s": 30
      },
      "status": "pending"
    }
  ]
}

TASK.md · Long-Horizon Task Decomposition

Breaks a natural language instruction into multiple sub-tasks and assigns them to different robots. This is what the orchestrator reads.

# Active Task

## Goal
Pick up the red apple from the table and place it into the blue basket in the kitchen.

## Sub-Tasks
| ID | Action | Target Device | Status | Depends On | Result |
|----|--------|---------------|--------|------------|--------|
| T1 | navigate_to | go2_edu_001 | completed | — | arrived |
| T2 | detect_target | go2_edu_001 | completed | T1 | found apple |
| T3 | pick_up | franka_001 | pending | T2 | — |
| T4 | navigate_to | go2_edu_001 | pending | T3 | — |
| T5 | place_in | franka_001 | pending | T4 | — |

SESSIONS.md · Runtime Session Queue

The runtime session loop replaces the old single-action ACTION.md queue for structured execution. Each session binds a target and a skill, with priority-based scheduling, timeouts, retry policy, and routing metadata.

# Runtime Sessions

```yaml
version: runtime_sessions_v1
sessions:
  - session_id: sess_pick_apple_001
    goal_id: goal_pick_apple
    target_ref: target://franka_lab_a
    skill_ref: skill://rekep_grasp
    task_description: "pick up the red apple on the table"
    status: pending
    priority: high
    timeouts:
      queue_timeout_s: 30
      execute_timeout_s: 120
      policy_timeout_s: 5
    retry:
      max_retries: 1
      attempted: 0
    routing:
      policy_endpoint: dummy://local
      adapter: franka_real_adapter
```

TARGETS.md · Runtime Target Registry

Declares available rollout targets (sim, real_robot) with perception config, adapter selection, and skill compatibility. Target observation and action dimensions are declared in config.

# Runtime Targets

```yaml
version: runtime_target_registry_v1
targets:
  - id: dummy_sim
    type: sim
    backend: dummy
    enabled: true
    workspace: workspaces/dummy_sim
    supported_skills:
      - openpi_sim_vla
    adapter: dummy_openpi_adapter
    perception:
      enabled: false
      strict_preflight: true
      sensor_config_ref: configs/runtime/sensors/dummy_sim.sensors.yaml
      perception_config_ref: configs/runtime/perception/dummy_sim.perception.yaml
    config:
      success_after_steps: 3
      observation:
        image_size: 16
        state_dim: 8
      action:
        action_dim: 7
        chunk_size: 4
```

SKILLS.md · Runtime Skill Registry

Skills declare what they need (sensors, environment outputs) and what runtime handles execution. Skills do not select perception plugins — the runtime resolves that from the perception config.

# Runtime Skills

```yaml
version: runtime_skill_registry_v1
skills:
  - id: openpi_sim_vla
    category: vla
    runtime: OpenPISimSkillRuntime
    supported_target_types:
      - sim
    policy_client: dummy
    supports_chunk: true
    default_replan_every: 4
    requires:
      sensors: []
      environment_outputs: []
      strict_environment_contract: true
```

ORCHESTRATOR.md · Global Dashboard

The "control tower" for multi-robot coordination. It doesn't store action details — only aggregates overall progress, active devices, and key decisions.

# Orchestrator Status

## Current Mission
Pick up the red apple from the table and place it into the blue basket.

## Progress
2/5 sub-tasks completed (40%)

## Active Devices
- go2_edu_001: navigating_to_kitchen
- franka_001: idle_awaiting_next_action

## Issues
- None

## Decision Log
- [10:05] Assigned T1 to go2_edu_001 because it has the locomotion capability.
- [10:12] Assigned T3 to franka_001 after apple was confirmed in scene_graph.

Who Reads, Who Writes · One Table