API Usage / Runtime Session

Runtime Session Loop Advanced

SESSIONS.md → WatchdogSupervisor → SkillRuntime → Target → Artifacts — the session-centered execution pipeline.

Runtime vs HAL: Two Execution Paths

PhyAgentOS provides two execution paths. The HAL path is simpler and action-oriented. The Runtime path is more structured, session-oriented, and adds priority scheduling, preflight checks, and perception pipelines.

FeatureHAL PathRuntime Path
Queue formatACTION.md (single actions)SESSIONS.md (session queue)
Executorhal_watchdog.pyWatchdogSupervisor + run_runtime_watchdog.py
SchedulingFIFO onlyPriority-based (high / normal / low)
Preflight checksMinimalStrict: sensor calibration, schema validation
Perception pipelineNot integratedBuilt-in plugin pipeline
Artifact outputInline in ACTION.mdepisode.json + LOG.md
Use caseSimple action executionStructured rollout with observability

SESSIONS.md In Detail

SESSIONS.md uses YAML frontmatter-style sessions. Each session includes priority, timeout, retry, and routing fields.

## session_001 priority: high queue_timeout: 300 execute_timeout: 600 policy_timeout: 30 max_retries: 3 attempted: 1 policy_endpoint: builtin adapter: simulation status: queued instruction: "navigate to the table and pick up the red cube"
FieldDescription
priorityhigh → normal → low. Higher priority sessions execute first. Same priority uses file order.
queue_timeoutMax seconds a session can wait in queue before being marked timeout.
execute_timeoutMax seconds for the entire session execution (across all steps and retries).
policy_timeoutMax seconds for a single policy inference call.
max_retriesMaximum retry attempts. Counter tracked in attempted field.
policy_endpointWhere the policy runs: builtin, remote, or custom adapter name.
adapterTarget adapter for execution (maps to a TargetAdapter instance).
statusCurrent state: queued, running, completed, failed, timeout, rejected.

Initialize Runtime Workspace

The runtime workspace must be initialized before the watchdog can begin processing sessions. Use the init script to scaffold all required files.

python scripts/init_runtime_workspace.py --workspace /tmp/test

Files generated:

FilePurpose
TARGETS.mdRegistry of available execution targets (simulators, real robots) with their capabilities and perception configs.
SKILLS.mdDeclared skills per target, including sensor requirements and policy endpoints.
SESSIONS.mdThe session queue. Initially empty, populated by the Agent or manually.
LOG.mdExecution log with per-session results (status, steps, return values).
configs/runtime/sensors/*.yamlSensor configuration files defining available sensors and their channels.
configs/runtime/perception/*.yamlPerception pipeline configuration: preprocessors, models, postprocessors.

Run Runtime Watchdog

The runtime watchdog polls SESSIONS.md, dispatches sessions to the WatchdogSupervisor, and records results.

Single pass (--once)

python run_runtime_watchdog.py \ --workspace /tmp/test \ --once

Continuous polling

python run_runtime_watchdog.py \ --workspace /tmp/test \ --environment-workspace /tmp/env

Expected outputs after each session:

  • SESSIONS.md status updated to completed, failed, or timeout
  • LOG.md appended with session_id, target_id, status, num_steps, and return_value
  • artifacts/runtime/<session_id>/episode.json created with full execution trace
Use --environment-workspace to write ENVIRONMENT.md v2 to a separate directory. This keeps environment observations decoupled from the runtime session state.

Perception Plugin Pipeline

Setting perception.enabled: true in TARGETS.md triggers strict preflight validation of sensor and perception YAML configs before any session executes.

Preflight Checks

Validates sensor availability, calibration file paths, and observation schemas. Fails fast if any requirement is unmet.

Plugin Pipeline

Runs after preflight passes. Chains sensor readers, preprocessors, model inference, and postprocessors.

Pipeline output: ENVIRONMENT.md v2 in the environment workspace with structured observations.

Common FailureCauseFix
missing sensor_config_refSKILLS.md references a sensor not defined in configs/runtime/sensors/Add the missing sensor YAML or fix the reference
missing observation channelSensor config declares channels the driver doesn't exposeAlign sensor config channels with driver output
dtype/shape mismatchPerception model expects different tensor shape than sensor producesAdd a preprocessor to reshape/downscale sensor output

Execution Results

Each completed session produces structured artifacts for debugging, evaluation, and replay.

LOG.md Format

| session_id | target_id | status | num_steps | return_value | |------------|-----------|---------|-----------|--------------| | sess_001 | sim_01 | completed | 12 | 0.87 | | sess_002 | sim_01 | failed | 3 | 0.12 |

episode.json Structure

{ "session_id": "sess_001", "target_id": "sim_01", "status": "completed", "steps": [ { "step": 0, "observation": {...}, "action": {...}, "reward": 0.5, "next_observation": {...} } ], "return_value": 0.87, "metadata": { "start_time": "...", "end_time": "...", "retries": 0 } }

Artifacts directory layout:

artifacts/runtime/ ├── sess_001/ │ └── episode.json ├── sess_002/ │ └── episode.json └── latest -> sess_002/

Enabling Perception Step-by-Step

Perception is disabled by default. Enable it through TARGETS.md and ensure all required configs are in place.

  1. Set TARGETS.md perception.enabled to true
    ## sim_01 type: simulation perception: enabled: true sensor_config_ref: sim_rgbd perception_config_ref: sim_basic
  2. Add sensor requirements to SKILLS.md

    Each skill must declare which sensors it needs. Example: pick_up requires rgb and depth channels.

  3. Ensure sensor/perception YAML configs exist
    ls configs/runtime/sensors/sim_rgbd.yaml ls configs/runtime/perception/sim_basic.yaml
  4. Run watchdog with --environment-workspace
    python run_runtime_watchdog.py \ --workspace /tmp/test \ --environment-workspace /tmp/env
  5. Verify ENVIRONMENT.md appears
    cat /tmp/env/ENVIRONMENT.md

    If empty or missing, check preflight output for errors.

Dry-run validation: Set perception.enabled: true but run the watchdog with --once --dry-run to validate the config pipeline without executing any sessions.
The Runtime Session loop is the future execution backbone of PhyAgentOS. HAL mode remains available for simple action execution; Runtime mode adds structure, observability, and perception for production-grade rollouts.