Three Colors, Three Statuses
Green = someone has already verified it, you can use it as a reference; Yellow = prototype or design doc exists, needs more testing and polish; Gray = completely empty — the perfect opportunity for your first PR.
Completed
Core framework, simulator, ReKep/VLN/Franka real-robot pipelines, Runtime Session Loop, Perception Plugin Pipeline verified. These are stable reference baselines.
In Progress
Expanding runtime registries to real benchmark and robot targets. Long-horizon orchestration, IoT integration, and Fleet multi-robot protocols have prototypes but need more scenario testing.
Planned
Concurrent multi-target scheduling, resource arbitration, Hiwonder driver, ROS2 deep guide. Clear entry points with minimal friction.
Three-Phase Roadmap
Phase 1 · Desktop Closed Loop
✓ VerifiedGet a single robot running end-to-end: perception → decision → action → state writeback. Proving that the Markdown protocol approach works.
Phase 2 · Multi-Robot Collaboration
⋯ In ProgressMultiple heterogeneous robots sharing an environment, task decomposition, and collaborative execution. Adding long-horizon memory and experience reuse.
Phase 3 · Advanced Coordination
○ PlannedComplex spatiotemporal constraints, resource scheduling, cross-scenario industrial-grade tasks. This is the longer-term vision.
8 Robot Integration Progress
Below is the current hardware support list. Want to connect a new device? See if there's a similar reference implementation.
Desktop robotic arm · ReKep + SAM3 full pipeline verified
Desktop robotic arm · ReKep deployment verified
Dual-arm robot · One-click deployment implemented
Industrial-grade · Visual Q&A + grasping implemented
Quadruped · Mobility + semantic navigation working, complex actions pending
Mobile manipulation · Locomotion adaptation in progress
IoT · Voice conversation working, action control protocol pending design
Educational robots · Suitable beginner-friendly project
Topic Gaps · High Priority
Fleet Orchestration & Multi-Robot Coordination
TASK.md and ORCHESTRATOR.md protocols exist, but scheduling strategies, conflict resolution, and robot selection logic need more scenario validation.
Runtime Registry Expansion
Session loop and perception pipeline are verified with DummySimTarget. Next: expand to real benchmark targets (LIBERO, RoboCasa), real robot targets, and concurrent multi-target scheduling.
ROS2 Integration Developer Guide
cmd_vel/lidar/odom/rgbd adapter code exists, but lacks a comprehensive onboarding guide for developers.
Perception Plugin Expansion
Built-in dummy plugins work. Next: integrate SAM3, YOLO, Open3D, RealSense as lazy-loaded plugins and validate the full closed loop with real sensor data.
Choose Your Entry Point by Background
| Your Background | Recommended Starting Point | Expected Output |
|---|---|---|
| Own a robot, familiar with its Python SDK | Write a Driver + Profile + minimal test | A new Driver that Watchdog can load directly |
| Experience with ROS2 / navigation / perception | Supplement ROS2 integration docs, extend existing adapters | A complete guide that lowers the barrier for ROS2 users |
| Frontend / DevOps / cloud-native background | Write gateway deployment docs, Dockerization plan | A production-ready deployment manual |
| No hardware, contributing purely in software | Simulator scenarios, test cases, docs site improvements | PRs that can be merged without real hardware |