RAI
New1. Overview
RAI (Robotec AI) is an open-source, vendor-agnostic agentic framework for Physical AI robotics, built on ROS 2. It lets a robot interpret natural-language instructions, reason about a task, plan a sequence of actions, and execute them on real hardware or in simulation. The project is maintained by Robotec.ai, a Warsaw-based deep-tech company that builds high-fidelity digital twins and physics-based simulation for autonomous systems, and is one of the flagship projects of the ROS 2 Embodied AI Community Group. RAI is model-agnostic: it orchestrates large language models and vision-language models through LangChain, so a user can choose frontier cloud models or open-weight models running on-premise or air-gapped. For a robotics team, RAI is the layer that turns a general-purpose model into a robot-executable behaviour with logging, voice interaction, and benchmarked evaluation. The accompanying paper, "RAI: Flexible Agent Framework for Embodied AI" (arXiv:2505.07532, 2025), documents the architecture and design rationale, and the project is mirrored by a companion company research report on this site (see /en/reports/robotecai-physics-ai-2026/).
RAI ships as a modular Python package set rather than a monolith. The core coordinates a multi-agent system, human-robot interaction, and multimodal inputs. Around it sit specialised packages: rai whoami extracts a robot's embodiment profile from documentation, images, and URDF files; rai_asr and rai_tts provide speech-to-text and text-to-speech; rai_sim connects RAI to simulation environments; rai_bench is a benchmarking suite for agents, models, tools, and simulators; rai_perception adds open-set object detection; and rai_finetune fine-tunes LLMs on embodied data. This site already covers the simulation backends RAI integrates with, such as NVIDIA Warp and MuJoCo, and the physics solvers behind them such as OpenFOAM; this entry documents the agentic orchestration layer that ties them together, and the broader Physical-AI context is discussed in the NVIDIA company report.
2. How It Works
RAI's design separates three concerns: a multi-agent coordination layer, a human-robot interaction layer, and a multimodal perception layer, all orchestrated by a reasoning core. A typical task begins with a natural-language instruction. The core parses the instruction, selects the appropriate agent or tool, and produces a plan expressed as a sequence of executable steps over the robot's ROS 2 interface. Because the orchestration backbone is LangChain, the planner can call LLMs or VLMs as reasoning components and chain their outputs into tool invocations such as navigation, manipulation, or perception queries. rai whoami supplies the robot's embodiment profile so the planner knows the hardware it is reasoning about; rai_perception contributes open-set object detection; rai_asr and rai_tts close the loop with speech. The result is a behaviour that can be driven by free-form language rather than a hardcoded state machine.
The framework exposes a first-class abstraction over ROS 2 Humble and Jazzy distributions, while an abstraction layer simplifies communication across networks and protocols and allows future protocol extensions. In practice a developer configures the robot, extracts its embodiment with rai whoami, then lets agents execute tasks and interact with humans in real or simulated environments. The M-Agent model under the hood defines an agent through its reasoning system, sensors, and actuators, and streamlines those abstractions into the agent's core logic and connectors. Because most features can run independently of a live ROS 2 stack, RAI is also usable for non-robotic orchestration experiments, which is part of why the project is framed as a general agent framework rather than a robotics-only library.
3. Scope and Assumptions
The first boundary is the execution environment. RAI is built for ROS 2 and expects a robot or simulator that exposes standard ROS 2 topics, services, and actions. A fleet running proprietary middleware needs an adapter before RAI can issue commands. The second boundary is the model layer. RAI is model-agnostic by design, which means task quality depends entirely on the chosen model and its grounding; the framework does not itself guarantee that a plan is physically valid or safe. The third boundary is evaluation. rai_bench measures agents, models, tools, and simulators, but a passing benchmark is a behaviour test, not a certification of real-world safety. The documentation is explicit that RAI is built to test agent-generated behaviour before deployment, not to replace verification. Teams should treat RAI output as a candidate behaviour to be validated, not as a certified control policy.
A fourth practical assumption is infrastructure. Running RAI requires a ROS 2 workspace on the host, Python 3.10 or 3.12 on Ubuntu 22.04 or 24.04, and network access to the chosen model. The framework's voice and vision features add their own dependencies, and rai_sim presumes a reachable simulator. None of these are exotic, but they mean RAI is a developer tool rather than an end-user application; the setup guide is the intended entry point, and the project deliberately does not ship a turnkey app. That positioning is consistent with Robotec.ai's services business, where RAI is the open component that customers extend under their own SOWs.
4. Getting Started
The fastest entry point is the documentation site at robotecai.github.io/rai, which provides a quick setup guide and tutorials. The framework installs from source or as Python packages and requires a ROS 2 workspace on the host. A first experiment is the Rosbot XL autonomous-mobile-robot demo, which shows RAI handling navigation and control through natural language, or the Franka Emika manipulation demo, where RAI combines language understanding with grounded segmentation (Grounded SAM 2) to complete flexible pick-and-place tasks. Running the same instruction in rai_sim before running it on hardware lets a team catch planning failures in simulation. Because RAI is model-agnostic, the first configuration decision is which model to bind: a frontier cloud model for capability, or an open-weight model for data sovereignty and air-gapped deployment.
Beyond the two headline demos, the project documents an autonomous tractor in a virtual orchard, where RAI reasons about the best course of action for unexpected obstacles rather than only detecting them, and an RB-KAIROS+ mobile manipulator with on-board execution. The ROSCon 2024 talk introduced RAI to the community, and the Embodied AI Community Group continues to host recurring meetings. For teams evaluating the framework, the recommended path is to reproduce one demo in simulation, then wire rai_bench to the specific models and simulator they intend to use, because the framework's value shows up most clearly when the same instruction can be run repeatedly across models and backends while metrics are recorded.
5. Comparison with Alternatives
RAI is not a simulator and not a single robot stack; it is an orchestration layer between models and robots. Against low-level frameworks such as MuJoCo or OpenFOAM, which provide physics, RAI sits one layer up and calls them through simulation bridges. Against ROS 2 navigation and manipulation stacks such as Nav2 or MoveIt, RAI adds a language-driven, model-backed planning layer on top, so classical pipelines can be triggered by intent rather than by fixed calls. The defining property is vendor and model agnosticism: the same agent description can run against different models and different robot bodies, which is what distinguishes it from vendor-locked agent offerings. Compared with other LLM-agent robotics frameworks, RAI's emphasis is multi-agent systems with explicit embodiment mechanisms and a simulation-first, benchmarked workflow.
The closest conceptual neighbours are general agent frameworks that happen to target robotics, but RAI's differentiators are its ROS 2-native integration, its rai_bench evaluation discipline, and its origin inside a simulation company that runs the same stack against Isaac Sim, O3DE, and Genesis. That lineage matters: RAI is built to be validated in simulation before deployment, not merely to generate plausible robot commands. For buyers choosing between RAI and a vendor's bundled agent layer, the trade is portability and transparency (RAI is open, Apache-2.0, model-agnostic) against the convenience of a vertically integrated stack that may lock behaviour to one model or one robot platform.
6. Benchmarks and Evidence
The primary public evidence is the accompanying paper (arXiv:2505.07532, 2025) describing the architecture and a set of embodied tasks, and a ROSCon 2024 talk presenting RAI. The rai_bench suite ships with the framework for measuring agent, model, tool, and simulator behaviour, but the project does not publish a single headline benchmark number; reported results are task-specific and should be reproduced locally. Robotec.ai's own case studies use RAI to coordinate agentic reasoning, perception, and manipulation on a warehouse mobile manipulator with AMD Ryzen AI edge hardware, and to drive a ROSbot XL and a Franka arm in demos. These are vendor demonstrations, not independent blind benchmarks, and should be validated before appearing in a procurement document.
The paper itself evaluates deployments on a physical Husarion ROSBot XL coupled with its digital twin, plus two simulations: a robot-arm manipulator and a tractor controller. Each deployment was assessed for control capability, embodiment effectiveness, and perception ability, and the authors note that the work also helped identify and address shortcomings of the generative models used for embodied AI. That is honest, task-scoped evidence rather than a cross-framework leaderboard. For due diligence, the useful signal is that RAI has been exercised on real hardware and in simulation with explicit metrics; the caution is that no third-party, head-to-head benchmark currently ranks RAI against alternative agent frameworks on a common task set.
7. Role in Physics-AI Workflows
In a Physical AI programme, RAI is the behaviour layer that connects a trained model or a general-purpose model to a robot's ROS 2 interface, with logging and benchmarking built in. During development it accelerates the loop from "a model can do X" to "a robot does X safely in a test scenario," because rai_sim and rai_bench let teams exercise agent-generated behaviour before hardware. During validation it provides a repeatable harness: the same instruction can be run across models and across simulation or hardware backends, and rai_bench records the outcome. During deployment it keeps the model choice swappable, so a team can move from a cloud model to an on-premise or air-gapped model without rewriting the robot integration. The NVIDIA company report gives the wider platform context in which ROS 2 and agentic robotics sit.
RAI's place in the stack is therefore complementary to, not competitive with, the simulation engines and silicon it orchestrates. A typical Robotec.ai workflow pairs RAI with Nav2 for navigation and MoveIt for manipulation, then validates the combined behaviour in RoSi or another simulator before field use. For organisations building Physical-AI products, the practical benefit is a standard way to express intent in natural language and to collect comparable evidence about how different models and robots perform on the same instruction. That evidence loop is what turns a one-off demo into a repeatable engineering process, and it is the reason RAI is positioned as the agentic layer of a broader simulation-and-digital-twin platform rather than a standalone chatbot for robots.
8. Limitations and Failure Modes
The most common failure is over-trusting a language model's plan. Because RAI is model-agnostic, a weak or poorly grounded model produces plausible but unsafe or infeasible plans, and the framework has no built-in physics check. A second failure mode is the ROS 2 assumption: a robot without a clean ROS 2 interface needs substantial adaptation before RAI is useful. A third is simulation-reality mismatch: behaviour that passes in rai_sim may fail on hardware because the simulator does not model the relevant contact, latency, or sensor noise. RAI is explicitly a pre-deployment testing tool, not a safety certifier, so any behaviour it generates still requires independent validation before field deployment. Licensing is permissive (Apache-2.0), which lowers adoption risk, but bundles or third-party models a user connects remain under their own terms.
A subtler limitation is evaluation depth. rai_bench can measure whether an agent completed a task, but it does not by itself certify physical safety, robustness under distribution shift, or the absence of jailbreak and hallucination risks that the paper itself flags as inherent to LLM-based agents. Because RAI orchestrates external models, its ceiling is set by those models; switching to a stronger model can improve behaviour without any change to RAI itself. For production use, teams should pair RAI with independent verification, human-in-the-loop oversight (the tablet UI in the AMD warehouse demo is one example), and conservative fallback policies, rather than treating the agent's first plan as the final action.
9. Version and Source Notes
Record one: RAI is maintained by Robotec.ai and published under the Apache-2.0 license; the repository is github.com/RobotecAI/rai and the documentation at robotecai.github.io/rai. Record two: the architecture and design are described in "RAI: Flexible Agent Framework for Embodied AI" (arXiv:2505.07532, 2025; authors Rachwał, Majek, Boczek, Dąbrowski, Liberadzki, Dąbrowski, Ganzha). Record three: RAI is one of the flagship projects of the ROS 2 Embodied AI Community Group, and Robotec.ai co-authored the ROS 2 Simulation Interfaces standard with NVIDIA and Intrinsic; these partner claims come from Robotec.ai's technology page and the ROS community, not from an independent audit. Record four: the framework supports speech interaction (rai_asr, rai_tts), simulation connection (rai_sim), and benchmarking (rai_bench), and is model-agnostic across frontier cloud and open-weight on-premise models; exact package maturity varies by component and should be checked against the repository at the time of use.
Record five: the package set has grown beyond the original core. In addition to rai core, rai whoami, rai_asr, rai_tts, rai_sim, rai_bench, rai_perception and rai_finetune, the project also documents rai_nomad for NoMaD-based navigation integration, reflecting an expanding scope from manipulation and dialogue toward full navigation. Record six: supported runtimes are Ubuntu 22.04/24.04 with Python 3.10/3.12 and ROS 2 Humble or Jazzy; these version constraints are the practical compatibility floor and should be confirmed against the repository's current setup guide before adoption. As with any fast-moving open-source agent project, pin versions and re-run rai_bench when upgrading models or dependencies.
10. References
- Robotec AI, RAI repository, github.com/RobotecAI/rai, README, components, demos and citation.
- Robotec AI, RAI documentation, robotecai.github.io/rai, setup, tutorials and configuration.
- Rachwał K., Majek M., Boczek B., Dąbrowski K., Liberadzki P., Dąbrowski A., Ganzha M., RAI: Flexible Agent Framework for Embodied AI, arXiv:2505.07532, 2025.
- Robotec.ai, Technology Stack, robotec.ai/technology, simulation orchestration, ROS 2, O3DE, Isaac Sim, Genesis, OpenUSD, RAI and the ROS 2 Simulation Interfaces standard.
- Robotec.ai, official site, robotec.ai, products AIKROS, SLINGSHOT, RoSi and RAI positioning, and the company research report on this site.
- ROS 2 Embodied AI Community Group, github.com/ros-wg-embodied-ai, RAI as a flagship project.
- ROSCon 2024, talk on RAI, vimeo.com/1026029511.
- Robotec AI, RAI citation (BibTeX), github.com/RobotecAI/rai, referencing arXiv:2505.07532.