🌊 Tide

Information moves like the tide. We pick the waves worth studying.

δΈ­ζ–‡

← Today's tide

Published 2026-07-02 Β· PhysicalAI System Group @ SAIL Lab, Southeast University Β· Ling Xu, Chuyu Han, Borui Li et al.

Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

arXiv Β· HuggingFace Β· β–²3

Abstract (verbatim)

Embodied AI models now span vision-language-action (VLA) models and world-action models (WAMs), but practical deployment remains fragmented across model-specific Python stacks, backend assumptions, and robot-side glue code, especially on heterogeneous edge devices. Existing inference runtimes are designed mainly for request-response serving and therefore do not satisfy the runtime contract of embodied deployment: multi-rate execution inside closed-loop control, latency-first batch-1 inference on heterogeneous hardware, and extensible embodied interfaces beyond fixed token I/O. We present Embodied.cpp, a portable C++ inference runtime for embodied models. Based on an architectural analysis of representative VLA models and WAMs, Embodied.cpp captures a shared execution path and organizes it into five layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters. The runtime provides modular multi-rate execution, latency-first fused inference, and extensible operator and I/O support, enabling deployment across heterogeneous devices, robots, and simulators through one backend abstraction. We evaluate Embodied.cpp on two VLA models, HY-VLA and pi0.5, and on a preliminary WAM benchmark using a LingBot-VA Transformer block. The VLA deployments achieve successful closed-loop execution with 100.0% and 91.0% task success rates, respectively. The WAM benchmark reduces block memory from 312.2 MiB to 88.1 MiB. These results show that Embodied.cpp improves deployment efficiency while preserving high accuracy across diverse embodied model architectures.

Background

Background Analysis

With the rapid development of artificial intelligence technology, embodied AI has become a hot spot in research and application. This technology is mainly used in robots and simulation environments, enabling machines to interact with the physical world through perception, decision-making, and execution. However, although academia and industry have developed a large number of embodied models, such as vision-language-action (VLA) models and world-action models (WAMs), there are still many challenges in practical deployment.

Firstly, the deployment of embodied models needs to meet the requirements of real-time and stability, which requires the inference system to execute components such as perception encoders, transformer backbones, and predictive branches at different rates within a closed-loop control. Secondly, due to the limitations of hardware resources, the inference system needs to achieve low latency, low jitter, and small-batch execution on heterogeneous edge devices. Finally, the interface of embodied models needs to be more flexible to adapt to multimodal inputs and diverse outputs.

However, previous inference runtimes are mainly designed for request-response serving and cannot meet the needs of embodied deployment. These systems usually have relatively uniform token interfaces and throughput-oriented optimizations, while embodied inference needs to handle dependencies on the robot and simulator sides, as well as custom operators and heterogeneous outputs. Therefore, even a powerful model needs to be stitched with Python research code, backend-specific inference paths, hand-written sensor wrappers, and platform-specific control logic before it can work on a robot.

To solve these problems, this paper proposes Embodied.cpp, a portable C++ inference runtime for embodied models. By analyzing representative VLA models and WAMs, this runtime reveals a shared execution path and organizes it into five layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters. This design enables Embodied.cpp to achieve efficient deployment on heterogeneous devices, robots, and simulators while maintaining high accuracy.

In summary, the key difference of Embodied.cpp is that it is optimized for the specific needs of embodied deployment, providing a modular multi-rate execution environment, latency-first fused inference, and extensible operator and I/O support. This enables embodied models to be efficiently deployed in various heterogeneous environments while maintaining high accuracy.

Method, Figure by Figure

M E T H O D1 / 2

Figure 2: Project overview of Embodied.cpp . Diverse sensors and datasets enter through input adapters, model execution is unified around one embodied-model runtime that covers VLA models, WAMs, and future variants, and outputs are bridged to simulators and real robots through deployment adapters. Beneath this shared path, three runtime capabilities directly address the core challenges: modular multi-rate execution, latency-first batch-1 execution on heterogeneous hardware, and an embodied AI kernel warehouse for reusable operators and model-specific kernels.

This figure presents the overall architecture of the Embodied.cpp project, which aims to provide a portable inference runtime for embodied AI models to address the current fragmentation in deployment.

First, let's look at the data input part. The "Sensor" module on the left lists various types of sensors, such as Camera, Force, Tactile, and IMU. The data from these sensors enters the system through the "Input Adapter (Sen./Data.)". The "Dataset" module below lists datasets like LIBERO, DROID, etc., which can also serve as input data sources. The input adapter's role is to convert these diverse sensor data and datasets into a unified format that the system can process.

Next is the core model execution part, enclosed in the red dashed box labeled "Embodied.cpp". It contains several key components: 1. Embodied.cpp Model Gallery and Embodied.cpp Runtime Lib.: This is the core of the entire system, responsible for loading and executing various embodied AI models, including Vision-Language-Action (VLA) models, World-Action Models (WAMs), and other types of embodied AI models (Other EAI Models). The model execution process starts from multiple "Input"s, goes through the processing of the model library and runtime library, and generates "Future World" and "Action". 2. Three key runtime capabilities: - Modular Multi-rate Execution: Supports multi-rate execution in closed-loop control to meet the different time step requirements of different tasks. - Batch-1 Hetero. HW Execution: Emphasizes low-latency inference on heterogeneous hardware (such as CPU, GPU, NPU, etc.), especially for the case where the batch size is 1, which is very important in real-time systems. - Embodied AI Kernel Warehouse: Provides reusable operators and model-specific kernels to support the execution of different models. 3. Hardware Backends: Including CPU, GPU, NPU, etc., it provides different hardware acceleration options to adapt to different computing needs and device capabilities.

Then there is the data output part, which sends the model's output (such as actions) to different targets through the "Output Adapter (Sim/Real)". The "Platform" module includes simulators (such as Isaac Sim, GAZEBO) and real-world devices (such as ROS, apollo cyber RT). The output adapter converts the processed data into a format that these platforms can understand, thus achieving control over robots or simulated environments.

This figure reveals the workflow of Embodied.cpp: first, diverse sensors and datasets enter the system through the input adapter; then, the core model execution part (Embodied.cpp runtime) processes these inputs, using its three key runtime capabilities (multi-rate execution, low-latency inference on heterogeneous hardware, kernel warehouse) to execute various embodied AI models; finally, the processed output is sent to simulators or real-world robot devices through the output adapter, realizing perception, decision-making, and action in the environment.

In summary, Embodied.cpp solves the fragmentation problem in the deployment of embodied AI models by providing a unified inference runtime, supporting efficient model execution and closed-loop control on heterogeneous devices and platforms.

Figure 1: Architectural taxonomy of embodied models. We first separate VLA models (a-d) from WAMs (e-h), and then organize each family by internal inference structure.

This figure (Figure 1) presents the architectural taxonomy of embodied models. We first separate Vision - Language - Action (VLA) models (subfigures a - d) from World - Action Models (WAMs) (subfigures e - h), and then organize each category according to their internal inference structures, which clearly shows the design logic and data flow of different embodied models:

Visual - Language - Action (VLA) Model Part (a - d)

  • (a) AR - Token VLA: The inputs (Inputs, light - blue boxes) include observations (Obs.) and instructions (Instr.). These inputs are fed into the "Auto Reg. Transformer" (model building block, gray box). After processing by this module, action tokens (Action Tokens, orange box, output) are generated. The data flow is: Obs. + Instr. β†’ Auto Reg. Transformer β†’ Action Tokens.
  • (b) VLM - Backboned VLA: The inputs include observations (Obs.), instructions (Instr.), and states (State). First, the inputs are sent to the "Pretrained VLM" (pre - trained Visual - Language Model, model building block), and then pass through the "Flow / Diffusion Head" (model building block). Finally, the output is generated at the "Action Trunk". The data flow is: Obs. + Instr. + State β†’ Pretrained VLM β†’ Flow / Diffusion Head β†’ Action Trunk.
  • (c) Hierarchical VLA: The inputs contain observations (Obs.), instructions (Instr.), and memories (Memory). First, the "High - level Planner (VLM)" (high - level planner using VLM, model building block) processes the inputs and generates "Subgoal / Option" (subgoal/option, intermediate representation, green box). Then, it is passed to the "Low - level Controller (VLA)" (low - level controller, VLA, model building block), and finally the output is action tokens or trunk (Action Tokens/Trunk). The data flow is: Obs. + Instr. + Memory β†’ High - level Planner (VLM) β†’ Subgoal / Option β†’ Low - level Controller (VLA) β†’ Action Tokens/Trunk.
  • (d) Asynchronous VLA: The inputs include observations (Obs.), priorities (Prio.), and instructions (Instr.). They are processed by "Enc.1", "Enc.2", and "Enc.3" (three encoders, model building blocks) respectively, and then sent to the "Action Expert" (action expert, model building block). Finally, the output is at the "Action Trunk". The data flow is: Obs. + Prio. + Instr. β†’ Enc.1/Enc.2/Enc.3 β†’ Action Expert β†’ Action Trunk.

World - Action Model (WAM) Part (e - h)

  • (e) Predict - then - Act WAM: The input is the combination of observation and instruction (Obs. + Instr.). It is sent to the "World Model" (world model, model building block), which generates "Predicted Future" (predicted future, intermediate representation, green box). Then, after passing through the "Policy/Inverse Dynamics" (policy/inverse dynamics, model building block), the output is action tokens or trunk (Action Tokens/Trunk). The data flow is: Obs. + Instr. β†’ World Model β†’ Predicted Future β†’ Policy/Inverse Dynamics β†’ Action Tokens/Trunk.
  • (f) Unified AR - Modeling WAM: The inputs include image tokens (Img. Tok.), text tokens (Tex. Tok.), and action tokens (Act. Tok.). They are sent to the "Unified AR Backbone" (unified autoregressive backbone, model building block), and the outputs are future images (Future Img.) and action tokens (Action Tok.). The data flow is: Img. Tok. + Tex. Tok. + Act. Tok. β†’ Unified AR Backbone β†’ Future Img. + Action Tok.
  • (g) Shared - Backbone WAM: The inputs include observations (Obs.), instructions (Instr.), and action contexts (Act. Ctx.). They are sent to the "Video/World Backbone" (video/world backbone, model building block), which generates "Future Video" (future video, intermediate representation) and action tokens (Action Tok.) through the "Act. Decoder" (action decoder, model building block). The data flow is: Obs. + Instr. + Act. Ctx. β†’ Video/World Backbone β†’ Future Video + Act. Decoder β†’ Action Tok.
  • (h) Latent - space WAM: The input is the combination of observation and instruction (Obs. + Instr.). It is sent to the "Latent World Model" (latent world model, model building block), which generates "Latent Future / Subgoal" (latent future/subgoal, intermediate representation, green box). Then, after passing through the "Action Expert" (action expert, model building block), the output is action tokens or trunk (Action Tokens/Trunk). The data flow is: Obs. + Instr. β†’ Latent World Model β†’ Latent Future / Subgoal β†’ Action Expert β†’ Action Tokens/Trunk.

Overall Architectural Logic and Method Operation

This figure reveals the internal structural differences between the two main families of embodied models (VLA and WAM): - VLA models: They usually combine the processing of vision, language, and action. Some use autoregressive Transformers (such as a), some are based on pre - trained VLMs (such as b), some have hierarchical structures (such as c) or asynchronous processing (such as d). The core is to convert multi - modal inputs (observations, instructions, etc.) into action outputs. In the middle, processes such as planning, control, or encoding and decoding may be involved. - WAM models: They focus more on the combination of world models (predicting future states) and actions. Some adopt the "predict - then - act" strategy (such as e), some use unified autoregressive modeling (such as f), some share backbones (such as g), or process in latent spaces (such as h). The core is to generate actions by predicting the future through the world model and combining action strategies, or directly model the action - related sequences.

The commonality of the data flow is: Input (observations, instructions, etc.) β†’ Model building blocks (process inputs, may generate intermediate representations) β†’ Output (action tokens, trunk, or future predictions, etc.). The difference between different models lies in the type of model building blocks (such as Transformer, VLM, world model, etc.), the processing of intermediate representations (such as subgoals, latent futures, etc.), and the type of inputs and outputs (such as whether they contain states, priorities, multi - modal tokens, etc.).

Through this classification, we can see the diversity of embodied models. The Embodied.cpp runtime in the paper is based on the analysis of these architectures, extracts the shared execution path, and organizes it into five layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters. This is to achieve deployment across heterogeneous devices, meet the runtime contract of embodied deployment (such as multi - rate execution within closed - loop control, latency - first batch - 1 inference on heterogeneous hardware, extensible embodied interfaces, etc.).

← β†’ flip Β· ↑ ↓ scroll notes Β· Esc exit