agents.components#

A Component is the main execution unit in EmbodiedAgents and in essence each component is synctactic sugar over a ROS2 Lifecycle Node. EmbodiedAgents provides the following components. These components can be arbitrarily combined to form an embodied agent graph.

Component Name

Description

LLM

Uses large language models (e.g., LLaMA) to process text input. Can be used for reasoning, tool calling, instruction following, or dialogue. It can also utilize vector DBs for storing and retreiving contextual information.

VLM

Leverages multimodal LLMs (e.g., Llava) for understanding and processing both text and image data. Inherits all functionalities of the LLM component.

VLA

Provides an interface to utilize Vision Language Action (VLA) models for manipulation and control tasks. It can use VLA Policies (such as SmolVLA, Pi0/Pi0.5, NVIDIA GR00T N1.7 etc.) served with HuggingFace LeRobot Async Policy Server and publish them to common topic formats in MoveIt Servo and ROS2 Control.

MoveIt

Provides classical, collision aware manipulation by driving a running MoveIt 2 move_group node. Plans and executes end-effector pose goals, joint goals, named targets from the robot’s SRDF and straight-line Cartesian paths, and controls the gripper. Complements the VLA component, which learns manipulation skills, by providing planned and collision checked motion.

SpeechToText

Converts spoken audio into text using speech-to-text models (e.g., Whisper). Suitable for voice command recognition. It also implements small on-board models for Voice Activity Detection (VAD) and Wakeword recognition, using audio capture devices onboard the robot.

TextToSpeech

Synthesizes audio from text using TTS models (e.g., TransformersTTS). Output audio can be played using the robot’s speakers or published to a topic. Implements say(text) and stop_playback functions to play/stop audio based on events from other components or the environment.

Memory

Provides a graph-based spatio-temporal memory primitive powered by eMEM. Encodes perception streams (e.g., VLM descriptions, detections) and interoception streams (internal body state) into a memory indexed by meaning, location, and time. Exposes structured retrieval tools (semantic, spatial, temporal, entity, episode) as component actions and supports episode-based consolidation with entity tracking.

SemanticRouter

Routes information between topics based on semantic content and predefined routing rules. Uses a vector DB for semantic matching, an LLM for decision-making, or a decision model that chooses the route in one forward pass. This allows for creating complex graphs of components where a single input source can trigger different information processing pathways.

DecisionComponent

Asks a decision model (served by a TypeSafe-compatible /v1/systemone API) typed questions about its inputs: yes/no, a choice among options, or a score on ordered levels, each answered in one forward pass with a probability for every option. Publishes each question’s answers on its own topic as Decision messages, so events can act on them with thresholds.

Vision

An essential component in all vision powered robots. Performs object detection and tracking on incoming images. Outputs object classes, bounding boxes, and confidence scores. It implements a low-latency small on-board classification model as well.

Cortex

An LLM-powered high level cognitive component that combines the roll of a long term task planner and executor and also serves as the system monitor. Receives high-level natural language goals, decomposes them into steps by inspecting available components, and executes them with per-step confirmation. Automatically discovers component actions and ROS entrypoints as callable tools.

MotionDetector

Detects motion from a stream of images or a stream of point clouds. Publishes the motion state on a Bool topic (useful as an event source for other components), video messages of coherent motion sequences (image inputs) or the coordinates of motion centers (point cloud inputs). Supports ego-motion handling through an optional odometry topic.

Submodules#