EmbodiedAgents
agents.components#
A Component is the main execution unit in EmbodiedAgents and in essence each component is synctactic sugar over a ROS2 Lifecycle Node. EmbodiedAgents provides the following components. These components can be arbitrarily combined to form an embodied agent graph.
Component Name |
Description |
|---|---|
Uses large language models (e.g., LLaMA) to process text input. Can be used for reasoning, tool calling, instruction following, or dialogue. It can also utilize vector DBs for storing and retreiving contextual information. |
|
Leverages multimodal LLMs (e.g., Llava) for understanding and processing both text and image data. Inherits all functionalities of the LLM component. |
|
Provides an interface to utilize Vision Language Action (VLA) models for manipulation and control tasks. It can use VLA Policies (such as SmolVLA, Pi0/Pi0.5, NVIDIA GR00T N1.7 etc.) served with HuggingFace LeRobot Async Policy Server and publish them to common topic formats in MoveIt Servo and ROS2 Control. |
|
Converts spoken audio into text using speech-to-text models (e.g., Whisper). Suitable for voice command recognition. It also implements small on-board models for Voice Activity Detection (VAD) and Wakeword recognition, using audio capture devices onboard the robot. |
|
Synthesizes audio from text using TTS models (e.g., TransformersTTS). Output audio can be played using the robot’s speakers or published to a topic. Implements |
|
MapEncoding (deprecated — use Memory) |
Provides a spatio-temporal working memory by converting semantic outputs (e.g., from MLLMs or Vision) into a structured map representation. Uses robot localization data and output topics from other components to store information in a vector DB. Deprecated: use the Memory component instead. |
Provides a graph-based spatio-temporal memory primitive powered by eMEM. Encodes perception streams (e.g., VLM descriptions, detections) and interoception streams (internal body state) into a memory indexed by meaning, location, and time. Exposes structured retrieval tools (semantic, spatial, temporal, entity, episode) as component actions and supports episode-based consolidation with entity tracking. |
|
Routes information between topics based on semantic content and predefined routing rules. Uses a vector DB for semantic matching or an LLM for decision-making. This allows for creating complex graphs of components where a single input source can trigger different information processing pathways. |
|
An essential component in all vision powered robots. Performs object detection and tracking on incoming images. Outputs object classes, bounding boxes, and confidence scores. It implements a low-latency small on-board classification model as well. |
|
An LLM-powered high level cognitive component that combines the roll of a long term task planner and executor and also serves as the system monitor. Receives high-level natural language goals, decomposes them into steps by inspecting available components, and executes them with per-step confirmation. Automatically discovers component actions and ROS entrypoints as callable tools. |
|
Detects motion from a stream of images or a stream of point clouds. Publishes the motion state on a Bool topic (useful as an event source for other components), video messages of coherent motion sequences (image inputs) or the coordinates of motion centers (point cloud inputs). Supports ego-motion handling through an optional odometry topic. |