agents.components.vla#

Module Contents#

Classes#

VLA

This component utilizes Vision-Language-Action (VLA) policies served on the LeRobot Async Policy Server (e.g. SmolVLA, Pi0/Pi0.5, NVIDIA GR00T N1.7, ACT, Diffusion) for robot manipulation and control tasks.

API#

class agents.components.vla.VLA(*, inputs: List[agents.ros.Topic], outputs: List[agents.ros.Topic], model_client: agents.clients.lerobot.LeRobotClient, config: agents.config.VLAConfig, component_name: str, **kwargs)#

Bases: agents.components.model_component.ModelComponent

This component utilizes Vision-Language-Action (VLA) policies served on the LeRobot Async Policy Server (e.g. SmolVLA, Pi0/Pi0.5, NVIDIA GR00T N1.7, ACT, Diffusion) for robot manipulation and control tasks.

The component runs as a ROS2 Action Server exposing the <component_name>/manipulate_with_vla action which takes a natural language task description as its goal. While a goal is active, the component continuously streams observations (mapped joint states and camera images) to the policy server at observation_sending_rate and publishes the received action chunks as joint commands at action_sending_rate. Overlapping action chunks from consecutive inferences are merged using the aggregation strategy set in the config (or a custom callable set with set_aggregation_function). Goal termination is configured with set_termination_trigger (after a number of timesteps, on a key press, or on an event).

Parameters:

  • inputs (list[Topic]) – The input topics for the VLA component. This should be a list of Topic objects, containing exactly one JointState topic and at least one Image (or RGBD) topic. The camera topics should be mapped to the dataset camera names in camera_inputs_map of the config.

  • outputs (list[Topic]) – The output topics for the VLA component. This should be a list of Topic objects. JointState, JointTrajectory and JointJog types are handled automatically, covering common input formats for MoveIt Servo and ROS2 Control.

  • model_client (LeRobotClient) – The model client for the VLA component. This must be an instance of LeRobotClient, connected to a running LeRobot Async Policy Server which serves the policy defined in a LeRobotPolicy model.

  • config (VLAConfig) – The configuration for the VLA component. This should be an instance of VLAConfig. joint_names_map and camera_inputs_map are required to map the dataset feature names to the robot’s joints and camera topics.

  • component_name (str) – The name of the VLA component. This should be a string.

Example usage:

joint_states = Topic(name="joint_states", msg_type="JointState")
camera = Topic(name="camera/image_raw", msg_type="Image")
joint_cmd = Topic(name="joint_cmd", msg_type="JointState")

policy = LeRobotPolicy(
    name="pick_policy",
    checkpoint="my_hf_user/smolvla_finetuned",
    policy_type="smolvla",
    dataset_info_file="https://huggingface.co/datasets/my_hf_user/my_dataset/resolve/main/meta/info.json",
)
model_client = LeRobotClient(model=policy, host="127.0.0.1", port=8080)

config = VLAConfig(
    joint_names_map={
        "shoulder_pan.pos": "Rotation",
        "elbow_flex.pos": "Elbow",
    },
    camera_inputs_map={"front": camera},
    robot_urdf_file="./my_robot.urdf",
)
vla_component = VLA(
    inputs=[joint_states, camera],
    outputs=[joint_cmd],
    model_client=model_client,
    config=config,
    component_name="vla",
)
vla_component.set_termination_trigger(mode="timesteps", max_timesteps=200)

A task can then be sent to the running component as a ROS2 action goal, e.g. from the command line:

ros2 action send_goal /vla/manipulate_with_vla automatika_embodied_agents/action/VisionLanguageAction "{task: 'pick up the orange'}"
custom_on_activate()#

Custom activation

custom_on_deactivate()#

Custom deactivation

set_termination_trigger(mode: Literal[timesteps, pynput.keyboard, event] = 'timesteps', max_timesteps: int = 100, stop_key: str = 'q', stop_event: Optional[agents.ros.Event] = None)#

Set the condition used to determine when an action is done.

Parameters:

  • mode – One of ‘timesteps’, ‘keyboard’, ‘event’.

  • max_timesteps – The number of timesteps after which to stop (used if mode=‘timesteps’ or ‘event’).

  • stop_key – The key to press to stop the action (used if mode=‘keyboard’).

signal_done()#

Signals that the action is complete. Can be used as an action for signaled events

set_aggregation_function(agg_fn: Callable[[numpy.ndarray, numpy.ndarray], numpy.ndarray])#

Set the aggregation function to be used for aggregating generated actions from the robot policy model

Parameters:

agg_fn (Callable[[np.ndarray, np.ndarray], np.ndarray]) – A callable that takes two numpy arrays as input and returns a single numpy array.

Raises:

TypeError – If agg_fn is not a callable or does not match the expected signature.

main_action_callback(goal_handle: agents.ros.VisionLanguageAction.Goal)#

Callback for the VLA main action server

Parameters:

goal_handle (VisionLanguageAction.Goal) – Incoming action goal

Returns:

Action result

Return type:

VisionLanguageAction.Result

property additional_model_clients: Optional[Dict[str, agents.clients.model_base.ModelClient]]#

Get the dictionary of additional model clients registered to this component.

Returns:

A dictionary mapping client names (str) to ModelClient instances, or None if not set.

Return type:

Optional[Dict[str, ModelClient]]

fallback_to_local() → str#

Switch from remote model_client to the built-in local model at runtime.

The local model is deployed on first call (lazy initialization) to avoid consuming GPU memory until actually needed. If enable_local_model is not already set in config, it is enabled automatically.

This is commonly used as a target for Actions in the Event system.

Returns:

A confirmation message describing the switch.

Return type:

str

Raises:

RuntimeError – If the local model could not be deployed.

Example:


    from agents.ros import Action

    # Define an action to switch to the 'local model' available in each component
    switch_to_local = Action(
        method=brain.fallback_to_local,
    )

    # Trigger this action if the component fails (e.g. internet outage)
    brain.on_component_fail(action=switch_to_local, max_retries=3)
change_model_client(model_client_name: str) → str#

Hot-swap the active model client at runtime.

This method replaces the component’s current model_client with one from the registered additional_model_clients. It handles the safe de-initialization of the old client and initialization of the new one.

This is commonly used as a target for Actions in the Event system.

Parameters:

model_client_name (str) – The key corresponding to the desired client in additional_model_clients.

Returns:

A confirmation message describing the swap.

Return type:

str

Raises:

RuntimeError – If no additional clients are registered, the requested client name is not found, or initialization fails.

Example:


    from agents.ros import Action

    # Define an action to switch to the 'remote_backup' client defined previously
    switch_to_backup = Action(
        method=brain.change_model_client,
        args=("remote_backup",)
    )

    # Trigger this action if the component fails (e.g. server down)
    brain.on_component_fail(action=switch_to_backup, max_retries=3)
inspect_component() → str#

Return component info including additional model clients.

custom_on_configure()#

Create model client if provided and initialize model.

property warmup: bool#

Enable warmup of the model.

create_all_subscribers()#

Override to handle trigger topics and fixed inputs. Called by parent BaseComponent

activate_all_triggers() → None#

Activates component triggers by attaching execution step to callbacks

destroy_all_subscribers() → None#

Destroys all node subscribers