Skip to main content

Key Concepts and Overall Architecture

Robot Layer​

The Robot Layer represents the physical robot or smart edge device running ROS2. It is responsible for exposing a set of topics for:

  • Sensory Data Streams
    Examples: depth cameras, microphone arrays, LiDAR sensors.

  • Actuations
    Examples: movement commands, playing sounds (MP3 files or streams), robot gestures, and other actuators present.

The names and configuration of these topics are specified via a JSON configuration file, allowing flexible adaptation across different platforms without modifying the core code.

HRI Framework​

The HRI Framework operates within a ROS2 Humble environment on the robot or edge device.

Its purpose is to manage consistent, reusable computations that are typically required across HRI applications, including:

  • Extracting and tracking human skeletons.
  • Transforming skeletons into a 3D coordinate space.
  • Fusing skeleton and sound source data.
  • Interpreting visual and vocal social cues.

The HRI Framework is structured into cognitive layers, each implemented as a main ROS node that communicates with others through ROS messages.

Each layer:

  • Has its own JSON configuration file that exposes its input and output topics.
  • Publishes and subscribes to ROS string messages formatted as JSON objects.
  • Structures these JSON objects according to classes defined in the HRI-LIB.

HRI-LIB​

The HRI-LIB is a pure Python library (non-ROS) containing:

  • HRI-Interfaces: Abstract definitions for different functionalities.
  • HRI-Implementations: Concrete classes that realize these interfaces.
  • Configuration Files: JSON-based settings for flexible control.

Implementations are divided into:

  • HRI Toolkit: Official components developed by the HRI Consortium and Cogniteam.
  • HRI App: User-defined extensions or replacements tailored to specific needs.

The HRI-LIB supports seamless integration of rapidly evolving technologies (e.g., new AI models for object detection, speech-to-text, text-to-speech, LLMs), following the Open-Closed Principle:

"Software entities should be open for extension, but closed for modification."

hri_framework_config.json​

The hri_framework_config.json file enables configuring complex application-level behaviors without the need for additional coding.

It allows developers to:

  • Define workflows based on primitive robot capabilities.
  • Create high-level interactions by composing cognitive functionalities.

Use of LLMs (Large Language Models)​

While modern LLMs offer powerful reasoning and language capabilities, the HRI Framework avoids using them as opaque decision-making engines.
Instead, LLMs are carefully applied as narrow, function-like entities for specific tasks:

  • Mapping natural language instructions to actionable commands.
  • Reasoning over complex queries not pre-programmed.
  • Rephrasing robot utterances based on the calculated and desired social context and interaction type.

This hybrid approach ensures:

  • Transparency: All critical decisions remain traceable.
  • Safety: Reducing unpredictability from black-box AI behaviors.

HRI-DB​

The HRI-DB is a document-based, dictionary-of-dictionaries database included in the HRI Toolkit.

It allows:

  • Natural language queries over structured robot knowledge.
  • Retrieving complex entities (e.g., "Find the beautiful but expensive water bottle Bob likes.").

When the database fails to retrieve an entity based on direct lookup, an LLM is engaged in applying common-sense reasoning to fulfill the query.
This enables complex interactions such as:

  • "Bring the bottle with my favorite color to Bob."
  • "Take the folder closest to me to Bob's office."

The HRI-DB bridges symbolic knowledge representation and natural language understanding in a safe and structured manner.

Summary

The HRI Consortium's architecture enables robust, socially aware human-robot interactions through:

  • Modular and configurable design.
  • Safe, explainable AI methods.
  • Seamless adaptability to future technological advances.

By blending classical robotics, modern AI, and careful application of LLMs, the HRI Framework lays the foundation for the next generation of natural, responsible human-machine collaboration.