Live Stream WebSocket

Local only

This feature is not available in the Cloud API.

The live stream WebSocket is the Local API Gateway connection for continuous robot data. It can:

• receive output such as pose, images, point clouds, and laser scans;
• send low-latency Twist or AckermannDriveStamped teleoperation commands.

For live Custom KPI values from local-fleet member robots, use the separate Fleet Custom KPI WebSocket. That route always uses JSON and has a different payload.

Basic Flow

1. Log in with POST /apigateway/v2/auth/login.
2. List robots with GET /apigateway/v2/robots.
3. List stream descriptors with GET /apigateway/v2/robots/{robotId}/streams.
4. Select an output descriptor to receive data, or a supported input descriptor to send commands.
5. Open one WebSocket for that descriptor, direction, and format.

To receive feedback while sending commands, open two WebSockets: one input connection and one output connection.

Endpoint

GET /apigateway/v2/robots/{robotId}/streams/connect

This route must be a WebSocket upgrade request.

Choose the Frame Format

Add these optional query parameters:

• direction=output|input
• format=protobuf|json

If they are omitted, the connection uses direction=output&format=protobuf. This keeps older output clients working without URL or decoder changes.

The four combinations are:

• output + protobuf: the Agent sends binary MessageStream envelopes. Decode the envelope, then decode its typed data field.
• output + json: the Agent sends the selected typed message directly as a JSON text frame. For example, a pose frame is the pose object, with no extra MessageStream wrapper.
• input + protobuf: send the raw serialized command message as a binary frame. Do not wrap it in MessageStream.
• input + json: send the command object directly as a JSON text frame.

JSON is convenient for browsers, scripts, pose, status, and other readable values. Protobuf is more efficient and is recommended for images, scans, point clouds, and other large or frequent output.

Query Parameters

• access_token: Local API Gateway token. Browser clients normally remove the leading bearer before placing the token in the URL.
• componentName: descriptor componentName; required for component streams.
• streamName: descriptor streamName; required for every source.
• source: descriptor streamSourceType.
• streamType: descriptor streamType.
• direction: output or input; default output.
• format: protobuf or json; default protobuf.
• url: descriptor sourceUrl for ONVIF, or a user-entered ONVIF/RTSP URL.
• endpoint: descriptor sourceEndpoint for a device source.
• username and password: optional ONVIF/RTSP credentials. Discovery does not return credentials.
• maxRate: optional output rate limit. Omit it or use 0 to let Agent defaults apply.
• withoutDownSampling: optional output point-cloud control; default false.

maxRate and withoutDownSampling apply to output connections. Input command rate is limited by the Agent as described under Teleoperation Safety.

Source Support

Output accepts:

• 1: Nimbus component stream;
• 2: ROS1 topic;
• 3: ROS2 topic;
• 4: ONVIF URL;
• 5: RTSP URL;
• 6: device endpoint, such as /dev/video0.

Input accepts component, ROS1, and ROS2 sources (1-3). Use the fields returned by stream listing; do not guess generated gateway component names.

JSON Output Example

This browser example prints a readable typed JSON frame, such as PoseStamped:

const query = new URLSearchParams({
  access_token: token.replace(/^bearer\s+/i, ''),
  componentName: stream.componentName || '',
  streamName: stream.streamName,
  source: String(stream.streamSourceType),
  streamType: stream.streamType,
  direction: 'output',
  format: 'json'
});

const socket = new WebSocket(
  `ws://agent-host:19992/apigateway/v2/robots/${encodeURIComponent(robotId)}/streams/connect?${query}`
);
socket.onmessage = event => {
  const value = JSON.parse(event.data);
  console.log('typed stream value', value);
};

Use wss:// only when the client trusts the Agent's local HTTPS certificate.

Protobuf Output

Every output WebSocket message is one serialized MessageStream envelope when format=protobuf. The envelope contains:

• stream identity;
• dataType, which names the Nimbus.Messages.* payload type;
• data, which contains the typed protobuf payload;
• compression information.

Decode it in this order:

1. Decode the WebSocket frame as MessageStream.
2. If it is gzip-compressed, inflate data.
3. Select the typed protobuf parser using dataType.
4. Parse data with that parser.

Common output types include CompressedImage, Image, LaserScan, PoseStamped, and PointCloud2.

Teleoperation Input

Input WebSockets initially accept these exact descriptor types:

• Nimbus.Messages.geometry_msgs.Twist
• Nimbus.Messages.ackermann_msgs.AckermannDriveStamped

JSON Twist example:

{
  "linear": { "x": 0.2, "y": 0, "z": 0 },
  "angular": { "x": 0, "y": 0, "z": 0.1 }
}

JSON Ackermann example:

{
  "drive": {
    "steeringAngle": 0.1,
    "steeringAngleVelocity": 0,
    "speed": 0.2,
    "acceleration": 0,
    "jerk": 0
  }
}

For format=protobuf, serialize the selected Twist or AckermannDriveStamped object and send those raw bytes. A binary input frame is not a MessageStream envelope.

Teleoperation Safety

• Only one input WebSocket can control the same resolved stream target. A second connection receives HTTP 409.
• The Agent publishes at most 20 commands per second. When frames arrive faster, the newest valid command replaces the older pending command.
• If no valid command arrives for 500 ms, the Agent publishes a type-correct zero command.
• Invalid data, disconnect, cancellation, host stop, and Agent shutdown also attempt a zero command.
• Each complete input message is limited to 64 KiB. Fragmented WebSocket frames are supported.
• Speed and steering values must be finite. The robot controller is still responsible for its own physical speed and steering limits.
• A cooperative client should send zero before disconnecting. The Agent deadman remains the final safety mechanism when client cleanup cannot run.

Use the joystick at /debug/local-api-gateway/streams to exercise discovered Twist and Ackermann inputs locally. It supports JSON and protobuf and sends zero on normal joystick release paths.

Python Examples

The Agent repository includes two relevant console samples:

# Read output as direct JSON, for example a pose stream.
nimbus-apigateway-streams <local-user-token> <base-url> pose --format json

# Send a finite one-second command after selecting an input descriptor.
nimbus-apigateway-teleoperation <local-user-token> <base-url> cmd_vel \
  --format json --speed 0.2 --turn 0.1 --rate 10 --duration 1

The teleoperation sample defaults speed and turn to zero, caps its rate at 20 Hz, caps duration at five minutes, and sends zero before and after the requested command interval.

Output Delivery and Size Limits

• Output is live, best-effort delivery, not guaranteed replay.
• Each output client has a bounded queue of one frame. Slow clients can lose older queued frames.
• JSON conversion allows at most 12 MiB of decoded protobuf and a 16 MiB UTF-8 JSON frame. The Agent closes oversized JSON output with code 1009; use protobuf for that stream.
• Input messages allow at most 64 KiB.

Errors

Before the WebSocket upgrade:

• 400: missing/invalid query, unsupported input type/source, or not a WebSocket request;
• 401: token missing, expired, or invalid;
• 404: local robot, stream, or message type not found;
• 409: another input WebSocket already controls the target;
• 501: the selected robot is a local-fleet member; live WebSocket tunneling is not implemented.

After the upgrade, the close code explains the failure:

• 1000: normal close;
• 1003: text/binary frame does not match format;
• 1007: malformed protobuf, JSON, or UTF-8;
• 1008: command violates input policy, such as a non-finite value;
• 1009: input frame or JSON output is too large;
• 1011: target publication or output conversion failed.

Browser code 1006 means the browser did not receive a close frame. With local HTTPS, a common cause is an untrusted local certificate.