Live Stream WebSocket

Local only

This feature is not available in the Cloud API.

The live stream WebSocket is the Local API Gateway transport for high-rate robot data. It sends binary MessageStream protobuf envelopes and is intended for video, point clouds, laser scans, poses, and other live streams.

For live Custom KPI values from local-fleet member robots, use the separate Fleet Custom KPI WebSocket. The Local Fleet KPI route uses data-only JSON text frames, not an outer MessageStream envelope.

Flow

1. Log in with POST /apigateway/v2/auth/login.
2. List robots with GET /apigateway/v2/robots.
3. List stream descriptors with GET /apigateway/v2/robots/{robotId}/streams.
4. Select an output descriptor.
5. Open the WebSocket with descriptor fields in the query string.

Endpoint

GET /apigateway/v2/robots/{robotId}/streams/connect

This route must be a WebSocket upgrade request.

Browser URL Example

ws://<agent-host>:19992/apigateway/v2/robots/{robotId}/streams/connect?access_token=<token>&componentName=camera&streamName=image&source=1&streamType=Nimbus.Messages.sensor_msgs.CompressedImage&maxRate=5&withoutDownSampling=false

Use wss:// only when the browser trusts the local HTTPS certificate. For HTTP development, an http:// base URL produces a ws:// WebSocket URL and avoids browser certificate rejection.

Query Parameters

• access_token: local API Gateway token. Browser clients usually strip the leading bearer from the login response before placing the token in the URL.
• componentName: descriptor componentName; required for component streams.
• streamName: descriptor streamName; required for all sources.
• source: descriptor streamSourceType.
• streamType: descriptor streamType; required for ROS streams and used by video source descriptors.
• url: descriptor sourceUrl for ONVIF sources or a user-entered ONVIF/RTSP URL.
• endpoint: descriptor sourceEndpoint for device endpoint sources.
• username and password: optional ONVIF/RTSP credentials. Discovery never returns these values; ask for them only when connecting.
• maxRate: optional requested maximum rate. Omit it or use 0 to let agent defaults apply.
• withoutDownSampling: optional point-cloud downsampling control. Default is false.

Source-Specific Requirements

• Source 1, component stream: componentName and streamName are required.
• Source 2, ROS1 topic: streamName and streamType are required.
• Source 3, ROS2 topic: streamName and streamType are required.
• Source 4, ONVIF URL: url is required. username and password are optional.
• Source 5, RTSP URL: url is required and must start with rtsp://. Credentials can be supplied separately or embedded in the URL.
• Source 6, device endpoint: endpoint is required.

Browser Example

async function login(baseUrl, userToken) {
  const response = await fetch(`${baseUrl}/apigateway/v2/auth/login`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ userToken })
  });
  if (!response.ok) throw new Error(`login failed: ${response.status}`);
  return (await response.json()).token;
}

async function getJson(baseUrl, path, bearerToken) {
  const response = await fetch(`${baseUrl}${path}`, {
    headers: { Authorization: bearerToken }
  });
  if (!response.ok) throw new Error(`${path} failed: ${response.status}`);
  return response.json();
}

async function connectFirstOutputStream(baseUrl, userToken) {
  const bearerToken = await login(baseUrl, userToken);
  const robots = await getJson(baseUrl, '/apigateway/v2/robots', bearerToken);
  const robotId = robots.robotDataList[0].basicData.id;
  const listed = await getJson(baseUrl, `/apigateway/v2/robots/${encodeURIComponent(robotId)}/streams`, bearerToken);
  const stream = listed.robotStreams.find(item => item.direction === 'output');
  if (!stream) throw new Error('no output stream descriptor found');

  const query = new URLSearchParams();
  query.set('access_token', bearerToken.replace(/^bearer\s+/i, ''));
  query.set('componentName', stream.componentName || '');
  query.set('streamName', stream.streamName || '');
  query.set('source', String(stream.streamSourceType || 1));
  query.set('streamType', stream.streamType || '');
  if (stream.sourceUrl) query.set('url', stream.sourceUrl);
  if (stream.sourceEndpoint) query.set('endpoint', stream.sourceEndpoint);
  query.set('maxRate', '5');
  query.set('withoutDownSampling', 'false');

  const url = new URL(baseUrl);
  url.protocol = url.protocol === 'https:' ? 'wss:' : 'ws:';
  const socket = new WebSocket(`${url.origin}/apigateway/v2/robots/${encodeURIComponent(robotId)}/streams/connect?${query}`);
  socket.binaryType = 'arraybuffer';
  socket.onmessage = event => {
    console.log('binary MessageStream bytes', event.data.byteLength);
  };
  return socket;
}

Payload Format

Every WebSocket message is one serialized MessageStream protobuf envelope.

The envelope contains:

• stream identity fields;
• dataType, which names the Nimbus.Messages.* payload type;
• data, which contains the typed protobuf payload;
• compression fields.

Decode in this order:

1. Decode the WebSocket frame as a MessageStream envelope.
2. If the envelope is compressed, inspect the compression type and inflate gzip payloads before payload decoding.
3. Use dataType to choose the typed protobuf parser for data.

Common renderer targets include:

• Nimbus.Messages.sensor_msgs.CompressedImage
• Nimbus.Messages.sensor_msgs.Image
• Nimbus.Messages.sensor_msgs.LaserScan
• Nimbus.Messages.geometry_msgs.PoseStamped
• Nimbus.Messages.sensor_msgs.PointCloud2

Delivery Semantics

• This is live best-effort delivery.
• It is not a replay API.
• It does not guarantee delivery after reconnect.
• Each client has a bounded outbound queue with capacity 1; slow clients can drop old queued frames instead of growing memory.
• Use maxRate when a UI cannot keep up with the stream.

Common Errors

• 400: request is not a WebSocket upgrade or descriptor query fields are invalid.
• 401: token is missing, expired, or invalid.
• 404: local robot or stream target was not found.
• 501: selected robot is a local-fleet member; WebSocket tunneling is not implemented.
• Browser close code 1006: the browser did not receive a close frame. With local HTTPS, this commonly means the browser rejected wss:// because the local certificate is not trusted.