JavaScript SDK / Media

Media#

Inbound audio and video arrive on the WebRTC media plane. The SDK can attach them to a <video> element for you, correct defaults included, or hand you the raw tracks.

The one-call path#

const { session, muted, reason } = await client.streamToVideoElement("stage");

target is a <video> element, its id, or a CSS selector. An id is tried first and a selector second, because both are things a developer will reasonably pass and guessing wrong should not be an error. Failing to resolve one throws ConfigurationError.

The call sets playsInline so iOS does not take over the whole screen, sets srcObject, and starts playback.

session.streamToVideoElement(target) does the same for a session you already have.

The autoplay problem#

Chrome and Safari will not start an unmuted media element without a prior user gesture. Muted autoplay is always allowed. So the SDK tries with sound, falls back to muted, and returns which happened.

if (muted) {
  unmuteButton.hidden = false;
  unmuteButton.onclick = () => session.unmute();
}

session.unmute()#

Must be called from inside a user gesture handler, a click or a tap. Called from anywhere else the browser simply refuses again, which looks like the method is broken when it is the policy doing exactly what it says. Throws SessionError if nothing has been attached yet.

The retry is attempted on any play failure rather than only on NotAllowedError: browsers are not consistent about which error name they use for the autoplay block, and the cost of a pointless second attempt is one rejected promise, while the cost of missing the block is a blank avatar.

Doing it by hand#

streamToVideoElement is connect() plus the attach. If you want the session without the element:

const session = await client.connect();
videoEl.playsInline = true;
videoEl.srcObject = session.mediaStream as MediaStream;
await videoEl.play();   // you now own the autoplay problem described above

The attach helpers are exported if you want the behaviour without the session wiring:

import { attachStream, resolveVideoElement, unmuteElement } from "@zeligate/zeli-avatar";
 
const el = resolveVideoElement("#stage video");
const { muted } = await attachStream(el, session.mediaStream);
// from a click handler:
await unmuteElement(el);

They work against a small structural VideoElementLike shape rather than HTMLVideoElement, so the package builds and runs under Node, where the DOM lib does not exist.

Tracks#

ZeliEvent.TRACK fires once per inbound track as it arrives, which is the hook to use if you want to route audio and video separately.

import { ZeliEvent, MediaKind } from "@zeligate/zeli-avatar";
 
client.on(ZeliEvent.TRACK, (track) => {
  if (track.kind === MediaKind.VIDEO) attachVideo(track);
  else attachAudio(track);
});

Choosing what to receive#

Pass SessionOptions to connect() or streamToVideoElement():

const session = await client.connect({
  receiveAudio: true,
  receiveVideo: true,
  idleAvatarId: "01-presenter-male",
});

receiveAudio and receiveVideo both default to true. idleAvatarId names a separate clip to hold between utterances; omit it and the box uses the requested avatar's own idle clip.