Zeli AvatarDeveloper docs

JavaScript SDK / Events & models

Events and models

Register handlers on the client, not the session, so they are in place before connect() opens anything. Handler arguments are typed per event through ZeliEventMap.

import { ZeliEvent, type ZeliEventMap } from "@zeligate/zeli-avatar";

ZeliEvent

EventHandler arguments
CONNECTION_ESTABLISHEDnone
SESSION_READYinfo: SessionInfo
TRACKtrack: MediaStreamTrackLike
MESSAGE_RECEIVEDmessage: Message
MESSAGE_STREAM_EVENT_RECEIVEDevent: MessageStreamEvent
MESSAGE_HISTORY_UPDATEDhistory: Message[]
AVATAR_SPEECH_STARTEDcorrelationId: string | null
AVATAR_SPEECH_ENDEDcorrelationId: string | null
TALK_STREAM_INTERRUPTEDcorrelationId: string | null
EMOTION_DETECTEDevent: EmotionEvent. Reserved, not currently emitted. No box sends an emotion frame
CREDENTIAL_STATElive: boolean. Whether the box accepts the credential now on the control socket
SESSION_EXPIREDevent: SessionExpiredEvent. The box retired this session; a close follows immediately
UNKNOWN_FRAMEevent: UnknownFrameEvent. A control frame this SDK has no branch for, handed over whole
SERVER_WARNINGmessage: string
ERRORerror: Error
CONNECTION_CLOSEDcode: ConnectionCloseCode, reason: string | null. The reason carries the box's own sentence when it gave one

From the microphone

These arrive only while session.startMicrophone() is running. The reply to something spoken is not among them, deliberately: it arrives on MESSAGE_STREAM_EVENT_RECEIVED and MESSAGE_RECEIVED above, exactly as a typed turn does, so one renderer serves both.

EventHandler arguments
MICROPHONE_READYinfo: MicrophoneReady. Recognition is attached and the box is listening
TRANSCRIPT_PARTIALevent: TranscriptPartial. The running guess, revised as somebody talks
TRANSCRIPT_FINALevent: TranscriptFinal. The turn settled and went to the model. It is in the transcript too
TURN_ENDPOINTevent: TurnEndpoint. How complete the turn looked and the wait it bought. Telemetry
MICROPHONE_STATEstate: MicrophoneState. thinking brackets the reply; bargein is somebody taking the floor back
TOOL_INVOKEDevent: ToolInvocation. A tool running right now, which is the only moment this is useful
PERSONA_ACTIONevent: PersonaAction. The persona did something rather than said something: it ended the call, skipped a turn, paused, or changed language. A pause carries paused, maxSeconds and, when it ends, reason (a PauseEndReason, UNKNOWN for a word this SDK has never heard of, kept in reasonRaw). A language change carries language, applied, and voiceId with voiceLabel only when the voice changed. Each field is absent unless the box sent it. The field table
SESSION_ENDED_BY_PERSONAevent: SessionEndedByPersona. The box ended the session after the persona's goodbye. Final: do not reconnect
REPLY_METRICevent: ReplyMetric. How long the first audio of the reply took
REPLY_FAILEDerror: Error. One turn failed. The socket is still good, so keep listening
VOICE_NOTICEnotice: string. The voice account this session speaks on changed. An empty string clears the warning
MICROPHONE_RECONNECTINGdelayMs: number, attempt: number. The socket dropped and is being redialled
MICROPHONE_CLOSEDevent: MicrophoneClosed. The microphone ended, and whether trying again could work
A spoken reply streams, and waiting for the end shows nothing

The box hands over the reply one clause at a time, as each clause is handed to the voice, and the finished reply arrives last. An application that renders only MESSAGE_RECEIVED is blank for the whole time the avatar is speaking, then fills in after it stops. Render MESSAGE_STREAM_EVENT_RECEIVED and let MESSAGE_RECEIVED close the turn.

Two of these carry failures, and they are not interchangeable

REPLY_FAILED is one turn failing on a microphone that is still listening: the next turn can still work. ERROR is the socket, and a fatal one is always followed by MICROPHONE_CLOSED carrying the same sentence and the code to branch on. Handle both.

The credential renews itself on the socket

A browser session token lives for minutes, and a WebSocket freezes the credential it opened with into its URL, so the box asks for a fresh one on the socket rather than answering 401. The SDK handles that for you when the client was built with a tokenProvider: it asks your backend, sends the renewed credential back down the same socket, and the media connection is never touched, so the avatar does not disappear and come back. Without a tokenProvider there is nothing to send, and that is reported as a SERVER_WARNING rather than left silent.

Always register SERVER_WARNING

A box that could not prepare your avatar streams a stand-in face and says so only here. The same channel reports a tone this avatar never advertised. Without a handler, the only symptom is the wrong face on screen.

From your custom tools

These four arrive whichever socket the call came in on, the control socket or the microphone socket, so a page draws one lifecycle. Answer a call with client.registerToolCallHandler(name, handler); the events are for showing what is happening. The full contract, the failure reasons and a worked example are under Tools.

EventHandler arguments
TOOL_CALL_STARTEDevent: ToolCallStarted. The persona decided to call one of your tools: callId, name, mode
TOOL_CALLcall: ToolCall. The call itself, with args, timeoutSeconds and handled, which is false when no handler is registered for the name
TOOL_CALL_COMPLETEDevent: ToolCallCompleted. Finished, with durationMs. For fire_and_forget that means sent, not carried out
TOOL_CALL_FAILEDevent: ToolCallFailed. reason is a ToolFailureReason; a reason from a newer server is UNKNOWN with the server's word on rawReason

Models

SessionInfo

Carried by SESSION_READY.

FieldTypeDescription
sessionIdstringBox-issued id for this session.
avatarstring | nullAvatar the gateway bound this session to, or null if it named none.
tonesstring[]Tone variants available for that avatar. The only place a caller can discover what talk({ tone }) accepts on this box, since it is per avatar.

Message

A complete message in the transcript.

FieldTypeDescription
idstringMessage id.
roleMessageRoleuser, assistant or system.
contentstringThe text.
tonestring | nullWhich face delivered this, when the box said. null on the user's own turns and on a reply the box sent no tone for. Worth reading rather than assuming: tone coverage is per avatar, so a reply can be neutral simply because that avatar has no clip for what the words are doing.

MessageStreamEvent

An incremental chunk of an assistant reply.

FieldTypeDescription
idstringId of the message being assembled.
roleMessageRoleAuthor of the message.
contentstringThis chunk.
contentIndexnumberPosition of this chunk in the reply.
endOfSpeechbooleanTrue on the final chunk.
correlationIdstring | nullTies the chunk to the utterance that produced it.
tonestring | nullThe face for this chunk. A reply may change tone between sentences.

EmotionEvent

FieldTypeDescription
emotionstringThe emotion the box classified for an utterance.
correlationIdstring | nullWhich utterance it classified.

SessionExpiredEvent

Why the box retired a session. Two clocks can end one and reason names which.

FieldTypeDescription
reasonstringidle or max_length today. A string rather than an enum, so a newer box's reason arrives in the box's own words.
messagestringThe sentence the box gave, written for a person.
secondsnumberThe limit that was crossed, so a caller can raise it.
codenumberThe close code carried on the close frame that follows.

UnknownFrameEvent

A control frame this SDK has no branch for. It is still ignored, so a newer box cannot break an older client; what this adds is that ignoring it is visible.

FieldTypeDescription
typestringThe frame's type, or "" when it carried none.
frameRecord<string, unknown>The frame exactly as it arrived.

MicrophoneReady

Carried by MICROPHONE_READY, and the only proof the microphone session works.

FieldTypeDescription
sampleRatenumberSamples per second the box reads. The SDK converts capture to it for you.
voiceOriginstringWhose provider account this session speaks on: box or user.

TranscriptPartial and TranscriptFinal

FieldTypeDescription
textstringWhat was heard. On a final where continues is true, this is only the new words, because the earlier half is already on screen as its own turn.
continuesbooleanFinal only. True when this turn resumes one that was interrupted.

TurnEndpoint

FieldTypeDescription
eotnumberHow complete the turn looked, from 0 to 1.
waitMsnumberHow long the box will wait before answering.
reasonstringThe box's own word for why it chose that wait.

MicrophoneState

FieldTypeDescription
thinkingboolean | nullTrue when a reply started, false when it ended either way, null when the frame was not about thinking.
speakingboolean | nullFalse when the avatar stopped speaking, null when not stated.
bargeinbooleanTrue when somebody took the floor back mid reply. TALK_STREAM_INTERRUPTED is emitted beside it.

ToolInvocation and ReplyMetric

FieldTypeDescription
namestringThe tool's identifier.
labelstringA sentence written for a person, ready to put in a caption.
firstAudioMsnumberMilliseconds from the end of speech to the first audio of the reply.
speculativebooleanTrue when the reply was started early, during the end of turn wait.

MicrophoneClosed

Carried by MICROPHONE_CLOSED, and by MicrophoneRefusedError.closed when the box refuses at the start.

FieldTypeDescription
codenumberThe close code. A number rather than an enum, because a newer box may close with one this SDK has never heard of.
reasonstringThe box's own sentence, written for a person.
retryablebooleanWhether dialling again, by itself, could succeed. The SDK's automatic redial follows this.
retiredbooleanTrue when the session was released rather than the microphone refused. Connect a new session.
attemptsnumberHow many automatic redials were spent before giving up.

MicrophoneCloseCode

CodeMeaningRedial
4001The box has no speech recognition configuredNo
4002There is no live session. Connect one firstNot automatically. A person acts, then start the microphone again
4003Another socket already holds the single microphone slotNot automatically. Once it closes, start again
4004This credential was revoked. Sign in againNo
4005Speech recognition would not startYes, automatically
4006The box could not set the session upYes, automatically
4007The session sat idle, so the box released itNo. Retired: connect a new session
4008The session reached the length it was allowedNo. Retired: connect a new session
A retirement is not a refusal

The last two released the session, so the avatar has left the stage and a new microphone socket has nothing to speak into. That is why retired is its own field rather than a shade of retryable: the fix is a new session, not a new microphone.

ConnectionCloseCode

The first argument of CONNECTION_CLOSED. These are four situations a caller reacts to differently, which is why they are not lumped together.

CodeMeaning
normalYou ended it, through stopStreaming().
server_closedThe far end hung up cleanly. Nothing is broken.
webrtc_failureThe peer connection reported failed. A broken transport, worth retrying or a network to check.
signalling_failureThe control gateway went while media was still up. Talk and interrupt are gone, frames may still arrive.

MessageRole

user, assistant, system.

Other exported enums

These describe the wire and are exported so a caller can name what it sees in a log, rather than because a typical app reaches for them.

EnumMembers
GatewayCommandmessage, talk, talkstream, interrupt, auth, heartbeat
GatewayEventsession_ready, user_message, message_stream, message, avatar_speech_started, avatar_speech_ended, emotion (reserved), interrupted, warning, error, auth_required, auth_state, session_expired
MicrophoneCommandconfig, mute, auth, bye
MicrophoneEventready, partial, endpoint, final, state, tool, action, session_end, reply_chunk, metric, reply, reply_error, voice_notice, auth_required, auth_state, error. Sixteen, and the SDK handles every one
PeerConnectionStateconnected, failed, closed, disconnected
IceGatheringStatecomplete
MediaKindvideo, audio
TransceiverDirectionrecvonly

Example

import { ZeliClient, ZeliEvent, MessageRole } from "@zeligate/zeli-avatar";
 
const client = new ZeliClient({ serverUrl, sessionToken });
 
client.addListener(ZeliEvent.SESSION_READY, (info) => {
  console.log("tones on this avatar:", info.tones.join(", "));
});
 
client.addListener(ZeliEvent.MESSAGE_STREAM_EVENT_RECEIVED, (chunk) => {
  process.stdout.write(chunk.content);
  if (chunk.endOfSpeech) process.stdout.write("\n");
});
 
client.addListener(ZeliEvent.MESSAGE_HISTORY_UPDATED, (history) => {
  const spoken = history.filter((m) => m.role === MessageRole.ASSISTANT);
  render(spoken);
});
 
client.addListener(ZeliEvent.CONNECTION_CLOSED, (code, reason) => {
  console.log("closed:", code, reason ?? "");
});
Zeli Avatar · real-time avatars over WebRTC · self-hostable · AU data residency · source