JavaScript SDK / Events & models
Events and models
Register handlers on the client, not the
session, so they are in place before connect() opens anything.
Handler arguments are typed per event through ZeliEventMap.
import { ZeliEvent, type ZeliEventMap } from "@zeligate/zeli-avatar";ZeliEvent
| Event | Handler arguments |
|---|---|
CONNECTION_ESTABLISHED | none |
SESSION_READY | info: SessionInfo |
TRACK | track: MediaStreamTrackLike |
MESSAGE_RECEIVED | message: Message |
MESSAGE_STREAM_EVENT_RECEIVED | event: MessageStreamEvent |
MESSAGE_HISTORY_UPDATED | history: Message[] |
AVATAR_SPEECH_STARTED | correlationId: string | null |
AVATAR_SPEECH_ENDED | correlationId: string | null |
TALK_STREAM_INTERRUPTED | correlationId: string | null |
EMOTION_DETECTED | event: EmotionEvent. Reserved, not currently emitted. No box sends an emotion frame |
CREDENTIAL_STATE | live: boolean. Whether the box accepts the credential now on the control socket |
SESSION_EXPIRED | event: SessionExpiredEvent. The box retired this session; a close follows immediately |
UNKNOWN_FRAME | event: UnknownFrameEvent. A control frame this SDK has no branch for, handed over whole |
SERVER_WARNING | message: string |
ERROR | error: Error |
CONNECTION_CLOSED | code: ConnectionCloseCode, reason: string | null. The reason carries the box's own sentence when it gave one |
From the microphone
These arrive only while
session.startMicrophone()
is running. The reply to something spoken is not among them, deliberately:
it arrives on MESSAGE_STREAM_EVENT_RECEIVED and MESSAGE_RECEIVED above,
exactly as a typed turn does, so one renderer serves both.
| Event | Handler arguments |
|---|---|
MICROPHONE_READY | info: MicrophoneReady. Recognition is attached and the box is listening |
TRANSCRIPT_PARTIAL | event: TranscriptPartial. The running guess, revised as somebody talks |
TRANSCRIPT_FINAL | event: TranscriptFinal. The turn settled and went to the model. It is in the transcript too |
TURN_ENDPOINT | event: TurnEndpoint. How complete the turn looked and the wait it bought. Telemetry |
MICROPHONE_STATE | state: MicrophoneState. thinking brackets the reply; bargein is somebody taking the floor back |
TOOL_INVOKED | event: ToolInvocation. A tool running right now, which is the only moment this is useful |
PERSONA_ACTION | event: PersonaAction. The persona did something rather than said something: it ended the call, skipped a turn, paused, or changed language. A pause carries paused, maxSeconds and, when it ends, reason (a PauseEndReason, UNKNOWN for a word this SDK has never heard of, kept in reasonRaw). A language change carries language, applied, and voiceId with voiceLabel only when the voice changed. Each field is absent unless the box sent it. The field table |
SESSION_ENDED_BY_PERSONA | event: SessionEndedByPersona. The box ended the session after the persona's goodbye. Final: do not reconnect |
REPLY_METRIC | event: ReplyMetric. How long the first audio of the reply took |
REPLY_FAILED | error: Error. One turn failed. The socket is still good, so keep listening |
VOICE_NOTICE | notice: string. The voice account this session speaks on changed. An empty string clears the warning |
MICROPHONE_RECONNECTING | delayMs: number, attempt: number. The socket dropped and is being redialled |
MICROPHONE_CLOSED | event: MicrophoneClosed. The microphone ended, and whether trying again could work |
The box hands over the reply one clause at a time, as each clause is handed to
the voice, and the finished reply arrives last. An application that renders
only MESSAGE_RECEIVED is blank for the whole time the avatar is
speaking, then fills in after it stops. Render
MESSAGE_STREAM_EVENT_RECEIVED and let
MESSAGE_RECEIVED close the turn.
REPLY_FAILED is one turn failing on a microphone
that is still listening: the next turn can still work.
ERROR is the socket, and a fatal one is always followed by
MICROPHONE_CLOSED carrying the same sentence and the code to
branch on. Handle both.
A browser session token lives for minutes, and a WebSocket freezes the
credential it opened with into its URL, so the box asks for a fresh one on the
socket rather than answering 401. The SDK handles that for you when the client
was built with a tokenProvider: it asks your backend, sends the
renewed credential back down the same socket, and the media connection is never
touched, so the avatar does not disappear and come back. Without a
tokenProvider there is nothing to send, and that is reported as a
SERVER_WARNING rather than left silent.
A box that could not prepare your avatar streams a stand-in face and says so only here. The same channel reports a tone this avatar never advertised. Without a handler, the only symptom is the wrong face on screen.
From your custom tools
These four arrive whichever socket the call came in on, the control socket or the
microphone socket, so a page draws one lifecycle. Answer a call with
client.registerToolCallHandler(name, handler); the events are for showing what
is happening. The full contract, the failure reasons and a worked example are
under Tools.
| Event | Handler arguments |
|---|---|
TOOL_CALL_STARTED | event: ToolCallStarted. The persona decided to call one of your tools: callId, name, mode |
TOOL_CALL | call: ToolCall. The call itself, with args, timeoutSeconds and handled, which is false when no handler is registered for the name |
TOOL_CALL_COMPLETED | event: ToolCallCompleted. Finished, with durationMs. For fire_and_forget that means sent, not carried out |
TOOL_CALL_FAILED | event: ToolCallFailed. reason is a ToolFailureReason; a reason from a newer server is UNKNOWN with the server's word on rawReason |
Models
SessionInfo
Carried by SESSION_READY.
| Field | Type | Description |
|---|---|---|
sessionId | string | Box-issued id for this session. |
avatar | string | null | Avatar the gateway bound this session to, or null if it named none. |
tones | string[] | Tone variants available for that avatar. The only place a caller can discover what talk({ tone }) accepts on this box, since it is per avatar. |
Message
A complete message in the transcript.
| Field | Type | Description |
|---|---|---|
id | string | Message id. |
role | MessageRole | user, assistant or system. |
content | string | The text. |
tone | string | null | Which face delivered this, when the box said. null on the user's own turns and on a reply the box sent no tone for. Worth reading rather than assuming: tone coverage is per avatar, so a reply can be neutral simply because that avatar has no clip for what the words are doing. |
MessageStreamEvent
An incremental chunk of an assistant reply.
| Field | Type | Description |
|---|---|---|
id | string | Id of the message being assembled. |
role | MessageRole | Author of the message. |
content | string | This chunk. |
contentIndex | number | Position of this chunk in the reply. |
endOfSpeech | boolean | True on the final chunk. |
correlationId | string | null | Ties the chunk to the utterance that produced it. |
tone | string | null | The face for this chunk. A reply may change tone between sentences. |
EmotionEvent
| Field | Type | Description |
|---|---|---|
emotion | string | The emotion the box classified for an utterance. |
correlationId | string | null | Which utterance it classified. |
SessionExpiredEvent
Why the box retired a session. Two clocks can end one and reason names which.
| Field | Type | Description |
|---|---|---|
reason | string | idle or max_length today. A string rather than an enum, so a newer box's reason arrives in the box's own words. |
message | string | The sentence the box gave, written for a person. |
seconds | number | The limit that was crossed, so a caller can raise it. |
code | number | The close code carried on the close frame that follows. |
UnknownFrameEvent
A control frame this SDK has no branch for. It is still ignored, so a newer box cannot break an older client; what this adds is that ignoring it is visible.
| Field | Type | Description |
|---|---|---|
type | string | The frame's type, or "" when it carried none. |
frame | Record<string, unknown> | The frame exactly as it arrived. |
MicrophoneReady
Carried by MICROPHONE_READY, and the only proof the microphone session works.
| Field | Type | Description |
|---|---|---|
sampleRate | number | Samples per second the box reads. The SDK converts capture to it for you. |
voiceOrigin | string | Whose provider account this session speaks on: box or user. |
TranscriptPartial and TranscriptFinal
| Field | Type | Description |
|---|---|---|
text | string | What was heard. On a final where continues is true, this is only the new words, because the earlier half is already on screen as its own turn. |
continues | boolean | Final only. True when this turn resumes one that was interrupted. |
TurnEndpoint
| Field | Type | Description |
|---|---|---|
eot | number | How complete the turn looked, from 0 to 1. |
waitMs | number | How long the box will wait before answering. |
reason | string | The box's own word for why it chose that wait. |
MicrophoneState
| Field | Type | Description |
|---|---|---|
thinking | boolean | null | True when a reply started, false when it ended either way, null when the frame was not about thinking. |
speaking | boolean | null | False when the avatar stopped speaking, null when not stated. |
bargein | boolean | True when somebody took the floor back mid reply. TALK_STREAM_INTERRUPTED is emitted beside it. |
ToolInvocation and ReplyMetric
| Field | Type | Description |
|---|---|---|
name | string | The tool's identifier. |
label | string | A sentence written for a person, ready to put in a caption. |
firstAudioMs | number | Milliseconds from the end of speech to the first audio of the reply. |
speculative | boolean | True when the reply was started early, during the end of turn wait. |
MicrophoneClosed
Carried by MICROPHONE_CLOSED, and by MicrophoneRefusedError.closed when the
box refuses at the start.
| Field | Type | Description |
|---|---|---|
code | number | The close code. A number rather than an enum, because a newer box may close with one this SDK has never heard of. |
reason | string | The box's own sentence, written for a person. |
retryable | boolean | Whether dialling again, by itself, could succeed. The SDK's automatic redial follows this. |
retired | boolean | True when the session was released rather than the microphone refused. Connect a new session. |
attempts | number | How many automatic redials were spent before giving up. |
MicrophoneCloseCode
| Code | Meaning | Redial |
|---|---|---|
4001 | The box has no speech recognition configured | No |
4002 | There is no live session. Connect one first | Not automatically. A person acts, then start the microphone again |
4003 | Another socket already holds the single microphone slot | Not automatically. Once it closes, start again |
4004 | This credential was revoked. Sign in again | No |
4005 | Speech recognition would not start | Yes, automatically |
4006 | The box could not set the session up | Yes, automatically |
4007 | The session sat idle, so the box released it | No. Retired: connect a new session |
4008 | The session reached the length it was allowed | No. Retired: connect a new session |
The last two released the session, so the avatar has left the
stage and a new microphone socket has nothing to speak into. That is why
retired is its own field rather than a shade of
retryable: the fix is a new session, not a new microphone.
ConnectionCloseCode
The first argument of CONNECTION_CLOSED. These are four situations a caller
reacts to differently, which is why they are not lumped together.
| Code | Meaning |
|---|---|
normal | You ended it, through stopStreaming(). |
server_closed | The far end hung up cleanly. Nothing is broken. |
webrtc_failure | The peer connection reported failed. A broken transport, worth retrying or a network to check. |
signalling_failure | The control gateway went while media was still up. Talk and interrupt are gone, frames may still arrive. |
MessageRole
user, assistant, system.
Other exported enums
These describe the wire and are exported so a caller can name what it sees in a log, rather than because a typical app reaches for them.
| Enum | Members |
|---|---|
GatewayCommand | message, talk, talkstream, interrupt, auth, heartbeat |
GatewayEvent | session_ready, user_message, message_stream, message, avatar_speech_started, avatar_speech_ended, emotion (reserved), interrupted, warning, error, auth_required, auth_state, session_expired |
MicrophoneCommand | config, mute, auth, bye |
MicrophoneEvent | ready, partial, endpoint, final, state, tool, action, session_end, reply_chunk, metric, reply, reply_error, voice_notice, auth_required, auth_state, error. Sixteen, and the SDK handles every one |
PeerConnectionState | connected, failed, closed, disconnected |
IceGatheringState | complete |
MediaKind | video, audio |
TransceiverDirection | recvonly |
Example
import { ZeliClient, ZeliEvent, MessageRole } from "@zeligate/zeli-avatar";
const client = new ZeliClient({ serverUrl, sessionToken });
client.addListener(ZeliEvent.SESSION_READY, (info) => {
console.log("tones on this avatar:", info.tones.join(", "));
});
client.addListener(ZeliEvent.MESSAGE_STREAM_EVENT_RECEIVED, (chunk) => {
process.stdout.write(chunk.content);
if (chunk.endOfSpeech) process.stdout.write("\n");
});
client.addListener(ZeliEvent.MESSAGE_HISTORY_UPDATED, (history) => {
const spoken = history.filter((m) => m.role === MessageRole.ASSISTANT);
render(spoken);
});
client.addListener(ZeliEvent.CONNECTION_CLOSED, (code, reason) => {
console.log("closed:", code, reason ?? "");
});