API reference / Voices
Voices
A voice on this service is a voice at your provider, not a record in our database. We hold the key that reaches your account, resolve the right engine per request, and pass the provider's catalogue through. That is why the read routes are shaped like a catalogue and there is no voice CRUD: there is nothing local to create, rename or delete.
Save a provider key and every reply on that credential speaks through
your account, with your catalogue and your billing. Without a
key, the box's own engine stands in. When a saved key cannot be used, the
fallback is reported rather than silently applied: read
voice_notice on
GET /api/chat/status.
| Method | Path | Purpose | Plane |
|---|---|---|---|
| GET | /api/voices | Every voice this caller can use, grouped by provider | Box, any |
| GET | /api/voice/preview | A short WAV sample of one voice | Box, any |
| POST | /api/voice/upload | Clone a voice from a reference clip | Box, full |
| POST | /api/voice/prepare | Warm a clone so the first reply is instant | Box, full |
| WSS | /api/voice/ws | Microphone audio in, driving the live reply path | Box, any |
| GET | /auth/voice/settings | Saved provider keys and engine selection | Portal |
| PUT | /auth/voice/key | Save a provider key | Portal |
| DELETE | /auth/voice/key/{provider} | Remove one | Portal |
| PUT | /auth/voice/engine | Choose the engine | Portal |
The catalogue
One list from every provider this caller can use: Zeli Voices always, plus
each provider they hold a saved key for (on their account, with their
catalogue), plus the on box clone engine where the box runs one. Providers are
fetched in parallel, each with its own timeout, and cached per provider and
caller for five minutes (a failed fetch for thirty seconds), so a second call
costs no provider round trip and one slow vendor never delays the rest. A
provider whose key is refused, missing or unreachable is still a group, with a
one line reason, rather than a silent omission.
Every row carries its provider. Send it back beside the id (voice_provider
next to voice on POST /api/chat, and on the mic socket's config message):
the box refuses, with a 400, to speak one provider's id through another's
engine.
Refetch from the provider instead of serving the cache. Use it after a provider switch or after cloning a voice.
curl "https://avatar.zelibot.xyz/api/voices" -H "X-Api-Key: zsk_live_..."const catalogue = await admin.management.listVoices();A bare object, with no data wrapper and no pagination:
{
"groups": [
{
"provider": "zelispeech",
"label": "Zeli Voices",
"state": "ready",
"reason": "",
"local": true,
"voices": [ { "provider": "zelispeech", "id": "default", "name": "Zeli", "...": "..." } ]
},
{
"provider": "elevenlabs",
"label": "ElevenLabs",
"state": "ready",
"reason": "",
"local": false,
"voices": [
{
"provider": "elevenlabs",
"id": "a0e99841-...",
"name": "Ariana",
"language": "en",
"gender": "female",
"description": "Warm, mid range",
"preview": "/api/voice/preview?provider=elevenlabs&voice=a0e99841-...",
"recommended": true
}
]
},
{
"provider": "cartesia",
"label": "Cartesia",
"state": "refused",
"reason": "Cartesia key was refused, replace it in the portal",
"local": false,
"voices": []
}
],
"voices": [ "...every row above, flattened, Zeli first then by name..." ],
"current": "a0e99841-...",
"current_provider": "elevenlabs",
"engine_ready": true,
"provider": "elevenlabs",
"provider_label": "ElevenLabs (hosted)",
"requires_env": "ELEVENLABS_API_KEY",
"voices_from": "hosted",
"local": false,
"byo": true,
"voice_notice": ""
}| Field | Meaning |
|---|---|
groups | One entry per provider, in a fixed order (Zeli Voices first). state is one of ready, empty, starting, refused, unreachable, timed_out, key_problem; reason is the one line to show when it is not ready |
voices | Every listed row, flattened. Each row carries provider and a preview path; the rest of the item shape is the provider's, not ours |
current, current_provider | The voice, and the engine, this session would speak with right now. Both come from the same resolution the reply uses, so they cannot disagree with the list |
engine_ready | false means the engine in use is still starting. Show a spinner and poll, rather than "no voices" |
provider, provider_label | The engine in use (the same as current_provider), so an empty state names the right one |
requires_env | The variable the box needs for this engine, or null when there is nothing to configure |
local | Whether the engine in use runs on the box itself |
byo | Whether this box resolves each caller's own saved provider keys |
voice_notice | Non empty only when the caller's own account could not be used for this session |
A provider that fails does not fail the call: its group carries the reason,
the others list as normal, and the status is still 200. Branch on each
group's state, not on the status code.
Which engine speaks is decided per session, in this order: a voice picked in the Live Studio (it names both the voice and its provider), else the engine chosen on the portal's Voices page (the default for new sessions), else the box's own engine.
name is a name and not a display name field, gender
is whatever lowercase string the provider uses, and language is
not a country code. There is no sample URL, no created or updated timestamp
and no provider model id on the item, because we are not storing a voice
record. Do not write a client that assumes a normalised schema here.
Preview a voice
The voice id to audition. Defaults to the session's current voice.
The provider the voice belongs to, as its row in GET /api/voices names it.
The sample is synthesised through that provider, so a voice from another of
the caller's providers previews correctly. Omit it to use the session's
engine. A provider this caller cannot use is a 404. Every row's preview
field is this URL, already filled in.
Returns WAV bytes of a short fixed sentence, cached per provider and voice.
Clone a voice
Multipart. Requires a full key.
The reference clip. Transcoded to clean mono and length checked before it reaches the engine, so a clip that is too short is refused with a readable message rather than producing a bad clone.
Display name for the cloned voice.
{ "ok": true, "voice_id": "cb_7f21...", "voices": [], "current": "a0e99841-..." }voices is the refreshed catalogue, so the picker needs no second call.
Cloning works only when the box is running the self hosted engine. On any other
provider the call is a 400 saying so. The check runs against the
engine the box booted with rather than the one this caller
selected, so a caller on their own self hosted engine can still be refused on a
box that booted on a hosted provider.
Warm a clone
Eagerly clones a voice on the engine so the first reply is instant instead of appearing to hang. Call it after an upload and on voice select. Requires a full key.
The voice to warm. Blank is a 400.
The engine's own response is passed straight through, so treat the body as opaque
and branch on the status code. A 502 means the engine failed. Same on instance
engine restriction as cloning, and the same 400 when it does not apply.
Microphone input
The browser streams 16 kHz mono PCM over this socket, speech recognition turns it into text, and that text is fed into the same live reply path the text box uses, so voice and typing share one conversation.
This route is why the query parameter form of the credential exists at all: a browser cannot set headers on a WebSocket upgrade.
Binary frames carry that audio, signed 16 bit little endian. Text frames are small JSON control messages:
type | Fields | What it does |
|---|---|---|
config | avatar, voice | Picks the face and the voice this socket speaks with. Send it on open, and again whenever the picker changes |
mute | muted | States the mic switch as a fact, so audio already in flight when somebody mutes does not still become a turn they did not want sent. Optional, and a socket that never sends it is unmuted |
auth | token | Hands the socket a fresh credential. The one in the URL is frozen for the life of the socket and a socket cannot answer with a 401, so this is the only way to renew it |
bye | none | Ends the session from the client side |
resume | none | Ends a pause the avatar began with pause_conversation. Safe at any time: with nothing paused it ends nothing and starts no turn, and is still answered with an action frame that says paused: false |
What comes back
Seventeen event types, not the handful a live caption needs, and the count is
the point: the reply itself arrives on reply_chunk, one clause at a time as
each clause is handed to the voice. reply closes the turn rather than being the
first place the words appear, so a client that switches on reply alone shows
nothing at all until the avatar has stopped speaking.
type | When it arrives | Fields |
|---|---|---|
ready | Once, when speech recognition is attached and the box is listening. This is the only proof the session works, so it is the right place to reset a reconnect budget | sample_rate, voice_origin (box or user) |
partial | Continuously while somebody is talking, as recognition revises the words | text |
endpoint | Beside each revision, carrying how complete the turn looked and how long the box will wait before answering. Telemetry for a debug overlay | eot, wait_ms, reason |
final | Once the turn has settled and is on its way to the model. continues is true when this turn resumes one that was interrupted, and text then holds only the new words, because the earlier half is already on screen | text, continues |
caption_clear | A final resolved to nothing worth a turn (a lone listening noise, held or dropped) after partial had already drawn it live, or a held word was still showing when a mute, a mic off or End session ended the turn with nothing to send. Settle the caption to its resting state | none |
state | thinking brackets the reply, true when it starts and false when it ends either way. speaking false with bargein true is somebody taking the floor back mid reply | thinking, or speaking and bargein |
tool | While a tool runs, not once it is over | name, label |
action | When the persona does something rather than says something: it ended the call, or stayed quiet for a turn because the person asked for time to think. end_call arrives while the goodbye is still playing, so end nothing on it; wait for session_end. See Tools | name (end_call or skip_turn), label |
reply_chunk | Once per clause, as that clause is handed to the voice | text, tone |
metric | When the first audio of the reply is ready. spec is true when the reply was started early, during the end of turn wait | first_audio_ms, spec |
reply | Last on a turn that produced words, carrying the whole reply as the moderation screen left it | reply, tone, avatar |
reply_error | In place of reply, on a turn that failed or that found chat unconfigured | error |
session_end | Once, when the box ends the session itself, after the persona's goodbye has played and just before the teardown. Treat it as final: end the session on your side and do not reconnect | reason (ended_by_persona_request) |
voice_notice | Whenever the account this session speaks on changes, including back to normal, where notice is empty and the warning should be cleared | notice |
auth_required | Once per lapse, when the socket's credential ages out. Answer it with an auth message | message |
auth_state | After an auth message, saying whether the credential now on the socket is live | live |
error | The socket is in trouble rather than the turn. Either a refusal, or the transcription stream ending under a socket that then closes normally | error, plus fatal and a 4000 range code on a refusal |
reply_chunk and tool are the same objects, field for field, that the NDJSON
form of POST /api/chat writes, so one renderer
serves both doors.
reply_error is one turn failing on a socket that
is still good: keep listening, because the next turn can still work.
error is the socket, and a fatal one
is always followed by a close carrying the same sentence and the same code.
Handle both. A client that handles only error silently drops every
failed turn, and one that handles only reply_error never learns
why the socket went away.
Treat an unrecognised type as ignorable rather than as an error. The list above
is what the box sends today, and a client that throws on anything else is a
client that breaks the next time one is added.
When the box says no
There are two different refusals, and they arrive in different places.
A credential the box will not accept is refused at the upgrade, as an HTTP
401 with the usual code and message body. No socket is opened, so a browser
sees only a failed connection: onopen never fires and there is nothing to read.
Renew the credential before dialling again, because the one in the URL is frozen
for the life of the socket.
A socket the box accepts and then closes always says why, on the close frame
itself, using the private 4000 range. The JSON error event is sent first and
carries the same sentence plus a matching code, so either signal is enough.
| Close code | Meaning | Worth retrying |
|---|---|---|
4001 | The box has no speech recognition configured | No |
4002 | There is no live session yet. Go Live first | Yes, once live |
4003 | Another socket already holds the single mic slot | Yes, once it closes |
4004 | This credential was revoked. Sign in again | No |
4005 | Speech recognition would not start | No |
4006 | The box could not set the session up | Yes, after a pause |
4007 | The session sat idle, so the box released it | No, go live again first |
4008 | The session reached the length it was allowed | No, go live again first |
The last two are retirements rather than refusals. The session itself was released and the avatar left the stage, so redialling the microphone reaches a box with no session to speak into: going live again is the whole of the fix.
Anything else, including the ordinary 1000, is a real disconnect rather than a
refusal, and reconnecting is the right response to it.
The provider key vault
Four routes on the portal plane. They take a signed in person rather than an API key, and they are callable from a browser only same origin.
Every one of them answers the same refreshed settings object, so a client needs one round trip rather than two:
{
"voice": {
"engine": "cartesia",
"engines": [
{ "id": "zeli", "label": "Zeli Voices", "byo": false, "provider": null, "note": "..." },
{ "id": "cartesia", "label": "Cartesia (your account)", "byo": true, "provider": "cartesia", "note": "..." }
],
"keys": [
{
"provider": "cartesia",
"last4": "9f2a",
"masked": "sk_...9f2a",
"created_at": 1769900000,
"updated_at": 1769900000,
"validated": true,
"validated_at": 1769900000
}
]
}
}The saved key entry structurally cannot hold the secret: there is no plaintext
and no ciphertext field on it, so no serialiser can leak one. You get
last4 and a mask, and that is all there will ever be. Store your
own copy if you need it.
validated is true when the key was verified with the provider, false when
the provider explicitly rejected it, and null when it was not checked. A
rejected key is never persisted.
Read the settings
Takes no parameters. The acting identity is the partition.
Save a key
elevenlabs or cartesia.
The provider key. Sent in the body of a PUT, never in a path or
a query string.
Remove a key
The provider is the last path segment.
Choose the engine
zeli, elevenlabs or cartesia. The two
bring your own engines need a saved key first.
Not served in this group
- A presigned upload. The reference clip must fit in the request body. There is no storage key handoff for a large file.
- Voice import from a provider. We store a provider key, never a provider voice as a local record, so there is nothing to import into.
GET,PUTandDELETEa voice by id. There is no local voice record. Display name, model, gender, country and description live at the provider and are not ours to change.- Search and pagination on the catalogue. It returns the complete list.