Python SDK / Voice input
Voice input
A VoiceSession is the microphone socket:
you push PCM up, the box pushes sixteen kinds of event down,
and the recognised speech is answered on the same conversation your typed turns
use. It is returned by
session.start_voice().
from zeli import VoiceSessionThe SDK owns the socket, you own the audio
This package does not capture a microphone, and that is a decision rather than an omission.
In a browser there is exactly one way to get a microphone, it is
getUserMedia, every page already has it, and it costs nothing to
depend on. So the
JavaScript SDK owns capture.
In Python there is no microphone. Reading one means PortAudio, through
sounddevice or pyaudio: a C library that has to build
or ship a wheel per platform, a device permission problem on macOS, and a class
of failure that has nothing to do with avatars, added to a package whose entire
runtime dependency list is four entries. And the microphone is not the common
case here. A Python caller is usually a server, where the audio arrives from a
telephony leg, a file, an upload or a queue.
So this package owns the socket and the protocol, and takes audio from you. The two SDKs are asymmetric deliberately, and this callout is the record of it.
Starting a session
async with client.connect() as session:
voice = await session.start_voice()The rate of the audio you are going to send. Declared, not detected: any other value is refused here.
Face to speak with. Defaults to the session's own avatar.
Voice to speak with. Defaults to the session's own voice.
Start with the microphone muted. The mute lives on the socket, so a fresh socket is unmuted unless you say otherwise.
How long to wait for the box to say it is listening.
start_voice returns once the box has attached speech recognition, not merely
once the socket is open. The box accepts the upgrade and then refuses, for
eight different reasons, so an open socket proves nothing. A refusal is raised as
a VoiceSocketError carrying the box's own close code.
Feeding audio
16 kHz, mono, signed 16 bit little endian. One frame at a time:
await voice.send_audio(pcm_bytes)Or as anything that is an async iterable of bytes, which is what a microphone, a telephony leg, a file and a test fixture all are:
await voice.stream_audio(microphone_frames())
# A source with no clock of its own, such as a file, needs pacing: recognition
# is a streaming service, and audio arriving at disk speed is transcribed badly
# rather than refused.
await voice.stream_audio(wav_frames("question.wav"), pace=True)You do not have to frame the audio yourself. A buffer that ends halfway through a sample has its dangling byte held back to lead the next one, and a buffer larger than a tenth of a second is split, so one careless call with a whole file in it arrives as a stream.
The socket carries raw PCM: no header, no rate, nothing that says what the bytes
are. Audio at 48 kHz is not rejected by the box. It is handed to speech
recognition as if it were 16 kHz and comes back as a transcript of confident
nonsense, with nothing anywhere logging a problem, and the first symptom is the
avatar answering a question nobody asked. Nothing on either side of the wire can
detect that, so the rate is declared when you start the session and any other
value raises a ConfigurationError before a socket is dialled.
Resample before you feed it in.
What you get back
Everything arrives on the events you already have, wherever one existed, so a transcript renderer written for typed turns works for spoken ones without changes.
| Frame | Reaches you as |
|---|---|
ready | VOICE_READY with a VoiceReady |
partial, final | MESSAGE_STREAM_EVENT_RECEIVED with the USER role. final also finalises a Message |
endpoint | VOICE_ENDPOINT with a VoiceEndpoint |
state | VOICE_STATE with a VoiceState |
tool | VOICE_TOOL with a VoiceTool |
reply_chunk | MESSAGE_STREAM_EVENT_RECEIVED with the ASSISTANT role, one clause at a time |
metric | VOICE_METRIC with a VoiceMetric |
reply | MESSAGE_RECEIVED, and the transcript grows |
reply_error | ERROR carrying a VoiceTurnError |
voice_notice | VOICE_NOTICE with the server's sentence, which is empty when the warning should be cleared |
auth_required, auth_state | Handled for you; auth_state also reaches you as CREDENTIAL_STATE |
error | ERROR carrying a VoiceSocketError |
| the close itself | VOICE_CLOSED with a VoiceClosed |
Both arrive on the same event and they revise in opposite directions.
Recognition keeps changing its mind about the whole utterance while somebody is
still talking, so each USER chunk is the turn so far and
content_index counts revisions: draw it over the last one. Each
ASSISTANT chunk is one new clause, appended, because the avatar
cannot unsay what it has already said.
The avatar speaks one clause at a time and each clause is sent as it is handed
to the voice. reply closes the turn rather than opening it, so a
client that waits for it shows nothing at all until the avatar has stopped
talking. Render the ASSISTANT role of
MESSAGE_STREAM_EVENT_RECEIVED and use
MESSAGE_RECEIVED to settle the line.
A VoiceTurnError is one turn failing on a socket
that is still good: keep listening, because the next turn can still work. A
VoiceSocketError is the socket. Both arrive as
ZeliEvent.ERROR, so handle both and tell them apart by type.
Driving the socket
await voice.mute() # and voice.mute(False)
await voice.configure(avatar="...", voice="...")
await voice.resume() # ends a pause the persona began
voice.paused # whether the conversation is on hold
await voice.close() # says goodbye, then hangs upresume is safe at any time and never raises. It returns True when the request
was sent and False when there is no live microphone to send it on. The box ends
the pause, so paused stays true until its answer arrives; it also clears on a
reply and when the microphone or the session ends. session.paused and
session.resume() are the same two on the session.
mute states the switch as a fact rather than leaving it to be inferred from
silence, so audio already in flight does not still become a turn somebody has
just said they did not want sent. Closing the session closes the microphone with
it, so close() is only needed to stop talking while the avatar stays.
When the socket ends
VOICE_CLOSED carries a VoiceClosed, and the two fields that matter are the
ones you cannot work out from the number:
| Field | Meaning |
|---|---|
code | The raw close code, or 0 when the transport gave none |
close_code | A VoiceCloseCode when the box named one of its own, None for an ordinary disconnect |
reason | The sentence the box wrote, for a person to read |
retirement | The session was released, not the microphone refused |
retryable | Dialling again, by itself, could succeed |
Close codes 4007 and 4008 mean the session sat idle,
or reached the length it was allowed. The box released the session and the
avatar left the stage, so dialling the microphone again reaches a box with no
session to speak into. The fix is a new session, which is why
retirement is a field rather than something to infer.
A redial is always a fresh start_voice, and that is deliberate: an upgrade
freezes the credential it presents for the life of the socket, and start_voice
renews before it dials. Reusing a frozen credential is how a box that had never
opened a single microphone socket still showed "reconnecting" all session.
Renewing a credential mid session
Nothing to do. A session token lives for minutes and a WebSocket cannot answer
401, so the box asks on the socket and the SDK answers on the same socket,
using the same renewal path the control channel uses. The media connection is
never touched, so the avatar does not disappear and come back. Without a
token_provider there is nothing to send, and that is reported as a
SERVER_WARNING rather than left silent. voice.credential_is_live is the box's
last word on this socket specifically.