Concepts / Streaming media
Streaming media
The avatar's audio and video arrive as real-time streams of PyAV frames over WebRTC. Consuming them is a matter of iterating two async iterators.
Frame iterators
The session exposes inbound media as async iterators:
Consume them concurrently. Each iterator runs until the session ends, so writing one loop after the other means the second never starts:
import asyncio
async def show_video(session):
async for frame in session.video_frames(): # PyAV VideoFrame
img = frame.to_ndarray(format="rgb24") # (H, W, 3) uint8
async def play_audio(session):
async for frame in session.audio_frames(): # PyAV AudioFrame
samples = frame.to_ndarray() # 48 kHz stereo int16
async with client.connect() as session:
await asyncio.gather(show_video(session), play_audio(session))Two sequential async for loops over these iterators is a trap, and this page
used to show exactly that. The first loop only exits at teardown, so the audio
loop below it never runs a single iteration, with no error to explain the
silence.
video_frames()yields PyAVVideoFrameobjects. Callframe.to_ndarray("rgb24")(or"bgr24"for OpenCV).audio_frames()yields PyAVAudioFrameobjects carrying 48 kHz stereo int16 PCM.
Both iterators end when the session closes, so async for loops exit cleanly on
teardown.
Keeping latency low
Frames arrive in real time. When a consumer falls behind, the SDK drops the oldest frame rather than growing latency, so a slow renderer degrades to skipped frames, not an ever-increasing delay.
Because the two iterators run concurrently, a renderer that blocks starves the other one as well, so keep per-frame work off the event loop. See Receiving media.
Streaming text in
Streaming isn't only inbound. When your text arrives incrementally (tokens from your own model), push it with a talk stream so the avatar starts speaking before the sentence is complete:
async with session.create_talk_stream() as talk:
await talk.send("Streaming ")
await talk.send("this out loud.", end_of_speech=True)See Talk streams and Driving the avatar.