API reference / Tools

Tools#

A turn may pause while the avatar looks something up or asks your application to do something. A page can explain that pause instead of showing an unexplained spinner, and for a custom tool it can take part in it.

What you can observe#

The live path's reply carries tools, empty on a turn that used none:

{
  "reply": "It is 4:20 in the afternoon in Sydney.",
  "tone": "neutral",
  "tools": ["Checking the time"],
  "streamed": true
}

The strings are labels, not identifiers. They are deliberately not the tool names: calculate and fetch_url are names for a model and a log, while somebody watching an avatar pause should read "Working out the answer". Present participles, because the label appears while the tool is running. Do not branch on these strings; they are copy and may be reworded. A custom tool has no written label, so its name is shown with the underscores removed.

On a typed turn the array arrives with the finished reply. On the voice path the same information arrives on the reply event.

The built in tools#

Five, fixed in code and the same for every caller. They run only on a deployment that has switched them on.

current_time answers in the timezone the browser sent on client_context and falls back to the deployment's zone, never silently to UTC. See Knowledge.

Custom tools#

A custom tool is a name, a description and a parameter schema. The persona decides when to call it from the description. Your application carries it out.

Define them#

Custom tools live under the tools key of your settings record, beside your prompt and model. The subject is the credential the request authenticated with. The write needs a full API key. Send the whole list each time; an empty list clears it.

{
  "tools": [
    {
      "name": "open_device_settings",
      "description": "Open the settings panel of the app. Call it when the person asks to change their microphone, speaker or camera.",
      "parameters": {
        "type": "object",
        "properties": {
          "section": { "type": "string", "enum": ["audio", "video", "general"] }
        },
        "required": ["section"]
      },
      "mode": "await_result",
      "timeout_seconds": 5,
      "enabled": true
    }
  ]
}

At most ten tools. These names are reserved and refused: pause_conversation, end_call, skip_turn, change_language, calculate, current_time, weather, lookup_wikipedia, fetch_url and speak. The parameter name response_to_user is added by the platform and cannot be defined. It is the line the avatar says while your tool runs; if the model leaves it empty and says nothing else first, the avatar says a short built in line instead ("One moment, let me check that.", in the call's language), which you can replace per tool with the persona's tool_phrases setting. A refused write answers 400 with error.code setting_refused and a message that says which rule was broken; nothing is saved from a refused request. The request body itself is limited to 256 KB: a larger one answers 413 with error.code body_too_large, and one that is not a JSON object, or is nested too deeply to parse, answers 400 with error.code invalid_body.

Returns your list under settings.tools. This is the enumeration of your custom tools. The persona only ever sees the entries that are enabled and valid.

Answer them#

When the persona calls a tool, the server sends a tool_call frame to the application that owns the session:

{
  "type": "tool_call",
  "call_id": "3f0c6c0f4b7e4d0f9a3e1b2c5d6e7f80",
  "name": "open_device_settings",
  "args": { "section": "audio" },
  "mode": "await_result",
  "timeout_seconds": 5
}

args has already been checked against your schema and carries declared properties only. For await_result, answer with the same call_id:

{ "type": "tool_result", "call_id": "3f0c6c0f4b7e4d0f9a3e1b2c5d6e7f80", "ok": true, "result": { "opened": true } }

{ "type": "tool_result", "call_id": "3f0c6c0f4b7e4d0f9a3e1b2c5d6e7f80", "ok": false, "error": "settings are locked" }

POST /api/chat/tool_result takes the tool_result body and answers {"ok": true, "disposition": "accepted"}, or ok: false with why the result was ignored. A typed turn is only offered custom tools when the caller is reading the event stream, because otherwise nobody could answer. Two streamed turns open at once under one credential (two tabs) are each answerable, up to eight at a time; the ninth stream is refused with a plain error.

Modes. await_result holds the turn for up to timeout_seconds; the persona says a short holding line as the call starts, then uses your result in its reply. No answer in time, or ok: false, and the persona says it could not be done. fire_and_forget returns at once; the persona is told the request was sent and nothing you send later reaches it.

Results are data. A result is capped at 8 KB of JSON (longer is truncated and marked so), an error at 200 characters, and both reach the model as tool output that is marked as application data, never as the person speaking and never as instructions.

Ignored results. An unknown call_id, a second result for the same call, a result after the timeout and a result for a fire_and_forget call are all ignored. A call_id is only looked up among the calls of the session it arrived on, so another session's id is an unknown id.

Answer them from an SDK#

Both SDKs do the frame work for you. Register one handler per tool name, before you connect, because the persona can call a tool on its first turn. What the handler returns is sent as the result. A thrown ToolCallError is sent as ok: false with its message, and the persona says it could not be done. Any other error is sent as the generic sentence "The tool failed." and handed to your application on ZeliEvent.ERROR instead, so a driver's error text never reaches the model or the transcript.

import { ZeliClient, ZeliEvent } from "@zeligate/zeli-avatar";
 
const client = new ZeliClient({ serverUrl, sessionToken });
 
const unregister = client.registerToolCallHandler(
  "open_device_settings",
  async ({ section }, call) => {
    // call.signal aborts when the answer can no longer matter.
    await settingsPanel.open(section, { signal: call.signal });
    return { opened: true };
  },
  { timeoutMs: 4000 }, // optional, MILLISECONDS: give up sooner than the box would
);
 
const session = await client.connect();
// later, for example in a framework cleanup hook
unregister();

from zeli import ZeliClient, ZeliEvent
 
client = ZeliClient(session_token=token, options=options)
 
# timeout is optional and in SECONDS: give up sooner than the box would
@client.tool_call_handler("open_device_settings", timeout=4.0)
async def open_device_settings(args, call):
    # Cancelled when the answer can no longer matter.
    await settings_panel.open(args["section"])
    return {"opened": True}
 
# The same thing without the decorator, which hands back the unregister function.
unregister = client.register_tool_call_handler("log_interest", log_interest)
 
async with client.connect() as session:
    await session.wait_until_closed()

The lifecycle arrives as four typed events in both SDKs: TOOL_CALL_STARTED, TOOL_CALL, TOOL_CALL_COMPLETED and TOOL_CALL_FAILED on ZeliEvent.

import { ToolFailureReason, ZeliEvent } from "@zeligate/zeli-avatar";
 
client.on(ZeliEvent.TOOL_CALL_STARTED, ({ name }) => showBusy(name));
client.on(ZeliEvent.TOOL_CALL_COMPLETED, ({ name, durationMs }) => clearBusy(name));
client.on(ZeliEvent.TOOL_CALL_FAILED, ({ name, reason, rawReason }) => {
  clearBusy(name);
  if (reason === ToolFailureReason.CANCELLED) return; // the person moved on
  report(name, reason === ToolFailureReason.UNKNOWN ? rawReason : reason);
});

from zeli import ToolFailureReason, ZeliEvent
 
@client.on(ZeliEvent.TOOL_CALL_STARTED)
def started(event):
    show_busy(event.name)
 
@client.on(ZeliEvent.TOOL_CALL_FAILED)
def failed(event):
    clear_busy(event.name)
    if event.reason is ToolFailureReason.CANCELLED:
        return  # the person moved on
    unknown = event.reason is ToolFailureReason.UNKNOWN
    report(event.name, event.raw_reason if unknown else event.reason.value)

Frame reference#

Five frames, the same on every door. Both SDKs declare exactly these (ToolFrameType in JavaScript, ToolFrame in Python), and a test in each holds the table below to the SDK and to the server.

Failure reasons#

The list is append only. A reason your SDK version does not know arrives as ToolFailureReason.UNKNOWN with the server's word on rawReason (raw_reason in Python). Treat it as a generic failure.

A worked example#

An order lookup, end to end: define the tool with a full key on your server, answer it where the session runs.

// 1. On your server, once, with a full key.
await admin.management.updateSettings({
  tools: [
    {
      name: "lookup_order",
      description: "Look up the status of an order. Call it when the person gives an order number.",
      parameters: {
        type: "object",
        properties: { order_number: { type: "string" } },
        required: ["order_number"],
      },
      mode: "await_result",
      timeout_seconds: 6,
    },
  ],
});
 
// 2. Where the session runs, before connect.
import { ToolCallError } from "@zeligate/zeli-avatar";
 
client.registerToolCallHandler("lookup_order", async ({ order_number }, call) => {
  const response = await fetch(`/api/orders/${encodeURIComponent(String(order_number))}`, {
    signal: call.signal,
  });
  // ToolCallError is the one error whose message reaches the persona. Anything
  // else that throws here (a network failure, a bug) is sent as "The tool failed."
  if (!response.ok) throw new ToolCallError("that order number was not found");
  const order = await response.json();
  // Return only what the persona should say. Never the whole record.
  return { status: order.status, expected_on: order.expectedOn };
});
 
const { session } = await client.streamToVideoElement("stage");
await session.startMicrophone();

# 1. On your server, once, with a full key.
await admin.management.update_settings({
    "tools": [
        {
            "name": "lookup_order",
            "description": "Look up the status of an order. Call it when the person gives an order number.",
            "parameters": {
                "type": "object",
                "properties": {"order_number": {"type": "string"}},
                "required": ["order_number"],
            },
            "mode": "await_result",
            "timeout_seconds": 6,
        }
    ]
})
 
# 2. Where the session runs, before connect.
from zeli import ToolCallError
 
@client.tool_call_handler("lookup_order")
async def lookup_order(args, call):
    order = await orders.find(args["order_number"])
    if order is None:
        # ToolCallError is the one exception whose message reaches the persona.
        # Anything else raised here is sent as "The tool failed."
        raise ToolCallError("that order number was not found")
    # Return only what the persona should say. Never the whole record.
    return {"status": order.status, "expected_on": order.expected_on.isoformat()}
 
async with client.connect() as session:
    voice = await session.start_voice()
    await session.wait_until_closed()

The person says "where is order 4417", the persona says a short holding line, your handler runs, and the reply uses what it returned: "It shipped yesterday and should arrive on Friday."

Security#

• A tool result is untrusted text to the model. It reaches the model as tool output marked as application data, never as instructions and never as the person speaking. Treat it the same way on your side: if a result can contain text somebody else wrote (a product review, a ticket body), that text can try to steer the persona. Return the smallest structured answer that does the job.
• Never put secrets in tool arguments, descriptions or results. Arguments are written by a language model from the conversation, and results are read by one. An API key, a session cookie or an internal URL in either can be spoken aloud or logged in a transcript. Keep credentials inside the handler.
• The model chooses the arguments, so the handler must check them. The schema guarantees shape, not permission. {"order_number": "4417"} is well formed whoever is asking: authorise against the signed in person in your handler, not against what the persona was told.
• Errors are private by default. What a database driver or an HTTP client throws routinely names a host, a port or a connection string, and a tool error is read by the model and kept in the transcript. So the SDKs send the generic sentence "The tool failed." for anything a handler throws, and hand the whole error to your application on ZeliEvent.ERROR, where it never leaves the process. To say something specific, throw the public error type on purpose: ToolCallError in both SDKs. Its message is sent, capped at 200 characters, so write it for a person and never put a secret in it.
• Results are bounded. The SDKs refuse to send a result over 256 KB of JSON (MAX_TOOL_RESULT_BYTES), because a socket message of a few megabytes ends the session. The server then shows the model only the first 8 KB. Return a summary, not a table.
• Handlers run where the SDK runs. In a browser that is the visitor's page, so a handler there can only do what that visitor is already allowed to do. Anything privileged belongs behind your own authenticated API.
• Provider keys never reach the browser. A page holds a session token, which cannot write settings. Defining tools needs a full key and belongs on a server.

Lifecycle events#

The same door also carries three events, so a page can show what is happening:

{ "type": "tool_call_started",   "call_id": "...", "name": "open_device_settings", "mode": "await_result" }
{ "type": "tool_call_completed", "call_id": "...", "name": "open_device_settings", "duration_ms": 412 }
{ "type": "tool_call_failed",    "call_id": "...", "name": "open_device_settings", "reason": "timeout" }

reason is one of timeout, host_error, invalid_args, not_enabled, no_host, session_closed and cancelled. cancelled means the person interrupted, or the session was ended, while the call was waiting; a result sent afterwards is ignored. For fire_and_forget, completed means the request was sent, not that your application carried it out.

System tools: what the avatar can do by voice#

A system tool changes the conversation itself instead of fetching a fact. The person asks by voice, the avatar acts. There are four:

They are always on for voice sessions, whether or not the five tools above are enabled on the box: a system tool reaches nothing outside the conversation. They are not offered on typed turns (POST /api/chat) or on the SDK control socket yet, because those doors do not act on them.

end_call is deliberately hard to trigger. The avatar is told to call it only once the person has clearly asked to finish, and to confirm first when it is unclear. If the person talks over the goodbye, the call stays open.

Pausing, and the three ways back#

A pause ends when the person says something real, when your page sends { "type": "resume" } on the voice socket, or after 90 seconds. Nothing is said when it lapses, and a lapse is not activity: the ordinary idle timeout counts from when the pause began. If the person talks over the holding line, no pause begins. resume with no pause running ends nothing and starts no turn, and is still answered with a paused: false frame, so a page that missed the end of a pause clears itself. Treat any reply from the avatar as the end of a pause too.

{ "type": "action", "name": "pause_conversation", "label": "On hold", "paused": true, "max_seconds": 90 }
{ "type": "action", "name": "pause_conversation", "label": "Welcome back", "paused": false, "reason": "spoke" }

reason is one of spoke, resumed and timed_out. Show a paused state only on a frame that carries paused. The same name also arrives without it while the holding line is still playing, and that turn can still be interrupted.

Changing language#

The languages are a closed set: en, es, fr, de, it, pt, hi, ja. A Cartesia voice keeps its voice and changes language. Any other voice engine keeps its voice when that voice speaks the language. Otherwise it switches to a voice from its whole catalogue that does, preferring the same gender as the current voice and then the lowest id, so the same request always picks the same voice. It refuses when it has none. When the voice changed, the frame also carries voice_id and voice_label. The switch lasts for the call. It is never saved to the persona, so the next call starts in the box's own language.

{ "type": "action", "name": "change_language", "label": "Now speaking Spanish", "language": "es", "applied": true }
{ "type": "action", "name": "change_language", "label": "This voice cannot speak that language", "language": "en", "applied": false }

language is the language the call is in after the request. The avatar then says one fixed line: a confirmation in the new language, or the refusal in the current one.

From an SDK#

Both SDKs type these frames for you, on the event they already raise for end_call and skip_turn. Every outcome field is absent (JavaScript) or None (Python) unless the box sent it, and a value of the wrong type is left out rather than guessed. pause_conversation and change_language each arrive twice: once with only a name and a label while the avatar is still speaking, and once more with the outcome. Act on the second, the one that carries paused or applied.

PauseEndReason is SPOKE, RESUMED, TIMED_OUT, SESSION_ENDED, UNKNOWN or CONNECTION_LOST. A reason a newer box sends that the SDK has never heard of becomes UNKNOWN, with the box's own word kept in the raw field, so it never throws. CONNECTION_LOST is the SDK's own and the box never sends it: the box keeps a pause per socket, so when the JavaScript SDK redials a dropped microphone socket it clears the hold and raises one action with this reason and no raw word. The Python SDK never redials in place, so there a dropped socket is a closed session and the hold clears with it.

import { PauseEndReason, PersonaActionName, ZeliEvent } from "@zeligate/zeli-avatar";
 
client.on(ZeliEvent.PERSONA_ACTION, (action) => {
  if (action.name === PersonaActionName.PAUSE_CONVERSATION && action.paused !== undefined) {
    showHold(session.isPaused, action.maxSeconds);
    if (action.reason === PauseEndReason.TIMED_OUT) note("The pause ended");
  }
  if (action.name === PersonaActionName.CHANGE_LANGUAGE && action.applied !== undefined) {
    note(action.label);
    if (action.voiceId) showVoice(action.voiceLabel ?? action.voiceId);
  }
});
 
resumeButton.onclick = () => session.resume();

from zeli import PauseEndReason, VoiceActionName, ZeliEvent
 
async def on_action(action):
    if action.name == VoiceActionName.PAUSE_CONVERSATION.value and action.paused is not None:
        show_hold(session.paused, action.max_seconds)
        if action.reason is PauseEndReason.TIMED_OUT:
            note("The pause ended")
    if action.name == VoiceActionName.CHANGE_LANGUAGE.value and action.applied is not None:
        note(action.label)
        if action.voice_id:
            show_voice(action.voice_label or action.voice_id)
 
client.add_listener(ZeliEvent.VOICE_ACTION, on_action)
 
await session.resume()

The hold state is kept for you. session.isPaused (JavaScript) and session.paused (Python) are true from the frame that says paused: true until the box says it is over. They are already up to date inside the action handler, they clear when a reply arrives (the frame that ends a pause can be lost), and they clear when the microphone or the session ends.

resume() is safe to call at any time and never throws. It sends { "type": "resume" } and resolves true. With nothing paused the box ends nothing, starts no turn, and still answers with a paused: false frame. With no live microphone there is nobody to ask, so it sends nothing and resolves false. The box ends the pause, not the call: the hold state stays true until the box's answer arrives.

On the wire#

Two frames on /api/voice/ws:

{ "type": "action", "name": "skip_turn", "label": "Take your time" }
{ "type": "action", "name": "end_call", "label": "Ending the call" }
{ "type": "session_end", "reason": "ended_by_persona_request" }

Branch on name; show label. An end_call action arrives while the goodbye is still playing, so end nothing on it. session_end follows once it has played, just before the box tears the session down: end the session on your side and do not reconnect. The JavaScript SDK raises PERSONA_ACTION and SESSION_ENDED_BY_PERSONA; the Python SDK raises VOICE_ACTION and VOICE_SESSION_ENDED.

Switching one off#

system_tools on POST /api/settings is the list of tools that are on. Leave it out and every one is on.

{ "system_tools": ["end_call"] }

Names outside the four, a repeated name, or anything that is not a list is refused with 400 setting_refused and nothing is changed.

Not served in this group#

• GET /tools. The built in set cannot be enumerated over HTTP. The table above is the enumeration. Your custom tools are read from GET /api/settings.
• Webhook and server side custom tools. The server never calls a URL for a custom tool. Only the connected application can answer.
• Read, update or delete one tool by id. There is no tool id. The tools list is written whole.
• Switching a built in tool on or off per caller. The five are a deployment setting. The system tools are the exception, through system_tools.
• System tools on typed turns, the SDK control socket or LiveKit. They act on voice sessions only. In a LiveKit session the model runs in your agent.
• LiveKit rooms. In a LiveKit session the language model runs in your agent, not on this server, so there is no tool_call to send. Define tools in your agent there.
• An SDK helper for typed chat. registerToolCallHandler answers calls on the two sockets. A call that arrives on a POST /api/chat event stream is answered with POST /api/chat/tool_result by your own code.