Build

Web and mobile clients

The caller’s side of a call is a small piece of code in your web page or mobile app: join a room, send the microphone, play the agent’s voice. Calls run on LiveKit, so any of its client SDKs will do.

The shape of it

1. Your app asks your server for a call. Your server calls Start a call with its API key.
2. Your server passes livekitUrl and token back to the app. The token opens that one room for that one participant and nothing else.
3. The app connects, turns the microphone on and plays the audio track the agent publishes.
4. To hang up, the app disconnects. The call ends with the reason user-left.

In a browser

npm install livekit-client

import { Room, RoomEvent } from 'livekit-client';

export async function joinCall({ livekitUrl, token }) {
  const room = new Room({ adaptiveStream: true, dynacast: true });

  room.on(RoomEvent.TrackSubscribed, (track) => {
    if (track.kind === 'audio') document.body.appendChild(track.attach());
  });
  room.on(RoomEvent.Disconnected, () => {
    // The call is over: the agent hung up, the time limit was reached, or the network dropped.
  });

  await room.connect(livekitUrl, token);
  await room.localParticipant.setMicrophoneEnabled(true);
  return room;
}

Start the call from a click or a tap. Browsers hold a page’s sound back until the visitor has interacted with it.

Live captions and what the agent is doing

While a call runs, the agent publishes messages on the data topic goblins-transcript. There are two kinds:

import { Room, RoomEvent } from 'livekit-client';

const decoder = new TextDecoder();
const room = new Room();

room.on(RoomEvent.DataReceived, (payload, _participant, _kind, topic) => {
  if (topic !== 'goblins-transcript') return;
  const message = JSON.parse(decoder.decode(payload));

  if (message.type === 'state') {
    // 'listening' | 'thinking' | 'speaking'
    showAgentState(message.state);
  } else if (message.role) {
    // role: 'user' | 'assistant'. The same words can arrive more than once: see below.
    showCaption(message.role, message.text, message.final);
  }
});

Caption messages favour speed over tidiness, and a display has to allow for that:

• An agent’s reply arrives one sentence at a time, each with final: false until its last, and then once more as the whole reply. If the caller interrupted, that whole reply holds only what was actually said.
• The caller’s words can be sent again as the transcript settles.
• There are no message IDs. Treat a message that starts like the line you are showing for that speaker, or the other way round, as that line again.

A talk button for your website has a caption function that handles all three. These messages are for your interface; the record of what was said is the call’s transcript, which you read from the API or receive by webhook.

Asking the agent to say that again

If the conversation seems stuck, your app can send repeat_request on the topic goblins-client-control, for example from a “Didn’t catch that” button. The agent speaks a short line asking the caller to repeat themselves. Requests sent in quick succession are ignored.

const encoder = new TextEncoder();

await room.localParticipant.publishData(
  encoder.encode(JSON.stringify({ type: 'repeat_request' })),
  { reliable: true, topic: 'goblins-client-control' },
);

Mobile apps

The flow is the same on a phone; only the SDK changes.

Each is documented at docs.livekit.io. Three things matter on a phone that do not in a browser:

• Ask for microphone permission before connecting, and explain why in the system prompt’s text.
• Keep the audio session alive when the screen locks or the app goes to the background, or the call will drop.
• On Android, a call that continues in the background needs a foreground service with the microphone type.

When the join fails