Stream Message Layer - Sanas Developer Hub

Documentation Index

Fetch the complete documentation index at: /llms.txt

Use this file to discover all available pages before exploring further.

The Stream message layer defines the application protocol shared by the WebSocket Stream API and the WebRTC API. It covers session setup, pipeline configuration, turn finalization, transcription and translation events, lifecycle events, errors, and usage modes. Transport mapping:

Do not send audio in JSON messages. The init, configure, flush, and server event shapes are the same across transports.

Protocol Overview

Connection Lifecycle

ServerClientServerClientalt[Manual turn finalization][Automatic turn finalization]opt[Start another turn]opt[End session]

1. Open authenticated Stream transport
2. Send init with session parameters
3. Send configure with languages and voice
4. Send configured
5. Stream input audio over transport
6. Send transcription events
7. Send translation events
8. Stream output audio over transport
9. Send output boundary and lifecycle events
10. Send flush
11. Detect end of speech with VAD
12. Send remaining results and flushed
13. Send new configure
14. Close transport

Key Concepts

• Init: Sent once per connection. Sets session-level parameters (sample rates, conversation ID).
• Configure: Sent after init and again at any point to change languages, voice, or glossary. Each configure starts a new pipeline configuration. The server responds with configured when ready.
• Flush: Signals the server to finalize the current input and produce all remaining output. The server responds with flushed when complete. Use this for manual turn-taking or to force processing of buffered audio.
• Utterance: A continuous segment of speech. The server assigns an incrementing utterance_idx to each detected utterance. When VAD (Voice Activity Detection) is active, the server automatically splits speech into utterances.

Client Messages (Client -> Server)

Client control messages are JSON objects with a "type" field. Send them as WebSocket text frames on the WebSocket transport, or as stringified JSON messages on the WebRTC data channel. Audio transport is described in Input Audio.

init

Required as the first message on every connection. Sets connection-level parameters that remain fixed for the session lifetime.

{
  "type": "init",
  "conversation_id": "optional-id",
  "session_name": "my-session",
  "input_sample_rate": 16000,
  "output_sample_rate": 16000,
  "realtime_playback": true
}

configure

Initializes or reconfigures the translation pipeline. Must be sent after init. Can be sent again mid-session to change languages, voice, glossary, or features without reconnecting.

{
  "type": "configure",
  "language_routes": [
    {"lang_in": "en-US", "lang_out": "es-ES"}
  ],
  "features": [],
  "voice_id": null,
  "glossary": [
    {"terms": {"en-US": "Sanas", "es-ES": "Sanas"}}
  ],
  "request_id": "optional-correlation-id"
}

The server responds with a configured message when the pipeline is ready to accept audio.

Language Routes

Language routes define the translation directions the pipeline supports. Each route specifies a source and target language.

{"lang_in": "en-US", "lang_out": "es-ES"}

Single route — translates one direction only:

"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"}
]

Bidirectional routes — the server detects which language is being spoken and translates to the other. Use this when both parties may speak either language (replaces the old can_lang_swap behavior):

"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"}
]

When multiple routes are provided, the server performs language identification on the incoming audio and selects the matching route. You will receive a language_route event indicating which route was chosen for each utterance. Auto-detect mode (any language to one target) — use "*" as lang_in to accept any supported language and translate it to a fixed target. The server identifies the spoken language automatically and translates to the specified output language:

"language_routes": [
  {"lang_in": "*", "lang_out": "en-US"}
]

This translates any incoming language to English. The server will send a language_route event with the detected language filled in:

{"type": "language_route", "utterance_idx": 0, "lang_in": "ja-JP", "lang_out": "en-US", "is_final": true}

In auto-detect modes, the server may send tentative language_route messages (is_final: false) while language identification is still refining, followed by a single is_final: true message once the route is locked in. Clients that only care about the final answer can ignore tentative messages and act on is_final: true. You can combine auto-detect with an explicit route. For example, to translate English to Spanish and auto-detect everything else to English:

"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "*", "lang_out": "en-US"}
]

NOTE: When multiple language routes are specified, they will be matched in list order. The first route to match wins. In the following language route configuration, the server will always choose the first route because * matches any detected language. In this case, the consequence is: English speech will remain as English.

"language_routes": [
  {"lang_in": "*", "lang_out": "en-US"},
  {"lang_in": "en-US", "lang_out": "es-ES"},
]

Wildcard catch-all route — add a "*" -> "*" route alongside your normal routes to handle speech in unexpected languages gracefully. When the detected language does not match any explicit route, the wildcard route is selected and the pipeline enters a transcription-only mode (no translation or TTS output). This is useful for prompting the user to switch languages:

"language_routes": [
  {"lang_in": "en-US", "lang_out": "es-ES"},
  {"lang_in": "es-ES", "lang_out": "en-US"},
  {"lang_in": "*", "lang_out": "*"}
]

When the wildcard catch-all route is triggered, you will receive a language_route event with the detected language echoed into bothlang_in and lang_out. For the configuration above, if the user speaks French you will see:

{"type": "language_route", "utterance_idx": 1, "lang_in": "fr-FR", "lang_out": "fr-FR", "is_final": true}

Note that the server then emits transcription / transcription_ended for the utterance but no translation, translation_ended, output_audio, output_speech_started, or output_speech_ended (transcription-only mode). You can use this language_route to show a prompt to the user, for example: “It seems like you are speaking French. Would you like to switch your language?” If the user confirms, send a new configure with the appropriate routes.

Glossary Entries

Glossary entries either (a) keep a phrase verbatim across languages or (b) lock in a specific translation between two languages. Each entry is a map of language codes to the desired term in that language. Keep a phrase verbatim across languages — use "*". For brand names, product names, and other proper nouns that should never be translated, use the "*" wildcard:

{"glossary": [
  {"terms": {"*": "Sanas"}},
  {"terms": {"*": "ACME"}}
]}

This tells the pipeline that “Sanas” should remain “Sanas” in every language, and “ACME” should remain “ACME” in every language. Lock in a specific translation between 2+ languages — use language codes. To force a phrase in one language to a specific phrase in another, list each phrase by its language code:

{
  "terms": {
    "es": "servicio al cliente",
    "fr": "service client",
    "en": "customer service"
  }
}

Here, “servicio al cliente” spoken in Spanish becomes “service client” when translating to French, and “service client” spoken in French becomes “servicio al cliente” when translating to Spanish. A glossary entry only takes effect on a language route when both the input and output languages are specified. If either side isn’t covered, the entry is ignored for that route.

Features

The features array controls which pipeline mode is used. By default, you can leave the features array empty or omitted. Note that feature names are case-sensitive:

Input Audio

Streams audio data to the server after receiving configured. Audio transport depends on the API you are using:

For WebSocket binary audio, use this format:

• Encoding: 16-bit signed integer PCM (little-endian)
• Channels: Mono (1 channel)
• Sample rate: Must match input_sample_rate from init

flush

Signals the server to finalize processing of all buffered audio and produce remaining results. The server will respond with a flushed message after all output has been sent.

{
  "type": "flush",
  "request_id": "optional-correlation-id"
}

Server Messages (Server -> Client)

Server control and text messages are JSON objects with a "type" field. Receive them as WebSocket text frames on the WebSocket transport, or as stringified JSON messages on the WebRTC data channel. Audio transport is described in Output Audio. Note on optional fields: Fields documented as “or null” (such as request_id, probability, utterance_idx on errors) are omitted from the JSON when not present, rather than sent as "field": null. Your client should treat a missing key the same as a null value.

configured

Confirms the pipeline is ready to accept audio after a configure message.

{
  "type": "configured",
  "request_id": "r1"
}

input_speech_started

Indicates the server’s VAD has detected the start of speech in the input audio.

{
  "type": "input_speech_started",
  "utterance_idx": 0,
  "input_time": 0.5
}

language_route

Indicates which language route was selected for the current utterance. Sent when the system identifies the spoken language, particularly useful when multiple language routes are configured.

{
  "type": "language_route",
  "utterance_idx": 0,
  "lang_in": "en-US",
  "lang_out": "es-ES",
  "is_final": true
}

transcription

Real-time transcription of the input audio. Complete words have been finalized and will not change. Partial words are in-progress and may be updated in subsequent messages.

{
  "type": "transcription",
  "utterance_idx": 0,
  "complete": [
    {"word": "Hello ", "start": 0.0, "end": 0.5, "probability": 0.98}
  ],
  "partial": [
    {"word": "world", "start": 0.5, "end": 0.8, "probability": 0.85}
  ]
}

Word object:

transcription_ended

Indicates that all transcription for a given utterance is complete. No further transcription messages will be sent for this utterance.

{
  "type": "transcription_ended",
  "utterance_idx": 0
}

translation

Translated text derived from the transcription. Like transcription, includes complete (finalized) and partial (in-progress) words.

{
  "type": "translation",
  "utterance_idx": 0,
  "complete": [
    {"word": "Hola ", "start": 0.0, "end": 0.5, "probability": 0.95}
  ],
  "partial": [
    {"word": "mundo", "start": 0.5, "end": 0.8, "probability": 0.85}
  ]
}

translation_ended

Indicates that all translation for a given utterance is complete.

{
  "type": "translation_ended",
  "utterance_idx": 0
}

input_speech_ended

Indicates the server’s VAD has detected the end of speech in the input audio. After this, the server will not accept further input audio for this utterance and will finalize the translation.

{
  "type": "input_speech_ended",
  "utterance_idx": 0,
  "input_time": 3.2
}

output_speech_started

Indicates the server has begun sending synthesized output audio for an utterance.

{
  "type": "output_speech_started",
  "utterance_idx": 0,
  "output_time": 0.0
}

Output Audio

Synthesized audio from the translated text. Audio transport depends on the API you are using:

For WebSocket binary audio, the format is:

• Encoding: 16-bit signed integer PCM (little-endian)
• Channels: Mono (1 channel)
• Sample rate: Matches output_sample_rate specified in init

output_text_boundary

Timing information that maps a position in the output audio stream back to positions in the transcription and translation text. Use this to synchronize text highlighting with audio playback. Sent by default. Clients that don’t need word-level highlighting can opt out by setting the "hide_output_text_boundary" feature in configure, which suppresses these messages.

{
  "type": "output_text_boundary",
  "utterance_idx": 0,
  "output_time": 1.5,
  "transcription": {"word_idx": 3, "char_idx": 15},
  "translation": {"word_idx": 2, "char_idx": 12}
}

TextPosition object:

output_speech_ended

Indicates all synthesized audio for an utterance has been sent.

{
  "type": "output_speech_ended",
  "utterance_idx": 0,
  "output_time": 4.2
}

flushed

Confirms that a flush has been fully processed and all pending output has been sent.

{
  "type": "flushed",
  "request_id": "f1"
}

error

Indicates an error occurred during processing.

{
  "type": "error",
  "message": "Invalid config message",
  "code": "CORE_ERROR",
  "utterance_idx": 3,
  "request_id": "r1"
}

Non-fatal errors (e.g., malformed JSON, unknown message type) are sent as error messages with code "STREAM_ERROR" but the connection remains open. You can continue sending messages after receiving an error. Errors originating from the backend translation pipeline will have their own error codes.