Using sync to avoid exhausting the audio buffer

The API allows you to buffer up to 30MB of audio data at any given time. The buffer size is chosen to be large enough that you don’t need to worry about it in most normal use cases — any individual audio file will comfortably fit inside the 30MB limit. However, this can become an issue if your application tries to play multiple large audio files in succession, or if you play the same audio file repeatedly in a loop.

In the following example, the application naively plays the same file again and again, looping forever. This will quickly fill the API’s audio buffer and cause the call to end with an error:

# NB: This code does not work
while True:
    with open("audio.wav", "rb") as f:
        await elks_ws.send(json.dumps({
            "t": "sending",
            "format": "wav"
        }))
        for chunk in iter(lambda: f.read(32 * 1024), b""):
            await ws.send(json.dumps({
                "t": "audio",
                "data": base64.b64encode(chunk).decode()
            }))

Instead of writing complex timing code in your application, or guessing how much data is in the API’s buffer at any given moment, you can use the sync message to synchronize the state between the API and your application.

The sync message acts as a “checkpoint” of sorts that lets your application know that the API has processed all messages up until that point. When the API receives a sync message it will reply back with its own sync — but only after all previous messages have been processed. In other words, if you send a sync after sending some audio, the API will not respond to it until all audio sent before the sync has been played.

You can use this to ensure your application doesn’t send “too much” audio at once, as in the following example.

Example: Looped playback with sync

This example plays a WAV file in a loop until the user hangs up.

async def looped_playback(ws):
    # Get the call metadata from the hello message
    hello = json.loads(await ws.recv())
    print(f"Received {hello['to']} <- {hello['from']} ({hello['callid']})")

    while True:
        # Tell the API the format we'll be sending audio in
        await ws.send(json.dumps({
            "t": "sending",
            "format": "wav"
        }))

        # Send the WAV file bytes in chunks
        with open("audio.wav", "rb") as f:
            for chunk in iter(lambda: f.read(32 * 1024), b""):
                await ws.send(json.dumps({
                    "t": "audio",
                    "data": base64.b64encode(chunk).decode()
                }))

        # Send the sync message
        await ws.send(json.dumps({
            "t": "sync"
        }))

        # Wait for the response from the API.
        msg = json.loads(await ws.recv())

        # Exit if the call has ended. Else, play the file again.
        if msg["t"] == "bye":
            print("Call ended:", msg["message"])
            break