How should an outbound audio queue be sized when relaying a bursty/faster-than-real-time streaming TTS source (e.g. ElevenLabs WebSocket TTS) into a real-time RTP sink (e.g. WhatsApp/WebRTC calling) that can only play back at wall-clock speed (50 packets/sec for 20ms Opus frames)?

webrtc · verified Jul 11, 2026

Fix: Size the buffer to hold at least one full TTS utterance/response (tens of seconds), not a few hundred milliseconds of audio. Streaming TTS providers like ElevenLabs generate and push audio over the WebSocket much faster than real-time playback consumes it, so the full response for a single turn arrives as a burst. Since the RTP sender can only drain at a fixed real-time rate (e.g. 50 pkt/s for 20ms Opus frames), a queue capacity of only a few seconds fills to capacity almost immediately for any non-trivial response and stays pinned at the cap for the rest of the turn. Every subsequent enqueue then evicts the oldest queued packet -- which is also the next packet due to be played -- causing continuous audible dropouts, while playback is permanently stuck behind by however many seconds the cap represents (observed: a 5-second/250-packet cap produced sustained 'queue overflow' warnings and a persistent ~5s lag for the entire call). Sizing the cap to a full utterance (e.g. 60s) is safe because: (1) it adds no latency to time-to-first-audio, since the sender still drains from the front of the queue every tick regardless of queue depth -- the extra capacity only holds audio that hasn't been due for playback yet; (2) the queue naturally drains to empty between turns since TTS stops producing once the response is fully generated; (3) any interrupt/barge-in path must still explicitly clear the queue on user speech detection to keep responsiveness, since a deep buffer only avoids the overflow-drop pathology, it does not by itself bound interrupt latency.

webrtcrtpelevenlabsttsbufferingreal-time-audiovoice-agent

References