In-depth analysis · About 6 min read
Cekura’s 26-model TTS benchmark: align the clocks before choosing a voice
The October 19 snapshot separates streamed-text latency, difficult readings, estimated naturalness and stopping. Its fastest number is not a complete voice-agent response time.
- Published here
- Source published
- Source checked
Cekura’s text-to-speech benchmark, first published on October 4 and last updated October 19, compares 26 model configurations. Reading the original tables and methodology shows why response speed, accurate codes and a natural voice require separate decisions. These are Cekura’s reported measurements, not AISubHub tests or new model launches.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- The page reports 104 agent sentences, three readings each and 30,564 total requests, from a single US East client region with one stock voice per configuration.
- Gradium’s −64 ms streamed-text median and Inworld TTS-2 Flash’s 46 ms full-text median use different starting points. Reading accuracy and UTMOS naturalness are explicitly estimates.
What the controlled comparison does, and does not, hold constant
Cekura calls each configuration from one cloud region in the US East and uses one stock voice. The same 104 agent sentences range from greetings to replies of roughly 500 characters, with three readings of each sentence. This helps compare a selected voice and connection path. It does not put every vendor’s inference servers in the same place, establish every voice’s performance or cover a customer’s actual telephone route. The visible examples also do not establish equivalent results for Chinese.
The request counts need their own boundary. Multiplying 26 models by 104 sentences by three readings produces 8,112, not the page’s reported 30,564 requests. The public page does not break that total into every condition, so we cannot invent a denominator or explain the difference as retries. Whole-text, streamed-text and interruption tests are distinct; eight configurations cannot accept streamed text and are absent from that chart. Neither the headline total nor an omitted row is evidence of one uniform sample.
Why a negative streaming latency is possible
Non-streaming latency starts when a whole sentence is sent and ends at the first audible sound. Connection establishment is excluded, while opening silence is included. The selected Inworld TTS-2 Flash voice, Sarah, records a 46 ms median on that measure and 45 ms with streamed text. Gradium’s Everly records 76 ms with whole text. These are measurements of the tested synthesis path, not the time from a caller finishing a request to an agent completing an answer.
For streamed text, the clock instead starts at the last word sent. Gradium’s −64 ms median means sound begins before that word arrives on most tested sentences, according to Cekura. It does not mean negative physical waiting time or knowledge of future text. A voice agent may also wait for speech recognition, reasoning, tools and playback buffers. Those stages can overlap, so adding isolated vendors’ median latencies would not produce a reliable end-to-end forecast either.
Separate pronunciation estimates from naturalness estimates
Reading accuracy concerns difficult spans such as numbers, codes, addresses, context-dependent abbreviations and medical terms. Recordings are transcribed by Deepgram Nova-3 and OpenAI Whisper; an accepted reading in the transcripts determines a pass, and their results are combined. No person checks those outcomes. Cekura notes that the two transcribers differ by ten percentage points on one model. A high score therefore estimates correct reading under this evaluator, rather than proving what every listener understood or whether a business task succeeded.
Inworld TTS-2 scores 93.4%, Flash 93.2% and Fish Audio S2.1-Pro 92.0%; the page advises treating leading scores within two points as a tie. Naturalness answers another question: Fish Audio Drama 3 preview, using Delia, leads its UTMOS estimate at 4.54. UTMOS predicts a listening-test score; this is not a human panel’s published preference vote. A pleasing timbre cannot certify a confirmation code, and a tiny numerical lead without the needed uncertainty analysis should not become a universal quality claim.
Stopping the generator and stopping playback are different jobs
Interruption exposes a separate boundary. Grok TTS with Carina reports 0 ms stopping, whereas Cartesia Sonic 3.6 with Skylar reports 764 ms. Those figures describe the benchmark’s stopping measurement, not a promise that a caller’s handset will become silent instantly. Some rows have no stop message; the application drops the stream instead. That is not a measured zero for native generator cancellation, and the application must still manage audio already queued for playback.
Cekura also marks configurations whose generated audio sometimes falls behind playback: ElevenLabs v4 Turbo does so on 5% of the tested sentences. That is different from a slow first sound or an interruption result. In an illustrative booking agent, reading the first phrase quickly is useful only if the remainder arrives smoothly and a corrected confirmation code replaces the old speech. Developers should test client playback queues and recovery as well as the synthesis endpoint; this example is an application design check, not a benchmark we performed.
Turn the snapshot into an application acceptance test
Price is another separate column. The page’s lowest displayed list rate is Soniox TTS RT v2 at $14.81 per million characters; Fish Audio Drama 3 preview has no listed price there. Missing price does not mean free access. Character-based list rates are not monthly subscriptions or completed-call quotations. A deployment budget must establish billing units, tiers and availability with the provider, then include recognition, reasoning, tools, retries and the actual amount of generated speech.
For acceptance, first rerun representative codes, names and long responses with the intended voice and user region; then stream partial text, interrupt it and inspect audible playback; finally compare quality and full-task cost against the application’s requirements. These are AISubHub recommendations, not newly measured results. Cekura discloses that it sells testing and monitoring in the market it evaluates and that its advisory board is still being formed. It links a pinned harness commit and publishes results under CC BY 4.0. That supports attribution and inspection, not an independent reproduction by us or a combined global model ranking.
Why it matters
AISubHub’s analysis is to select a voice against an application’s actual input and interruption pattern, then verify difficult content and complete-call cost. A single winning column does not establish the best conversational system.
Limits and availability
We read the complete public page and its About and governance material, but did not run the harness or listen to its recordings. This snapshot does not establish Chinese performance, all voices, every region, live service guarantees or current purchasing eligibility.
Sources and evidence
Source published: · Source checked:
Further reading
Source checked:
Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.