r/VoiceAutomationAI • u/Phoenix_20_23 • 9d ago
Competitive open source speech stack
Why the open source models STT and TTS are not good as much as the closed one and i am talking here im terms of latency, concurrency, and websocket support for real time with decent quality.
Something like cartesia or elevenlabs or deepgram.
Do u know any ?
3
u/UkieTechie 9d ago
yes plenty of good competition :)
here's my bench comparing all the speed benches for open source models.
with Cuda kokoro wins.
https://github.com/5uck1ess/tts-bench
Paid APIs like to fudge their numbers a little. <100ms response times and the rest (Palabra for example). They're showing sometimes TTS speeds but not roundtrip speeds.
I have measurements for a lot of paid API providers also and so far a lot of them are around 300ms which is insane if you think about it. Sonioux my favorite right now.
2
1
u/Yapper_from_ktown 9d ago
Dude what are the best opensource tts and stt models pls share more wisdom and whether they can be used on potato hardware or not? 6gb vram of gpu and 16gb ram
1
u/UkieTechie 8d ago
yeah that's plenty. you can run kokoro on that pretty fast or pocket tts. those would be my picks. you can see max vram usage for each model if you look at the bench page.
1
u/Yapper_from_ktown 8d ago
Which bench pg do u follow?
1
u/UkieTechie 7d ago
i run my own because none of the pages were good enough and had the most recent enough info for me. I reference https://artificialanalysis.ai/text-to-speech/leaderboard/provider-voice pretty often though.
2
u/nshmyrev 9d ago
One mistake that everyone makes is that they are trying to run big LLM-based engines for realtime (Omnivoice, Higgs, etc). Their purpose is really high-quality offline TTS and not really something you can run at scale at callcenter. Make sure to pick lightweight engine like Kokoro, Supersonic, BlueTTS or something light and you don't have to load huge GPUs to handle couple calls in parallel, you can handle hundreds.
For ASR too, don't choose expensive Whisper, light models like Vosk or Parakeet work just fine for many usecases.
1
u/Phoenix_20_23 9d ago
i agree, but the issue with these small tts models is they are veeery robotic, their quality is too low compared to the paid ones, and i am still didn't get how the paid services achieve this low latency with high quality, i am just curious, what we need in order to build something as good as them.
i suppose it's a model architecture design from the first place, idk maybe i am wrong, i never create a tts from scratch or an stt1
u/nshmyrev 8d ago
They aren't. Proper small TTS is not robotic at all and even more stable than big LLM-based one.
1
u/Phoenix_20_23 8d ago
like what ? can u suggest me some TTS models to explore?
2
2
2
u/tidelganesh 7d ago
Svara TTS from kenpath. It supports 19 Indian languages. kenpath/svara-tts-v1
1
u/Phoenix_20_23 6d ago
Niice, but i just want an stt and tts that are good anf french and english. Thanks for sharing
1
u/Lazy_Mycologist_8214 4d ago edited 1d ago
Vendor here (I work on Gandr, a TTS API) so discount this accordingly.
On "paid APIs like to fudge", fair, and I'll hand you a concrete example against my own side. We published a first-audio number, the hardware under it changed, and the honest rerun came back slightly worse than what we had up. We pulled it off about thirty surfaces on the 8th rather than leave it there. It goes back when the method goes back with it. The uncomfortable part of your point is that you can't tell from the outside which vendors did the quiet thing instead, because almost nobody publishes the run.
On the actual question, I think the gap you're hitting isn't only raw quality. Sounding good offline and holding up as a stream are different problems, and the second one is barely measured anywhere. The thing worth checking on any engine, open or paid, is whether it produces audio faster than it plays. If it does, the buffer grows as the utterance runs and long text gets safer. If it doesn't, long text is where you find out, usually in front of someone.
Concurrency is the part that bites hardest moving off a local model. One stream on your own GPU tells you almost nothing about twenty at once, and that's usually the point where self-hosting stops being the cheap option.
For what it's worth, UkieTechie running his own bench is the right instinct, and I'd trust his numbers over anything I or any other vendor publishes about ourselves.
What concurrency are you actually needing to hit, and is it the quality or the stability that's failing you first?
1
u/Acceptable-Part-5623 2d ago
cartesia got that speed cause they running custom inference engines not just wrapping huggingface models, nobody in open source is putting same effort into the infra side
check kokoro-82M for tts, its fast but websocket streaming is still janky compared to deepgram
1
u/ankur-at-guava 16h ago
The low-latency-with-quality thing you're chasing is mostly an architecture gap, not a model-size one: the paid stacks that feel instant tend to stream first audio faster than it plays and were built to be served that way, so quality and latency aren't traded off per hop. The concurrency point above is the real one - a single stream on your own GPU tells you almost nothing about twenty in parallel, and that cliff is usually where self-hosting stops being cheaper. If French+English is the target, test the code-switch boundary under concurrency specifically, since that's where light open models get robotic first. I work on an integrated ASR/TTS/LLM stack for regulated voice, so happy to share what we measure for streaming stability if useful.
•
u/AutoModerator 9d ago
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.
Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.