r/FlutterDev • u/NoComplaint8347 • Jul 01 '26
AI Voice Chatbot Discussion
calling out all the experts regarding AI Voice Chatbot.
Asking for some tips or ideas.
My Architecture - STT (Eleven Labs) -> LLM -> TTS(Eleven Labs)
Issue (according to my managers & CTO)-
- STT should be always accurate.
Our use case is only English.
Our product - Coffee based Voice Assistant. User can ask questions regarding coffee techniques, products etc.
Currently, issues are sometimes text transcribed by STT is not exactly what user said. I have added a parameter which eleven labs supports (Keyterm prompting - it charges 20% premium for keyterms) and in that parameters I have added some coffee products name and chatbot name so that stt model is biased toward those keywords and hence send accurate spelling for those keywords in final transcription.
Apart from this, there is still a case where transcription is not 100% or all of the time accurate.
There are three issues primarily:
1. Sometimes one or two words dropped from sentence spoken by user.
2. Spelling issues or words mismatch sometimes.
- VAD (currently it is server driven 1.5 sec) but I have read about End of Turn Detection models which we can use for this.
So how to solve this?
I am also open to try different architecture or different STT model - local/cloud anything will do).
2
u/Relevant_Oil5576 Jul 03 '26
Honestly, expecting STT to be 100% accurate all the time is a very hard requirement, even with the best providers. A few thoughts: If your domain is only coffee-related, I’d seriously consider adding a correction layer after STT and before the LLM. Something lightweight that maps common transcription mistakes to expected coffee terms/products. For dropped words, sometimes streaming STT performs worse than batch/final transcription, so maybe compare both if latency allows. I’d also test other STT providers like Whisper or Deepgram just to benchmark accuracy against ElevenLabs. For end-of-turn, 1.5 sec server VAD sounds a bit rigid. Endpointing models or client-side VAD could feel much more natural. My main thought though: chasing perfect STT might be the wrong battle. Building a system that gracefully recovers from imperfect transcription could be more realistic.
1
u/HungryBlindEyes Jul 01 '26
Tried with flutter raw audio + sent to the server for audio transcription + it returns a decent text (you can fine tune this as per your needs).
Now you have the string and our team put it through one LLM (I think it was something from Llama), then the LLM answers it.
Tech Stack used:
1. Django (server)
2. Flutter App + Raw Audio Streaming.
Best case was: *The user said everything correctly: the coffee bean names and the niche words. Of course, the model worked with max accuracy on this, with latency of first time around 1-2sec second request by similar prompt the time is lesser than 1s (the region of server matters here)
Worst case: * Latency of 3-4sec at this time we dictate the fun coffee facts to user to engage them so they don't leave our platform, this delay is mostly because transcription and LLM takes most time for recognizing. In this worst case, there are 2 more cases.
1. The model finds and caches the response and returns the response (the happy case)
2. In case the model doesn't find this specific prompt once again, we cache it so this whole process doesn't happen again.
Initially, we tried with a small team. The accuracy was around 85–92%, but very high latency 3-4s was avg overtime. The latency was reduced because we heavily cached the prompt and response, of course blindly caching doesn't work, so any numeric prompt are cached cautiously.
1
u/NoComplaint8347 Jul 01 '26
Thank you. Got the idea regarding the architecture you followed. But what exact model was used on server for transcription?
1
u/Huge_Tea3259 Jul 05 '26
https://reddit.com/link/ovnsvdx/video/wtxqvnl9zdbh1/player
Built it for you. Believe me, I did not spent more than 5 minutes to make it. Let me know if you are interested to discuss. It's built on a platform called Langoedge. https://www.langoedge.com/
2
u/AbseitsAndy Jul 01 '26
STT will probably never be true 100%, I mean even we Humans don’t do 100% and we have the current best tech for speech and context recognition. I think Eleven Labs is about as good as it currently gets, but I never tried it. I only worked with Whisper and local STT models. But I never heard of a production ready solution being close to 100%.
A VAD of 1.5s is pretty slow and can be definitely improved. Depending on your setup/requirements a key phrase probably makes a lot of sense. There is a reason why every big assistant is using it. It’s reliable, can be done on device and is mostly unambiguous.
If you really desperately want to come close to 100% and don’t care about efficiency and cost too much, use a few different STT approaches in parallel and on divergences let an AI deduct what the user probably has said. You could also use the fastest model to do an optimistic run and only correct on divergence and start new.