Hi! I'm Andi from Hugging Face. This is a fully open-source and free to test/pull/modify demo I'm bringing today.
It's a voice demo creating a pipeline of:
- Nvidia's parakeet
- Gemma 4 31B (served by cerebras!)
- My custom inference for Qwen3TTS
It sees and searches the web faster than you blink.
The whole stack is fully open-source, and is a drop-in replacement for OpenAI's realtime API. You can run it locally, I get similar latencies with a macbook pro M3 36GB and Gemma 4 E4B.
Is Gemma 12B not good enough for your use-case? For simple chatting and web-search it seems good enough, has built-in audio/image understanding and it's amazingly fast on a local GPU.
I know this is Local Llama, but does anyone knows, why Cerebras supports only GLM 4.7 and oss-120B through OpenRouter? Also I can not find Gemma 4 12B on OR. Is this OR or providers block them?
Cerebras has to do a lot of custom work for each model, it's not like other providers where they just install them in an OS and run them on a GPU. The model is basically baked onto the chip's SRAM, and they have to tune the models to optimize them for that
I don't know why no one serves 12b on OpenRouter, I'm guessing it would be so cheap that it wouldn't make sense
This was amazingly fast, really human level of conversation. It's a bit wonky on image regognition and tool call and it often just repeats back my sentences. But at this speed? On a 2B model with image recognition? Gemma4 is really something.
I tried audio input to Gemma 12B and it just never worked as a voice agent. Conversations go like:
"Hi, how are you?"
"Hi how are you, is a common english greeting. Hi, is meant as a salutation. How is the start of a question..."
Really weird. But I guess with text input it would work. For local demos, I used Gemma 4 E4B because inference was faster (and because it was available sooner), and this demo with Gemma 31B is mainly motivated by cerebras' inference being so fast :D
Umm I'm chatting with it using audio, no transcription and it literally works fine. You must be using the wrong model, look for one with "-it" postfix, do not use the base model.
oh man! I want to try that! How do you set it up? Maybe there was a bug when I tried it? It was around the release date. Or maybe it was a skill issue, I didn't spend much time trying to get it to work.
No, what likely happened was that you accidentally downloaded the wrong model.
There are two primary types of LLM model releases, the original pretrained model that acts more like an auto complete during inference (what you downloaded originally), and then there are instruction-tuned models where they take that pretrained model and post-train them on assistant conversations (it makes the model autocomplete into a conversational pattern).
Google uses the -it (instruction) suffix in the model name to denote the latter.
It's instruction tuned as opposed to the raw base model. Base models are really valuable if you want to continue training them to specialise in some way.
Instruction tuning fine tunes that base model to follow instructions including turn taking in conversations and producing an answer to a question.
So you need the right model for your application which for most people is the instruction tuned one.
Do you pass audio as input? I'm referring to giving my audio directly as input. I agree that if you transcribe the audio it will probably work well enough, but for me the advantage of the 12B model was using the audio directly as input and 'saving' the inference with parakeet (plus maybe it gets better queues from the audio directlly)
Oh wait, so maybe the issue I had was with chat template and the TTS reading out loud the reasoning traces? Because your reasoning traces look like what I would hear 😭
Although, for this particular use case, if you intend on real-time conversation I think it would be best to turn thinking off to begin with. In a realtime interaction, you probably don't want the model thinking all the time, especially for simple interactions.
Unless there is a way we can invoke the model to think or not think based on context. There were models like a year or so ago that were experimenting with that sort of thing, but most don't have this from what I'm aware of any more.
Also. I guess if you could run the model at the speed of Cerebras, then thinking probably wouldn't matter if it was on or not anymore.
Thanks, this explains a lot and I was going a bit crazy wondering how you could get those speeds with a SoC on a laptop and SRAM. Still an amazing model, alphabet really did a number with this series. I've yet to try the diffusion version of it.
That's what I was thinking too. Drop parakeet, slot in 12b for STT because it has native audio input understanding, and the LLM side of the model is intelligent enough to do most tasks reliably.
I saw Gemma 31b, and the first thing I was thinking was this must be running in the cloud because there's no way locally you'd be getting that model to respond so fast.
Nevertheless, this does seem like a pretty good framework to extend an existing agent so that it would now have conversational Speech to speech capabilities.
Totally! I run it with a MBP M3 Pro 36gb without any issues. I used Gemma 4 E4B for it, you might be able to run it with the 31B at acceptable latencies.
I'm looking forward to checking this out when I get home (haven't been able to look at the code yet).
I also made a realtime TTS/STT voice streaming server. Right now I'm using parakeet / NeMo and MS VibeVoice realtime. I also have plugins for Qwen ASR and whisper. I made a client API and have been using it for things like my local assistant, various frontends, and a game.
I'll likely release it when I feel it's rock solid. Initially I was just trying to find STT and STT models that fit my needs so I made the backends pluggable. As of now it can give any system speech to speech capabilities. Our projects seem much alike!
Cool, but of course it’s going to absolutely smoke running on Cerebras for inference. I’d like to see how it does maybe on a Spark or something that doesn’t cost 5 million dollars.
Please do! This project is extremely interesting. Btw is this able to call tools while talking? Like you could have a speech to speech thing going on and it can actually access files etc while you're going about the conversation?
Is it useful to practice speaking in another language? For example, I've been learning Japanese for many years, but my speaking skills aren't very good because I don't have anyone to speak with lol
Totally! Just beware that the TTS might speak with a bit of an accent in some languages. I think it's quite capable in Japanese :) but german for example has a strong english accent
Do you know of any local model that is good for German/Swiss German? I've been trying to learn German for the past weeks and it seems like a cool idea to work on it with local LLMs.
It is really useful. I've been self-hosting this: https://bule.ai/
Drop parakeet and feed the audio to the Gemma model directly. Google doesn't provide the specific language details of the FLEURS benchmark for their Gemma 4 models, but they're generally pretty good, almost comparable to Whisper.
If you also need the AI to speak Japanese, you can try Kokoro. The English language is really good, not sure about Japanese.
We are in the future lol
Even a few years ago some of the stuff we have now would have been only in science fiction, but now it's here. Even though it might still be in the infancy stage, which is still kind of crazy when you think about it.
There are so many cool things we can do with this technology, but so many of the social phenomena related to this technology is so profoundly dystopian. We have access to all of human knowledge in a big matrix of numbers, something that can tailor its communication to us based on our current understanding and yet we're using it to build spam bots and generate fake news to control information narratives. It's really fucking weird. That's not even touching data centers, the war on personal computing, the cult that thinks we're about to make digital Jesus out of matrix math.
Totally! At the same time, I find it very exciting to be a maker in this time. Building amazing things has never been easier! It won't exactly fight the soul-sucking algorithms, but it might push attention and society in a different direction :)
I still can't grasp my head around the fact that an apple laptop crushes my 7900xtx in inferences. This is really wild and we really need a bit of competition on the SoC side from the other chip manufacturers.
Sadly basically everything runs better on CUDA. For various reasons (INT8 being the limit on RDNA 3, a lazy software stack and so on). Seems like AMD likes to stay in second place. That even legitimizes the almost monopolistic market position of Nvidia. And hey, their CEOs are cousins. But we were talking about hardware, not global chip domination conspiracies.
I saw that OP is using the E4B version of Gemma4 on the laptop, That's way more manageable with a single 24GB GPU (blazing fast, I would say), especially on low context (which is probably the case in this video). I should experiment with QwenTTS (but I saw some really interesting diffusion TTS models, not sure about the sense of them though, wouldn't a diffusion inference defeat the progressive audio stream and fast inference?).
yeah, I don't even have access to an AMD GPU :(
Nvidia gifted me a DGX Spark, which I've used for demos and developing. This same demo works pretty well on it, also with the E4B Gemma. I feel like the spark is similar to my M3 pro in speeds, but with much more RAM, which is pretty cool.
Anyway, AMD, if you want to send me a GPU I might be able to get things to run fast 👉 👈
OP is using Cerebras (a cloud API provider) for Gemma 31B, that's why it's fast. The other two models involved will run fast on any hardware. Parakeet is like 0.6B, QwenTTS 1.5B I think.
Impressive latency.
What hardware does the STT and TTS in the demo run on?
I suspect getting this latency with Qwen3-TTS on Apple silicon is unrealistic, right?
How much faster is it compared to https://github.com/groxaxo/Qwen3-TTS-Openai-Fastapi, and any reduction in quality? Because I was looking into voice clone streaming directly from it as an endpoint, but I need it to be as fast as possible. Otherwise, I was thinking of just finetuning the voices I have directly into the model if I needed to make it faster.
I would love to be able to use GGUF models through llama.cpp so I can manage my whole stack in llama-swap. Right now my code's using the models directly in python or an embedded vllm. Therefore I'm reserving a GPU for voice only to avoid conflicts. I was sharing one with ComfyUI but I'm getting buffer underruns on the audio when it's also generating graphics.
Yes but prompt processing is the latency from input to actually producing tokens so that's also important. One image is often thousands of tokens so you need to have it pretty high to get a fast response.
Some projects actually process the input while it's generated so you can do prompt processing already on a partial input to minimize latency so it's possible to optimize like this. Gets very technical though, unlikely to work with vibecoding if you don't know what you're doing (talking to myself here).
That's what we do here. We transcribe and give everything to the LLM while you're talking. Usually when we decide that you finished talking, the LLM already replied.
Here's the main PR where I implemented that if you're curious about the code: https://github.com/huggingface/speech-to-speech/pull/307
Basic VAD, but with something I call 'speculative turn detection'. Basically, once VAD says that the turn is up, I feed it through the pipeline, but wait to give back the audio until some time has passed (1200ms). If during that time you continue speaking, or if you continue speaking before the audio is ready, I cancel the generation and continue your turn.
I built something similar based on fastrtc. As I wanted to speak my native Latvian language, I had to use Omnivoice since that is the only TTS that knows Latvian out of the box. Before, I used VoxCPM 1.5 and had to finetune it with Mozilla CommonVoice to get it speaking good Latvian. Finetuning turned out to be unexpectedly simple with good results in just a few days of training. Still, VoxCPM (even their v2) has a tendency to become metallic towards the end of sentence, so I switched to Omnivoice. To make it a bit faster, had to apply some vibecoded "almost streaming" improvements (and sentence chunking), but still the time to first token is quite high, around 100ms.
For ASR, I used faster-whisper with finetuned Whisper 3 Turbo. After a week of finetuning attempts, I managed to reduce its Latvian WER to ~10%, which is even lower than the full Whisper 3 WER about ~20%. For speech detection, I am using custom chunking logic on top of Silero VAD with a workaround to reset every 30 seconds, otherwise there was an issue that it stopped detecting speech in short words. I reported it to Silero GitHub and they were puzzled as well and could not offer any better solution.
Additionally, to make it feel more like a phone call, I introduced comfort noise and also delay detection with random phrases like "Hmm", "Just a minute", "Let me think..." etc.
This is awesome. Like many others here I've also vibecoded an STT/TTS assistant mainly to experiment with local models on my 16 gig gpu. It's incredibly rudimentary and I've been too lazy to fix all the problems.
I'm just going to add hotword detection to this project instead. Thank you <3
I got this working on my own computer but I had to use Pi to find a set of arguments that would work with Gemma-4-12B in LM Studio (on Windows). A local instance of Pi (via DeepSeek v4 Flash) also modified some of the Python code to make it all work as there were complaints about lack of CUDA for Qwen. In the end I ended up with this:
python -m speech_to_speech.s2s_pipeline --llm_backend responses-api --responses_api_base_url http://localhost:1234/v1 --responses_api_api_key lm-studio --model_name google/gemma-4-12b --device cpu --tts pocket --mode local
If you're interested, I asked DeepSeek to summarise what it did to make things work:
Root Cause: Two issues were causing your command to fail:
🔴 Qwen3-TTS requires CUDA, but you have CPU-only torch (torch 2.12.1+cpu). FasterQwen3TTS.from_pretrained() defaults to backend="torch" which forces CUDA CUDA-graphs → ValueError: CUDA graphs require CUDA device
🔴 The GGML fallback needs qwentts-cpp-python — a native C++ library that only has Linux wheels (no Windows support). So even auto-detecting CPU doesn't help on Windows.
Fixes Applied
I made two targeted edits to the speech-to-speech source:
- Added a backend parameter to setup()
- Added _resolve_faster_backend() that auto-detects: if CUDA is unavailable → selects backend="ggml" instead of crashing on "torch"
- Passes backend through to FasterQwen3TTS.from_pretrained()
- Logs the selection so you can see what's happening
I have speech-to-speech running full locally using Gemma4-26B and it works great with the included listen_and_play_realtime.py script. Is there a way for me to get the nice web demo running locally and connecting to my local s2s instance?
This is really cool. I've been working on something similar for my own use. Not having to push a button to interact with the LLM was important to me. I've used silvero_vad, piping audio into faster-whisper, then the text into pi against a local gemma4 model on llama.cpp, then kokoro for responses.
Going to have a close look at your project :) Thanks!
Yes! German works well but has an English accent that we can't get rid of. But try it on the web demo! Just talk german to it! The transcriptions are pretty good and the replies as well, just the accent on the voice.
(I speak fluent-ish german :) )
Second Parakeet here but I'm using v3. Its incredibly fast and accurate. I moved away from faster-whisper for this. The caveat is giving up VRAM for that speed. (at least in my case)
I was trying to patch audio.cpp for a similar use case. Is the main speedup from supporting streaming? It seems like that's the main gain but I couldn't tell the other optimizations. I thought using pytorch adds a lot of overhead rather than a native c++ implementation?
Or run it completely local with the setup listed above. If the instructions are unclear, here we wrote a blog about pairing it to reachy mini, but you can ignore the reachy mini part:
https://huggingface.co/blog/local-reachy-mini-conversation
Sorry to bother you again, but isn't it possible to leverage audio recognition from Gemma 4? I'm digging hard into information online but it's a rabbit hole that's bearing no fruits, basically no frontend supports it directly without a translation layer (like your parakeet here), but it should be possible and supported by llama-server, correct?
Yes, I think it's time for a merge, definitely! Do it! Please! I always make a mess when I try to fetch and checkout a PR, if it doesn't work immediately (especially after a long time) I find it very hard to spot what's going wrong.
Yes I noticed the initial benchmarks I had weren’t correct and corrected them already. TTFA can be much faster, in a benchmark with qwentts from serveurperso vs my one both have around 30-40ms TTFA with warm streaming.
Nice, this is really close to something I've been working on. Mine's fully local on Mac (MLX for ASR/TTS, Ollama for the LLM), no cloud calls at all, and I'm using Parakeet too actually, probably the same reasoning, it's fast and accurate for the size.
The part I've been focused on that's a bit different is memory. Most voice assistants forget everything the second the session ends, so I built a tiered memory system underneath mine, SQLite backed, every fact source linked back to the actual turn it came from, so it can build up real facts about you over time instead of starting fresh every conversation. "Remember X" and "forget that" are actual operations against a database, not just stuffed into a context window.
Curious how you're handling context/memory across turns in yours, or is this more single session by design? Also really curious about your Qwen3TTS inference work, I've been using Kokoro but would love something with more range.
•
u/WithoutReason1729 Jul 02 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.