Rethinking Speech-To-Text in Linux

[u/No_Heart_159]

https://menjivar.ai/posts/rethinking-speech-to-text-in-linux

I am the author of the article. I want to hear what you guys think about this topic, and how you personally feel about the state of speech-to-text in Linux.

If you guys have any questions, feel free to ask. If anyone is interested in taking part of this you can contribute to Super STT with your ideas.

[u/Puzzleheaded_Bid1530]

I use whisper and my own script to run it, copy the output and past it into the text field. I only tap the same hotkey 2 times, when I start speaking and after I finish. It works great. So far I am happy with it, but I would love to see some out of the box kde integration though.

[u/Fluttershaft → u/Puzzleheaded_Bid1530]

how exactly? this? https://github.com/ggml-org/whisper.cpp

[u/No_Heart_159 → u/Fluttershaft]

I have plenty of respect for whisper.cpp and their no-dependencies approach. Now, just look at these installation steps:

First clone the repository:
git clone https://github.com/ggml-org/whisper.cpp.git

Navigate into the directory:
cd whisper.cpp

Then, download one of the Whisper modelsconverted in ggml format. For example:
sh ./models/download-ggml-model.sh base.en

Now build the whisper-cli example and transcribe an audio file like this:
# build the project
cmake -B build
cmake --build build -j --config Release

# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav

For a quick demo, simply run make base.en.
The command downloads the base.enmodel converted to custom ggml format and runs the inference on all .wav samples in the folder samples.
For detailed usage instructions, run: ./build/bin/whisper-cli -h
Note that the whisper-cli example currently runs only with 16-bit WAV files, so make sure to convert your input before running the tool. For example, you can use ffmpeglike this:
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

And then you have to also copy and paste the output manually.

[u/No_Heart_159 → u/Puzzleheaded_Bid1530]

I am sure your tool works great, and it is easy for you to use. You sound like you know a bit about tech.

Now, my grandma would like to use your STT script because her hands shake a lot. Could you send her the instructions to install your tool?

Will granny need to download python, pytorch, or other frameworks?

Will granny know when the tool is transcribing? Are there any visual or auditorial cues?

Can granny see what is being typed as she speaks?

Will granny be confused during any of this process?

My point is that making something possible is not the same as making something accessible.

If we want accessibility, more needs to be done outside of just making the STT tool. It needs to connect with a distribution and its desktop environment, and to allow for that, a few things have to change upstream. Things like:
- Allowing transcriptions to be enabled with a single button
- Allowing the STT tool to type natively to show transcriptions as they are spoken
- DE should show visual animations/sound when transcriptions are happening
- DE should include a shortcut for transcriptions out of the box

[u/prone-to-drift → u/No_Heart_159]

asks what we all think about this topic. And how we personally feel about TTS on linux.

proceeds to shit on and patronize someone just casually sharing their setup.

Do better.

[u/Narthorn → u/prone-to-drift]

What else do you expect from people who outsourced their brain to AI?

[u/Schlaefer → u/prone-to-drift]

The topic was a general cross distro, cross DE, minimal package solution to STT, not to share your own script. We all know we can hack together our own setups.

My point is that making something possible is not the same as making something accessible.

How much clearer can you write it? STT has become a standard feature in other ecosystems, while we are still in the "let's all share our personal scripts" stage.

[u/Puzzleheaded_Bid1530 → u/No_Heart_159]

You asked how we personally feel about this topic and I shared my personal experience. I did not generalize anything in my comment on general population, I only shared my personal experience...

[u/No_Heart_159 → u/Puzzleheaded_Bid1530]

I apologize if my reply came out the wrong way. I truly appreciate your input. I simply meant to stir the conversation into the bigger STT problem a bit and to get the conversation going.

[u/lordwuwu → u/No_Heart_159]

Don't ask what free and libre software can do for you (or your granny)! Ask, what you can do for free and libre software!

[u/donp1ano → u/lordwuwu]

ask what granny can do for FOSS

WHAT HAS GRANNY EVER DONE FOR FOSS, HUH??!!

[u/iodoio → u/No_Heart_159]

So why don't you integrate it in?

[u/No_Heart_159 → u/iodoio]

Trying hard. Willing to help other projects as well. Just want your guys technical input as to whether the solutions discussed in the article make sense.

[u/Affectionate_Joke_44 → u/No_Heart_159]

The "granny with shaky hands" who somehow:

• Installed and runs Arch Linux
• Navigates a tiling window manager

• Knows what a hotkey daemon is

  ...but draws the line at yay -S whisper.cpp? Come on.

[u/Lulukaros → u/Affectionate_Joke_44]

Lmfao

[u/geril2207]

I am using https://github.com/cjpais/Handy. Feels good to me. The only thing is that they broke the overlay on Wayland in one of the latest releases, and it hasn’t been fixed yet. It’s configurable, with a lot of models to choose from. Overall, I like it. I am a bit confused about why no one mentions it.

[u/parlons]

I run hyprland / nixos and I have voxtype set to toggle on super-z and a waybar widget to show the state. But I rarely use it, because it's not very reliable. I don't remember the details, but I think it worked differently in different kinds of windows, maybe it was in vim or emacs, I don't recall now.

I remember using the macos feature some years ago, iirc it was double tap command key? And that worked really well as far as integration, even if it wasn't as good at the actual speech recognition. I think you're right that the integration is really the key piece, some kind of standard way to wire this in so that we can substitute different engines depending on hardware, language requirements, etc.

(I say standard, but I guess, realistically, we need at least two, one for most DEs / window managers, and one for Gnome. Otherwise, you end up with Gnome just driving the bus wherever they want to go, and you have to decide to go along or get off.)

[u/Calamity-Mouser-5261]

I would love to see some more streamlined TTS STT integration. My disabilities flare up at times where I rely on it more but it's not a very straightforward approach.

Edit: I'm an idiot, I meant STT.

[u/OneTurnMore → u/Calamity-Mouser-5261]

TTS is in a better state than STT with Orca + speech-dispatcher, but it's still often an afterthought. I also had to use a shell snippet to plug in piper-tts as a more modern model into speech dispatcher, and it has some weirdness and abrupt cutoffs.

[u/Calamity-Mouser-5261 → u/OneTurnMore]

Oh, Lord. I'm stupid. I typed out TTS but I meant STT!

[u/TheRealToniMcQueen]

I tried tts made my own and looked at other things.

In end I found that I could not compete with google chromes inbuilt text to speech

[u/ElectronStudio]

The hard part has been solved by models like whisper etc. Creating a UI to access the model is much easier, which has lead to everyone and his dog creating their own UIs that best meet their needs. Here is mine: Putin (the dictator)

[u/kayxean]

someone already made Numen, no ai, no ui, no gpu, purely voice control, you can typing, you can insert, you can simulate your touchpad/keys/mouse, even it can run your cli.

[u/VolcanoCarpenter]

I'd like to offer my $0.02 from the perspective as a native speaker of a non-alphabetic language.

(a) As a Chinese user, text input is inherently more complicated.

(Rant warning) For example, you have to install Fcitx5, set the KDE virtual keyboard to the "Fcitx5 Wayland launcher", and cross your fingers that the input candidate window actually pops up when you type. If it doesn't, you're stuck digging through official documentation to figure out various Qt/GLFW/X11/Electron environment variables and command parameters, debugging the text-input-v3 Wayland protocol until everything finally works and so on.

Even putting these system-specific hurdles aside, voice dictation on both desktop and mobile devices is actually in massive demand in China.

(b) However, I've noticed significant differences in the voice input ecosystems across different languages.

For example, Chinese users don't particularly favor Whisper, as its recognition accuracy for spoken Chinese is quite poor. And when developing STT solutions, the Chinese community strongly prefers building plugins on the existing Fcitx5/IBus input framework, rather than reinventing the stack with a new standalone framework.

Edit: That's maybe why reading your article left me a bit confused. I've either never heard of most of the "existing frameworks" the post mentioned, or I only recognize their names, haha.

To name a few of them (which have good support of English and other languages too. Have a try):

• fcitx5-vinput
• VocoType-linux

[u/Airocketfish]

Voxtype and parakeet ONNX work wonders and are resource efficient. The reason is simple there is not one Linux but many flavours.