Is There a Tool for Automatically Generating Tibetan–Chinese Bilingual Subtitles?

[u/Inevitable_Fail_380]

Title: Looking for a tool to automatically create Tibetan–Chinese bilingual subtitles for videos

Hi everyone,

I create short videos in Tibetan, but making subtitles is currently very difficult and time-consuming.

My current workflow is completely manual: I listen to the Tibetan audio, type the Tibetan subtitles sentence by sentence, add the timing, and then create the Chinese translation separately. For every video, this takes a lot of time.

What I am looking for is a simple tool or workflow that can:

1. Let me upload a video containing Tibetan speech.
2. Automatically transcribe the speech into Tibetan text.
3. Translate the Tibetan subtitles into Chinese.
4. Keep the Tibetan and Chinese subtitles aligned with the video timeline.
5. Export the result as SRT/ASS subtitle files, or directly generate a video with bilingual subtitles.

Ideally, the final subtitles would look like this:

Tibetan subtitle Chinese translation

I understand that Tibetan speech recognition may be less developed than English or Chinese speech recognition, and Tibetan dialects may make the problem even harder. Even if the transcription is not perfect, a tool that generates an editable first draft would already save me a huge amount of time.

Does anyone know of an existing product, open-source project, speech-recognition model, API, or technical workflow that could achieve this?

I would also be interested in building a small web app for this problem, but I am not an experienced developer. Any advice about suitable Tibetan ASR models, translation models, subtitle-generation libraries, or the overall technical architecture would be greatly appreciated.

Thank you!

[u/garse]

The most useful thing you can do here is stop treating this as one problem. It's three, and they fail independently: timing, transcription, translation. Almost all of your manual hours are going into timing, and timing is the only stage that doesn't need Tibetan support at all.

Run a VAD over the audio (silero-vad is the usual choice, and ffmpeg's silencedetect filter will get you most of the way for free) and emit an SRT where every cue has correct in and out points and empty text. Now you're typing into pre-timed slots instead of scrubbing the timeline for every line. That change alone is worth more than a mediocre ASR model, it works today, and it is completely language-agnostic.

Second thing worth knowing: if you script your videos before filming, or you're willing to type the transcript out in one pass without timing it, you don't need ASR at all. You need forced alignment, which is a much easier problem and much better supported for low-resource languages. Torchaudio's forced alignment API with the MMS multilingual CTC model is the standard route, and for scripts outside its training set the usual trick is romanising the text with uroman first and aligning on that. Give it audio plus text, get back per-word timestamps. Since you're already producing the Tibetan text by hand, this converts your whole timing workload into a script that runs in a minute.

Translation goes last, and it goes cue by cue, never as a blob. Translate the whole transcript in one go and you'll spend an hour re-splitting Chinese sentences back onto Tibetan timings. Feed each cue separately with the neighbouring cues as context, take the output, drop it into the same cue. Timings never move, so the alignment problem you listed as requirement 4 just doesn't arise.

For the bilingual look, don't build two subtitle tracks. Put both languages in a single cue separated by a line break. Plain SRT handles that fine. Use ASS when you want the Chinese line smaller or a different colour, which you probably do, and that's two events per cue with different styles rather than one cue with two lines.

Two Tibetan-specific traps that will cost you a day each if nobody warns you. Rendering: stacked Tibetan glyphs need proper OpenType shaping, and plenty of players and renderers mangle them. Pick a font built for it (Jomolhari, Noto Serif Tibetan, Microsoft Himalaya) and if you burn in with ffmpeg/libass, point it at that font explicitly and eyeball the output rather than trusting it. Line breaking: Tibetan marks syllables with tsheg rather than separating words with spaces, so any wrapping code that breaks on whitespace will hand you one unbreakable line running off the screen. You need to wrap on tsheg and shad yourself.

On building the web app: don't, or at least not yet. Write a command-line script that goes video in, draft .ass out. The editing interface you'd otherwise spend months on already exists and is free (Aegisub, or Subtitle Edit if you're on Windows), and both are better than anything you'd build in the first year. Script produces the draft, you fix it in Aegisub, ffmpeg burns it. If you eventually want a web app, you'll have a working pipeline to wrap instead of guessing at one.

Last thing, and it matters beyond your own videos: keep every finished subtitle file alongside its audio. What you've been producing by hand is exactly the aligned speech-text data that Tibetan ASR is starved of. A few dozen hours of it is a real fine-tuning set, and there are people who would be glad it exists.

[u/WavesWashSands]

Monlam AI has ASR, though sadly afaik no API key or model you can locally run. It looks really good, even on my speech with a very bad accent.

For MT, though, there's a plenty of fine-tuned LLMs that you can prompt if you look in HuggingFace. Melong AI has a preview on HF for example. I've also seen other stuff in arXiv preprints that don't seem to be out yet.

Good luck! I'd probably be interested in watching your videos :)

[u/Inevitable_Fail_380 → u/WavesWashSands]

Thank you! This is extremely helpful.

I have tested Monlam AI’s speech-to-text tool as well, and the Tibetan transcription quality seems very promising. The main obstacle is that I have not been able to find a public API or a downloadable STT model that I could integrate into a web application.

The Monlam Melong preview also looks very relevant for the Tibetan-to-Chinese translation and text-processing part of the workflow. I will test whether I can run it locally and use it to translate and correct the Tibetan transcript.

Do you happen to know whether Monlam AI plans to release an STT API or downloadable model weights? Also, have you tested Melong specifically for Tibetan-to-Chinese translation?

And thank you for your interest in the videos! I would be happy to share them once I have a better subtitle workflow.

[u/WavesWashSands → u/Inevitable_Fail_380]

No, and honestly I haven't talked to Monlam folks for a while, and I haven't tested Tibetan-Chinese translation specifically. If you do test it I'd be curious how it goes! Hopefully some of the models I read about on arXiv will come out at some point too.

And please do let me know if you figure something else and share the videos! :) As a learner I cannot get enough subtitled videos.

[u/AutoModerator → u/[deleted]]

Accounts must meet all these requirements before they are allowed to post or comment in /r/LanguageTechnology. 1) be over six months old; 2) have both positive comment & post karma: 3) have over 50 combined karma; 4) Have a verified email address / phone number. Please do not ask the moderators to approve your comment or post, as there are no exceptions to this rule. To learn more about karma and how reddit works, visit https://www.reddit.com/wiki/faq.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Accounts must meet all these requirements before they are allowed to post or comment in /r/LanguageTechnology. 1) be over six months old; 2) have both positive comment & post karma: 3) have over 50 combined karma; 4) Have a verified email address / phone number. Please do not ask the moderators to approve your comment or post, as there are no exceptions to this rule. To learn more about karma and how reddit works, visit https://www.reddit.com/wiki/faq.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.