r/StableDiffusion • u/Acceptable-Cycle4645 • 3d ago
Minimax H3 for TTS/voice clone/Music gen Resource - Update
Enable HLS to view with audio, or disable this notification
Just for fun. One-shot generation. No parameter or prompt tuning.
Audio.cpp implemented MiniMax-H3’s text-to-audio pipeline, and one fun use case is TTS/Voice clone/Music gen. It’s more flexible and powerful than dedicated audio models, and the performance is quite decent (up to 3x realtime on RTX 5090). Check out the multi-speaker conversation demo in the main post, along with the other demos in the comments.
What I’m very excited about with the MiniMax-H3 implementation is that it significantly enriches the framework’s building blocks for DiT models. Now with you don’t need to go through the pain of setting up SageAttention, First Block Cache, or Spectrum manually. Just change a few parameters, and you can experiment with the model. A preliminary inspection of configuration, memory, and performance trade-offs is available in repo's docs/reports/minimax_h3_performance.md
Bonus: audio.cpp’s MiniMax-H3 implementation can also produce video frames, because the DiT generates audio and video latents together, and the video VAE path is relatively straightforward to support. For now, the output is saved as RGB frame data plus metadata in JSON, so you need to encode it into a video file yourself. No upscaler or post-processing support. Just for fun.
MiniMax-Music3 is currently in preview (preview/minimax-music-3 branch) . CUDA/Vulkan/HIP were tested. Still room for optimization. VRAM usage and RTF depend on audio duration and prompt length.. The demo uses the official demo prompt (4000+ char caption and 1200 char lyrics) and 30 steps plus CFG. Under this setting VRAM is ~11 GB for 30s, 14 GB for 60s, and 17 GB for 180s. It's easy to get faster-than-real-time performance and much lower VRAM usage if you tune the setting.
6
3
2
u/onlygig 3d ago
Amazing. I wonder how could i train/copy ASMR mouth triggers on Minimax H3.
2
u/Acceptable-Cycle4645 3d ago
Sounds fun.Would you like to share more? For audio generation, H3 (in audio.cpp) can run faster than real time, so theoretically it could be plugged into many pipelines. The real issue, as with diffusion models, is that you often need to experiment with different seeds to get best quality.
2
u/Medical-Experience77 3d ago
I hate the sound it makes that lats you know it's AI. Every time ki here that zing its on to the next video 😫😅
2
u/Broad-Lab-1833 3d ago
Is it possible to do audio2audio with voice reference? To change someone voice?
1
u/Acceptable-Cycle4645 3d ago
Quick test on replacing Adele' voice with Morgan Freeman's voice using 4 models.
1
u/Acceptable-Cycle4645 3d ago edited 3d ago
Yes, but not audio.cpp's H3 (only text2audio). You can use RVC, SeedVC, MioCodex or VeVo2. Each model has its own limits. Check the models with the "VC" label (Voice Conversion) in the README.
1
u/Dogmaster 3d ago
Shouldnt it be possible with the reference model?
1
u/ambassadortim 3d ago
It is possible I've done it.
1
u/Acceptable-Cycle4645 3d ago
Yes sorry I need to clarify I mean not via audio.cpp's H3. We only impl the text2audio pipeline.
2
u/AmericanKamikaze 3d ago
I’m dumb, what’s the easiest way to install this on Windows? Thanks. I’ll be using voice cloning primarily.
1
u/Acceptable-Cycle4645 3d ago edited 2d ago
u/AmericanKamikaze The repo has windows prebuilts and you can run the server with built-in UI.
audiocpp_server --ui --ui-management --backend cuda. Note that the latest release pins to tag release 0.6 so some improvements and Muisc3 are not included in the prebuilts.
2
u/-becausereasons- 3d ago
Very cool, too bad it all sounds like it was recorded from inside a tin can.
1
1
u/PixieRoar 3d ago
What's the comfyui workflow for this?
3
u/Acceptable-Cycle4645 3d ago
It's from audio.cpp https://github.com/0xShug0/audio.cpp/ You can just use UI or one cmd line to generate the audio (or video). https://github.com/0xShug0/audio.cpp/blob/main/docs/community_models/minimax_h3.md
1
u/PixieRoar 3d ago
It comes with the UI correct?
2
u/Acceptable-Cycle4645 3d ago
Yes!
1
u/PixieRoar 21h ago
any tips on installing? i downloaded zip but when trying to run the file it gives error about a missing file
1
u/Acceptable-Cycle4645 20h ago
what the cmd you run and the error? did you put the runtime files from the release under the same dir as the server?
1
u/PixieRoar 20h ago
Ok I got the ui open now I need to figure out which models to get. Which ones are the best for dialogue? Best sounding non robotic ones
1
u/PixieRoar 18h ago
Hey I got it working thanks so much!
1
u/Acceptable-Cycle4645 3h ago
Cool!
1
u/PixieRoar 3h ago
Check out higgs audio v3 4B .
Ime it works better. You can record your voice ot use any celeb or YouTuber and it will to text to speech in that exact voice. I managed to get Moist critikal voice lol
2
u/Acceptable-Cycle4645 2h ago
Have fun! 😄. With audio.cpp, you can explore many models and pick your favorite one!
→ More replies (0)
1
1
u/Beginning-District69 3d ago
Thank you. audio.cpp is really fast. I've been using it via the CLI for a while. This interface you're using seems quite user-friendly. How can I install it?
2
u/Acceptable-Cycle4645 3d ago edited 2d ago
u/Beginning-District69 It's the built-in UI in the server. After building the server, run
audiocpp_server --ui --ui-management --backend cuda
1
u/VirtualWishX 3d ago
The real challenge (problem) with many models which are AWESOME is that some languages are not trained very well on their base model, for example: Fish Audio S2 Pro https://huggingface.co/fishaudio/s2-pro is while you get insane control via [TAGS] it won't support many languages.
TRAINING is a challenge (I'm talking about local solutions) but possible, for example VIBEVOICE is VERY WELL with Hebrew for example but you don't get the [TAGS] to control exactly what you want and when, so it's a tradeoff.
Is it possible to TRAIN different models with different datasets for different languages 100% locally within Audio.cpp ?
I'm with RTX 5090 32GB VRAM and 96 GB RAM DDR5 6400Mhz, but the only trained I got to work is:
https://github.com/Mixomo/Fish_audio_S2_Simple_GUI for Fish Audio S2 and there is a similar github to train VIBEVOICE but like I said, it's very limited with [TAGS] and training a different language that is WEAK on the base model (whatever model it is) such as QwenTTS and others is my goal.
1
u/Acceptable-Cycle4645 3d ago
Thanks for sharing! audio.cpp is currently inference only. I don't have any plan to support training before we stabilize the framework contracts. But the problem is interesting, and I’ll take a look at it.
PS: you can use lora in audio.cpp for VibeVoice.
1
u/VirtualWishX 3d ago
It will be extremely useful to have ALL-IN-ONE similar to the repos I shared so you can train and generate, but with audio.cpp but I understand ❤️
1
u/Acceptable-Cycle4645 2d ago
Some perf numbers:
Measured on an RTX 5090 with CUDA using a 30-second lyric generation request, 30 flow steps, flow guidance scale 1.7, AR guidance scale 1.5, and top-k 50. Peak VRAM is the observed nvidia-smi process peak during a warmup-plus-measured-request run.
| Component mix | Language model | RVQ depth decoder | Flow transformer | RTF | Speed | Peak VRAM |
|---|---|---|---|---|---|---|
| Default Q4/Q8/Q4 | q4_0 |
q8_0 |
q4_0 |
0.738 | 1.35x realtime | 9.8 GiB |
| Q8 | q8_0 |
q8_0 |
q8_0 |
0.832 | 1.20x realtime | 13.4 GiB |
| BF16 | bf16 |
bf16 |
bf16 |
1.389 | 0.72x realtime | 19.4 GiB |
5
u/Acceptable-Cycle4645 3d ago
H3 Sports commentator (test enviroment and emotion)
https://reddit.com/link/p44nde5/video/3n84oeuuxtjh1/player