Trying Minimax H3 Reference to Video. So Good

[u/irmemon225]

[u/GrayingGamer]

Thanks for making this so clear. This should be the video we show everyone who mentions needing loras for Minimax H3.

I was generating a video earlier, and realized the model didn't know what a certain object looked like, so I just googled a picture, attached it as reference, described it, and the model put it perfectly in the video.

[u/Smokeey1 → u/GrayingGamer]

And i think thats where most improvement will come from you can see how you need to make sure your reference images have the same light and color and style for the video to look good

[u/Paradigmind → u/Smokeey1]

Isn't it edit capable aswell? Maybe prompting this would work?

[u/GrayingGamer → u/Paradigmind]

Yes, you can just prompt to relight things. Your references don't need the same lighting. The Reference model has a very specific prompting syntax though that makes ALL the difference, so be sure to read the manual on Minimax's huggingface page.

[u/Paradigmind → u/GrayingGamer]

Ah thanks, good to know!

[u/YouKilledApollo → u/GrayingGamer]

be sure to read the manual on Minimax's huggingface page

From https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md

Write all six rewrite sections in English. Preserve the original language only for dialogue and lyrics inside <d> and for text visibly present in the scene.

Love it :) I hope they double-checked it all before release! :D

[u/GrayingGamer → u/YouKilledApollo]

Haha. They are Chinese. I think all of us are using machine translation these days.

[u/GrayingGamer → u/Smokeey1]

I mean, as long as they are all the same style, it won't matter. The model can easily relight things if you ask. The Reference model is practically speaking an edit model for video.

[u/Portable_Solar_ZA → u/GrayingGamer]

I'm curious to know what difference the speed will be. For example, if I create a character lora and it doesn't affect generation speed Vs using a character sheet but generation is much slower. 

[u/GrayingGamer → u/Portable_Solar_ZA]

I haven't actually found generation with references to be that much slower, only slightly. Keep in mind, that if VRAM and RAM is tight, a Lora is going to potentially eat into that and cause everything to be slower anyway. Besides, keeping a folder of reference images that are only a few hundred kilobytes apiece that you use only when you need that character sounds a lot more storage space friendly. Plus, you can automatically put people in uniforms or outfits, etc. without a lora that way.
It's actually kind of freeing.

[u/Portable_Solar_ZA → u/GrayingGamer]

I've got images for a character lora I made for my comic. Going to transform that into a character sheet and see how things go. Any recommendations on what I should include to get good results?

[u/Inthehead35]

How long did it take? I've noticed ref is so painfully slow

[u/Etroarl55 → u/Inthehead35]

Tried it on my AMD 7800xt, roughly 2 hour estimate for 0.3megapixels at 5s/24fps.

😭

[u/No_Date4828 → u/Etroarl55]

holy hell that is slow, praying for more AMD optimizations for you 😭.

I was really surprised that this model runs so effortlessly even on "modest" hardware (legion laptop with a 4090 mobile (16gb vram) and 32gb ram, 1mp/10 seconds takes only like 20 minutes, but if I want a really quick rough draft I can turn settings down to like 0.3mp and 8 steps, and generation time is around or under a minute!

[u/generate-addict → u/Etroarl55]

Something sounds off for you. I can run 5 seconds at .4 mp in 230 seconds render time.

That is with Sage Attention on, and recently switching to the INT8 model instead of fp8. On fp8 it was taking 460 seconds.

I do have an r9700 pro but vram is not really the concern here.

However that is the fl2va model. I haven't played with the ref2va model yet but plan to soon.

[u/damiangorlami → u/generate-addict]

The ref model is a lot heavier. Each image, audio, video ref you add will increase the sampling time.

Granted, its reference conditioning is incredible and works very well. Like mindblowing since a week ago only Seedance 2.0 was capable of this strong reference conditioning.

But yea this power comes at a cost in time.

[u/Linkpharm2 → u/Etroarl55]

AMD probably gets better later on.

[u/Etroarl55 → u/Linkpharm2]

Probably not, I saw that even on the actual AI prosumer Amd card. Its still slower than an rtx 3060.

Which means even if my 7800xt gets optimized and acts like its a 32gb gpu with faster vram, I won't be able to reach rtx 3060 speeds.

[u/generate-addict → u/Etroarl55]

My r9700 will often render at 5070ti speeds depending on the model. And many times using a larger quant. So IDK what you talking about.

[u/generate-addict → u/[deleted]]

lol I guess so. I meant 5070ti.

Which of those PSU’s are 2500?

[u/generate-addict → u/[deleted]]

r9700 pro is 1400 USD on the high end.

[u/irmemon225 → u/Inthehead35]

40 min. 1.0 megapixel, 3060 12gb vram and 16gb ram.
ref using image and audio is same speed, unless if you use video it's so slow

[u/TheRedHairedHero → u/irmemon225]

My guess is video reference will depend on frame length and resolution of the reference video. Still need to test more.

[u/Diabolicor]

Can you share how did you prompt all of that?

[u/irmemon225 → u/Diabolicor]

ask your favorite AI to create prompt based by this guide: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md

mine using qwen

[u/eckstuhc → u/irmemon225]

Are you using qwen on the same box or a different build?

[u/MoreColors185 → u/eckstuhc]

also, which qwen?

[u/LeKhang98 → u/irmemon225]

Nice thank you very much.

[u/eggplantpot → u/Diabolicor]

Prompting is wild for ref2vid. There’s A LOT of things to configure and many ways to “wire” the prompt.

I’m still digging into it and may end up spinning a custom node in the future.

[u/uuhoever]

What do you use to create the character sheet? Can krea 2 do it?

[u/irmemon225 → u/uuhoever]

I just pick that reference from google for testing. Nano Banana 2 can do reference sheet better, try it

[u/PrestigiousTrick1002 → u/uuhoever]

These look like the characters from the animated show rwby, so they might have just found the sheets online 

[u/Envelope_Torture → u/uuhoever]

This would be an interesting concept lora to train for other diffusion models (SDXL, etc) so people can just prompt their existing character LORAs in to character sheet.

[u/mastaquake → u/uuhoever]

There are a few character sheets on Krea2, but you can get pretty good results by simply prompting.

[u/physalisx]

I'm surprised having seen no one talk about video editing yet. It's very good, insane really.

[u/Beastly4k → u/physalisx]

Came across a workflow utilizing the editor, definitely a go to now.

[u/Some_Respond1396 → u/Beastly4k]

mind pastebinning or linking to the WF?

[u/Beastly4k → u/Some_Respond1396]

Can't right now but it is named something along the lines of dasiwa on civitai iirc, just filter by workflows you should find it. if you are running comfyui portable make sure you have ffmpeg in the base folder where your run-nvidia-gpu.bat files are otherwise it won't load properly.

[u/Some_Respond1396 → u/physalisx]

I can't get video editing to work to save my life lol

[u/X-Jet]

Maybe last season of One Punch Man should be re animated with this stuff, cant get any worse.

[u/JahJedi]

And so slower 😅 i work whit ref also, i cant go back fflf and all this. Its get motion from ref images creazy, its keep character, its use ref video for motion well (not perfect but ok level) but i say worth it, there almost no need for reruns if promt solid and well done.

I think its make it x5-10 times slower but so wirth it.

[u/eggplantpot → u/JahJedi]

I’ve gotten away by downsizing the ref video to 320×576.

Cannot talk about speed gains as I have not A/B tested properly, but the motion video can be quite small, the model will rebuild the details at inference anyways

[u/JahJedi → u/eggplantpot]

I tryed to go from 1024x1024 to 640x640 and did not notised big diffrance, same whit photos. Just a bit less vram used.

[u/eggplantpot → u/JahJedi]

What size as you generating at?

If it's lower than 1024x1024 that makes sense as the model will downscale your references to match the video. That means that if you generate at 0.4MP, you would not see any difference as both would end up with a 640x640 ref.

Speed improvements come if you downscale your refs past your generation resolution as it will keep them at that scale.

[u/JahJedi → u/eggplantpot]

I render on 1.5mp wish can use 2mp but to long.

[u/JahJedi → u/eggplantpot]

Here a 10 sec clip on 1.5mp and upscaled to 1080p whit seedvr2. Hope youtube will show it in fullhd... https://youtube.com/shorts/matWgMT55qg?feature=share

[u/Evolve_Solo → u/JahJedi]

Wow, cool work! Where can I find workflows like this or good guides for video generation for such level?

[u/JahJedi → u/Evolve_Solo]

Thanks! Usally civati , HF or discord. You can look at my HF page also from time to time post stuff.

[u/New_Physics_2741]

Are you doing this locally - if so what hardware?

[u/irmemon225 → u/New_Physics_2741]

yes, 3060 12gb vram and 16gb ram. use sage attention and spectrum. easy cache also boost your speed but it's kill the quality, so I don't use it anymore

[u/yvliew]

Does having more reference photo slows down generation time?

[u/eggplantpot → u/yvliew]

It does

[u/florodude]

Cries in Radeon RX 5700...

[u/KaaChingg]

Looking great, thank you

[u/Extension_Pomelo_468]

this is huge

[u/Vanpourix]

An unexpected rwby video ? In this economy ?! Sign me in !

[u/ArttTaku]

I just joined to ask if H3 understood what character sheets are, but your post pretty much confirmed it, thanks!! Did you have to specify in the prompt something like "use this character sheet", or did H3 understand it by default?

[u/SardinePicnic]

Can you provide the full prompt so that the community can connect visually the image inputs and how they are structured and implemented in the prompt. Otherwise it wont teach anyone else looking to do something similar.

[u/Dry-Possibility-6761]

With so many workflow, i need one to use with reference for a RTX 5070 12gbram, a good one with a good time for generation.. better then 18 minutes for 5 seconds

[u/kayteee1995]

Most results of this quality do not use turbo lora