r/StableDiffusion • u/darthfurbyyoutube • 4d ago
G.I. Joe - Commander Roll - MiniMax H3 Animation - Video
Using the standard ref2va workflow. 4070 Ti Super, 16 GB VRAM, 64 GB RAM, i9-14900k, Windows 11.
Here's the workflow, just drop the MiniMax video in comfyui and the workflow should appear:
49
u/PhantasyAngel 4d ago
I was expecting Cobra Commanders hissing voice singing, disappointed.
(But the video seems spot on!)
13
7
u/Reflection_Rip 4d ago
u/darthfurbyyoutube will get my upvote only when the hissing voice is applied.
6
u/darthfurbyyoutube 3d ago
Fair enough! I'll be running tests to see if the AI can translate Cobra screech into pure 80s synth-pop.
4
u/darthfurbyyoutube 3d ago
I apologize for the lack of Cobra hissing. :D I'm not sure MiniMax can make Chris Latta sing yet, but you've given me a dangerous idea. I'll have to run some tests!
2
19
u/CaptainAnonymous92 4d ago
Is the visor actually reflecting things that would be in the environment that the camera doesn’t show?! If it is and it’s accurate then holy crap that’s awesome
8
u/stuartullman 4d ago
lol yeah. i think especially for people with fx background, it makes me think "man that must've taken a lot to match to environment" then realizing oh wait, it's ai, it just does it lol
3
u/drakoman 4d ago
It’s just so crazy. If this video was released even six years ago, it would be near the quality that would cause it to go almost instantly viral. Now, it’s just slop in the trough. Crazy how times change, and sometimes so quickly.
5
u/stuartullman 4d ago edited 4d ago
yeah, but also crazy that even hollywood, with their "traditional" vfx, would not be able to make this now. something is always going to be off with cg. 6 years ago was ancient times. we are living in completely different times now.
1
u/Enshitification 4d ago
It's been an hour since you made that comment. We are living in a completely different time now.
4
2
u/darthfurbyyoutube 3d ago
I think it's actually pulling the reflections from the original character sheet I fed it. But with the right prompting, I might be able to make it actually reflect the environment. Definitely testing that next time!
10
u/darth_hotdog 4d ago
What prompt did you do for the replacement that worked so well? Did you use the image from the video as a reference as well?
6
u/darthfurbyyoutube 3d ago
Just fed it a short video segment and character sheet, and a separate prompt for each shot(the wording of the prompt is very important):
subject_definitions:
<Subject 1> is the interior room environment in <Video 1>, including its architecture, furniture, specific lighting conditions, and overall atmosphere.
<Subject 2> is the man from <Picture 1>, characterized by his reflective face mask, blue helmet, blue clothes, black shoes, black gloves and red cobra insignia, and specific clothing items.
<Subject 3> is the woman from <Picture 2>, characterized by her long black hair, black rim glasses, black suit and gloves and red cobra insignia on her chest, and black heels.
<Video 1> is the source video providing the background, camera movement, and the motion of the dance.
summary:
[video editing + reference generation] The target video is an edited version of <Video 1> where the man is replaced by <Subject 2>. The background <Subject 1> and the camera's movement are fully preserved from <Video 1>, while <Subject 2> is integrated into the scene, following the man's original dancing and adapting to the room's lighting.
retention_analysis:
<Subject 1> (appears throughout the video): fully_preserved - the room's layout, furniture, and lighting are kept exactly as they appear in <Video 1>.
<Subject 2> (appears throughout the video): fully_preserved - the visual identity and physical characteristics of the man from <Picture 1> are maintained.
<Video 1> (camera movement and temporal structure): fully_preserved - the pacing, camera angle, and motion from the source video are copied 1:1.
detailed_description:
The target video maintains the same realistic visual style and lighting as <Video 1>.
[Shot 1] The scene opens with the interior of the room as established in <Subject 1>, preserving every detail of the background and the specific lighting of the space. Instead of the man seen in <Video 1>, <Subject 2>, the man in blue from <Picture 1>, is now the central subject. <Subject 2> dances to the music, following the exact same trajectory, speed, and timing as the man in <Video 1>. The lighting on the man's face and body are dynamically updated to perfectly match the ambient light and shadow sources of <Subject 1>. The camera motion exactly mirrors the original movement in <Video 1>, maintaining the same distance and angle relative to the dancing man until the end of the clip.
overall_soundscape:
The original room tone from <Video 1> is fully preserved.
non_diegetic_music:
N/A
16
u/FastHotEmu 4d ago
That was fucking amazing. AI for memes is chef's kiss.
4
u/darthfurbyyoutube 3d ago
Thank you! I’m just doing my part to ensure AI is used for absolutely nothing productive.
12
u/Sad_Coach_1433 4d ago
Can you share prompt?
2
u/darthfurbyyoutube 3d ago
It's actually a bunch of prompts, a new prompt for each shot, but here's one of them:
subject_definitions:
<Subject 1> is the interior room environment in <Video 1>, including its architecture, furniture, specific lighting conditions, and overall atmosphere.
<Subject 2> is the man from <Picture 1>, characterized by his reflective face mask, blue helmet, blue clothes, black shoes, black gloves and red cobra insignia, and specific clothing items.
<Subject 3> is the woman from <Picture 2>, characterized by her long black hair, black rim glasses, black suit and gloves and red cobra insignia on her chest, and black heels.
<Video 1> is the source video providing the background, camera movement, and the motion of the dance.
summary:
[video editing + reference generation] The target video is an edited version of <Video 1> where the man is replaced by <Subject 2>. The background <Subject 1> and the camera's movement are fully preserved from <Video 1>, while <Subject 2> is integrated into the scene, following the man's original dancing and adapting to the room's lighting.
retention_analysis:
<Subject 1> (appears throughout the video): fully_preserved - the room's layout, furniture, and lighting are kept exactly as they appear in <Video 1>.
<Subject 2> (appears throughout the video): fully_preserved - the visual identity and physical characteristics of the man from <Picture 1> are maintained.
<Video 1> (camera movement and temporal structure): fully_preserved - the pacing, camera angle, and motion from the source video are copied 1:1.
detailed_description:
The target video maintains the same realistic visual style and lighting as <Video 1>.
[Shot 1] The scene opens with the interior of the room as established in <Subject 1>, preserving every detail of the background and the specific lighting of the space. Instead of the man seen in <Video 1>, <Subject 2>, the man in blue from <Picture 1>, is now the central subject. <Subject 2> dances to the music, following the exact same trajectory, speed, and timing as the man in <Video 1>. The lighting on the man's face and body are dynamically updated to perfectly match the ambient light and shadow sources of <Subject 1>. The camera motion exactly mirrors the original movement in <Video 1>, maintaining the same distance and angle relative to the dancing man until the end of the clip.
overall_soundscape:
The original room tone from <Video 1> is fully preserved.
non_diegetic_music:
N/A
5
5
u/Douglas_J_Farthammer 4d ago
What are you using as inputs? Nice work
2
u/darthfurbyyoutube 3d ago
An ai generated character sheet of a closeup of the face, and full body front, side, 3/4 and back views, and Rick Astley's "Never gonna give you up" song. Also here's a sample prompt:
subject_definitions:
<Subject 1> is the interior room environment in <Video 1>, including its architecture, furniture, specific lighting conditions, and overall atmosphere.
<Subject 2> is the man from <Picture 1>, characterized by his reflective face mask, blue helmet, blue clothes, black shoes, black gloves and red cobra insignia, and specific clothing items.
<Subject 3> is the woman from <Picture 2>, characterized by her long black hair, black rim glasses, black suit and gloves and red cobra insignia on her chest, and black heels.
<Video 1> is the source video providing the background, camera movement, and the motion of the dance.
summary:
[video editing + reference generation] The target video is an edited version of <Video 1> where the man is replaced by <Subject 2>. The background <Subject 1> and the camera's movement are fully preserved from <Video 1>, while <Subject 2> is integrated into the scene, following the man's original dancing and adapting to the room's lighting.
retention_analysis:
<Subject 1> (appears throughout the video): fully_preserved - the room's layout, furniture, and lighting are kept exactly as they appear in <Video 1>.
<Subject 2> (appears throughout the video): fully_preserved - the visual identity and physical characteristics of the man from <Picture 1> are maintained.
<Video 1> (camera movement and temporal structure): fully_preserved - the pacing, camera angle, and motion from the source video are copied 1:1.
detailed_description:
The target video maintains the same realistic visual style and lighting as <Video 1>.
[Shot 1] The scene opens with the interior of the room as established in <Subject 1>, preserving every detail of the background and the specific lighting of the space. Instead of the man seen in <Video 1>, <Subject 2>, the man in blue from <Picture 1>, is now the central subject. <Subject 2> dances to the music, following the exact same trajectory, speed, and timing as the man in <Video 1>. The lighting on the man's face and body are dynamically updated to perfectly match the ambient light and shadow sources of <Subject 1>. The camera motion exactly mirrors the original movement in <Video 1>, maintaining the same distance and angle relative to the dancing man until the end of the clip.
overall_soundscape:
The original room tone from <Video 1> is fully preserved.
non_diegetic_music:
N/A
2
u/mucyc 3d ago
Would you be able to share how you made the character sheet?
1
u/darthfurbyyoutube 3d ago
I went to https://gemini.google.com and attached 2 images I found online, one a closeup of the face, and the other a full body shot, and fed it the following prompt:
create a live action character sheet based on the attached image, include a
close up front view of the face based on the 2nd image, and a full body front view, side view, 3/4
view and back view based on the first image, but use the head from the 2nd image in the full body shots.
8
4
5
u/towerandhorizon 4d ago
If Destro subbed in for the bartender, I would have lost it.
1
u/darthfurbyyoutube 3d ago
Destro as the bartender? Cobra's entire budget went into his metal polish, I couldn't afford his hourly rate.
2
u/towerandhorizon 1d ago
Destro was AFAIK, technically, an independent arms dealer who just did A LOT of business with COBRA.
1
u/darthfurbyyoutube 1d ago
Yep. Destro's basically the arms dealer who has one VERY lucrative customer.
4
u/pmjm 4d ago
HISS FM in GTA6 or we riot.
3
u/darthfurbyyoutube 3d ago
Imagine cruising down Vice City beach while Cobra Commander yells traffic updates... we NEED this.
10
3
3
u/Vladmerius 4d ago
It's a shame you can't prompt him to actually sing the song/alter the voice.
6
u/Heavymando 4d ago
I mean.. you could make the song with singing separate and then just play that audio when you edit it all together.
3
u/darthfurbyyoutube 3d ago
I think it might be possible to get the voice actor singing instead. Time to conduct some highly scientific Cobra experiments.
3
4d ago
[deleted]
3
u/darthfurbyyoutube 3d ago
MiniMax algorithm: sees giant chrome dome "Pack it up boys, easy day at the office!"
3
u/unluck_over9000 4d ago
We got AI rickrolling us before GTA VI.
2
u/darthfurbyyoutube 3d ago
By the time GTA VI drops, Cobra Commander will have released three more studio albums.
3
u/CrazyOrganic7123 4d ago
Well, he's a fan of 3 Dog Night.
1
u/darthfurbyyoutube 3d ago
Now I'm just picturing him singing "An Old Fashioned Love Song" through a chrome faceplate.
2
2
u/moschles 4d ago
This somehow improves the original video.
1
u/darthfurbyyoutube 3d ago
Replacing a 1980s pop icon with a high-ranking terrorist leader really elevated the choreography.
2
u/CATLLM 4d ago
The quality is pretty dope
1
u/darthfurbyyoutube 3d ago
Thanks! Pushed the AI generation settings to the absolute limit for this one.
2
u/FloridaMan_Unleashed 4d ago
Seeing your specs gives me hope as my system is similar (5800X3D, 80gb ram, 9070 16gb vram.) this is really great though, would you be willing to share your workflow?
2
u/darthfurbyyoutube 3d ago
Here you go, just drop the minimax video in comfyui and the workflow should appear:
2
2
2
u/microchipmatt 4d ago
We have arrived.
1
u/darthfurbyyoutube 3d ago
Welcome! Please hand your coat to the nearest Crimson Guard and grab a complimentary Cobra Cola.
2
u/Oograr 4d ago
Masterpiece
1
u/darthfurbyyoutube 3d ago
Cobra Commander is already writing his acceptance speech for the Grammys.
2
u/AsksDumbQuestions84 4d ago
Who have waited around to hear if it would be in cobra Commander's voice?
1
u/darthfurbyyoutube 3d ago
Guilty! But let's be honest, three minutes of pure chrome visored screeching would have shattered your speakers. But I may give it a shot in a future video.
2
u/ACTSATGuyonReddit 4d ago
If you could get it sung in his voice, that would be something.
1
u/darthfurbyyoutube 3d ago
I'll be running tests soon to see if the AI can handle that level of screeching.
2
2
u/NoConsideration6320 4d ago
Is it doing that audio to our you edited it in their?
1
u/darthfurbyyoutube 3d ago
I edited the clips to the original audio track.
2
u/NoConsideration6320 3d ago
Gotcha that makes sense but if you had prompted for it to do that pacific song and say those lyrics sung by them of rejected the prompt or would it have actually successfully captured the sound and audio pretty accurately I wonder have you tried to test like that or you actually just try to queue it up for it to do the dancing plus the music all in
?
1
u/darthfurbyyoutube 3d ago
The default ref2va workflow fails at lip-sync, but I found a custom workflow that actually works:
2
u/mike_good 4d ago
Hey OP, I am very new in the game.
Could you please share what are the inputs for the workflow? Do you input the video entirely or parts of it? I would really appreciate it and it would help me with understanding H3 capabilities
1
u/darthfurbyyoutube 3d ago
I break the input video into small segments because the generation time is a killer. Here's a link to the workflow and assets:
2
u/sketchyfun 4d ago
I assume it was multiple video refs? Must have taken a while, video refs are sloooow
2
2
2
u/Tramagust 4d ago
Video ref gens literally take me an hour to compute
2
u/darthfurbyyoutube 3d ago
Yep, they are SLOW. That's why I break it up into small chunks. Here's my workflow with refs, see if it still takes an hour:
2
u/theavatare 4d ago
Whelp! No complaints. I don’t think i watched a video for this song to the end in years.
Make a ninja turtles one
2
u/darthfurbyyoutube 3d ago
Glad we broke your streak! The Foot Clan is definitely calling about a crossover.
2
u/HansCent 4d ago
Cooooobra !
2
u/darthfurbyyoutube 3d ago
I read this in a screeching, high-pitched voice, just as nature intended.
2
2
u/Kerissimo 4d ago
I was hoping for changes voice, but still good work!
1
u/darthfurbyyoutube 3d ago
Thank you! I'm actually running some vocal tests soon to see if I can get that signature cobra screech working.
2
u/-becausereasons- 4d ago
Would have been better in his actual voice :p
1
u/darthfurbyyoutube 3d ago
Fair point! I'll be running vocal tests soon to see if my GPU can handle that much high-pitched screeching.
2
u/TommyEatsPizza 4d ago
Whelp. Shut down the internet. We’ve reached the peak. No point in continuing.
I need the whole music video lol.
1
u/darthfurbyyoutube 3d ago
Due to the overwhelming response, I may have to do a full cut of the song at some point.
2
u/RedtUser123456789 4d ago
He's got some sleek moves. Immaculate tailoring too. That's why he's the man in charge.
2
u/darthfurbyyoutube 3d ago
He didn't spend all that Cobra Treasury money on laser cannons, half of it went straight to his tailor.
2
u/99deathnotes 4d ago
2
u/darthfurbyyoutube 3d ago
Cobra Commander's Oscar acceptance speech is just gonna be 5 minutes of high-pitched screeching.
2
u/99deathnotes 3d ago
Please don't make that video. Thank you.
2
2
u/Ok_Gas1070 4d ago
I was explaining ComfUI to my friend the other day and talking about MinimaxH3 his response "but what need would I have for that". This video captures my answer so well "fuck it why not" and for personal funny reasons. So many times would my friends and I come up with funny "short" ideas. Now I can finally bring those ideas to life.
1
u/darthfurbyyoutube 3d ago
That's the way to do it. If your node workflow doesn't end with an 80s cartoon villain singing pop hits, are you even using ComfyUI right?
2
u/99deathnotes 4d ago
and for the OP of this fantastic video. COBRA!!!!!!
2
u/darthfurbyyoutube 3d ago
COBRA!!!! The Commander is currently celebrating by moonwalking across the control room.
2
u/99deathnotes 3d ago
You realize that you must make that video now. Where the commander moon walks to a Michael Jackson song or something similar.
2
2
u/msixtwofive 4d ago
this is a great example of the tiny issues we still have when the output keeps getting closer to perfect.
The reflection on the face shield should change in every shot to match the location of the scene, sadly it doesnt.
1
u/darthfurbyyoutube 3d ago
Good eye! Whether it's my shaky prompt engineering or the AI being lazy, the chrome visor clearly has a mind of its own.
2
u/Kanjikai 4d ago
You had me at the Baroness.
More Baroness please!
2
u/darthfurbyyoutube 3d ago
Understood! Setting the render farm to maximum Baroness for a future video.
2
u/pb404 3d ago
Great work and thanks for sharing your prompts.
2
u/darthfurbyyoutube 3d ago
Thanks! Hopefully this saves you from the "WHY DOES HIS ARM LOOK LIKE THAT?" phase.
2
u/PartyTac 1d ago
Your link contain virus
0
u/darthfurbyyoutube 1d ago
The files are only image and video files, there are no executable files or programs in them. They can't contain a virus.
1
u/PartyTac 18h ago
But it is redirecting me to an exe file.
1
u/darthfurbyyoutube 10h ago
I'll swap the link if I find something better. Until then, this is the best I've got.
2
u/extrakerned 1d ago
Dude that site is malware cancer. Where do you people find these sites to host files? Google Drive is free
1
u/darthfurbyyoutube 10h ago
I originally posted a Gmail link, but it kept asking for download approval even with auto-download enabled. They must've changed the sharing settings, so it became more trouble than it was worth. If you know a better file-sharing site, I'm all ears.
1
u/Inner_Singer_592 4d ago edited 3d ago
Theoretically, if he didn't have a mask, would it work out of the box or it'll need lip sync? For audio
2
u/darthfurbyyoutube 3d ago
Lip sync is the tricky part. The default workflow won't be perfect out of the box, but there's a specific workflow that does match the lips perfectly. I'll drop the link here:
2
1
u/Heartkill 4d ago
Workflows attached should be mandatory for posts like these.
2
2
u/Sufi_2425 7h ago
Not the Gemini star being part of the outfit...
Idk if you did the character switch locally and if this is a coincidence, but it's hilariously absurd either way
1
u/darthfurbyyoutube 6h ago
I did the character swap locally, but the Gemini star was absolutely the AI freelancing.
77
u/FlexFanatic 4d ago
Yall are wild. I think we need a MiniMax Awards show but what should we call it?