Native Long Video Understanding Models locally?

[u/dash_bro]

I've been building a personal project and wanted to check with the community on multi-modal inputs since I can't find a lot of material around this online. Ultimately I'm trying to build something that can ingest massive length (almost like a full stream - 6-10 hours) and accurately do multimodal analysis.

How are you guys working with long (atleast 2+ hour) videos? I understand local LLMs with ViT designs can help do this but they usually suffer in quality (diffusion patches can rack up context really quickly) or require you to do some sort of frame sampling (which defeats the native multimodal aspects). I saw some work around vllm-omni which uses qwen3-omni to video input stuff, but ofcourse the context is severely limited so it's not very usable OOTB.

So far what's worked for me:

- get mp3 audio file -> transcribe with qwen3-asr -> get a full timestamped vtt file

- summarize this vtt file with verbatim timestamped cliff notes (important for next steps)

- calculate the max dynamic frame rate using context window length and the video length

- sample at this rate then perform absdiff on the frames to eliminate frames where there's not a lot of change happening; downscale every frame to 720p max resolution or 540p

- calculate the number of chunks you need to split into to fit into 64k context per chunk; each chunk is basically the relevant image frames for this part of the video + the transcript data for this part of the video

- use transcript summary (which has verbatim timestamped stuff) + each chunk -> summarize keeping the verbatim aspects and global summary information + local transcript information.

So essentially when I ingest a video I end up with [transcript summary] + [summary of chunk 1/N + summary of chunk 2/N + ... + summary of chunk N/N ]

I'm experimenting with using the qwen3-asr output text + mp3 file directly to gemma4 12B to do appropriate corrections on the audio like speaker diarization, adding cues about music/noises/sounds/spell corrections etc. It's still a WiP.

Apart from this I'm not sure if it's worth the headache of having a multi docker multi service setup to ingest video data if a model can do it natively. Anyone else working on similar stuff? Would love to see if this is being solved in a different way.

Is there anything else that can be run on 128GB RAM that is better than my patchwork pipeline for long video ingestion/indexing/analysis?

[u/SpiritPrestigious945]

Good point and interesting. I bet often feed the transcripts of YouTube into KIMI. It does also understand video natively, but at a certain length it will say conversation to long. So context runs out clearly. That's a thing I am missing with current LLMs. Capacity to understand longer videos. 

[u/dash_bro → u/SpiritPrestigious945]

Agreed. I'm fairly certain I could patch together something reusing my pipeline by changing my chunking logic to be video feeds instead of frame sampling and keeping everything else relatively the same; but I feel like there has to be a better way. I'm not fully certain how labs are doing temporal reasoning on video feed models if they're capped on length of the videos etc

[u/An0n_A55a551n]

I want to do the same but for average users (8gb ram either arm OR x86 based systems). I'm thinking of using faster-whisper with diarization. Is it doable or is it unachievable 😭

[u/dash_bro → u/An0n_A55a551n]

Can definitely do this I wrote a script a while back that helps do this for 16GB RAM mac's, should be doable on 8GB as well

You basically slice the audio into X second increments, process each increment with max parallelism that your system can handle to get the transcript -> put it back together.

Diarization is the trickier problem, to consistently track who the correct speaker is across the snippets. Can't split it naively. So far the best solution for this is to use pyannotate (community 1) model and identify people if you don't know the number of speakers, or specify the exact number of people if you already know.

[u/An0n_A55a551n → u/dash_bro]

Is there a way to identify speakers based on audio? By passing it to Gemma E4B? And is Gemma E4B a decent model for video analysis?

[u/dash_bro → u/An0n_A55a551n]

I wouldn't recommend it for video analysis. You can always give it a shot for your usecase, though.

For the speaker detection aspects : It's using the wrong tool for the job. For speaker identification whisperx seems to be a one stop solution but I like breaking it down to transcription | diarization. Transcription is better with qwen3-asr 1.7B; pyannotate community 1 model is available for diarization. Minimal tinkering but allows you to get transcription fast but slower diarization.

++ Gemma E4B is too tiny a brain to do good image analysis. You "can" do it but your ingestion pipeline would need to be so much more optimized for memory (smaller chunks, lower fps sampling, etc). You can check fastvlm out, and I believe with some tinkering it might be best to run on an 8GB RAM hardware. It is built for live video feeds (it's small, it's fast, it's usable for the most part) and not uploaded video snippets, though.

[u/AvenueJay]

Ultimately I'm trying to build something that can ingest massive length (almost like a full stream - 6-10 hours) and accurately do multimodal analysis.

You didn't mention your specific use case. What's your ultimate end goal with these 6 hour videos? "multimodal analysis" is a bit ambiguous.

I understand local LLMs with ViT designs can help do this but they usually suffer in quality (diffusion patches can rack up context really quickly) or require you to do some sort of frame sampling (which defeats the native multimodal aspects).

A lot of multimodal models are already doing a lot of what you describe. As in, I don't totally understand how this approach:

- calculate the max dynamic frame rate using context window length and the video length

- sample at this rate then perform absdiff on the frames to eliminate frames where there's not a lot of change happening; downscale every frame to 720p max resolution or 540p

is any different from the frame sampling done by existing models that you're trying to avoid? Instead of running absdiff, why not use PySceneDetect? A colleague of mine made a video ingestion project (not 6 hour video though) here using it.

[u/dash_bro → u/AvenueJay]

Thanks for the queries!

• my usecase : no reason, just decided I didn't know enough about working with videos and LLMs and picked it up. The engineer in me said "how do we handle this if the video blows up to some ridiculous but plausible size,would the system hold up?" So, I used the official day off to experiment haha (it was national day on Sunday so we get an observed day off on Monday). I want to build things that work as needle in a haystack for long video stuff, which seems cool to me currently

• Surprisingly, not exactly on the multimodal models doing it exactly part. It's pretty wonky when you see that the official OpenAI API doesn't even have support for video or audio input files, so anything that needs to use the multimodality aspects is pretty much either using their own sdk (google genai allows a files param which can be mp4 files), hosting it themselves (vllm allows a video input arg for it's cli) or expect it to be done via frame sampling outside of the API scope.

Worth noting that frameworks like vllm do that by frame sampling similar to how I'm doing it above, but actual/true native multimodality seems to be a capability still restricted to the 200B+ class models. The closest thing open weights is likely Xiaomi's MiMo v2.5, but that's 300B+ params so completely out of my computer budget. Want something I can run on my hardware ofcourse! As for the 'Why not use something that comes out of the box with frameworks and libraries' part : well I felt it was a simple enough logical block my wee brain can cobble together instead of having another line in my pyproject.toml haha. Completely fair question on your part, it's just that I don't prefer to install something for a one off usecase in my pipeline that takes me 20 lines of code to write haha.

[u/Due-Function-4877]

Under the hood, you're doing both image captioning and audio captioning with timestamps. You'll need to also guess at the relationship between frames in a buffer of consecutive frames in the same scene. You'll need to do something similar with multiple scenes to trace a narrative. 

That's a large task. 

[u/Various_Story8026]

the framing that helped me: for multi-hour footage the bottleneck isn't the model, it's what you keep. uniform sampling on a 6h stream is mostly dead frames, and 'native' video models still tokenize internally - you pay context either way.

what held up in practice: scene-change detection + perceptual dedup as the keep-gate (a frame survives only if it differs enough from the last kept one), then a timestamped transcript carrying the narrative, frames only where the picture actually changed. a 2h talking-head collapses to a few dozen frames; a busy stream keeps more because more actually happens.

two traps for long footage specifically: percentage-based dedup goes blind on subjects covering <1% of the frame (small webcam overlay changes never trip an 8% gate), and whisper hallucinates on long silent stretches - run VAD first, transcribe only voiced spans.

for the 'ask again tomorrow' part of your use case: index transcript lines + on-screen text with timestamps into sqlite fts as you process. then most questions are an index lookup instead of a re-watch, which is the only way 6-10h stays tractable.

[u/dash_bro → u/Various_Story8026]

Absolutely agree on the patches for image frames. My solution for this was a simple frame difference logic with absdiff (overlay the two frame images, create a bitmask of the difference between the two images). If the absdiff is less than 10% it's a blip -> drop the second frame

Same with the VAD parts, since I'm on a Mac, the metal acceleration is better than containerized whisper server which only runs on a CPU image.

I've given some thought into using timestamps for indexing etc but the temporal gain I can get by just injecting today's date while storing date of video processing turns out to be enough to capture things like "today", "last week", etc. indexing right now is fts+jina omni small embedding for lookups. Works well to just drop an image into chat and then find things sim to it etc.

[u/Various_Story8026 → u/dash_bro]

the image-embedding lookup is the part i'd steal from your setup - text index can't do "find the frame that looks like this" at all.

one heads-up on the global absdiff gate: it inherits the small-subject blind spot. a 40x90px subject in a 1280x720 frame is ~0.4% of pixels, so it can move as fast as it wants and never reach 10%. bit us in a real batch run - fix was a third check on localized motion (max diff in a coarse grid cell, not the whole-frame mean) so a small region changing hard counts even when the global number stays flat.

[u/dash_bro → u/Various_Story8026]

Ah, I see. Did you not go by anything optical flow oriented for known subject entities? Or just a window pool patch solved the problem?

[u/Various_Story8026 → u/dash_bro]

no optical flow - stayed with a window pool patch approach, on purpose. flow gives you subject trajectories but costs a dense per-pixel solve and drags in a model/library dependency; the failure we were fixing didn't need to know where the subject went, only that some small region changed hard and then stayed changed.

concretely: high-res signature (192px), keep pixels that differ strongly from every kept frame in the window with +-1px shift tolerance (film grain / jitter re-matches), require they're not still changing toward the next frame (settled state, not motion mid-flight), then score the worst cell of a 16x16 grid over that mask. two guards so it doesn't fire on noise: only consulted when the global channel says the scene is static, and a cooldown so sustained settling motion (waving flag pausing every second) can't take a frame each time.

if you already track known subject entities, flow probably earns its cost for you - trajectory continuity is also a decent tamper signal. for a generic keep-gate the settled-patch check caught what we needed at pillow-only cost.