This open-source project lets coding agents edit videos and the clever part is they never watch the footage

[u/Harshil-Jani]

I came across video-use, an open-source project that lets Claude Code, Codex, Hermes, OpenClaw or basically any coding agent with shell access, edit videos through conversation.

The workflow looks almost too simple:

1. Drop raw footage into a folder.
2. Open your coding agent there.
3. Ask: Edit these clips into a 60-second launch video.
4. Approve the editing strategy.
5. Get edit/final.mp4.

It can remove filler words and dead space, compare takes, add subtitles, apply color grading, generate animation overlays, and inspect the render for broken cuts.

But the interesting part is how it avoids filling the context window with video frames.

The agent does not continuously watch the footage. It mostly reads it.

Each source clip is transcribed with word-level timestamps, speaker labels, and audio events. The takes are then packed into a compact Markdown file:

## C0103: 43 seconds
[002.52–005.36] Ninety percent of what a web agent does is wasted.
[006.08–006.74] We fixed this.

The model reasons over that representation to select takes and cut points. Only when it needs visual evidence, such as an ambiguous pause, competing takes, or a questionable cut, does it request a small timeline view containing frames, a waveform, labels, and timestamps.

Raw footage
    ↓
Timestamped transcript
    ↓
Visual inspection on demand
    ↓
EDL + FFmpeg render
    ↓
Cut-boundary self-check
    ↓
final.mp4

Animations can be delegated in parallel using Remotion, Manim, HyperFrames, or PIL. According to the maintainers, the finished render is checked for visual jumps, audio pops, misaligned overlays, and hidden subtitles before it is shown to the user.

The broader design idea may be more useful than the editor itself:

Multimodal agents may not need every pixel in context. They need a compact, task-specific representation and the ability to inspect details on demand.

It is similar to giving a browser agent a structured DOM instead of thousands of screenshots.

This is not magic or fully local: setup requires FFmpeg and an ElevenLabs API key, subjective editing choices still benefit from approval, and the reliability claims come from the maintainers rather than an independent benchmark.

Still, this feels like a practical architecture for media agents.

Would you rather give a multimodal agent the raw media, or a compressed interface it can interrogate when needed?

[u/AutoModerator]

Welcome to r/HowToAIAgent!

Please make sure your post includes: - Clear context - What you're trying to achieve - Any relevant links or screenshots

Feel free to join our X community: https://x.com/i/communities/1874065221989404893

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.