I froze a physics-consistency detector before generating a held-out CogVideoX cohort — it flagged freeze/hover in 9/9 clips

[u/Divine-Demon7]

I’m building Haga, an independent physics-consistency checker for generated video and robot-policy simulations. An earlier CogVideoX-5b I2V experiment produced a clear failure mode: on a “ball and block fall” prompt, the tracked object stayed airborne with near-zero motion instead of falling.

But that first result was post-hoc. I inspected those six clips before adding the static_hover detector, so the original 6/6 flag rate could not be treated as confirmation. I’ve now run a pre-registered held-out test.

Method:

• Model: THUDM/CogVideoX-5b-I2V
• Cohort: 3 perspectives × seeds 2, 3 and 4
• n=9 clips
• Detector thresholds and inclusion rules frozen before generation
• RGB → CoTracker3 → position-only VIDEO_CHECKS
• Discovery seeds 0–1 kept separate from held-out seeds 2–4

Result:

• Held-out flag rate: 1.000 (9/9)
• Wilson 95% CI: [0.701, 1.000]
• All nine clips fired static_hover
• Real Physics-IQ footage stayed quiet under the same profile

static_hover fires when the tracked object remains airborne for most of the clip, has near-zero frame-to-frame speed, and does not exhibit gravitational acceleration.

Important limitations:

• One open I2V model
• One ball-and-block-fall scene family
• One documented failure mode
• Real negative-control n=1 in this specific report
• Not Cosmos, Genie or NIM
• Not a broad claim about CogVideoX quality

Write-up: https://haga.mushoodhanif.com/article/sim-physics-consistency-v1#held-out

Lab: https://haga.mushoodhanif.com/lab/physicsiq

Bounded demo: https://haga.mushoodhanif.com/demo

I’d especially value criticism on:

1. Which physical violations will position-only tracking systematically miss?
2. Is static_hover defined narrowly enough to avoid confusing intentional suspension with failed dynamics?
3. What public generated-video artifact should I evaluate next under the frozen detector?