fix(cli): select containing source frames in snapshots

[Dante-dan]

What

Snapshots select the source video frame whose presentation interval contains the resolved source time. For a 24 fps source sampled at 0.4, 22/30, and 10/30 seconds, that means frames 9, 17, and 8.

Why

Seeking FFmpeg directly to an off-grid time discards the containing frame and returns the next frame. This shifts a baked video plate by one source frame relative to other renderers.

Related work

Fixes #4763. Implements the source-presentation-timestamp approach approved in the issue discussion.

PR #4767 addresses a separate VFX post-injection capture issue (#4762); this change addresses source-frame selection.

How

1. Pick the frame. ffprobe reads packet timestamps and keyframe flags, with no decoding, up to one second past the requested time. The read is measured from the stream's start, so it also covers an edit-list trim whose preroll begins before the first visible frame. Times are measured on the file's clock from the stream's exact integer start. The selected frame is the last one whose presentation time is at or before the requested time, with a small floating-point tolerance. A time before the first frame (audio leading the video) holds the first frame, as the engine does.
2. Find a decode start. Demuxers do not always land where they are asked: an MPEG-TS seek can land a whole keyframe group later, and an open-GOP keyframe is not a clean decode start. So, starting from the latest keyframe at or before the frame (at most three candidates), the extractor runs a stream-copy probe (-c copy, no decode). That probe reads which keyframe the seek actually lands on. The first candidate that lands at or before the frame, and lands on that same keyframe when sought again, is used.
3. Extract. FFmpeg seeks accurately to exactly that keyframe, which trims its open-GOP leading frames. It then selects the frame by its time relative to the first decoded frame (select=gte(t-start_t, …)), so the result does not depend on where FFmpeg puts zero for a given container. If no candidate qualifies, it decodes from the start of the file.

Both rendered-video injection and --against reference extraction use this rule.

Trim, playback-rate, automation, loop, and clip visibility time mapping remain in their existing shared runtime paths. A held tail samples the actual last containing frame.

Cost per sample: one ffprobe call for stream metadata and one for packets, up to six short stream-copy landing probes, and one decode from the chosen keyframe. On a 5-minute 1080p clip, a sample near the end took under 0.8 s in both MP4 and MPEG-TS. The 30-second extraction bound remains, and probing has its own 30-second/32 MiB output bound.

Without ffprobe, extraction falls back to a plain seek (the first frame at or after the time), as before. One case is not covered on that path: a held tail past the last frame returns no frame.

Test plan

• Snapshot suite: 70 tests passed. They use real FFmpeg extraction, each compared against a full-decode select=eq(n,k) frame. They cover:

  • frames 9/17/8 on a 24 fps grid, exact boundaries, an irregular VFR interval and held tails;
  • fragmented MP4, MPEG-TS and MP4 with B-frames, and open-GOP MPEG-TS;
  • a start time that rounds down, and an edit-list stream-copy trim;
  • MPEG-TS with leading audio, MP4 with late video, and a TS whose timestamps start below zero (with and without ffprobe);
  • an open-GOP TS with a keyframe every 15 frames;
  • unreadable sources, which blank only that video.

• Each regression test was confirmed to fail on the commit before its fix.

• The producer ffprobe argv contract passes (87 tests).

• CLI typecheck passed.

• Repository pre-commit checks passed: typechecks, lint/format, fallow (no new findings), tracked artifacts, large files, and commitlint.

• No browser pixel reproduction is claimed for this source-extractor change.

• Unit tests added/updated

• Manual testing performed

• Documentation updated (if applicable)

• Comments follow CONTRIBUTING.md "Comments"

[jrusso1020]

Requesting changes at d61d4951. The frame math is right when the video stream starts where the container starts. It goes wrong when the video starts later than the container does, which happens with an MPEG-TS that carries audio, or any file where the audio leads the video. I reproduced this with real FFmpeg/ffprobe 8.0.1.

Repro

ffmpeg -f lavfi -i testsrc2=d=3:r=30:s=160x90 -f lavfi -i sine=d=3 \
  -c:v libx264 -bf 3 -pix_fmt yuv420p -c:a aac av.ts

ffprobe reports the format start_time as 1.443444 and the video start_pts as 132000 (1.466667). So video frame n sits at 0.0232 + n/30 on the container timeline. I identified frames by comparing each output against select=eq(n,k) extractions:

time	containing frame	this PR	main
0, 0.01	n0 (before the first frame)	null	n0
0.05	n0	n1	n1
0.1	n2	n3	n3

I also made an MP4 with audio from 0 and the video shifted +0.5 s (setpts=PTS+0.5/TB). For times 0, 0.01 and 0.1 this PR returns null and main returns n0. From 0.5 s on, the PR is correct.

Two causes

1. Before the first frame there's no frame (regression). containingSourceFrameIndex returns -1, so the probe returns null. In the snapshot path, if (!png) continue then leaves the plate as whatever headless Chrome painted. --against prints "has no frame" and skips the reference pair. Main extracted the first frame. The engine also holds the first frame here: videoFrameExtractor.ts samples with fps=…:start_time=0, and on the +0.5 s MP4 that path writes 105 frames from 90 source frames, with the gap filled by frame 0. I tried a local patch that clamps the index to 0 when timestamps is non-empty. With it, the +0.5 s MP4 gives n0 at all three times, and snapshot.test.ts still passes 66/66.
2. The select threshold assumes FFmpeg's output timeline starts at the container start. On av.ts it doesn't when the seek is 0. With -ss 0, or no -ss at all, showinfo puts the first video frame at pts_time:0, not 0.0232. With -ss 0.05 the timeline does start at the container start (pts_time:0.00656 = 0.0566 − 0.05). So whenever seek comes out as 0, the threshold is off by the video's lead and the next frame is chosen. Any time inside the first GOP gives a seek of 0, and that's the whole clip here. The clamp alone doesn't fix this: with it, av.ts at t=0 gives n1. The PR's MPEG-TS test passes because its fixture has no audio, so the container start equals the video start.

Suggested fix: add audio to the TS fixture and add a late-video MP4 case. Then pick the target frame in a way that doesn't depend on where FFmpeg puts zero. One option is select=eq(n,k), with k counted in presentation order from the seek keyframe. Another is to read the offset from the first decoded frame. I haven't tried either.

Everything else checks out

• Boundaries: <= with a 16·ε tolerance selects the frame that starts exactly at the requested time. I confirmed that and frames 9/17/8 on the 24 fps grid, the irregular VFR interval, and the held last frame (with ffprobe) at head.
• ffprobe argv: runCancellableProcess spawns without a shell, and the argv is a fixed literal ending in -- before the path. I checked that a relative path literally named -i.mp4 still extracts. On the ffmpeg side the path is only ever the value of -i. Inputs are still project-local absolute paths or http(s) URLs, the same as main.
• Minor, not blocking: without ffprobe, a held tail now returns null, because main's -sseof -1 fallback was removed. I checked this by stubbing HYPERFRAMES_FFPROBE_PATH to a missing binary: the same call gives a frame with ffprobe and null without it. The body says ffprobe is required, but it doesn't mention this case. Either name it there or keep the tail fallback.

Tests at head: snapshot.test.ts 66/66.

— Rames

[jrusso1020]

When the requested time is before the first video frame (a video that starts after the container does), this is -1 and the extraction returns null. Main returned the first frame, and the engine's fps=…:start_time=0 path holds it too. I tried clamping to 0 when timestamps is non-empty: that fixes the late-video MP4, and snapshot.test.ts still passes 66/66.

[jrusso1020]

This threshold is measured from the container start, but FFmpeg's t isn't always measured from there. On an MPEG-TS with audio leading the video by 23 ms, -ss 0 puts the first video frame at t=0, so every seek-0 extraction picks the next frame (t=0.05 gives n1, t=0.1 gives n3). With -ss 0.05 the timeline does start at the container start. The table in the review body has the full repro.

[miguel-heygen]

@jrusso1020 thanks for the repro, both causes are fixed at 7c2ec71.

1. Before the first frame. A time before the video's first frame now holds that frame, as main and the engine do.

2. Where FFmpeg puts zero. Frame selection no longer assumes where FFmpeg puts zero after a seek. The extractor now:

   • reads which keyframe the seek actually lands on (stream copy, no decode);
   • seeks exactly to that keyframe, so its leading frames are trimmed;
   • selects relative to it (t - start_t).

   If no nearby keyframe lands at or before the wanted frame, it tries an earlier one, then decodes from the start.

Your av.ts repro and a late-video MP4 are now tests. So are two cases that came up while fixing this:

• a TS whose timestamps start below zero, with and without ffprobe;
• an open-GOP TS with a keyframe every 15 frames, where a seek to 0.5 s lands on the keyframe at 1 s.

Each test fails on the commit before its fix.

On your fixtures: av.ts at 0 / 0.01 / 0.05 / 0.1 / 1 gives n0 / n0 / n0 / n2 / n29. The +0.5 s MP4 holds n0 until the video starts, then gives n1 at 0.55.

Still as you noted: without ffprobe, a held tail past the last frame returns null, because the -sseof fallback is gone. I can restore it if you want it in this PR.

[jrusso1020]

Approving at 7c2ec71c, which replaces my changes request at d61d4951. Both blockers are fixed.

The two blockers, re-run on the same clips with real FFmpeg/ffprobe 8.0.1

clip	time	containing frame	d61d4951	7c2ec71c
av.ts (TS, audio leads by 23 ms)	0, 0.01	n0	null	n0
av.ts	0.05	n0	n1	n0
av.ts	0.1	n2	n3	n2
late.mp4 (video +0.5 s)	0, 0.01, 0.1	n0	null	n0

1. Before the first frame, the index now clamps to 0, as the engine does.
2. t-start_t together with the observed seek landing makes the threshold independent of where FFmpeg puts zero.

Wider sweep

I added five clips:

• gop.ts: MPEG-TS, audio leading, -g 15 -bf 3.
• The same TS with open-gop=1.
• An MP4 with the video late and -g 15.
• A fragmented open-GOP MP4.
• VP9 WebM.

Each clip was sampled at fixed early and mid times, then at ±2 ms around every 7th frame boundary across the whole clip. Frames were identified by rgb24 framemd5 against a passthrough decode. The result was 0 mismatches in 407 samples across the 7 clips, including the times where the TS seek lands on a later keyframe and the candidate loop has to walk back.

Other checks

• snapshot.test.ts passes 70/70 and tsc --noEmit is clean for the CLI.
• The new fixtures cover audio-leading TS, late-video MP4 and open GOP. Those are the cases from my repro.

Non-blocking:

• A held tail without ffprobe still gives null. That's the limit the body already names, and it's unchanged.
• Each extraction can now spawn up to six short -c copy ffmpeg runs: three candidates, each landing plus a confirm. Without ffmpeg, the code falls back to decoding from the file start. Both outcomes are correct. On a long file with many samples the extra runs could add up, but that's not a correctness issue.

— Rames