People on stage and camera motion, read off the video - #382
Merged
Merged
Conversation
A person detector on a concert video finds the audience too: heads and shoulders in the lower part of the frame, cut off by the bottom edge. Performers are on a raised stage, so their heads are in the upper half, and that one geometric fact separates the two well enough to count a soloist, a duo and a five-piece band from an operated camera. The camera moves, so no frame shows everyone; the span statistic is a high percentile of per-frame counts, the wide shots, reported with the median and maximum so the framing's variation is visible. `detect_people` samples the video at a low rate through an ffmpeg pipe and runs YOLO (the existing `ultralytics` extra); `on_stage` is the filter; `people_track` and `performer_count` summarise. A choir is under-counted, which the docstring says. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
An operated camera pans, tilts and zooms, and a multi-camera edit cuts between angles; both inflate frame-difference measures and change who is in the picture. `_camera.camera_motion` labels each sample of a small 2 fps proxy still, moving or cut from ORB matches and a partial-affine RANSAC fit between consecutive frames, and returns cuts, shots and the share of time in each state; `still_runs` gives the framings, the still runs between moves and cuts, and `camera_state_at` samples the state so motion measures can leave camera motion out. `performer_count(..., camera=)` counts per framing: the 75th percentile of each framing's per-frame counts, the widest framing being the estimate. That is what makes the count right with a moving camera: the percentile over all frames took the audience in the band's wide shot for two more performers, and a per-shot median took the dominant close framing for the band. Also the mypy notes from CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
…ode on the GPU Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two new modules, exported from the package root.
_performersdetect_people(video, fps=1, width=640, model="yolo11n.pt", ...): person boxes per sampled frame via an ffmpeg pipe and YOLO (ultralyticsextra), normalised coordinates, GPU when torch sees one.on_stage(box): keeps boxes whose heads are in the upper half of the frame and drops audience members cut off by the bottom edge.people_track/performer_count(detections, start_s, end_s, camera=None): how many perform in a span. Without camera analysis, a high percentile of per-frame counts (the wide shots). With it, one count per framing (see below), the widest framing being the estimate._cameracamera_motion(video): each sample of a 2 fps proxy labelledstill/moving/cutfrom ORB matches and a partial-affine RANSAC fit; returns cuts, shots and the time share per state.make_proxy,camera_state_at,still_runs(framings) alongside.Validated on an IMV concert recording with an operated camera (14% of the time moving, 25 cuts): the framing rule is exact for a solo pianist, a guitar/bass duo and a five-piece band, where the frame percentile counted 10 for the band and a per-shot median 4; a 17-voice choir comes out as 7, which the docstrings state.
Test plan
tests/test_performers.py: stage filter on real box geometries, audience exclusion, the wide-shot percentile, span bounds, and a smoke test ofdetect_peoplethat skips without weights/network.tests/test_camera.py: synthetic footage with a static scene, a pan and a cut → states, cut time, shots, framings,camera_state_at;performer_countper framing ignores the swing past a crowd.🤖 Generated with Claude Code
https://claude.ai/code/session_01PcjBSXbXjqdj1tmYHoLiKd