Build static camera web app with teacher tracking - #1
Axoled-Student with Copilot wants to merge 6 commits into
Conversation
Co-authored-by: Axoled-Student <235039939+Axoled-Student@users.noreply.github.com>
Co-authored-by: Axoled-Student <235039939+Axoled-Student@users.noreply.github.com>
Co-authored-by: Axoled-Student <235039939+Axoled-Student@users.noreply.github.com>
|
@copilot You are a senior web engineer. You must pull and inspect the repository Camera-Track on branch copilot/build-static-camera-web-app and then implement fixes. Do not guess the codebase—read it first, identify current behavior, and patch precisely. Problems to Fix (Observed) Camera choice is not remembered across sessions User selects a camera/lens (front/back or a specific device), but after reopening the site it “forgets” and uses a different camera. Camera + tracking are not auto-enabled next session After returning to the app (new session), it does not automatically start the camera and tracking, even if user previously had them enabled. Tracking box jitters, causing zoom/crop to jitter The detected person bounding box fluctuates frame-to-frame, and the virtual PTZ (auto crop/zoom) follows that jitter, making output unstable. Tracked target frequently switches when other people appear If a second person enters, the “teacher” target often changes unexpectedly. We need stronger target identity continuity (lock + re-ID style scoring). Need ability to set tracking FPS down to 1 FPS Add a setting allowing inference at 1 FPS (and up), without making framing jumpy. Non-Negotiable Constraints This is a static app (no backend). Must remain iOS Safari compatible. Must respect iOS camera permission rules: camera start generally requires a user gesture. If true auto-start is blocked, implement “one-tap resume” that starts camera+tracking immediately with a single tap, using remembered settings. Do not regress recording, downloading, model caching, or UI responsiveness. Step 1 — Repo Recon (Required) After checking out the branch: Identify where camera selection, tracking enable/disable, settings UI, and persistence are implemented. Find the code paths for: device enumeration (enumerateDevices) getUserMedia constraints building camera start/stop lifecycle tracking loop / inference scheduling box smoothing / crop smoothing target selection logic (largest box, etc.) local storage usage (if any) Produce a short “Findings” note in the PR description: which files/functions control each area. Step 2 — Persist Camera + Settings Correctly Implement a single versioned persisted state object in localStorage, e.g.: preferredCamera: { deviceId, labelHint, facingMode, groupIdHint } autoStartCamera: boolean autoStartTracking: boolean trackingFps: number (1–30) smoothing: config object (see below) targetLockMode: "auto-largest" | "locked" | "tap-to-lock" appearanceReidEnabled: boolean Add a small migration function for future changes (version number). B) Camera Persistence Behavior On next session: After permission is granted (so device labels exist), enumerate devices and attempt to restore: Exact deviceId match If missing, match by groupId or label keywords (back, front, ultra, wide, tele, etc.) if present If still missing, fallback to saved facingMode (environment preferred) Update the UI selector to reflect the restored camera. Important: Some browsers rotate deviceIds across sessions. That’s why you must keep a label/group/facing fallback path. C) Auto-Start Behavior (iOS-safe) If autoStartCamera is true, show a full-screen “Tap to Resume” overlay on load (only if needed). On the first user tap, immediately: start camera with the restored device enable tracking if autoStartTracking is true hide overlay If the browser allows immediate start without a gesture (some desktop cases), start automatically. Step 3 — Stop Jitter: Stabilize Both Detection and Framing Implement two loops: Inference loop at trackingFps (1–30 FPS) — updates the “latest measurement” Render/framing loop at display rate (requestAnimationFrame or requestVideoFrameCallback) — smoothly moves the crop target toward the latest measurement This ensures that at 1 FPS the framing transitions smoothly instead of “jumping” once per second. B) Add a Robust Smoothing Filter Implement one of: One Euro filter (preferred for jittery tracking), OR Critically damped spring smoothing with time-based dt, plus velocity clamp Apply smoothing to: bbox center (cx, cy) bbox size / scale (or target crop scale) Add: dead zone: ignore tiny bbox changes under a threshold (px and %) confidence gating: if confidence drops, slow movements and avoid zoom thrash zoom hysteresis: avoid continuous tiny zoom in/out Expose “Stability” slider in settings that adjusts filter parameters: higher stability = more smoothing, larger dead zone, lower max velocity Step 4 — Fix Target Switching (Target Identity Continuity) Current “largest box wins” is insufficient. Implement a target tracker with hysteresis + appearance matching: A) Target Lock State Machine Maintain currentTarget with: bbox confidence lastSeen timestamp lostFrames counter appearance signature (color histogram or simplified descriptor) Rules: Do not change target just because a new person appears. Only switch if: current target is lost for > T_lost seconds OR confidence is consistently low AND another candidate is strongly better for > T_switch seconds B) Matching / Scoring (Lightweight Re-ID) For each detection candidate i, compute a combined score: IoU with previous bbox (continuity) distance of centers (motion continuity) area similarity (teacher distance consistency) confidence appearance similarity: color histogram match on torso region (HSV histogram is fine) Combine with weights and choose best candidate, but with hysteresis: if current target has “good enough” match, keep it only switch when new candidate exceeds current by a margin for multiple inference ticks C) Appearance Signature Implementation Notes Use a small offscreen canvas crop of the target ROI from the raw video (not the cropped output) for stable sampling. Compute a low-res HSV histogram (e.g., 12×4×4 bins) or even simpler dominant color clusters. Keep it performant: downscale ROI to ~32×32 before histogram. Add a setting: “Use appearance lock (clothing color)” toggle “Tap to lock target” mode: user taps bbox to lock teacher Step 5 — Tracking FPS Setting Down to 1 FPS Add a UI control: “Tracking FPS” slider (1–30) default ~10–15 When set low (e.g., 1 FPS), the crop/framing should still look smooth due to render-loop smoothing. Make sure internal timers use real-time dt so behavior is consistent across devices. Step 6 — Playwright MCP Tests (Required) Update/add tests to cover the regressions: Settings persistence Select a camera option in UI Enable autoStartCamera + autoStartTracking Reload page Assert persisted state restored (UI reflects selection; internal state uses same device preference) One-tap resume flow Simulate page load where autoStart is enabled Assert overlay appears Click overlay Assert camera start + tracking start functions were called (stub/mocks if needed) Target lock stability (mock detector mode) Provide deterministic frames where two targets appear Assert target remains the same across frames unless “lost” threshold is exceeded Low-FPS smoothness sanity Set trackingFps = 1 Feed sparse bbox updates Assert crop position changes gradually (e.g., no single-frame jump exceeding a threshold) If the project already has a testing harness, integrate with it. If not, add minimal mock injection points without affecting production. Step 7 — Deliverable Expectations Make a PR with clear commits: persistence + auto-start overlay inference/render decoupling + smoothing target continuity + appearance lock tracking FPS control tests Update README: explain iOS gesture requirement and “Tap to Resume” document persistence keys and settings explain target lock + appearance mode explain tracking FPS Acceptance Criteria After selecting a camera and enabling auto-start + tracking, reopening the app: restores the same camera selection (or best fallback) starts camera+tracking via one tap (iOS) or automatically where allowed Bounding box and auto-framing are stable and do not jitter noticeably. Teacher target does not switch just because another person enters. Tracking FPS supports 1 FPS and still yields smooth framing. Playwright tests pass and cover persistence + target stability. |
… tracking support Co-authored-by: Axoled-Student <235039939+Axoled-Student@users.noreply.github.com>
Implemented all requested fixes in commit de07dc1: Camera Persistence: Camera selection now persists across sessions with smart fallback (deviceId → groupId → label → facingMode). Auto-start settings with "Tap to Resume" overlay for iOS. Jitter Reduction: Added One Euro filter, dead zones, zoom hysteresis, and new "Stability" slider (0-100%). Target Continuity: Implemented appearance-based re-ID using HSV color histogram. Target only switches after 2s lost or 1s consistently-better candidate. 1 FPS Tracking: Inference rate now supports 1-30 fps with smooth time-based interpolation. Tests: 47 tests passing (12 new tests).
Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
|
@copilot You are a senior web engineer. Fix a production bug in an existing repo (do not “warn it may freeze”; you must make it not freeze). Repo: https://github.com/Axoled-Student/Camera-Track Problem (real iPhone, 後置三鏡頭相機 / rear triple-camera): You MUST check out and inspect the repo/branch first. Identify the current camera-selection logic, any auto lens switching logic, zoom/crop logic, and recording pipeline. Then implement the fixes. Do not guess. A. Required Outcomes (no excuses / no warnings)
B. Implementation Strategy (must implement; do not downgrade features) B1) Introduce a “Lens & Zoom Controller” state machine (centralize all zoom/lens decisions)
B2) Prefer optical/hardware zoom when available; do not use crop as primary zoom if optical is supported
B3) Completely eliminate the lens-switch thrash that causes freezesRoot cause is frequent getUserMedia restarts / device switching due to zoom threshold oscillation and/or unstable target detection.
B4) Make tracking smooth and stop jitter (bbox + framing)
B5) Orientation-aware recording (must auto match rotation)
B6) Persist behavior across sessions
C. Tests (Playwright) — must add/extend Required tests:
D. Deliverables
Acceptance criteria: Verified on real iPhone rear triple-camera: enabling tracking + zoom no longer freezes/black-screens/shutters; framing is smooth; rotation produces upright recordings; no regressions. |
…nd orientation-aware recording Co-authored-by: Axoled-Student <235039939+Axoled-Student@users.noreply.github.com>
Fixed in commit 22af5db. Root Cause: The previous code called Fixes Implemented:
60 tests passing, CodeQL scan clean. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Fix: iOS Lens-Switching Freeze (iPhone Triple-Camera)
This commit addresses the production bug where tracking + auto zoom on iPhone rear triple-camera caused freezes, black frames, and jitter loops.
Root Cause
The previous code called
applyConstraints({ zoom })every frame during tracking, which caused iOS Safari to repeatedly restart the camera stream, leading to freezes and black frames.Changes Made
1. Zoom Controller (Anti-Thrash)
applyConstraintscalls per second2. Stream Locking
3. Orientation-Aware Recording
matchMediaandorientationchangeeventsTechnical Details
CONFIG.zoom.maxApplyRate = 100msCONFIG.zoom.rampSpeed = 0.15CONFIG.zoom.deadband = 0.03CONFIG.zoom.maxDeltaPerUpdate = 0.1CONFIG.stream.switchCooldown = 4000msCONFIG.stream.minDwellTime = 2500msTesting
Expected Behavior After Fix
Original prompt
Role
You are a senior web engineer building a production-quality static HTML camera web app. Prioritize iOS Safari compatibility. Deliver a working project with clean code, clear documentation, and automated tests.
Goal
Build a static website (plain HTML/CSS/JS, no backend) that:
1. Opens the device camera (especially iPhone).
2. Runs on-device person tracking (teacher tracking) in the browser using a lightweight web ML model (e.g., MediaPipe Tasks Vision person/pose detection or TFJS MoveNet—choose the most reliable for iOS Safari).
3. Automatically auto-frames the teacher: smooth pan + digital zoom (crop) to keep them centered with headroom.
4. Attempts to use the iPhone’s multi-camera lenses (0.5× / 1× / 3×) by switching cameras when possible, and/or by using track zoom constraints when available.
5. Records video easily (with audio option), then provides instant download of the recording.
6. Caches the ML model in the browser so it does not re-download every time (Service Worker + Cache API / IndexedDB).
7. Is mobile-friendly, with a clear Settings UI and a live overlay box around the tracked person.
8. Is optimized (battery/performance) and includes Playwright MCP tests to catch regressions.
Important Reality Constraints (must handle gracefully)
• iOS Safari limitations: you usually cannot directly “force lens 0.5×/1×/3×” with a simple API. Sometimes lenses appear as separate camera devices; sometimes only one back camera is exposed. MediaStreamTrack.getCapabilities() may or may not include zoom.
Requirement: implement best-effort lens selection:
• Enumerate devices after permission is granted (labels available).
• Prefer back cameras and detect “Ultra Wide / Wide / Telephoto” if labels exist.
• If zoom capability exists, use applyConstraints({ advanced: [{ zoom: ... }] }).
• If neither is possible, implement high-quality digital zoom via canvas crop.
• Always show a UI status telling the user what’s actually supported on their device.
• App Clip: Apple App Clips are native iOS features. A pure static website cannot “be an App Clip” by itself.
Requirement: deliver one of these approaches (choose the most realistic and document it clearly):
1. A minimal native iOS App Clip wrapper (Swift + WKWebView) that loads the static web app bundled or hosted, OR
2. If native is out of scope, deliver a “Web App Clip-like” experience: minimal installable PWA (Add to Home Screen) with offline caching and a short launch path, and clearly document that true App Clip requires native.
Functional Requirements
Camera & UX
• Single page app: index.html with:
• Live camera preview.
• A visible overlay (canvas) showing bounding box around the tracked person.
• Controls:
• Start/Stop camera
• Toggle tracking
• Toggle overlay box
• Start/Stop recording
• Download button appears immediately after recording stops
• Camera selector (front/back and any detected lenses)
• Resolution/FPS selector (safe presets)
• Audio on/off
• Stabilization/smoothing slider
• Framing mode: “Center”, “Rule of Thirds”, “Headroom”
• “Target lock” option (pick the largest person or tap-to-select target)
• Cache status: model cached / not cached + “Clear cache” button
• iOS-friendly UI:
• Large tap targets, bottom-sheet style settings panel, safe-area insets, no tiny controls.
• Works in portrait and landscape.
Person Tracking
• Use a robust browser ML solution:
• Preferred: MediaPipe Tasks Vision (WASM) for PoseLandmarker or ObjectDetector/PersonDetector (choose the best for stable person boxes).
• Must run fully in the browser and support caching of model assets.
• Tracking behavior:
• Detect people each frame (or at a controlled rate like 10–15 fps).
• Choose the “teacher” target:
• Default: largest bounding box (area) in the frame.
• Optional: tap on a detected person to lock target; keep lock until lost for N frames.
• Provide bounding box coordinates and confidence.
• Overlay:
• Draw bounding box + confidence.
• Show “Locked” indicator if target locked.
Auto-Framing (Smooth Pan/Zoom/Crop)
• Implement “virtual PTZ” by rendering the camera frame to an offscreen canvas:
• Compute a target crop rectangle that keeps the person centered with padding.
• Apply smoothing (e.g., exponential smoothing or critically damped spring) to avoid jitter:
• Smooth both center point and zoom level.
• Clamp zoom range to avoid excessive pixelation.
• Maintain aspect ratio matching the output recording size.
• The user should see a stable framed view that follows the teacher smoothly as they move left/right.
Multi-Lens / Camera Switching (0.5× / 1× / 3×)
• Best-effort strategy:
• Enumerate navigator.mediaDevices.enumerateDevices() after permission.
• Build a list of candidate video inputs and label them (Front / Back / Ultra Wide / Wide / Telephoto when available).
• If zoom capability exists, treat it as “lens-like” and map:
• 0.5× ≈ zoom 0.5 if supported (often minimum is 1; handle gracefully)
• 1...
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.