Immersive OpenXR goggle mode for Quest / Android XR - #2
Closed
iflyhere wants to merge 24 commits into
Closed
Conversation
Presents the stream as a head locked compositor quad layer instead of a 2D panel that the headset resamples. The decoder writes straight into an Android surface swapchain (XR_KHR_android_surface_swapchain), so the video path loses the view hierarchy, SurfaceFlinger and the panel's system-chosen resolution. There is no GL blit and no readback: the EGL context only exists because OpenXR requires a graphics binding. New module :app:xr - XrGoggleSession (native): loader init, instance, session, LOCAL + VIEW reference spaces, the android surface swapchain, action set, frame loop. - Head locked by default (quad in VIEW space); world locked mode anchors a yaw-only pose in LOCAL space and can be recentered. - XR_FB_composition_layer_settings quality sharpening on the video layer. - XR_FB_passthrough as an underlay so the pilot can see the radio, a spotter and the ground without taking the headset off. - XR_FB_display_refresh_rate, "auto" picks the highest the headset offers. - Touch controller bindings: A recenter, B passthrough, X record, Y head lock toggle, right stick screen size, left stick distance. Simple controller bindings as a fallback. - Controller haptics on link degradation - a warning a goggle cannot give. - Every optional extension is probed; a runtime without them still flies. - OpenXR loader comes from org.khronos.openxr:openxr_loader_for_android via prefab, so no vendor SDK is vendored into the tree. New XrVideoActivity - Resolves USB permission while still flat (a permission dialog is much easier to deal with before the compositor takes over), then starts the session. - Sizes the swapchain from the last seen stream resolution so the compositor never resamples twice; updates the layer's imageRect live. - DVR toggle on X using the already granted folder; picking a folder still needs flat mode. - No teardown in onPause/onStop: the runtime pauses and resumes an immersive session for its own reasons, and dropping USB there costs seconds of black. - Any failure falls back to VideoActivity with the reason. Supporting changes - WfbLinkManager reported status through ActivityVideoBinding, which only the flat activity has. It now takes a LinkStatusView, implemented by both. - DVR file naming/creation moved from VideoActivity into DvrFiles so both activities share one implementation. - Settings -> VR goggles (OpenXR): enter, head lock, passthrough, sharpening, refresh rate. The whole submenu is hidden unless the device looks like a headset, so nothing changes on a phone.
Passthrough was started at session begin and merely withheld from the frame's layer list when disabled, so the cameras and the reconstruction pipeline kept running for the whole flight. On a headset that is already thermally tight and on battery that is a real cost for nothing. xrPassthroughStartFB / xrPassthroughPauseFB now follow the setting, checked once per frame before the layers are submitted.
wirelessInfo() is static and read the static WifiManager field, which only
VideoActivity.initializeUI() ever assigns. WfbLinkManager calls it from the
"no compatible wifi adapter found" branch, so any other host of
WfbLinkManager - the immersive activity - crashed there with
NullPointerException: WifiManager.getConnectionInfo() on a null object
at VideoActivity.wirelessInfo(VideoActivity.java:183)
at WfbLinkManager.refreshAdapters(WfbLinkManager.java:174)
Confirmed on a Quest 3 (Horizon OS, Android 14): launching XrVideoActivity
with no adapter attached killed the process in onResume before the OpenXR
session could start.
wirelessInfo() now takes a Context, fetches the service itself and null
checks both the manager and the WifiInfo, and the static field is gone - so
this cannot come back through a different entry point.
Horizon OS routes an activity declared with com.oculus.intent.category.VR
through RequiresControllersLaunchInterceptor and refuses to start it while the
controllers are off:
ActivityLaunchInterceptorController: RequiresControllersLaunchInterceptor:
com.openipc.pixelpilot/.XrVideoActivity ->
com.oculus.vrshell/...LaunchCheckControllerRequiredDialogActivity
An app is treated as controller-required unless it advertises hand tracking.
For a ground station that is the wrong default: the pilot is holding a radio,
and video, link and DVR do not need a controller - only the optional in-flight
shortcuts do. Declaring oculus.software.handtracking as required="false" lets
the session start regardless, and the controller bindings still work when the
controllers are on.
The generic Khronos loader cannot find a runtime on a Quest 3, verified on the
device (Horizon OS, Android 14):
- /system/etc/openxr/1/active_runtime.json does not exist
- the Khronos broker authorities are registered but resolve to
internal.horizonos.openxr.DisabledRuntimeBroker /
DisabledRuntimeSystemBroker, and querying
content://org.khronos.openxr.runtime_broker/openxr/1/abi/arm64-v8a/runtimes/active
returns "No result found"
- libopenxr_loader.so from openxr_loader_for_android has no reference to
libopenxr_forwardloader.so at all
The runtime is reached through libopenxr_forwardloader.so in the VrDriver
APEX. Its lib directory is added to every app's linker namespace, and its only
exports are xrNegotiateLoaderRuntimeInterface and xrInitializeLoaderKHR - so
as far as a loader is concerned it *is* the runtime.
resolveRuntime() therefore writes a runtime manifest into the app's files dir
and sets XR_RUNTIME_JSON, trying in order:
1. nothing at all, so a device whose broker does work is left alone
2. library_path "libopenxr_forwardloader.so" (bare name = system search path)
3. the absolute APEX path, for a loader that insists on one
Each candidate is probed by asking for the instance extension list and
requiring XR_KHR_android_surface_swapchain to be in it, so "the call did not
fail" is not mistaken for "a runtime answered". Which candidate won is logged.
Two things VideoActivity.onCreate() does that XrVideoActivity did not, both
found on a Quest 3:
1. gs.key. The native aggregators open files/gs.key in their constructor and
throw if it is missing. VideoActivity copies the bundled key there before
constructing WfbNgLink; the immersive activity did not, so a fresh install
aborted the process on launch:
libc++abi: terminating due to uncaught exception of type
std::runtime_error: Unable to open
/data/user/0/com.openipc.pixelpilot/files/gs.key
Fatal signal 6 (SIGABRT) ... XrVideoActivity.onCreate
2. Adaptive link, TX power and FEC/LDPC/STBC. The native side does not read
these itself, so without initDefaultOptions() they were silently ignored in
immersive mode.
Both moved into GsKey and WfbOptions and used from both activities, so there is
one implementation rather than a copy that can drift.
Also hardened, because a missing key should never be fatal: WfbngLink::initAgg()
catches the aggregator constructor's exception, logs the key path and leaves the
aggregators null instead of letting it escape a JNI call and abort the process.
nativeRefreshKey() re-runs initAgg(), so fixing the key recovers without a
restart.
And XrGoggleSession now logs the runtime's full instance extension list, which
is the only way to tell from a bug report what a given headset offers.
Two findings from a Quest 3 (Horizon OS, Android 14). 1. xrCreateSwapchainAndroidSurfaceKHR returned XR_ERROR_VALIDATION_FAILURE. The extension documents format/sampleCount/faceCount/arraySize/mipCount as ignored for an Android surface swapchain, but Meta's runtime validates them, and the obvious 1s are rejected. Rather than swap one guess for another, createSwapchain() now tries the plausible shapes - ignored fields zeroed, set to 1, with and without COLOR_ATTACHMENT, with and without an explicit RGBA8 - and logs which one the runtime accepted. If all fail it dumps xrEnumerateSwapchainFormats so the next attempt has data instead of guesses. 2. passthrough was missing from the instance extension list on a Quest 3. Horizon OS only exposes XR_FB_passthrough to an app that declares com.oculus.feature.PASSTHROUGH in its manifest. Added as required="false", so nothing changes for a phone. Also corrects the reasoning in resolveRuntime(): the Khronos loader *does* resolve the Horizon runtime on its own, through the org.khronos.openxr.runtime_broker provider, and the "change nothing" candidate wins on a Quest. The earlier conclusion came from querying that provider with `adb shell content query`, which returns no rows only because the shell does not hold org.khronos.openxr.permission.OPENXR. The fallback is kept for a device that offers neither the broker nor active_runtime.json, but it is a fallback, not a workaround for the Quest.
Flying with hand tracking only and no controllers is a normal way to use a headset as a ground station, and the Quest 3 runtime offers XR_EXT_hand_interaction (confirmed in the instance extension list on Horizon OS 14, alongside XR_MSFT_hand_interaction and XR_META_hand_tracking_microgestures). The four boolean actions now also get bindings on /interaction_profiles/ext/hand_interaction_ext. A boolean action may be bound to a float source and the runtime does the thresholding, so no gesture code and no second input path is needed: right pinch -> recenter right grasp -> passthrough left pinch -> toggle DVR left grasp -> head lock Pinch is index-to-thumb and grasp is a whole-hand squeeze, far enough apart not to trigger each other. Controllers keep their A/B/X/Y and thumbstick bindings; whichever is active wins. Screen size and distance stay on the thumbsticks - there is no comfortable continuous hand equivalent. Also adds com.oculus.permission.HAND_TRACKING, which the handtracking feature needs before Horizon OS will actually hand the app tracked hands, and records the now-confirmed swapchain finding in the comment: Meta's runtime wants the "ignored" image description fields zeroed.
With no stream the video quad is an opaque black rectangle, which from inside the headset is indistinguishable from "nothing is being composited at all" - exactly the situation on a first run with no adapter attached, and impossible to diagnose while wearing the headset. The swapchain's producer side is an ordinary Android Surface, so before the decoder takes it over the activity paints into it with lockHardwareCanvas(): a border and a centre cross, so the panel's real extent, centre, size and distance can be judged with nothing else on screen, plus the current link status, channel/bandwidth and the UDP address a stream can be pushed to. It redraws once a second and stops on the first decoded frame, since a buffer queue has a single producer and MediaCodec becomes it. If the canvas is refused it gives up quietly and leaves the layer to the decoder.
Anything producing into an Android Surface - MediaCodec, and the Canvas used for the status screen - has its origin at the top left, while the compositor samples a swapchain image from the bottom left. Confirmed on a Quest 3: the layer composited correctly, in the right place and the right size, just vertically mirrored. XR_FB_composition_layer_image_layout exists for exactly this. The quad now carries XrCompositionLayerImageLayoutFB with VERTICAL_FLIP, chained together with the sharpening settings when both apply. Gated on the extension being present, like the others.
With a head-locked panel its height relative to your natural line of sight matters - too high or too low and you fly with your neck craned - and there was no way to change it. Pinch and grasp are already spent on recenter, passthrough, DVR and the head lock toggle, so the nudge goes on XR_META_hand_tracking_microgestures instead: a thumb swipe along the index finger, which is still an index-finger/thumb control but costs nothing that was already bound. right thumb swipe forward -> panel up right thumb swipe backward -> panel down That extension defines no structs or entry points, only binding paths, so it is suggested as part of the hand profile. Suggestions are atomic per profile, so a rejected microgesture path would take the working pinch/grasp bindings with it; if the call fails the microgesture entries are dropped and it is retried, and which variant took is logged. The controller mapping is reorganised so the same adjustment exists there and is intuitive - the left stick now moves the panel, the right stick sizes it: left stick up/down -> panel up/down (was: distance) left stick left/right -> nearer/farther right stick up/down -> panel size (unchanged) The offset is applied to the quad pose in both head-locked and world-locked mode and persisted as xr_quad_height.
Reported from the headset: the hand gestures work but the right thumbstick does nothing, and the trigger/grip do not move the panel. The likely cause of the dead stick is that only one interaction profile is live at a time. With hands tracked the runtime picks the hand profile and the touch controller bindings never fire, which looks exactly like a broken stick. The runtime offers XR_META_simultaneous_hands_and_controllers, so that is enabled and xrResumeSimultaneousHandsAndControllersTrackingMETA is called once the session starts. To make this visible rather than guessed at, the active interaction profile per hand is now logged on XR_TYPE_EVENT_DATA_INTERACTION_PROFILE_CHANGED and at session start. The mapping is also mirrored so a single controller in either hand is enough, and the trigger/grip take over distance: thumbstick up/down screen up/down thumbstick left/right screen smaller/bigger trigger (analog) pull the screen closer grip (analog) push the screen away A / X (lower button) recenter B / Y (upper button) toggle passthrough thumbstick click toggle recording A/B and X/Y sit in the same places on their respective controllers, so binding both to one action gives the same thumb positions whichever controller was picked up. Trigger and grip are float actions, so the movement is proportional to how far they are pulled. Head lock loses its controller binding - there is no button left that exists on both controllers - and stays on the left-hand grasp gesture and in the flat settings menu, which is where a set-once option belongs.
iflyhere
force-pushed
the
feature/xr-goggle-mode
branch
from
September 2, 2026 16:30
f18fa27 to
6cf71ba
Compare
… of the way Three problems reported from a session with the antenna connected. 1. Gestures triggered by accident - picking up the drone turned passthrough on. A whole-hand squeeze is exactly what grabbing a drone or a radio looks like, so the grasp bindings are gone. Pinch survives but now has to be *held*: separate hand-only actions fire once after 700 ms, so a momentary squeeze while handling the aircraft does nothing. The controller buttons are deliberately left immediate - they cannot be pressed by accident - which is why the hand variants are separate actions rather than a hold on the shared ones. Losing passthrough to a stray grab mid-flight is worse than not having the shortcut at all. Head lock loses its last input binding as a consequence and is now only in the flat settings menu. It is a set-once option, not an in-flight one. 2. Replugging the antenna opened the flat activity next to the running immersive one. The USB_DEVICE_ATTACHED filter lives on VideoActivity, so the system starts it on every attach. Two activities then fight over one adapter and one decoder and neither gets video. VideoActivity now steps aside when an immersive session is active and the launch came from that intent; the immersive activity already handles the attach itself through its own receiver. 3. No video, in either mode, with no crash. Diagnosis needs data the log did not have, so the whole chain is now visible: gs.key logs the byte count it wrote, and the wfb counters (all / decOk / decErr / fecRecovered / lost / bad / rssi) are logged every two seconds and drawn on the status screen. That separates "nothing is arriving from the air" from "packets arrive but nothing decodes", which are completely different bugs. Note for that hunt: WfbngLink::initAgg() no longer aborts the process when gs.key is unusable, so what used to be a loud crash is now silent - the log line is the only signal.
Diagnosed from a thread dump on a Quest 3 while the app was unresponsive:
3 S wfb-001/002 <- three live RX loops, one dongle
1 S nipc.pixelpilot <- main thread asleep in join()
Two libusb handles on the same interface means neither gets usable video, which
is why no picture arrived in either the flat or the immersive mode, and the
unbounded join() in a lifecycle callback turned it into a five second ANR.
How three threads happen:
- activeWifiAdapters in WfbLinkManager was static, so the flat and the immersive
activity shared it while each owned its own WfbNgLink. startAdapters() only
guards on wfbLink.isRunning(), which is per instance, so both decided the
adapter still needed starting.
- linkThreads.put(usbDevice, t) overwrites the previous entry, so the older
thread is orphaned and never joined, and stopAll() only ever joins the last.
- nothing refused a second RX loop for a device that already had one.
Fixes:
- a process-wide claim on the device name, keyed on the owning thread, so a
second RX loop is refused no matter which WfbNgLink instance asks. Keying on
the thread means a claim left behind by a thread that died is ignored instead
of locking the adapter out permanently; the claim is released in a finally
block around nativeRun().
- a per-instance check for an already-live thread, so the common case does not
even reach the claim.
- joins are bounded to 1500 ms and log when a driver thread does not come back.
These calls come from Activity lifecycle callbacks on the main thread, where
an unbounded join is an ANR waiting to happen.
- activeWifiAdapters is no longer static.
The real answer is a single owner for the link - a foreground service - but this
stops the immediate damage.
"No compatible wifi adapter found." was shown whenever no adapter ended up running, which covers three unrelated problems: nothing plugged in, a dongle whose id is not in usb_device_filter.xml, and a dongle that was found but failed to start. Only the middle one is what the message claims, and the last one became reachable when OpenIPC#116 stopped recording an adapter as active unless it actually came up. - getAttachedAdapters() logs every attached device with VID:PID, manufacturer and product name, and whether the filter matched. That id is exactly what a bug report needs in order to add an adapter, and there is no other way to read it - sysfs is not accessible to the shell on Horizon OS and dumpsys usb does not print host devices there. - the message now distinguishes "none attached", "found but could not be started - see the log" and "waiting for permission". Also: joinBounded() now releases the device claim even when the driver thread outstays the timeout. Holding it would leave the adapter unopenable until the process restarts, and the user would be told there is no compatible adapter - a worse failure than the duplicate RX loop it guards against, which the per-instance check and the single-owner handoff already cover.
Flat mode has video again once USB permission is granted, but entering immersive mode then reported that the adapter could not be opened. The handoff is the problem. VideoActivity.onPause() calls stopAll(), which asks the driver to stop and waits for the thread. StopRxLoop() only breaks the receive loop - the thread then still has to stop the TX frame, stop the adaptive link, power the chip down, release the USB interface and exit libusb. That takes longer than the 1500 ms bound added with the ANR fix, so the wait gave up, the immersive activity tried to claim an interface that was still held, failed once, and gave up too. - the join bound goes to 3000 ms, which is about what the unwind needs, while still being bounded so a stuck driver cannot freeze the UI. - the immersive activity retries bringing the adapter up every 700 ms for up to 14 s instead of trying once, and logs each attempt. The session is started regardless, so the pilot gets the status screen with a reason rather than a silent flat activity. - onXrReady() no longer calls startAdapters(): refreshAdapters() already brings adapters up and now retries, and two paths racing for one dongle is what caused the duplicate RX loops in the first place. - WfbLinkManager gained hasActiveAdapter() so a caller can tell "up" from "not up yet" rather than inferring it. Also: VideoActivity.showLinkMessage() now logs. It only ever reached the overlay, so a bug report had no record of what the app said about the link - a gap that cost real time in this hunt.
… by nobody With video working in flat mode, switching to immersive mode and back left no video in either. The activity dump explains it: topResumedActivity: .VideoActivity t5249 topResumedActivity: .XrVideoActivity t5253 threads: xr-goggle (no wfb-* at all) Both activities were alive in separate tasks - Horizon OS keeps the immersive one in its own task, so its onDestroy(), and with it stopAdapters(), never ran - while the flat one had also given the adapter up. Each activity constructs its own WfbNgLink, so with two of them the adapter belongs to neither. ModeOwner makes that impossible: whoever starts last evicts the other, whose onDestroy() releases the adapter. A WeakReference so a finished activity is not retained. The real answer is a single owner outliving both - a foreground service, which would also fix the flat mode dropping the link in onPause - but this removes the failure without that rework. Also corrects a wrong reading in the previous commit: three threads named "wfb-001/002" is *normal* for one adapter, because the TX and adaptive-link threads inherit the RX thread's name - nothing renames them. It is not three RX loops. The duplicate guard is still correct, since two activities each with their own WfbNgLink genuinely can double-start, but the evidence quoted for it was misread. Confirmed on hardware in the same run: the MediaCodec low-latency keys from OpenIPC#113 are accepted by Meta's H.265 decoder - "low-latency: 1, vendor.qti-ext-dec-low-latency.enable: 1, priority: 0" followed by "AMediaCodec_configure: OK".
The adapter, the wfb-ng link and the decoder belonged to whichever Activity was on screen. That does not survive contact with a ground station: - the flat activity tore the link down in onPause(), so a system menu, a file picker or taking the headset off cost seconds of black screen and a fresh USB enumeration - with a second presentation mode there were two owners for one adapter. Both activities stayed alive in separate tasks (Horizon OS keeps the immersive one in its own task, so its onDestroy never ran) and the adapter ended up owned by neither - no video in either mode - every mode switch reopened the adapter, which is also what made the handoff racy enough to need a claim guard, a bounded join and a retry loop LinkService now owns all three and outlives both activities. Activities bind for their whole lifetime, register as a LinkService.Client and hand over a Surface. Switching modes only swaps the decoder's output surface: reception and the adapter are never touched, so there is no reconnect and no second USB permission prompt. It is a foreground service with a Stop action, because it has to keep running while the pilot is looking at something else. - LinkClientActivity: shared bind/unbind, bound in onCreate and unbound in onDestroy rather than around start/stop, which is the point. - VideoPlayer gained attachSurface()/detachSurface(): a surface change rebuilds the decoder without stopping the receivers. stopAndRemoveReceiverDecoder() stopped the receivers too and nothing restarted them, which is why a hot surface swap was not possible before. - VideoPlayer and WfbNgLink take a Context instead of an AppCompatActivity. A decoder and a radio link have no business requiring an Activity, and the service cannot provide one. - the USB attach/detach receiver moved to the service: WfbLinkManager is that receiver, and it belongs to whoever owns the adapter. Registered by an activity, plug events were missed whenever none was around. - the flat activity no longer stops the link in onPause/onStop, and no longer tears it down to show a folder picker. - the service replays link status, resolution and stats to a client on attach, so a freshly opened mode never starts out blank. ModeOwner stays: one mode at a time is still the right model, and two surfaces on index 0 would fight.
Smoke test of the service refactor: the link came up and packets arrived, but the decoder logged "inputPipeClosed." on every NALU - it had no output surface. bindService() is asynchronous and the view hierarchy does not wait for it, so SurfaceView.surfaceCreated() fires before onServiceConnected(). The callback found link == null and silently dropped the surface, and nothing handed it over afterwards. The surface is now parked and attached either immediately or as soon as the service connects, whichever comes second. The immersive activity is unaffected - its surface comes from the XR session, which is only started from onLinkServiceConnected().
There was none: every controller button and both pinch gestures were already bound, so the only way back to the flat view was launching it from the launcher and letting ModeOwner evict the session. - left menu button, the conventional app button and the last one still free - right thumb swipe left, so it also works without controllers, next to the swipe forward/backward that moves the panel A single gesture rather than a held one is fine here because leaving is now cheap: the link lives in LinkService, so switching back to the flat view keeps the adapter open and the video running - a stray trigger costs a mode switch, not a reconnect. The status screen also lists the controls now. It is the only surface the immersive mode has, and with everything on unlabelled buttons there was no way to discover any of them.
BufferedPacketQueue reorders the RTP stream before it reaches the parser. When a packet is missing it holds everything behind the gap, which is right for a reorder and wrong for a loss - and past wfb-ng, on a loopback socket, a gap is almost always a loss that FEC could not recover. Three things made that wait longer than it needs to be. The monotonic-increase escape hatch, which exists precisely to notice "the sequence numbers keep climbing but the gap is not filling", never fires. calculateDistance(a, b) is how far b is ahead of a - seqLessThan reads it that way - but the call site asks for the distance from the incoming packet to the last delivered one. For a gap ahead of us that is negative, so the else branch clears the counter on every single packet. Fixing the argument order alone is not enough: the counter is only incremented while abs(dist) < MONOTONIC_THRESHOLD, and dist grows by one with every packet held back. Compared against the same constant that gates it, the counter tops out at three and can never reach five. The gate is now its own constant, so a run past a gap releases the buffer after five packets instead of never. That leaves MAX_BUFFER_SIZE as the only bound, and how much latency fifteen packets is depends entirely on the packet rate: about 20ms on a 1080p video stream, but roughly 300ms on the audio stream, which runs at a fraction of it. The buffer is now also bounded in time, so the worst case is the same on both. Twenty milliseconds is a little over one frame at 60fps and several orders of magnitude more than a loopback socket needs to reorder anything. Nothing is dropped by any of this - the buffered packets are still delivered, just without waiting on one that is not coming. The test target could not be built on a case-sensitive filesystem, since CMakeLists.txt spelled the source BufferedPacketqueue_test.cpp. The existing two cases both turn out to feed a strictly in-order stream, so neither reached the buffer at all; the new ones cover a permanent gap, the age bound, a reorder that resolves inside it, and a jump too large to be one. They drive the clock themselves rather than reading it, so the age bound cannot make them flaky on a loaded machine.
iflyhere
force-pushed
the
feature/xr-goggle-mode
branch
from
September 2, 2026 18:14
e147b32 to
7d693e4
Compare
The immersive activity carries its own MAIN/LAUNCHER entry so Horizon OS can start the app straight into VR. On a phone that is a second icon which leads to an activity that cannot create a session and immediately hands back, and there is no manifest condition for "this device has an XR runtime". So it ships disabled and VideoActivity.onCreate switches it on where isSupportedDevice() agrees. A phone install is then indistinguishable from before this feature existed. On a headset the first launch is the flat activity, which enables the entry, and the VR entry is in the library from then on. Verified on a Quest 3: after install only .VideoActivity resolves for MAIN/LAUNCHER and nothing resolves for the VR category; after one flat launch .XrVideoActivity resolves for both, and a second launch is a no-op rather than another package-manager write. The menu item and the entry are gated on the same predicate, so an XR device that isSupportedDevice() fails to recognise gets neither - widening that check is then the fix.
Two problems with the flush, both found in review. restartBuffering() set mLastPacketIdx to the packet that triggered the flush rather than the newest one delivered. With 1..5 delivered, 6 lost, then 7 8 9 12 10, the flush hands over 7 8 9 10 12 but leaves the pointer on 10 - so 11 would be delivered after 12, and 13..16 are held until 17 arrives. The queue re-created the stall it exists to prevent. bufferPacket() always runs before the flush, so the triggering packet is in the buffer either way and mLastPacketIdx = drainBufferInOrder() is all that was needed; restartBuffering() is gone. That alone would have broken the resync on a large jump, because drainBufferInOrder() seeded `highest` from mLastPacketIdx and compared with calculateDistance(), which reads as negative past half the sequence space. RTP starts at a random sequence number, so a VTX that reboots mid-session lands exactly there: fed 1, 2, then 40000, 40001, ..., neither the seed nor the comparison ever moves and every packet comes out one flush late, forever. On video the fifteen-packet cap used to paper over it by rewinding to currPacketIdx; on audio nothing did. `highest` is now seeded from a packet that is actually in the buffer. While sorting the flush: by raw value a block straddling the wrap point (65534, 65535, 0, 1) sorts to 0, 1, 65534, 65535 and was handed to the parser in that order. Sorting by distance from the last delivered packet fixes the order and makes the last element the newest. Three tests added, one per problem. The large-jump case uses 40000 - the existing LargeJumpFallsBackToTheBufferCap uses 30000, which is still positive as an int16 and does not reach any of this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Working on a Quest 3, not yet flown. Session, layer orientation, head lock,
passthrough and hand-tracking input are all confirmed by eye on hardware (see Hardware
status). What is still untested is a live video stream through the layer - that needs the
wifi adapter. Fork-only; not for upstream until it has flown.
Why
On a headset PixelPilot runs as a 2D Android panel. That costs three things at once:
1080p is downsampled to the panel, then the compositor resamples that quad onto the
display — two resampling steps for a stream that was already the bottleneck.
glued to your view like a pair of goggles.
What this does instead
xrCreateSwapchainAndroidSurfaceKHRhands out an AndroidSurfacethat is the producerside of a compositor swapchain. MediaCodec decodes straight into it and the runtime
composites it as a quad layer. The view hierarchy, SurfaceFlinger and the panel resolution
leave the video path entirely. There is no GL blit and no readback — the EGL context exists
only because OpenXR requires a graphics binding for the session.
New module
:app:xrXrGoggleSession(native, ~950 lines) + a thin Java wrapper that owns the frame thread.XrCompositionLayerQuad, head locked inXR_REFERENCE_SPACE_TYPE_VIEWLOCALwith a yaw-only pose, recenterableXR_FB_composition_layer_settings,QUALITY_SHARPENING_BIT_FBXR_FB_passthroughas an underlay — see the radio, a spotter, the ground. Started on demand, not at session begin, so the cameras are not running all flightXR_FB_display_refresh_rate; "auto" takes the highest offered/interaction_profiles/oculus/touch_controller,khr/simple_controllerfallbackControls: A recenter · B passthrough · X DVR record · Y head-lock toggle ·
right stick screen size · left stick distance.
Every optional extension is probed at instance creation and the session runs without any of
them. The three that are genuinely required (
XR_KHR_android_create_instance,XR_KHR_opengl_es_enable,XR_KHR_android_surface_swapchain) produce a clean error and afallback instead of a crash.
The OpenXR loader comes from
org.khronos.openxr:openxr_loader_for_androidthrough prefab,so no vendor SDK is vendored into the tree and the ABI set already matches the project's
(
arm64-v8a,armeabi-v7a).New
XrVideoActivitya permission dialog is much easier to deal with before the compositor takes over.
flight the swapchain matches the stream exactly and nothing is resampled twice. The
layer's
imageRectis updated live from the decoded resolution.result, so that stays in flat mode.
onPause/onStop. An immersive runtime pauses and resumes the app forits own reasons (system menu, proximity sensor), and dropping the USB link there costs
seconds of black screen on the way back.
VideoActivitywith the reason in a toast.Supporting changes
WfbLinkManagerreported status throughActivityVideoBinding, which only the flatactivity has. It now takes a
LinkStatusView; both activities implement it. No behaviourchange for the flat path.
VideoActivityintoDvrFilesso bothactivities share one implementation rather than duplicating it.
Nothing changes on a phone
like a headset (
android.hardware.vr.headtracking,android.software.xr.immersive, orthe Horizon shell package).
android.hardware.vr.headtrackingis declaredrequired="false", so the APK stillinstalls and runs everywhere.
libXrGogglefailing to load is caught, and the flat app carries on.Known rough edges
com.oculus.intent.category.VRcategory without disturbing the flat activity'sUSB_DEVICE_ATTACHEDfilter. On a phone that means two icons. Before this could goupstream it wants a product flavour so the mobile APK only ships one.
before putting the headset on. A second android-surface swapchain fed by
Surface.lockHardwareCanvas()would let the existing OSD views be composited as their ownquad at native resolution — that is the obvious next step.
unverified.
Hardware status
Working on a Quest 3, Horizon OS, Android 14 (API 34). Confirmed by eye in the headset:
XR_SESSION_STATE_FOCUSED, frame loop runsoculus.software.handtracking+com.oculus.permission.HAND_TRACKINGWhat the hardware brought up
wirelessInfo()NPEWifiManageronly ever assigned byVideoActivity.initializeUI(); any other host ofWfbLinkManagercrashed inonResumeSIGABRTon launchfiles/gs.keyin their constructor and throw; onlyVideoActivitywas provisioning itinitDefaultOptions()had no equivalent in the immersive pathxrCreateSwapchainAndroidSurfaceKHR→XR_ERROR_VALIDATION_FAILUREformat/sampleCount/faceCount/arraySize/mipCountas ignored, but Meta's runtime validates them and wants them zeroedSurfacehas its origin top-left, the compositor samples bottom-left. Fixed withXR_FB_composition_layer_image_layout+VERTICAL_FLIPXR_FB_passthroughabsent from the extension listcom.oculus.feature.PASSTHROUGHRequiresControllersLaunchInterceptorOne earlier conclusion in this branch's history was wrong and is corrected in the code: the
Khronos loader does resolve the Horizon runtime by itself, through the
org.khronos.openxr.runtime_brokerprovider. Querying that provider fromadb shellreturns no rows only because the shell does not hold
org.khronos.openxr.permission.OPENXR.resolveRuntime()is kept as a fallback for adevice offering neither the broker nor
active_runtime.json; on a Quest its firstcandidate ("change nothing") wins.
Update — launcher entry and packet queue
The immersive activity's launcher entry now ships disabled (
android:enabled="false") andVideoActivity.onCreateturns it on only whereXrGoggleSession.isSupportedDevice()agrees.That was the last thing keeping this branch from being upstreamable: a phone install is now
indistinguishable from before the feature existed, with no second icon.
Verified on a Quest 3 — after install only
.VideoActivityresolves forMAIN/LAUNCHERandnothing resolves for
com.oculus.intent.category.VR; after one flat launch.XrVideoActivityresolves for both, and a second launch is a no-op rather than another package-manager write.
Also merged in the
BufferedPacketQueuefix, which went upstream separately asOpenIPC#120 — a lost packet used to hold back everything behind it until the buffer
hit fifteen packets, because the escape hatch meant to catch exactly that had its arguments
reversed and was gated on the constant it was compared against.
Measured, for the record
Glass-to-glass, photographed through the lens with a millisecond counter visible both in the
FPV image and in passthrough: the FPV image runs ~87 ms behind passthrough. Relative, not
absolute — passthrough has its own latency on top.
The decoder's own averages over a 138k-frame session on this branch:
Parsingis not CPU time —creationTimeis set when the first RTP packet of a NALUarrives, so it is one frame's packetization spread on the wire.
WaitInputBufferat 0.55 msmeans the decoder is never starved for input buffers. So the decode path totals ~19 ms and has
no large win left in it; the rest sits upstream of this app, in the camera encoder and the
wfb-ng FEC block.