Skip to content

Bound how long the packet queue holds a frame back - #120

Merged
vertexodessa merged 2 commits into
OpenIPC:masterfrom
iflyhere:fix/buffered-queue-latency
Sep 2, 2026
Merged

Bound how long the packet queue holds a frame back#120
vertexodessa merged 2 commits into
OpenIPC:masterfrom
iflyhere:fix/buffered-queue-latency

Conversation

@iflyhere

@iflyhere iflyhere commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

BufferedPacketQueue reorders the RTP stream before it reaches the parser. When a packet is
missing it holds back everything behind the gap — correct for a reorder, wrong for a loss. And
by the time packets reach it they have already been through wfb-ng's FEC and a loopback
socket, so a gap here is almost always a packet FEC could not recover, not one that is about
to turn up. Three separate things made that wait longer than it needs to be.

The monotonic-increase escape hatch never fires

It exists precisely to notice "the sequence numbers keep climbing but the gap is not filling":

auto dist = calculateDistance(currPacketIdx, mLastPacketIdx);
if (std::abs(dist) < MONOTONIC_THRESHOLD)
{
    if (dist > 0) { mMonotonicOutOfOrderIncreaseCount++; ... }
    else          { mMonotonicOutOfOrderIncreaseCount = 0; }

calculateDistance(a, b) returns how far b is ahead of a — that is how seqLessThan just
below it reads the same function. So asking for the distance from the incoming packet to the
last delivered one gives a negative number for a gap ahead of us, which is the only case
this heuristic exists for. The else branch runs instead, and clears the counter on every
single packet.

Swapping the arguments back is not enough on its own. The counter is only incremented while
abs(dist) < MONOTONIC_THRESHOLD, and dist grows by one with every packet held back — so
gated on the same constant it is compared against, the counter tops out at three and can never
reach five. The gate now has its own constant.

Fifteen packets is a different amount of time on each stream

That left MAX_BUFFER_SIZE as the only bound, and what it costs depends entirely on the
packet rate: roughly 20 ms on a 1080p video stream, but around 300 ms on the audio stream,
which runs at a fraction of it. The buffer is now bounded in time as well, so the worst case
is the same on both. 20 ms is a little over one frame at 60 fps, and several orders of
magnitude more than a loopback socket needs to reorder anything.

Nothing is dropped by any of this — the buffered packets are still delivered, just without
waiting on one that is not coming.

The tests

CMakeLists.txt spelled the source BufferedPacketqueue_test.cpp, so the target could not be
configured on a case-sensitive filesystem. With that fixed, both existing cases turn out to
feed a strictly in-order stream, so neither ever reached the buffer they were meant to cover.

The new cases cover a permanent gap, the age bound, a reorder that resolves inside it, and a
jump too large to be one. They drive the clock themselves instead of reading it, so the age
bound cannot make them flaky on a loaded runner.

The permanent-gap case is a straight regression test. Against unmodified master — one packet
lost, then six more arriving in order:

Expected equality of these values:
    Which is: { 1, 2, 3, 4, 5 }
    Which is: { 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12 }

Nothing behind the gap comes out at all. With this change all six pass:

1/6 Test #1: BufferedPacketQueueTest.WrapAroundDelivers ..................................   Passed
2/6 Test #2: BufferedPacketQueueTest.ReorderedDeliversInOrder ............................   Passed
3/6 Test #3: BufferedPacketQueueTest.PermanentGapDoesNotHoldTheStreamForFifteenPackets ...   Passed
4/6 Test #4: BufferedPacketQueueTest.StaleBufferIsFlushedOnTimeout .......................   Passed
5/6 Test #5: BufferedPacketQueueTest.ReorderWithinTimeoutIsStillPutBackInOrder ...........   Passed
6/6 Test #6: BufferedPacketQueueTest.LargeJumpFallsBackToTheBufferCap ....................   Passed
100% tests passed, 0 tests failed out of 6

The suite is a plain host gtest target — no NDK, no submodules, about a minute:

cmake -S app/videonative/src/main/cpp/tests -B build && cmake --build build -j && ctest --test-dir build

Scope

This is a jitter fix, not a throughput one. On a clean link the queue delivers every packet
straight through and none of this runs; it is the behaviour on loss that changes. For
reference, the decoder's own averages over a 138k-frame session on a Quest 3 are
Parsing: 11.9 ms | WaitInputBuffer: 0.55 ms | Decoding: 6.4 ms, so the decode path itself is
not where time goes — but the queue sits in front of all three and its holding time is not
counted in any of them.

Compile tested for arm64-v8a + armeabi-v7a.


Part of a series of independent fixes found while building an immersive (OpenXR) mode on a
Quest 3, each standalone and mergeable in any order:

#113 and #116 are now confirmed on hardware (Quest 3, Horizon OS, Android 14).

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

Bound BufferedPacketQueue latency on unrecoverable RTP gaps

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Caps packet hold latency at 20 ms before draining stalled reorder buffers.
• Repairs monotonic gap detection to release streams after five advancing packets.
• Adds deterministic loss, timeout, reorder, and jump tests; fixes case-sensitive test builds.
Diagram

graph TD
  A["RTP Packet"] --> B{"Buffer stale?"}
  B -- Yes --> C["Drain buffer"] --> D{"Packet expected?"}
  B -- No --> D
  D -- Yes --> E["Deliver in order"]
  D -- No --> F["Buffer packet"] --> G{"Release bound hit?"}
  G -- Yes --> C
  G -- No --> H["Await packet"]
Loading
High-Level Assessment

The synchronous, arrival-driven timeout is appropriate for this queue: it bounds observable packet latency without adding timer threads, asynchronous callback delivery, or lifecycle coordination. Retaining monotonic and size bounds provides fast loss recovery and a fallback for large sequence jumps, while the injected time point keeps tests deterministic.

Files changed (3) +159 / -12

Bug fix (1) +85 / -9
BufferedPacketQueue.hBound reorder stalls by time and repair monotonic gap detection +85/-9

Bound reorder stalls by time and repair monotonic gap detection

• Adds a 20 ms steady-clock age limit and drains stale buffered packets before processing new arrivals. Corrects sequence-distance direction, separates the monotonic distance gate from its release threshold, and centralizes ordered buffer draining while retaining the size cap.

app/videonative/src/main/cpp/BufferedPacketQueue.h

Tests (1) +73 / -2
BufferedPacketQueue_test.cppCover packet loss, timeout, reordering, and sequence jumps +73/-2

Cover packet loss, timeout, reordering, and sequence jumps

• Injects a fixture-controlled clock so timeout behavior is deterministic. Adds regression coverage for permanent gaps, stale-buffer flushing, successful reordering within the timeout, and large jumps falling back to the buffer-size cap.

app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp

Other (1) +1 / -1
CMakeLists.txtCorrect BufferedPacketQueue test source capitalization +1/-1

Correct BufferedPacketQueue test source capitalization

• Fixes the test source filename casing so CMake can configure the target on case-sensitive filesystems.

app/videonative/src/main/cpp/tests/CMakeLists.txt

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Wraparound flush misorders packets 🐞 Bug ≡ Correctness
Description
The timeout path calls drainBufferInOrder, which numerically sorts buffered uint16_t sequence
numbers, so a block spanning 65535→0 is delivered as 0 then 65535. This violates the queue's
ordering guarantee and can advance mLastPacketIdx incorrectly.
Code

app/videonative/src/main/cpp/BufferedPacketQueue.h[90]

+            mLastPacketIdx = drainBufferInOrder(callback);
Evidence
The new timeout branch invokes the drain at line 90. That drain first handles contiguous successors
using modulo arithmetic, but then sorts all remaining keys with raw numeric comparison before
callbacks; the class already provides wraparound-aware distance semantics, proving raw ordering is
unsuitable. For example, after sequence 65533 with 65534 missing, buffered 65535 and 0 are emitted
as 0,65535 on timeout.

app/videonative/src/main/cpp/BufferedPacketQueue.h[80-91]
app/videonative/src/main/cpp/BufferedPacketQueue.h[185-205]
app/videonative/src/main/cpp/BufferedPacketQueue.h[335-351]
app/videonative/src/main/cpp/BufferedPacketQueue.h[368-391]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Timeout flushing numerically sorts RTP sequence numbers, which misorders buffered packets across the uint16 wraparound boundary.

## Issue Context
Use wraparound-aware sequence distance relative to the last delivered sequence when ordering the remaining buffered packets and selecting the new last sequence.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[316-359]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[41-54]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Late packets replay out-of-order 🐞 Bug ≡ Correctness
Description
After a timeout or monotonic flush advances mLastPacketIdx, a subsequently arriving missing packet
is still buffered even though it is now behind the delivered position. A later timeout drains that
stale packet through the callback after newer packets, corrupting parser input order.
Code

app/videonative/src/main/cpp/BufferedPacketQueue.h[90]

+            mLastPacketIdx = drainBufferInOrder(callback);
Evidence
The new flush advances the delivery position, but every non-successor is routed to
handleOutOfOrderPacket. That handler only checks whether the sequence already exists in the map
and still buffers packets behind mLastPacketIdx; drainBufferInOrder later invokes the callback
for every remaining entry. Thus, after delivering 1,3,4 because 2 timed out, a late 2 remains
buffered while 5 onward are delivered and is eventually emitted after them.

app/videonative/src/main/cpp/BufferedPacketQueue.h[80-113]
app/videonative/src/main/cpp/BufferedPacketQueue.h[226-240]
app/videonative/src/main/cpp/BufferedPacketQueue.h[276-294]
app/videonative/src/main/cpp/BufferedPacketQueue.h[318-357]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Packets arriving behind `mLastPacketIdx` after a gap flush are buffered and eventually replayed after newer RTP data.

## Issue Context
Distinguish stale packets behind the delivery point from packets ahead across wraparound. Drop stale packets rather than inserting them into the reorder buffer, while preserving valid wraparound handling.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[80-113]
- app/videonative/src/main/cpp/BufferedPacketQueue.h[216-268]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[91-116]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Duplicates trigger monotonic flush 🐞 Bug ≡ Correctness
Description
The activated monotonic counter tests each packet only against mLastPacketIdx, not against the
previous buffered arrival, so duplicates or a non-monotonic reorder increment it repeatedly. Five
copies of the same ahead-of-gap sequence therefore force an unwarranted flush.
Code

app/videonative/src/main/cpp/BufferedPacketQueue.h[R239-240]

+        auto dist = calculateDistance(mLastPacketIdx, currPacketIdx);
+        if (static_cast<size_t>(std::abs(dist)) < MONOTONIC_MAX_DISTANCE)
Evidence
The changed distance direction makes the previously ineffective counter active, but the distance is
always calculated from the unchanged last delivered sequence. The duplicate check does not return,
and every positive distance increments the counter, so repeated identical packets or arrivals such
as 5,4,3 still satisfy the purported monotonic threshold.

app/videonative/src/main/cpp/BufferedPacketQueue.h[126-129]
app/videonative/src/main/cpp/BufferedPacketQueue.h[226-249]
app/videonative/src/main/cpp/BufferedPacketQueue.h[276-294]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The monotonic escape counter counts any packet ahead of the last delivered sequence, including duplicates and decreasing reordered arrivals.

## Issue Context
Track the previous out-of-order arrival and increment only for a genuine forward sequence progression. Duplicates and backward movement should reset or leave the run unchanged as appropriate.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[126-129]
- app/videonative/src/main/cpp/BufferedPacketQueue.h[226-259]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[71-86]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources

Grey Divider

Tip of the day
💡 Did you know, you can turn on the rule miner and Qodo learns your standards from review history

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

mPackets.size(),
(long long) MAX_BUFFER_AGE.count(),
static_cast<unsigned>(static_cast<SeqType>(mLastPacketIdx + 1)));
mLastPacketIdx = drainBufferInOrder(callback);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Wraparound flush misorders packets 🐞 Bug ≡ Correctness

The timeout path calls drainBufferInOrder, which numerically sorts buffered uint16_t sequence
numbers, so a block spanning 65535→0 is delivered as 0 then 65535. This violates the queue's
ordering guarantee and can advance mLastPacketIdx incorrectly.
Agent Prompt
## Issue description
Timeout flushing numerically sorts RTP sequence numbers, which misorders buffered packets across the uint16 wraparound boundary.

## Issue Context
Use wraparound-aware sequence distance relative to the last delivered sequence when ordering the remaining buffered packets and selecting the new last sequence.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[316-359]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[41-54]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

mPackets.size(),
(long long) MAX_BUFFER_AGE.count(),
static_cast<unsigned>(static_cast<SeqType>(mLastPacketIdx + 1)));
mLastPacketIdx = drainBufferInOrder(callback);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Late packets replay out-of-order 🐞 Bug ≡ Correctness

After a timeout or monotonic flush advances mLastPacketIdx, a subsequently arriving missing packet
is still buffered even though it is now behind the delivered position. A later timeout drains that
stale packet through the callback after newer packets, corrupting parser input order.
Agent Prompt
## Issue description
Packets arriving behind `mLastPacketIdx` after a gap flush are buffered and eventually replayed after newer RTP data.

## Issue Context
Distinguish stale packets behind the delivery point from packets ahead across wraparound. Drop stale packets rather than inserting them into the reorder buffer, while preserving valid wraparound handling.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[80-113]
- app/videonative/src/main/cpp/BufferedPacketQueue.h[216-268]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[91-116]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment on lines +239 to +240
auto dist = calculateDistance(mLastPacketIdx, currPacketIdx);
if (static_cast<size_t>(std::abs(dist)) < MONOTONIC_MAX_DISTANCE)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

3. Duplicates trigger monotonic flush 🐞 Bug ≡ Correctness

The activated monotonic counter tests each packet only against mLastPacketIdx, not against the
previous buffered arrival, so duplicates or a non-monotonic reorder increment it repeatedly. Five
copies of the same ahead-of-gap sequence therefore force an unwarranted flush.
Agent Prompt
## Issue description
The monotonic escape counter counts any packet ahead of the last delivered sequence, including duplicates and decreasing reordered arrivals.

## Issue Context
Track the previous out-of-order arrival and increment only for a genuine forward sequence progression. Duplicates and backward movement should reset or leave the run unchanged as appropriate.

## Fix Focus Areas
- app/videonative/src/main/cpp/BufferedPacketQueue.h[126-129]
- app/videonative/src/main/cpp/BufferedPacketQueue.h[226-259]
- app/videonative/src/main/cpp/tests/BufferedPacketQueue_test.cpp[71-86]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@vertexodessa

Copy link
Copy Markdown
Collaborator

@iflyhere thank you for the PR! I checked Qodo's comments they seem to be correct. could you please fix and I'll merge the PR

@vertexodessa

Copy link
Copy Markdown
Collaborator

@iflyhere additional findings:

  1. restartBuffering ignores the highest returned by drainBufferInOrder and still sets mLastPacketIdx = currPacketIdx. That's the packet that triggered the flush, not the highest one delivered. With 1..5 delivered, 6 lost, then 7 8 9 12 10: the flush delivers 7 8 9 10 12 but the pointer lands on 10, so 11 goes to the parser after 12, and 13..16 are held until 17 arrives (4 packets on video, the full 20 ms on audio). The queue creates the stall it is meant to prevent. Since bufferPacket always runs before restartBuffering, the triggering packet is already in the buffer, so mLastPacketIdx = drainBufferInOrder(callback) is enough.

  2. A jump of more than 32767 never resyncs on a slow stream. highest is seeded from the old mLastPacketIdx and compared via calculateDistance, which is negative past half the sequence space, so neither one moves. Fed 1, 2, then 40000, 40001, ... at 25 ms: 40 packets came out as 40000..40038, each one arrival late, 40039 still held, no recovery. At 1 ms spacing the 15 packet cap fires first and resyncs, so video is fine, audio is not. LargeJumpFallsBackToTheBufferCap uses 30000, which is still positive as int16, so it doesn't catch this. RTP starts at a random seq, so a VTX reboot mid session gets you here. Seed highest from a packet in the buffer instead of mLastPacketIdx, and maybe add a test with a jump of 40000 on the timeout path.

BufferedPacketQueue reorders the RTP stream before it reaches the parser. When a packet is
missing it holds everything behind the gap, which is right for a reorder and wrong for a loss
- and past wfb-ng, on a loopback socket, a gap is almost always a loss that FEC could not
recover. Three things made that wait longer than it needs to be.

The monotonic-increase escape hatch, which exists precisely to notice "the sequence numbers
keep climbing but the gap is not filling", never fires. calculateDistance(a, b) is how far b
is ahead of a - seqLessThan reads it that way - but the call site asks for the distance from
the incoming packet to the last delivered one. For a gap ahead of us that is negative, so the
else branch clears the counter on every single packet.

Fixing the argument order alone is not enough: the counter is only incremented while
abs(dist) < MONOTONIC_THRESHOLD, and dist grows by one with every packet held back. Compared
against the same constant that gates it, the counter tops out at three and can never reach
five. The gate is now its own constant, so a run past a gap releases the buffer after five
packets instead of never.

That leaves MAX_BUFFER_SIZE as the only bound, and how much latency fifteen packets is depends
entirely on the packet rate: about 20ms on a 1080p video stream, but roughly 300ms on the
audio stream, which runs at a fraction of it. The buffer is now also bounded in time, so the
worst case is the same on both. Twenty milliseconds is a little over one frame at 60fps and
several orders of magnitude more than a loopback socket needs to reorder anything.

Nothing is dropped by any of this - the buffered packets are still delivered, just without
waiting on one that is not coming.

The test target could not be built on a case-sensitive filesystem, since CMakeLists.txt
spelled the source BufferedPacketqueue_test.cpp. The existing two cases both turn out to feed
a strictly in-order stream, so neither reached the buffer at all; the new ones cover a
permanent gap, the age bound, a reorder that resolves inside it, and a jump too large to be
one. They drive the clock themselves rather than reading it, so the age bound cannot make them
flaky on a loaded machine.
Two problems with the flush, both found in review.

restartBuffering() set mLastPacketIdx to the packet that triggered the flush rather than the
newest one delivered. With 1..5 delivered, 6 lost, then 7 8 9 12 10, the flush hands over
7 8 9 10 12 but leaves the pointer on 10 - so 11 would be delivered after 12, and 13..16 are
held until 17 arrives. The queue re-created the stall it exists to prevent. bufferPacket()
always runs before the flush, so the triggering packet is in the buffer either way and
mLastPacketIdx = drainBufferInOrder() is all that was needed; restartBuffering() is gone.

That alone would have broken the resync on a large jump, because drainBufferInOrder() seeded
`highest` from mLastPacketIdx and compared with calculateDistance(), which reads as negative
past half the sequence space. RTP starts at a random sequence number, so a VTX that reboots
mid-session lands exactly there: fed 1, 2, then 40000, 40001, ..., neither the seed nor the
comparison ever moves and every packet comes out one flush late, forever. On video the
fifteen-packet cap used to paper over it by rewinding to currPacketIdx; on audio nothing did.
`highest` is now seeded from a packet that is actually in the buffer.

While sorting the flush: by raw value a block straddling the wrap point (65534, 65535, 0, 1)
sorts to 0, 1, 65534, 65535 and was handed to the parser in that order. Sorting by distance
from the last delivered packet fixes the order and makes the last element the newest.

Three tests added, one per problem. The large-jump case uses 40000 - the existing
LargeJumpFallsBackToTheBufferCap uses 30000, which is still positive as an int16 and does not
reach any of this.
@iflyhere
iflyhere force-pushed the fix/buffered-queue-latency branch from faed15f to ca01d43 Compare September 2, 2026 20:40
@iflyhere

iflyhere commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Both correct, and the second one would have bitten anyone whose VTX reboots mid-flight. Fixed
in the pushed commit, rebased onto master.

1. restartBuffering is gone. bufferPacket() always runs first, so the triggering packet
is in the buffer either way and mLastPacketIdx = drainBufferInOrder(callback) is all that
was needed at both call sites.

2. highest is now seeded from sortedPackets.front()->first rather than
mLastPacketIdx. Worth spelling out why the two findings had to be fixed together: the
fifteen-packet cap was only resyncing on a large jump because it rewound to currPacketIdx.
Fixing (1) on its own would have taken that away and left video in the same state as audio.

3. While in there — the flush sorted by raw value, so a block straddling the wrap point
came out as 0, 1, 2, 65534, 65535. It now sorts by distance from the last delivered packet,
which fixes the order and makes the last element the newest one, so highest falls out of it.

Three tests added, one per problem. Against the previous commit:

7/9 FlushAdvancesPastEverythingItDelivered ...***Failed
    Which is: { 1, 2, 3, 4, 5, 7, 8, 9, 10, 12 }
    Which is: { 1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13 }
8/9 StreamRestartFarAheadResyncs ............***Failed
    Which is: { 1, 2, 40000 }
    Which is: { 1, 2, 40000, 40001, 40002 }
9/9 FlushAcrossTheWrapDeliversInOrder .......***Failed
    Which is: { 65532, 0, 1, 2, 65534, 65535, 3 }
    Which is: { 65532, 65534, 65535, 0, 1, 2, 3 }
67% tests passed, 3 tests failed out of 9

and with the fix, 9/9. StreamRestartFarAheadResyncs uses 40000 for exactly the reason you
gave — 30000 is still positive as an int16 and never reaches any of this.

Compile tested for arm64-v8a + armeabi-v7a.

@vertexodessa
vertexodessa merged commit 3d6ca17 into OpenIPC:master Sep 2, 2026
@iflyhere
iflyhere deleted the fix/buffered-queue-latency branch September 3, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants