mirror of
https://github.com/nestriness/nestri.git
synced 2026-11-17 12:41:45 +02:00
Hub side of nescore's 2026-10-16 video-rate spec. nesprotocol: a per-frame feedback datagram (kind 3, client to hub on the video connection) carrying bytes, fragments, send time, first-fragment arrival, arrival spread and first-fragment queueing delay. The hub stats breakdown gains measured capacity and delivered share. neshub: - rate.rs, a per-client path model. Capacity is measured from each frame's train dispersion. The bottleneck counts as busy when the next train lands no later than its first fragment's own serialisation after the last one, and as overloaded when, during a busy run, the first fragments' queue grows and more is sent than delivered. Overload drops the rate at once to drain at the rate the queue built, then settles at the measured share. Steps up go to the measured capacity, or the highest rate the path has proven quiet at for longer than its own longest gap between overloads at or below it. Silence holds at the last share. Tested against a modelled FIFO bottleneck: above the path, below it, CBR keyframes, a capacity drop and recovery, recurring cross traffic. - control.rs: the slowest client's path decides, per frame as feedback arrives. Gone: the loss thresholds, back-off factor, climb fraction, deadband and queue thresholds. Ceiling, floor, manual and constant quality behave as before. - congestion.rs: an endpoint congestion controller that holds nothing back (window above u32::MAX, where noq stops pacing), so frames leave as trains for the client to measure. media-transport.md now says the hub turns transport pacing off, and why. The client does not send feedback yet (nescore step 2). Until it does, the rate stays where the session started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
229 lines
12 KiB
Markdown
229 lines
12 KiB
Markdown
# The media transport, as it actually behaves
|
|
|
|
Media rides one QUIC connection: video delta frames and audio as datagrams,
|
|
keyframes as short-lived unidirectional streams, cursor and stats and input as
|
|
streams of their own. Several things about that arrangement are not what they
|
|
look like, and we designed against the wrong model twice before reading the
|
|
source.
|
|
|
|
This file records what the transport *does*, with somewhere to check each
|
|
claim, and what follows from it. It is deliberately about behaviour rather than
|
|
about our code, because our code changes and this does not — except when the
|
|
dependency does, which is why the version is named.
|
|
|
|
Read against **iroh 1.1, which uses `noq` and `noq-proto` 1.3 — a quinn fork,
|
|
not quinn itself.** Paths below are inside `noq-proto` unless stated. Reading
|
|
upstream quinn instead is a mistake worth naming, because we made it: the
|
|
behaviour turned out to agree on every point here, but "turned out to" is
|
|
precisely the standard this file exists to replace. If the iroh or noq version
|
|
moves, re-read before trusting any of this.
|
|
|
|
## The rule
|
|
|
|
**A path is the unit of congestion, not a stream and not a connection.**
|
|
Everything else here follows from that one sentence, and almost every wrong
|
|
assumption we made came from forgetting some part of it.
|
|
|
|
## Congestion is per path, and shared by everything on that path
|
|
|
|
Each path carries its own congestion controller and its own pacer
|
|
(`connection/paths.rs`, `PathData::congestion` and `PathData::pacing_delay`),
|
|
and `poll_transmit` consults both for the path it is about to send on
|
|
(`connection/mod.rs`, in the transmit blocking checks). Within a path, every
|
|
stream and every datagram shares them, whatever the packet ends up carrying.
|
|
|
|
The "per path" part is not pedantry here: a connection commonly holds a relay
|
|
path and a direct path at once, they have different windows and different
|
|
round-trip times, and only the *selected* one describes where media is
|
|
currently going. Reading connection-level aggregates instead will average two
|
|
unrelated network conditions together.
|
|
|
|
So moving keyframes onto their own streams — which we did, and which was right
|
|
— changed the *delivery guarantee* and nothing else. A lost fragment no longer
|
|
destroys a keyframe, because QUIC retransmits stream data. But that keyframe
|
|
still competes for the same congestion window as everything else on the path,
|
|
and still queues in the same scheduler. Stream-versus-datagram is a reliability
|
|
distinction, not an isolation one.
|
|
|
|
The corollary is worth stating because it is tempting and wrong: **opening a
|
|
second connection does not create bandwidth.** At a 3 Mbps bottleneck a 150 KB
|
|
keyframe occupies about 400 ms of link time however it is carried. A second
|
|
connection turns strict queueing into competition between two flows over the
|
|
same bottleneck, which is sometimes what you want — but it is a change in
|
|
*fairness*, not in capacity, and it costs a second NAT traversal, a second
|
|
handshake and a second set of paths to manage.
|
|
|
|
## Datagrams are written before stream data
|
|
|
|
`populate_packet` fills each packet in a fixed order, and DATAGRAM frames come
|
|
before STREAM frames. Whatever datagrams are queued take their space first;
|
|
stream data gets the remainder.
|
|
|
|
Two consequences, and they point in opposite directions:
|
|
|
|
**Delta frames cannot be starved by keyframes.** A large keyframe simply
|
|
stretches out over many packets while deltas keep flowing beside it. This is
|
|
the behaviour we nearly built by hand before checking.
|
|
|
|
**Keyframes can be starved by delta frames.** On a saturated link a steady
|
|
stream of datagrams fills every packet, and the keyframe on its stream gets
|
|
leftovers. If the client is waiting on that keyframe to resynchronise, the
|
|
frames starving it are the very frames it cannot decode without it. That is a
|
|
feedback loop with no bottom, and it is what produced "26 keyframe fallbacks,
|
|
19 IDR requests" in the field: the recovery frame could not get out past the
|
|
frames that needed it to arrive first.
|
|
|
|
The fix is not to reorder the transport. It is to stop sending data that
|
|
depends on data the receiver does not have — see `dgram::ResyncGate`.
|
|
|
|
## The send path has no backpressure signal
|
|
|
|
`send_datagram` calls `datagrams().send(data, true)` (`noq/src/connection.rs`),
|
|
and that `true` is a `drop` flag: the queue makes room by **evicting the oldest
|
|
queued datagrams** via `make_space_for` and then returns `Ok`
|
|
(`connection/datagrams.rs`). It errors only for `TooLarge`, `UnsupportedByPeer`,
|
|
`Disabled`, and `ConnectionLost` — that is, for a datagram that was malformed
|
|
or a connection that is already gone, never for one the path could not carry.
|
|
|
|
(The same function with `drop: false` returns `Blocked` instead and is what
|
|
`send_datagram_wait` uses. That would give backpressure, at the cost of
|
|
prioritising old frames over current ones, which for live media is the wrong
|
|
trade.)
|
|
|
|
The hub sends each frame with `send_many_datagrams` (iroh 1.3), which is the
|
|
same call over a batch: `send_many(datagrams, true)`, evicting exactly as
|
|
above and returning the full count. What it adds is that the batch is queued
|
|
under one lock hold, and a `TooLarge` on any fragment rejects the whole batch,
|
|
so a frame goes out whole or not at all. noq-proto's `send_many` with
|
|
`drop: false` would queue until full and return a partial count — a real
|
|
backpressure signal — but iroh does not expose that mode.
|
|
|
|
A sender overrunning the path is therefore told nothing at all. Any counter
|
|
fed by that return value cannot be non-zero however badly things are going,
|
|
which is exactly how a field report came to contain "encoder perfectly healthy,
|
|
zero frames dropped" beside a client receiving almost nothing. `send_datagram`
|
|
is still the right call for live media — waiting for buffer space prioritises
|
|
old frames over current ones — but its silence has to be designed around
|
|
rather than read as good news.
|
|
|
|
What *can* be read: `datagram_send_buffer_space()` for remaining room, and the
|
|
selected path's `stats()` for `cwnd`, `rtt` and `lost_packets`. Note "selected":
|
|
a connection commonly holds a relay path and a direct path at once, and only
|
|
the selected one describes where media is going.
|
|
|
|
## quinn already paces
|
|
|
|
`poll_transmit` blocks on a token-bucket pacer (`connection/pacing.rs`, reached
|
|
through `PathData::pacing_delay`) sized from that path's congestion window and
|
|
round-trip time, before every packet. Handing 150 fragments over in a tight
|
|
loop does *not* put 150 packets on the wire at once.
|
|
|
|
Application-level pacing on top of this is redundant, and worse than
|
|
redundant: it would spread transmission according to our guess at the available
|
|
rate, while the pacer underneath is using the measured window and RTT. If the
|
|
application is producing faster than the pacer drains, the answer is to produce
|
|
less — which is the bitrate controller's job — not to add a second pacer.
|
|
|
|
**Since 2026-10-16 the hub turns this pacer off** (`neshub/src/congestion.rs`).
|
|
Its controller reports a window above `u32::MAX`, past which noq does not pace,
|
|
and ignores loss. The video rate is decided per frame from client feedback
|
|
(`neshub/src/rate.rs`), and that needs each frame to leave as a back-to-back
|
|
train, so the bottleneck spaces the fragments by its own rate and the client can
|
|
measure it. The transport pacing by its window estimate would smear exactly that
|
|
spacing. The rate is still produced to fit the path; it is the media's
|
|
controller that decides it, on the path's queueing delay, which leads loss and
|
|
RTT by tens of milliseconds. Transport settings are per endpoint, so the hub's
|
|
audio, input and control connections are unpaced too: they are small flows the
|
|
video rate leaves room for.
|
|
|
|
## Loss is a lagging indicator; the queue is the leading one
|
|
|
|
A path that is overrun does not necessarily drop anything. It can simply be
|
|
slower than the offer, in which case the pacer above meters packets out at the
|
|
rate it measures and the difference accumulates in the datagram send buffer.
|
|
Nothing is lost. Everything is late.
|
|
|
|
This was measured over a 1000-mile link: every frame arrived, every frame
|
|
completed, zero QUIC loss, zero congestion events, and a **flat** 180 ms round
|
|
trip for an entire session — while the picture ran up to eight seconds behind.
|
|
A flat RTT is the tell that the queue is *ours* and not the network's; a
|
|
standing queue in the path would have shown up as rising round-trip time.
|
|
|
|
The trap is that every health signal reads clean. Loss only appears once the
|
|
buffer finally overflows, long after the session became unplayable, and by then
|
|
the queue is deep enough that the only way out is the floor. A controller
|
|
driven by loss alone therefore climbs to its ceiling against a path it is
|
|
already overrunning, collapses when the buffer spills, drains, and climbs
|
|
again — a sawtooth with a period of tens of seconds, in which the stream is
|
|
never once at a rate the path can carry.
|
|
|
|
`DGRAM_BUFFER_BYTES` is 4 MiB. That was chosen as headroom for a keyframe
|
|
burst, which is a real requirement and still correct. But a buffer sized in
|
|
bytes is a queue whose *duration* depends on the drain rate: at a couple of
|
|
megabits, 4 MiB is several seconds of video. Sizing for one dimension silently
|
|
set a bound in the other.
|
|
|
|
The signal that does work was available the whole time and simply never read:
|
|
|
|
```
|
|
backlog_bytes = DGRAM_BUFFER_BYTES - conn.datagram_send_buffer_space()
|
|
backlog_ms = backlog_bytes * 8 / drain_kbps
|
|
```
|
|
|
|
Divide by the rate the queue actually drains at — what is getting through — and
|
|
not by what is being asked for, because the latter is the number that is too
|
|
high whenever this matters.
|
|
|
|
## What follows
|
|
|
|
These are the design rules we arrived at, and the reasoning is above rather
|
|
than in the code that implements them.
|
|
|
|
**Do not send what the receiver cannot use.** Frames that depend on a keyframe
|
|
the receiver is still waiting for are not merely wasted bandwidth; on this
|
|
transport they actively delay the keyframe.
|
|
|
|
**Do not leave a hole for something you chose not to send.** A receiver counts
|
|
a missing sequence number as loss, and that count feeds our own congestion
|
|
response. A sender that skips a frame *and* burns its sequence number makes the
|
|
controller lower the bitrate in response to the sender's own decision. Reusing
|
|
the number is safe precisely because nothing went out under it.
|
|
|
|
**Bound every suppression.** Withholding is correct only while the assumption
|
|
behind it holds. If the keyframe never comes, withholding forever converts a
|
|
recoverable freeze into a permanent black screen, so the suppression expires
|
|
and says that it did.
|
|
|
|
**Measure the queue, not just the casualties.** Loss says a path was overrun
|
|
some time ago. A standing send queue says it is being overrun right now, while
|
|
the frames are still only late. On a transport that drops the oldest datagram
|
|
rather than refusing the newest, the queue is the only signal that arrives in
|
|
time to act on — and climbing is what digs it, so climbing needs the queue to
|
|
be empty and not merely the loss to be low.
|
|
|
|
**Trust the receiver over the sender.** The sender's view of this path was
|
|
measured reporting 182 ms RTT with zero packet loss while the client received
|
|
almost nothing. Some of that gap is structural, per the section above. The far
|
|
end's report of what actually arrived is the only measurement that cannot be
|
|
quietly wrong in our favour.
|
|
|
|
## The method
|
|
|
|
Every item here was found by reading the dependency's source, and every one of
|
|
them contradicted something we believed. The first draft of this very file
|
|
cited the wrong crate — upstream quinn rather than the fork iroh actually
|
|
builds against — which is a good illustration of how little intuition is worth
|
|
here even when the conclusions survive. The same thing happened one layer
|
|
down, in Vulkan Video: rate control turned out to be session state rather than
|
|
per-frame state, `gopFrameCount` of zero means "implementation chooses" rather
|
|
than "infinite" (`UINT32_MAX` means infinite), and a hardcoded QP ceiling made
|
|
low bitrate targets silently unreachable. None of that was in any comment we
|
|
had written, and two of those we had confidently asserted the opposite of.
|
|
|
|
So: **when behaviour depends on a dependency's internals, read them, and write
|
|
down what you found with a path to check it against.** Not the documentation,
|
|
which is frequently about intent; not the interface, which is frequently about
|
|
what you may call rather than what will happen. A claim in this file that
|
|
nobody can verify in an afternoon is a claim that will be wrong within a year
|
|
and believed anyway.
|