netris-nestri/docs/media-transport.md

The media transport, as it actually behaves

Media rides one QUIC connection: video delta frames and audio as datagrams, keyframes as short-lived unidirectional streams, cursor and stats and input as streams of their own. Several things about that arrangement are not what they look like, and we designed against the wrong model twice before reading the source.

This file records what the transport does, with somewhere to check each claim, and what follows from it. It is deliberately about behaviour rather than about our code, because our code changes and this does not — except when the dependency does, which is why the version is named.

Read against iroh 1.1, which uses noq and noq-proto 1.3 — a quinn fork, not quinn itself. Paths below are inside noq-proto unless stated. Reading upstream quinn instead is a mistake worth naming, because we made it: the behaviour turned out to agree on every point here, but "turned out to" is precisely the standard this file exists to replace. If the iroh or noq version moves, re-read before trusting any of this.

The rule

A path is the unit of congestion, not a stream and not a connection. Everything else here follows from that one sentence, and almost every wrong assumption we made came from forgetting some part of it.

Congestion is per path, and shared by everything on that path

Each path carries its own congestion controller and its own pacer (connection/paths.rs, PathData::congestion and PathData::pacing_delay), and poll_transmit consults both for the path it is about to send on (connection/mod.rs, in the transmit blocking checks). Within a path, every stream and every datagram shares them, whatever the packet ends up carrying.

The "per path" part is not pedantry here: a connection commonly holds a relay path and a direct path at once, they have different windows and different round-trip times, and only the selected one describes where media is currently going. Reading connection-level aggregates instead will average two unrelated network conditions together.

So moving keyframes onto their own streams — which we did, and which was right — changed the delivery guarantee and nothing else. A lost fragment no longer destroys a keyframe, because QUIC retransmits stream data. But that keyframe still competes for the same congestion window as everything else on the path, and still queues in the same scheduler. Stream-versus-datagram is a reliability distinction, not an isolation one.

The corollary is worth stating because it is tempting and wrong: opening a second connection does not create bandwidth. At a 3 Mbps bottleneck a 150 KB keyframe occupies about 400 ms of link time however it is carried. A second connection turns strict queueing into competition between two flows over the same bottleneck, which is sometimes what you want — but it is a change in fairness, not in capacity, and it costs a second NAT traversal, a second handshake and a second set of paths to manage.

Datagrams are written before stream data

populate_packet fills each packet in a fixed order, and DATAGRAM frames come before STREAM frames. Whatever datagrams are queued take their space first; stream data gets the remainder.

Two consequences, and they point in opposite directions:

Delta frames cannot be starved by keyframes. A large keyframe simply stretches out over many packets while deltas keep flowing beside it. This is the behaviour we nearly built by hand before checking.

Keyframes can be starved by delta frames. On a saturated link a steady stream of datagrams fills every packet, and the keyframe on its stream gets leftovers. If the client is waiting on that keyframe to resynchronise, the frames starving it are the very frames it cannot decode without it. That is a feedback loop with no bottom, and it is what produced "26 keyframe fallbacks, 19 IDR requests" in the field: the recovery frame could not get out past the frames that needed it to arrive first.

The fix is not to reorder the transport. It is to stop sending data that depends on data the receiver does not have — see dgram::ResyncGate.

The send path has no backpressure signal

send_datagram calls datagrams().send(data, true) (noq/src/connection.rs), and that true is a drop flag: the queue makes room by evicting the oldest queued datagrams via make_space_for and then returns Ok (connection/datagrams.rs). It errors only for TooLarge, UnsupportedByPeer, Disabled, and ConnectionLost — that is, for a datagram that was malformed or a connection that is already gone, never for one the path could not carry.

(The same function with drop: false returns Blocked instead and is what send_datagram_wait uses. That would give backpressure, at the cost of prioritising old frames over current ones, which for live media is the wrong trade.)

The hub sends each frame with send_many_datagrams (iroh 1.3), which is the same call over a batch: send_many(datagrams, true), evicting exactly as above and returning the full count. What it adds is that the batch is queued under one lock hold, and a TooLarge on any fragment rejects the whole batch, so a frame goes out whole or not at all. noq-proto's send_many with drop: false would queue until full and return a partial count — a real backpressure signal — but iroh does not expose that mode.

A sender overrunning the path is therefore told nothing at all. Any counter fed by that return value cannot be non-zero however badly things are going, which is exactly how a field report came to contain "encoder perfectly healthy, zero frames dropped" beside a client receiving almost nothing. send_datagram is still the right call for live media — waiting for buffer space prioritises old frames over current ones — but its silence has to be designed around rather than read as good news.

What can be read: datagram_send_buffer_space() for remaining room, and the selected path's stats() for cwnd, rtt and lost_packets. Note "selected": a connection commonly holds a relay path and a direct path at once, and only the selected one describes where media is going.

quinn already paces

poll_transmit blocks on a token-bucket pacer (connection/pacing.rs, reached through PathData::pacing_delay) sized from that path's congestion window and round-trip time, before every packet. Handing 150 fragments over in a tight loop does not put 150 packets on the wire at once.

Application-level pacing on top of this is redundant, and worse than redundant: it would spread transmission according to our guess at the available rate, while the pacer underneath is using the measured window and RTT. If the application is producing faster than the pacer drains, the answer is to produce less — which is the bitrate controller's job — not to add a second pacer.

Since 2026-10-16 the hub turns this pacer off (neshub/src/congestion.rs). Its controller reports a window above u32::MAX, past which noq does not pace, and ignores loss. The video rate is decided per frame from client feedback (neshub/src/rate.rs), and that needs each frame to leave as a back-to-back train, so the bottleneck spaces the fragments by its own rate and the client can measure it. The transport pacing by its window estimate would smear exactly that spacing. The rate is still produced to fit the path; it is the media's controller that decides it, on the path's queueing delay, which leads loss and RTT by tens of milliseconds. Transport settings are per endpoint, so the hub's audio, input and control connections are unpaced too: they are small flows the video rate leaves room for.

Loss is a lagging indicator; the queue is the leading one

A path that is overrun does not necessarily drop anything. It can simply be slower than the offer, in which case the pacer above meters packets out at the rate it measures and the difference accumulates in the datagram send buffer. Nothing is lost. Everything is late.

This was measured over a 1000-mile link: every frame arrived, every frame completed, zero QUIC loss, zero congestion events, and a flat 180 ms round trip for an entire session — while the picture ran up to eight seconds behind. A flat RTT is the tell that the queue is ours and not the network's; a standing queue in the path would have shown up as rising round-trip time.

The trap is that every health signal reads clean. Loss only appears once the buffer finally overflows, long after the session became unplayable, and by then the queue is deep enough that the only way out is the floor. A controller driven by loss alone therefore climbs to its ceiling against a path it is already overrunning, collapses when the buffer spills, drains, and climbs again — a sawtooth with a period of tens of seconds, in which the stream is never once at a rate the path can carry.

DGRAM_BUFFER_BYTES is 4 MiB. That was chosen as headroom for a keyframe burst, which is a real requirement and still correct. But a buffer sized in bytes is a queue whose duration depends on the drain rate: at a couple of megabits, 4 MiB is several seconds of video. Sizing for one dimension silently set a bound in the other.

The signal that does work was available the whole time and simply never read:

backlog_bytes = DGRAM_BUFFER_BYTES - conn.datagram_send_buffer_space()
backlog_ms    = backlog_bytes * 8 / drain_kbps

Divide by the rate the queue actually drains at — what is getting through — and not by what is being asked for, because the latter is the number that is too high whenever this matters.

What follows

These are the design rules we arrived at, and the reasoning is above rather than in the code that implements them.

Do not send what the receiver cannot use. Frames that depend on a keyframe the receiver is still waiting for are not merely wasted bandwidth; on this transport they actively delay the keyframe.

Do not leave a hole for something you chose not to send. A receiver counts a missing sequence number as loss, and that count feeds our own congestion response. A sender that skips a frame and burns its sequence number makes the controller lower the bitrate in response to the sender's own decision. Reusing the number is safe precisely because nothing went out under it.

Bound every suppression. Withholding is correct only while the assumption behind it holds. If the keyframe never comes, withholding forever converts a recoverable freeze into a permanent black screen, so the suppression expires and says that it did.

Measure the queue, not just the casualties. Loss says a path was overrun some time ago. A standing send queue says it is being overrun right now, while the frames are still only late. On a transport that drops the oldest datagram rather than refusing the newest, the queue is the only signal that arrives in time to act on — and climbing is what digs it, so climbing needs the queue to be empty and not merely the loss to be low.

Trust the receiver over the sender. The sender's view of this path was measured reporting 182 ms RTT with zero packet loss while the client received almost nothing. Some of that gap is structural, per the section above. The far end's report of what actually arrived is the only measurement that cannot be quietly wrong in our favour.

The method

Every item here was found by reading the dependency's source, and every one of them contradicted something we believed. The first draft of this very file cited the wrong crate — upstream quinn rather than the fork iroh actually builds against — which is a good illustration of how little intuition is worth here even when the conclusions survive. The same thing happened one layer down, in Vulkan Video: rate control turned out to be session state rather than per-frame state, gopFrameCount of zero means "implementation chooses" rather than "infinite" (UINT32_MAX means infinite), and a hardcoded QP ceiling made low bitrate targets silently unreachable. None of that was in any comment we had written, and two of those we had confidently asserted the opposite of.

So: when behaviour depends on a dependency's internals, read them, and write down what you found with a path to check it against. Not the documentation, which is frequently about intent; not the interface, which is frequently about what you may call rather than what will happen. A claim in this file that nobody can verify in an afternoon is a claim that will be wrong within a year and believed anyway.