Follow-up to #423 from its re-review. The dgram writer now raises
awaiting_keyframe when it reaches the Resubscribe marker, in queue
order, and restart_picture sets the flag before queuing the marker.
Before this, an old keyframe still queued ahead of a primary switch
could clear the flag, letting the new picture's first delta out as the
first frame an old client saw after the switch. Needs a writer stalled
for at least 1 s to trigger.
Tested: cargo test -p neshub on nestripc-1, 78 passed.
The box half of phase 1 of window streams: two games in one box, each
its own stream. Wire formats are nesprotocol's `window` module (#421).
## What each app does now
**nescope**
- Gives every toplevel an id on first map: 0 for the first window, never
reused, none past 65535.
- Tells the hub, on the existing input socket, when a window opens,
changes title or size, and closes, in nesprotocol's window encoding. It
also sends a box-internal `MSG_WINDOW_PID` saying which process owns the
window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id
comes from that process's `NESTRI_LAUNCH_ID`.
- Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and
`INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means
window 0. There is still one output: a window's size applies while that
window has focus and is remembered otherwise. On every input connection,
input resets to window 0.
- Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they
describe changes.
**nescapture**
- Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in
the reserved byte). An older reader never sees it.
- Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared
`/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there
still reaches the newest encoder, as before. The shared socket was the
only resource two capturing processes fought over: the second one took
it, and commands went only to that one. The video and stats sockets are
send-only datagram sockets and are safe to share. Capture slots and
devices are per process.
**neshub**
- Joins frames to windows by pid. An untagged frame is window 0. A pid
with no window falls back to the newest window of its launch, then to
window 0 while no other window is open. A pid whose window closed
belongs to nobody.
- A connection that never subscribes gets the box's primary picture in
the old encoding. The primary picture is the frames of the first encoder
to draw, until it has been silent for a second. A subscribed connection
gets only its window's frames, windowed for any window but 0, with
per-window seq. PyroWave stays window 0 only.
- `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the
writer's seq and resync gate in line with the frames, then requests a
keyframe. The first subscribe opens `STREAM_WINDOWS` on the video
connection, starting with `OPENED` (and the icon, if known) for every
open window. Later lifecycle events go to every subscribed connection.
- Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window
changes, and the hub does no merging across tags. A gamepad message
moves focus only when its window differs from the current one and no
other client sent keyboard or pointer input in the last second. Focus
tags move keyboard and pointer focus only: nescope neither raises the
window nor resizes the output for one. The output changes size only on
an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it,
last writer wins with 500 ms hysteresis between windows. A client's
`INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except
for window 0.
- A cursor update goes to the clients subscribed to the window it
describes. Clients that never subscribed get every update, as before.
- When nescope reconnects (a new compositor), the hub sends `CLOSED` for
every window and every subscription goes dead until the client
subscribes again.
- `DatagramSender` sizes fragments with `max_fragment_payload` and
writes headers with `write_video_datagram_header`.
- Commands go to every known encoder on its own socket, or to the shared
socket while none is known (as today). An encoder that appears later is
sent the commands so far (the latest of each message type) and then a
keyframe request. The first encoder is not sent them again.
- A keyframe on a client's loss or subscribe goes only to that client's
window's encoder (the primary's, for a client that never subscribed).
- Known limit: bitrate is box-wide. One controller sets one target for
every window's encoder, so one client's congestion sets every window's
rate. Per-window rate is follow-up work.
- An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with
the same id. nescope no longer exits when its first game exits; nesinit
owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs
as another uid): nescapture sends its launch id.
## Shipping
nescope and neshub ship together in the guest image and must not ship
apart.
## Compatibility
One game with an old client sends the same bytes as before. The client
never subscribes, so it gets the primary picture, which is that game, as
`DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows
stream. Cursor updates reach it as before. Focus tags appear only if the
game's window is not 0, and window 0 resizes use the old form. Mixed
versions also work:
- An old nescapture is window 0.
- An old nescope never sends window messages. Every frame is then window
0, or the primary picture.
- An old hub sends no tags, which nescope reads as window 0, and it
still reaches nescapture through the shared socket.
## Tests
`cargo test` on nesprotocol (122), nescope (17+5), nescapture (161),
neshub (74), all passing. New tests:
- The window table: joining pids to windows, the launch fallback,
retired pids, the primary picture, the snapshot.
- Focus tagging and resize rewriting.
- The command fan-out.
- nescope's id registry (first window 0, ids never reused, refused past
65535) and per-window focus and sizes.
- The IPC pid trailer.
- Two QUIC end-to-end tests:
- Two tagged encoders: an unsubscribed client sees only the first game
in the old encoding. After subscribing to window 1 it sees only the
second game, windowed, with seq starting again, and a windows stream
that starts with both `OPENED` events.
- An untagged encoder gives the old stream.
Not run on a box: no neslet, nestri-host or box was touched.
A new input event, INPUT_OUTPUT_SIZE [4][w u16][h u16], rides the input
path to nescope, which resizes its output and every window on it. The
size is clamped to the launch size, so the box's encode budget holds,
and kept even for 4:2:0. apply_resize existed but nothing called it.
Old hubs stop a batch at an unknown type, so clients send it alone.
The first session record on the new controller (wired fiber, 8 Mbps
ceiling) put capacity at 9 Gbps. The client reads fragments in batches
that share a timestamp, so on a fast path one batched train looks
instantaneous, and the highest sample is exactly that train. The
ceiling capped it there. On a path near our rate it would have sent
every step-up straight to the ceiling and back.
Dispersion error is one-sided at each end (batching reads high,
interleaved cross traffic reads low), so the estimate is now the median
of the samples since the last overload, kept as two heaps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hub side of nescore's 2026-10-16 video-rate spec.
nesprotocol: a per-frame feedback datagram (kind 3, client to hub on the
video connection) carrying bytes, fragments, send time, first-fragment
arrival, arrival spread and first-fragment queueing delay. The hub stats
breakdown gains measured capacity and delivered share.
neshub:
- rate.rs, a per-client path model. Capacity is measured from each
frame's train dispersion. The bottleneck counts as busy when the next
train lands no later than its first fragment's own serialisation after
the last one, and as overloaded when, during a busy run, the first
fragments' queue grows and more is sent than delivered. Overload drops
the rate at once to drain at the rate the queue built, then settles at
the measured share. Steps up go to the measured capacity, or the
highest rate the path has proven quiet at for longer than its own
longest gap between overloads at or below it. Silence holds at the
last share. Tested against a modelled FIFO bottleneck: above the path,
below it, CBR keyframes, a capacity drop and recovery, recurring cross
traffic.
- control.rs: the slowest client's path decides, per frame as feedback
arrives. Gone: the loss thresholds, back-off factor, climb fraction,
deadband and queue thresholds. Ceiling, floor, manual and constant
quality behave as before.
- congestion.rs: an endpoint congestion controller that holds nothing
back (window above u32::MAX, where noq stops pacing), so frames leave
as trains for the client to measure.
media-transport.md now says the hub turns transport pacing off, and why.
The client does not send feedback yet (nescore step 2). Until it does,
the rate stays where the session started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A client's session is created with its first connection, and its media
channels with it, but each channel's writer only starts when that
carrier's own connection attaches. Every packet broadcast in between
piled up and went out in one burst once the connection was ready.
Client session records showed it as about 160 ms of audio arriving at
once at session start, and over half a second of video.
A session now takes a stream's media only once that stream's writer
exists. Before then there is nobody to give it to, and media is live or
nothing. No threshold: the rule is the attach event.
A video attach also asks for a keyframe and holds deltas until one
goes out. A fresh writer has sent no reference frame, so nothing else
is decodable, and the client no longer has to fail a decode and ask, a
round trip later.
New e2e test: audio broadcast before the audio carrier attaches never
reaches the client (fails without the fix: first position 0), and a
video attach asks for a keyframe. The pipeline-delay tests now build v4
payloads; they had been passing on v3 byte offsets by accident.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of nescore's 2026-10-16 media timing spec. Media carried no
send time and audio no time at all, so every layer had to infer lateness
from gaps between arrivals.
- clock: one sender clock, CLOCK_MONOTONIC in u64 microseconds, shared
by every process that stamps media. Off unix the client uses its own
process clock, which is all a receiver needs.
- Datagram headers (and PyroWave's) carry sent_us, stamped by neshub at
hand-off to QUIC. Keyframe streams open with it too.
- Video carries captured_us (from nescapture's present time) in place of
milliseconds since the encoder started. Audio carries its first
sample's position on the PipeWire graph clock, which neswire already
read to fill missed cycles. The IPC stamp is a u64 carrying either.
- delay: microseconds. The baseline no longer expires on a 30 s timer:
clock skew is measured as the least-squares slope of each second's
minimum one-way delay, and only a path change resets the baseline.
observe() returns each unit's queueing delay.
- ReceiverReport carries video and audio queueing delay summaries for
the rate control that will consume them.
- Shared encode/decode helpers for video and audio payloads, so the hub
and receivers stop hand-building byte offsets.
No compatibility kept. STREAM_VERSION is 4. nescore needs a pin and the
client side (step 2) before it can talk to this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
neslet now puts nestri.port= on the boot line and installs its forward
and pinned masquerade for that port before the box boots. nesinit hands
it to neshub as NESTRI_BIND_PORT, and neshub binds 0.0.0.0:<port> in
place of iroh's random default. Every flow, the relay's included, then
leaves on one public port, and the address the relay reports is the one
peers can punch through to. No nestri.port= keeps the old behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The codec the clients agree on goes to the encoder over a socket that does
not exist until the game draws, and a client can connect first. The command
was dropped, the encoder started on AV1, and a client without AV1 sat on a
splash with sound and no picture. Commands now wait for the socket, and a
frame in a codec no client decodes sends the agreed set again, which also
covers a launcher handing over to a game with an encoder of its own.
A host cannot tell a run somebody is playing from one nobody has looked
at in an hour, so it cannot end the second without guessing.
The hub now answers the address socket with a second line,
`viewers <n>`, counting clients that are streaming. Init reads it
alongside the address and sends a new `viewers` message to the host
when the count changes. A reader that takes only the first line gets
the address exactly as before, a server that never writes the second
line is still read, and a host that does not know the message skips it,
so either end can ship first.
Each client states what it decodes when it connects, and the hub passed
each statement to the encoder as it arrived. The newest client decided, so
a client that only decodes H.264 lost its picture as soon as one that
decodes AV1 or H.265 joined.
The hub now keeps every client's statement and gives the encoder their
intersection, recomputed when a client joins, when one leaves, and when the
controller returns to automatic. A room settles on the best codec all of
its clients share, and moves up again once the client holding it back has
left. Manual mode still leaves the encoder alone; the change is applied
when the mode goes back to automatic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
send_many_datagrams queues a frame's fragments under one lock hold and
one driver wake instead of one per fragment. It also rejects the whole
frame on TooLarge, so a path that shrinks mid-frame no longer leaves half
a frame on the wire for the client to time out. Eviction behaviour is
unchanged. A refused PyroWave frame now keeps its sequence number, since
nothing went out under it. pyro_bench mirrors both ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At this point, why not..
Calling it `nespyro` since it's slightly modified, uses slang shader
language instead, along other things..
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Adds `nesgamepad`, basically "diet-coke vimputti" for Nestri's microVM
needs here.
Comes with direct mapping for:
- Sony
- DualShock 4 (v2), DualSense
- Microsoft
- Xbox 360 pad, Xbox One S pad
- Nintendo
- Pro Controller
- Generic pad for unknown controllers
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Four findings from review, all of them real.
The relay's directory was mounted on the tree a session's shares live in. A
fresh tmpfs there hides every directory the image prepared underneath it: the
install, the user state, the work directory, and the mount point the log share
is attached to from fstab. A box would have come up with a socket and without
any of the places its workload looks for its files, and the exact-path check
could not notice, because what fstab mounts is a directory inside that tree
rather than the tree itself. It moves to /run, which is where a runtime socket
belongs, is a tmpfs already, and has nothing else mounted inside it.
It was also owned by this process and closed to everyone else, which stopped
the workload traversing it to reach the relay at all. The directory is now
readable and searchable, and still writable by nothing but this process, which
is what makes the socket in it unreplaceable; the socket itself is what the
workload is allowed to connect to. The permission belongs on the socket rather
than on the path.
The address served to a reader was built once at startup and served forever, so
a reader that polls for a better one could only ever get the first. An endpoint
does not know all of its own addresses when it binds: the first is the one that
works on the same network and fails from anywhere else. It is now rebuilt per
read, which is what makes polling for it worth doing.
And the address was taken from whoever held a path in a directory the workload
can write. Workload code could unlink the socket a service was listening on,
bind its own, and every read afterwards would hand the client an address of its
choosing -- a session given to somebody else rather than a session that fails.
The peer's credentials are now checked before a byte is read, from the kernel
rather than from anything the peer says about itself, and an address served by
the workload's own user is refused and said loudly.
That check is only worth something while the workload has a user of its own, so
the image grows one. Two users, and they must stay two: one runs the services
that ship in the image, the other is who a workload runs as. Sharing one does
not weaken the check, it makes every session fail it.
A workload running as root is every user at once and cannot be told apart from
anything; the check stands down there and says so at boot instead, because
refusing root would refuse whatever legitimately serves the address as well.
Also bumps tinyvec by a patch release. It does not build on this toolchain --
`vec` resolves to the module and not the macro -- which made every crate that
depends on an endpoint, including this one, unbuildable. Pre-existing and
nothing to do with this change; the lockfile said the same version before it.
The component nescapture, neswire and nescope all talk to, and the only
thing in the guest that speaks to the client. It muxes their frames into
one iroh QUIC endpoint and fans input back.
Renamed from nestri-guest-hub, which named a location rather than a job.
Four files came across unchanged -- session.rs, ipc_listener.rs,
ticket.rs, screenshot.rs. Between them they mention Steam zero times, and
they import only nesprotocol's open modules; the control feature carrying
LaunchIntent and SteamIdentity is used exclusively by the three files that
are staying closed. The two clusters shared a main.rs and nothing else, so
there was no untangling to do -- only a cut.
main.rs loses --proton, --steamclient-so, --root and the game uid/gid,
and no longer ends by handing the process to a controller. It runs until
it is stopped. Deciding when the box is finished belongs to nesinit.
The ticket used to leave via that controller, so it needed a new way out:
neshub now serves it on a socket and nesinit dials for it. Listening
rather than dialling matches every other socket here and means no startup
ordering to get wrong.
Three tests, where there were none -- the ticket crosses a process
boundary as text now, so a round trip that drops a field would otherwise
be found by whoever cannot connect.
Wanjohi
DatCaptainHorseandClaude Opus 5.5
