Three additions, all of them plumbing for a bitrate that something
actually decides.
`BootDescriptor` gains `VideoLimits`. It rides the boot document rather
than a kernel command line because `nesinit` handles `Boot` by mounting
and *then* bringing the stack up, so the value is in hand before `neshub`
is spawned -- no parsing, no window where the service is running without
its configuration. It is on the descriptor rather than a launch because
its consumer is a service that comes up with the box; geometry went the
other way for the same reason, its consumer being started per launch.
`bitrate_kbps` is an `Option` and the distinction is load-bearing.
"Nobody said" is not zero and is not unlimited, and a reader that
conflates the first with the last reproduces the bug exactly: every
session offered 10 Mbps because no number had ever been chosen and the
encoder's own default stood in for one. `neshub` now says which it got,
and falls back to something modest rather than to whatever it finds.
Note what `deny_unknown_fields` means here, since it is deliberate: a
host that sends `video` to a guest too old to know the field is refused
rather than quietly served. That is the right direction to fail -- the
alternative is a box that boots, streams, and ignores its ceiling -- and
it means the guest image is rebuilt before a host starts sending one.
`MSG_RECEIVER_REPORT` carries what a second looked like from the far
end: goodput actually released to the decoder, frames released,
incomplete and never-arrived, and the receiver's own RTT. The hub cannot
work any of this out for itself. Its own view was measured saying the
path was healthy while almost nothing was arriving, and one reason is
structural -- `send_datagram` evicts the oldest queued datagrams and
returns `Ok`, so the send side has no backpressure signal at all.
`MSG_CONTROL_MODE` says who is choosing the bitrate. Manual exists
because it is how this class of bug gets found: the original report said
the bitrate had already been lowered, and the only way anyone
established otherwise was by setting one by hand and watching the
picture come back.
Both decoders refuse what they cannot read rather than guessing.
`loss()` returns `None` for a second that accounted for no frames at
all, because a second with nothing sent and a second with nothing
arriving are indistinguishable from there, and answering either 0% or
100% would tell a controller something nobody knows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Get this thing going..
<!-- greptile_comment -->
<!-- greptile_summary -->
<h2><a
href="https://app.greptile.com/api/retrigger?id=63134761"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/RetriggerDark.svg?v=1"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"><img
alt="Retrigger"
src="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"
align="right"></picture></a>Confidence Score: 5/5</h2>
The PR appears safe to merge; all previous findings are resolved and the
latest readiness change introduces no established actionable regression.
<h3>Summary</h3>
- Establishes required guest filesystems, runtime directories, device
permissions, and service processes.
- Reports initialization and service deaths over the lifecycle channel.
- Supports launch, restart, and shutdown commands for a resident guest.
- Separates service and workload identities and configures per-launch
runtime environments.
- Removes the currently inactive nescope screenshot option and makes
capture-chain verification fail explicitly when compositor readback is
unavailable.
- Reworks the guest image around `nesinit` as PID 1 without a
distribution service manager.
<h3>Diagram</h3>
```mermaid
sequenceDiagram
participant Host
participant Init as nesinit
participant FS as Guest filesystems
participant Services as Service stack
participant Workload
Init->>Host: Ready(protocol version)
Host->>Init: Boot(mount descriptors)
Init->>FS: Establish and mount shares
Init->>Services: Spawn services in order
Services-->>Init: Required sockets ready
Init->>Host: Initialized(service names)
Host->>Init: Launch(id, exec, on_exit)
Init->>Workload: Spawn with isolated UID/runtime
Init->>Host: Started(id)
Workload-->>Init: Exit status
Init->>Host: WorkloadExited(id, status)
Host->>Init: Launch / Restart / Shutdown
```
<sub>Reviews (4) · Last reviewed commit: ["fix(nesinit): readiness is a
socket
that..."](731d34df9d)</sub>
<!-- /greptile_comment -->
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Four findings from review, all of them real.
The relay's directory was mounted on the tree a session's shares live in. A
fresh tmpfs there hides every directory the image prepared underneath it: the
install, the user state, the work directory, and the mount point the log share
is attached to from fstab. A box would have come up with a socket and without
any of the places its workload looks for its files, and the exact-path check
could not notice, because what fstab mounts is a directory inside that tree
rather than the tree itself. It moves to /run, which is where a runtime socket
belongs, is a tmpfs already, and has nothing else mounted inside it.
It was also owned by this process and closed to everyone else, which stopped
the workload traversing it to reach the relay at all. The directory is now
readable and searchable, and still writable by nothing but this process, which
is what makes the socket in it unreplaceable; the socket itself is what the
workload is allowed to connect to. The permission belongs on the socket rather
than on the path.
The address served to a reader was built once at startup and served forever, so
a reader that polls for a better one could only ever get the first. An endpoint
does not know all of its own addresses when it binds: the first is the one that
works on the same network and fails from anywhere else. It is now rebuilt per
read, which is what makes polling for it worth doing.
And the address was taken from whoever held a path in a directory the workload
can write. Workload code could unlink the socket a service was listening on,
bind its own, and every read afterwards would hand the client an address of its
choosing -- a session given to somebody else rather than a session that fails.
The peer's credentials are now checked before a byte is read, from the kernel
rather than from anything the peer says about itself, and an address served by
the workload's own user is refused and said loudly.
That check is only worth something while the workload has a user of its own, so
the image grows one. Two users, and they must stay two: one runs the services
that ship in the image, the other is who a workload runs as. Sharing one does
not weaken the check, it makes every session fail it.
A workload running as root is every user at once and cannot be told apart from
anything; the check stands down there and says so at boot instead, because
refusing root would refuse whatever legitimately serves the address as well.
Also bumps tinyvec by a patch release. It does not build on this toolchain --
`vec` resolves to the module and not the macro -- which made every crate that
depends on an endpoint, including this one, unbuildable. Pre-existing and
nothing to do with this change; the lockfile said the same version before it.
There was a cap on how much this would read and none on how long it would
wait. The far end can accept a connection and then write nothing, and a
read with no deadline turns that into a poll loop that never runs again:
the address already forwarded stays correct, and the better one that
arrives afterwards is never seen. That is the failure this polls to avoid,
reached by a different route.
Five seconds, against a two second interval, so an answer that is merely
slow still lands and one that is never coming is abandoned.
The test hangs against the code as it was, which is the whole point of it.
The root arrives read-only and this process is PID 1, so until it mounts
them there is no /proc and nowhere in the filesystem to put a socket.
Nothing else in the guest is an init system, so nothing else was going to.
The symptom was three failures that look unrelated and share one cause. On
a real box the payload relay could not bind, with EROFS; whatever serves
the session's address could not bind either, the same way; and this process
could not make itself ineligible for the OOM killer, because /proc was not
there to write to. What the caller saw was a workload that ran and
published nothing, which is true and says nothing about why.
/proc is mounted first and unconditionally: finding out what an image
already mounted requires it, and it is therefore the one entry that cannot
be checked that way itself. Everything after it is skipped when it is
already present, so an image that does this properly is not mounted over.
Failures warn rather than abort. Refusing to boot would replace a session
that fails with a reason by a guest that never dialled out at all, and the
second is harder to diagnose from the outside.
The relay's directory is named by the module that owns the socket rather
than spelled again here, with a test tying the two together: a rename that
reached one and not the other would put the relay back exactly as it was.
Whatever serves media in a box knows how it can be reached, and the person who
needs to know is not in the box. Standard output here is a log file inside a
VM, so the control channel is the delivery path rather than a convenience --
which makes this init's job and not a detail of whichever component happens to
bind the port. The socket it reads was already documented as being read this
way; nothing read it.
Polled rather than read once, because an address is not a value but the best
answer so far. An endpoint discovers more ways to reach it after it binds, so
the first answer is the one that works on a local network and fails from
anywhere else. Only a changed answer is forwarded.
It dials rather than listens, which is the opposite of the relay next door and
deliberate: there the guest listens because the workload starts later, and here
the server is the long-lived one. Dialling also makes a server that has not
bound yet something to retry rather than something to wait for without knowing
whether it is coming.
The address itself is never logged. It is a capability to reach the session,
and a log inside the guest is the one place it has no reason to be.
A carrier that stops does not end a session: whatever was already reported is
still correct, and the workload's exit still has to be.
Three problems in the relay, all of them found in review.
Handing an envelope over waited for room. That loop also carries stop,
shutdown and the workload's exit, so a workload slow to read its own mail — or
one that never connected — could hold the lifecycle layer still behind it. It
never waits now: an envelope that will not fit is dropped, which costs nothing,
because what crosses this layer is re-sent when it changes.
Envelopes were queued for a workload that was not there. The queue filled with
copies that would be stale by the time anyone connected, and filling it was
what stalled the session. Nothing is held while the socket has nobody on it.
A frame had no maximum length. The workload can write for as long as it likes
without ever sending a newline, and the process assembling that is the one the
kernel has been told not to kill, so the memory it takes comes out of
everything else in the guest. Past 64 KiB the connection is dropped and the
relay waits for the next one; the failure says how long the frame got and
nothing about what was in it.
Also, a tag or a mount point with a nul byte in it was quietly turned into an
empty string, so an unmountable descriptor arrived later as a mount failure
about something else, after the mount point had already been created. It is
refused by name now, before anything is created.
The relay's tests grew a harness that waits for the connection to be carried
before sending anything down it, because dropping what arrives with nobody
connected made "connected" something a test has to establish rather than
assume.
Builds on the previous change, which had PID 1, the channel and the trait but
mounted nothing.
The shares are mounted now: a tag names an export, the descriptor names where
it lands, and every share goes on nosuid and nodev whether or not it is
writable — a share is data handed to the guest, and no descriptor has a way to
ask for a setuid binary or a device node in one. Mounting needs privileges a
test does not have, so the arguments and flags are derived by a function the
tests can assert, which is where the read-only decision lives.
Progress is reported in two messages rather than one. A share that did not
mount and a command that did not run are not the same incident, and each
carries the reason the operating system gave and the path it happened on: a
permission error on a named directory can be acted on, where "the share did not
mount" cannot.
The second layer is relayed and never read. Bytes arrive on the channel in an
envelope, cross a unix socket to the workload, and come back the same way. The
body is a string rather than nested JSON on purpose: a document this component
can index into is a document it can grow to depend on, and then the layer is no
longer opaque and the boundary it exists to draw is gone. An envelope is never
logged — not the body, not truncated, not at debug level — and the channel name
with a byte count is the whole of what may be said about one. The type's Debug
is written by hand for the same reason, because a derived one puts the body one
careless format string away from a log line.
A write to a channel nobody is reading now ends the session the same way a
closed read does. A caller that has stopped listening has also stopped being
able to say stop, which is one situation and was two outcomes.
The guest listens on the relay socket and the workload dials in, which is the
convention the other guest sockets already use and removes the startup ordering
problem: a workload that is not running yet has simply not connected yet.
A pid is only a name for a process until that process is reaped; after that
the kernel may hand the same number to something else. The handle kept the
number, so a stop or a kill issued during shutdown — which every session
outcome reaches — could land on a process nobody meant, and the one aimed at
the workload would have been a SIGKILL.
The registry now clears the flag as it delivers the exit, in the same call, and
nothing signals a pid whose flag is down. Waiting for the workload on the way
out takes the same answer: already reaped is already gone.
There is a window left, between the reap and the delivery, and closing it
entirely needs a handle the kernel keeps rather than a number. Recorded rather
than papered over.
Three problems in the shutdown and reaping paths, all of them found in review.
Reaping and waiting cannot be two mechanisms. `waitpid(-1, ...)` collects any
child, so the reaper and a caller waiting on its own child race for the same
status, and whichever loses gets nothing — losing the workload's exit, which is
the one thing this component exists to report. The reaper is now the only
waiter and hands each exit to whoever asked for that pid. Registering interest
holds the same lock the delivery takes, so a child that exits before its caller
is registered is still delivered rather than dropped; there is a test that
fails without that.
Killing the workload killed everything. `kill(-1, SIGKILL)` is every process
init may signal, so a workload that overstayed its grace period took the
guest's services with it, before the ordered stop the shutdown promises them
had even started. It signals the one pid now.
The shutdown had no workload to stop. It built a fresh handle with no pid, so
the graceful stop was a no-op and the workload only died in the sweep that
follows — which is exactly the order this was written to avoid. The handle the
session used is now the handle the shutdown uses, and waiting for the workload
waits for that pid rather than for any child to leave.
A microVM has no init unless something is it, and three of the jobs belong to
nothing else in the guest: reaping whatever the workload orphans, turning a
signal into an ordered shutdown, and being the guest end of the one channel
out.
None of it knows what it is running. The guest dials out on a fixed vsock port,
says its protocol version first, is handed one boot descriptor — a command
line, shares, output geometry, and what an exit means — and carries that out.
There is no code path that branches on which workload started, which is the
property the component exists to keep.
It reports and does not supervise. When the workload ends, the exit goes up the
channel and the session is over; `on_exit` says what the exit means, and
starting something again is a decision for the end that can see whether
restarting is repair or a loop. A signalled workload is reported as signalled
with no exit code, because reporting 0 for a killed process makes a kill look
like a clean run.
Two seams keep this testable without a VM, which is the reason for both of
them. Reaping runs against real forked children, with the subreaper bit making
a test process inherit orphans the way PID 1 does. The channel is generic over
the byte stream, so the exchange is driven over an in-memory pipe — the
transport contributes nothing to the protocol beyond ordering and framing.
The lifecycle types live in nesprotocol behind a feature, off by default: both
ends of the channel read one definition and cannot drift from it silently,
while the media components keep building without serde.
Mounting shares is not implemented in this build. The descriptor's mounts are
refused rather than ignored — a workload started without the shares it was
promised fails later, somewhere else, for a reason nobody can see from here.
