Runtime half of the window streams contract (joins 1 and 6).
- nesinit runs any number of launches side by side, keyed by launch id.
Each process gets NESTRI_LAUNCH_ID
(nesprotocol::lifecycle::LAUNCH_ID_ENV), set after the caller's
environment so it cannot be overridden. Stop and restart reach one
launch only. A second launch under an id that is already running is
refused.
- A terminal launch ends the box only when no other launch is running,
so the box stops when its last launch exits. A box with one launch
behaves as before.
- New lifecycle messages, additive, no CONTROL_VERSION bump: host to
guest `mount {id, images, overlays}`; guest to host `launch_mounted
{id}` / `launch_mount_failed {id, reason}`. A failed mount does not take
the box down.
- A mount source that is a regular file (in a virtiofs share) is
attached to a loop device first (LOOP_CONFIGURE, autoclear), because
nesbox cannot hot-add block devices to a running VM. The guest kernel
fragment gains CONFIG_BLK_DEV_LOOP=y, so this needs a new guest
kernel/image to work end to end.
Tests: cargo test -p nesinit (117 passed) and the nesprotocol lifecycle
tests on nestripc-1. New session tests cover two launches side by side,
refusing a duplicate id, stopping one launch, the box ending with its
last launch, and mount success/failure/before boot. Not run in a real
box.
Companion: nestrilabs/nescore feat/window-streams-runtime (neslet
launch.start/launch.stop, nesteam --join-compositor).
Content layout and trust:
- Game and compat images come from a read-only virtiofs share
`launch-images` (mounted read-only at
/nestri/launch-images/<launchId>/build.erofs and compat.erofs). The
per-launch writable layer is in the read-write share `launches`
(/nestri/launches/<launchId>/upper.ext4). Which shares exist is decided
host-side in the boot descriptor; nesinit mounts what it is told.
- The share's only guest-writable files are the per-launch upper.ext4
files.
- Mount sources are opened with openat2(RESOLVE_NO_SYMLINKS): a link at
any component is refused, and only block devices and regular files are
accepted. Loop attach and mounts run on the blocking pool, so other
launches' exits and stops are not stalled.
- CONFIG_BLK_DEV_LOOP must ship in the guest kernel together with this
nesinit (build/kernel/nestri.fragment has it); without it a mount into a
running box fails with a launch_mount_failed.
With RLIMIT_RTPRIO alone, neswire's client loop went SCHED_FIFO 83 and
the PipeWire server's graph loop stayed SCHED_OTHER. The server's
module-rt asks for nice -11 first, and when that is refused (Max nice
priority 0) it hands off to rtkit, which grants nothing in a box. The
client config asks for no nice level, which is why only neswire got it.
RLIMIT_NICE 39 (nice down to -19) next to rtprio 95, as PipeWire's own
25-pw-rlimits.conf grants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside a box, every PipeWire, wireplumber and neswire thread was running
SCHED_OTHER. The rtkit -> polkit -> system bus chain started but granted
nothing. With a game keeping the vCPUs busy, the graph's 5 ms cycles ran
late. pw-top counted no xruns, but neswire sent packets in bursts, and
the client saw 15-25 ms gaps in audio alone (video arriving on time in
the same seconds), which emptied its 10 ms cushion every 30 s or so.
pipewire and neswire now exec with RLIMIT_RTPRIO 95, set before the uid
drop. module-rt checks that first and sets SCHED_FIFO itself, which is
how PipeWire is meant to run outside a desktop session. The workload
doesn't get the grant, so a game can't take the CPU away from the graph.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
neslet now puts nestri.port= on the boot line and installs its forward
and pinned masquerade for that port before the box boots. nesinit hands
it to neshub as NESTRI_BIND_PORT, and neshub binds 0.0.0.0:<port> in
place of iroh's random default. Every flow, the relay's included, then
leaves on one public port, and the address the relay reports is the one
peers can punch through to. No nestri.port= keeps the old behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel's default overcommit policy refuses any single allocation
larger than RAM plus swap. A box has no swap, so the ceiling is the
box's own size, and an engine or driver that reserves address space it
never touches is refused before it uses any of it. Dome Keeper asked
for 9 GiB on an 8 GiB box and never started.
Set vm.overcommit_memory=1 alongside vm.max_map_count, so a request is
granted and the OOM killer acts on what is actually used. In a box that
is the game: init has already exempted itself.
A host cannot tell a run somebody is playing from one nobody has looked
at in an hour, so it cannot end the second without guessing.
The hub now answers the address socket with a second line,
`viewers <n>`, counting clients that are streaming. Init reads it
alongside the address and sends a new `viewers` message to the host
when the count changes. A reader that takes only the first line gets
the address exactly as before, a server that never writes the second
line is still read, and a host that does not know the message skips it,
so either end can ship first.
The host can now ask a box for a shell on a pseudo-terminal, the way a container
runtime attaches to a container. `HostToGuest::Attach` starts the process, and
the guest then dials the host on a vsock port of its own (7001) and carries the
terminal's bytes over a small framed stream: data both ways, a resize, a hangup,
and the exit status last. A screenful of output on the JSON-lines control channel
would hold every other message in the box behind it, which is why it is not
there. A failure to start is the exception, and is said on the control channel
because nothing has connected to the data port yet.
A shell is not a launch: it never ends the box, and the box ending ends it. It
is started through the same reaper as everything else, so it cannot become the
zombie that exists to be prevented. With a terminal on the other end the PTY is
ordinary; without one it is put in raw mode so a command run for its output gets
its bytes back unrewritten. Closing the stream hangs the shell up, process group
and all.
The frame codec is a pure bytes-in, frames-out module in nesprotocol with a
payload cap and no skipping of unknown tags, since a length read from a guest is
read across a trust boundary. No version bump: adding a message does not need
one, and a guest that predates it simply never dials.
The runtime stage gains strace, lsof and less, so a root shell in a box is not
one that cannot trace a process.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There is no service manager in a box, so nothing applies the distribution's
sysctl configuration, and the guest ran on the kernel's own defaults. Two of
them are too low for a modern engine under Proton:
- vm.max_map_count defaults to 65530. A large Unreal Engine 5 title maps far
more regions than that, and the failure is a null allocation deep inside the
engine or Wine rather than a message about mappings. It is set to 2147483642,
the value SteamOS and the distributions that package Proton use. It is read
back after the write, because an accepted write that did not take reports
success while the box is still unconfigured.
- RLIMIT_NOFILE is raised on init, which every process in the box inherits. A
child can lower a limit but cannot raise its hard one, so the unprivileged
workload could never get above what it started with. 524288 is the hard limit
Proton's documentation asks for. It is only ever raised.
Both are best effort and each failure says what it costs.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The boot descriptor gains `images`: read-only EROFS images mounted on their
own, with no writable layer over them. nesinit mounts them first, before
overlays and shares, with the same nosuid/nodev guards every layer has and
without noexec because what is in one is run. CONTROL_VERSION moves to 5: the
descriptor refuses fields it does not know, and a guest image without Proton is
useless to a host that does not send one, so the mismatch is named at the
handshake.
The guest rootfs no longer contains Proton. The checks that used to guard the
in-image copy (every library its programs and Wine's core link, an interpreter
behind the entry point) move to a compat_check stage that verifies the Proton
named by PROTON_IMAGE and leaves a marker in the image saying which one. The
leaf stages copy that marker out of it so the stage cannot be skipped as
unused. With nothing at a Proton path, a box started without one fails saying
where it looked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
At this point, why not..
Calling it `nespyro` since it's slightly modified, uses slang shader
language instead, along other things..
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Adds `nesgamepad`, basically "diet-coke vimputti" for Nestri's microVM
needs here.
Comes with direct mapping for:
- Sony
- DualShock 4 (v2), DualSense
- Microsoft
- Xbox 360 pad, Xbox One S pad
- Nintendo
- Pro Controller
- Generic pad for unknown controllers
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Get this thing going..
<!-- greptile_comment -->
<!-- greptile_summary -->
<h2><a
href="https://app.greptile.com/api/retrigger?id=63134761"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/RetriggerDark.svg?v=1"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"><img
alt="Retrigger"
src="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"
align="right"></picture></a>Confidence Score: 5/5</h2>
The PR appears safe to merge; all previous findings are resolved and the
latest readiness change introduces no established actionable regression.
<h3>Summary</h3>
- Establishes required guest filesystems, runtime directories, device
permissions, and service processes.
- Reports initialization and service deaths over the lifecycle channel.
- Supports launch, restart, and shutdown commands for a resident guest.
- Separates service and workload identities and configures per-launch
runtime environments.
- Removes the currently inactive nescope screenshot option and makes
capture-chain verification fail explicitly when compositor readback is
unavailable.
- Reworks the guest image around `nesinit` as PID 1 without a
distribution service manager.
<h3>Diagram</h3>
```mermaid
sequenceDiagram
participant Host
participant Init as nesinit
participant FS as Guest filesystems
participant Services as Service stack
participant Workload
Init->>Host: Ready(protocol version)
Host->>Init: Boot(mount descriptors)
Init->>FS: Establish and mount shares
Init->>Services: Spawn services in order
Services-->>Init: Required sockets ready
Init->>Host: Initialized(service names)
Host->>Init: Launch(id, exec, on_exit)
Init->>Workload: Spawn with isolated UID/runtime
Init->>Host: Started(id)
Workload-->>Init: Exit status
Init->>Host: WorkloadExited(id, status)
Host->>Init: Launch / Restart / Shutdown
```
<sub>Reviews (4) · Last reviewed commit: ["fix(nesinit): readiness is a
socket
that..."](https://github.com/nestrilabs/nestri/commit/731d34df9df30463f67963363424cb9487e89196)</sub>
<!-- /greptile_comment -->
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Four findings from review, all of them real.
The relay's directory was mounted on the tree a session's shares live in. A
fresh tmpfs there hides every directory the image prepared underneath it: the
install, the user state, the work directory, and the mount point the log share
is attached to from fstab. A box would have come up with a socket and without
any of the places its workload looks for its files, and the exact-path check
could not notice, because what fstab mounts is a directory inside that tree
rather than the tree itself. It moves to /run, which is where a runtime socket
belongs, is a tmpfs already, and has nothing else mounted inside it.
It was also owned by this process and closed to everyone else, which stopped
the workload traversing it to reach the relay at all. The directory is now
readable and searchable, and still writable by nothing but this process, which
is what makes the socket in it unreplaceable; the socket itself is what the
workload is allowed to connect to. The permission belongs on the socket rather
than on the path.
The address served to a reader was built once at startup and served forever, so
a reader that polls for a better one could only ever get the first. An endpoint
does not know all of its own addresses when it binds: the first is the one that
works on the same network and fails from anywhere else. It is now rebuilt per
read, which is what makes polling for it worth doing.
And the address was taken from whoever held a path in a directory the workload
can write. Workload code could unlink the socket a service was listening on,
bind its own, and every read afterwards would hand the client an address of its
choosing -- a session given to somebody else rather than a session that fails.
The peer's credentials are now checked before a byte is read, from the kernel
rather than from anything the peer says about itself, and an address served by
the workload's own user is refused and said loudly.
That check is only worth something while the workload has a user of its own, so
the image grows one. Two users, and they must stay two: one runs the services
that ship in the image, the other is who a workload runs as. Sharing one does
not weaken the check, it makes every session fail it.
A workload running as root is every user at once and cannot be told apart from
anything; the check stands down there and says so at boot instead, because
refusing root would refuse whatever legitimately serves the address as well.
Also bumps tinyvec by a patch release. It does not build on this toolchain --
`vec` resolves to the module and not the macro -- which made every crate that
depends on an endpoint, including this one, unbuildable. Pre-existing and
nothing to do with this change; the lockfile said the same version before it.
There was a cap on how much this would read and none on how long it would
wait. The far end can accept a connection and then write nothing, and a
read with no deadline turns that into a poll loop that never runs again:
the address already forwarded stays correct, and the better one that
arrives afterwards is never seen. That is the failure this polls to avoid,
reached by a different route.
Five seconds, against a two second interval, so an answer that is merely
slow still lands and one that is never coming is abandoned.
The test hangs against the code as it was, which is the whole point of it.
The root arrives read-only and this process is PID 1, so until it mounts
them there is no /proc and nowhere in the filesystem to put a socket.
Nothing else in the guest is an init system, so nothing else was going to.
The symptom was three failures that look unrelated and share one cause. On
a real box the payload relay could not bind, with EROFS; whatever serves
the session's address could not bind either, the same way; and this process
could not make itself ineligible for the OOM killer, because /proc was not
there to write to. What the caller saw was a workload that ran and
published nothing, which is true and says nothing about why.
/proc is mounted first and unconditionally: finding out what an image
already mounted requires it, and it is therefore the one entry that cannot
be checked that way itself. Everything after it is skipped when it is
already present, so an image that does this properly is not mounted over.
Failures warn rather than abort. Refusing to boot would replace a session
that fails with a reason by a guest that never dialled out at all, and the
second is harder to diagnose from the outside.
The relay's directory is named by the module that owns the socket rather
than spelled again here, with a test tying the two together: a rename that
reached one and not the other would put the relay back exactly as it was.
Whatever serves media in a box knows how it can be reached, and the person who
needs to know is not in the box. Standard output here is a log file inside a
VM, so the control channel is the delivery path rather than a convenience --
which makes this init's job and not a detail of whichever component happens to
bind the port. The socket it reads was already documented as being read this
way; nothing read it.
Polled rather than read once, because an address is not a value but the best
answer so far. An endpoint discovers more ways to reach it after it binds, so
the first answer is the one that works on a local network and fails from
anywhere else. Only a changed answer is forwarded.
It dials rather than listens, which is the opposite of the relay next door and
deliberate: there the guest listens because the workload starts later, and here
the server is the long-lived one. Dialling also makes a server that has not
bound yet something to retry rather than something to wait for without knowing
whether it is coming.
The address itself is never logged. It is a capability to reach the session,
and a log inside the guest is the one place it has no reason to be.
A carrier that stops does not end a session: whatever was already reported is
still correct, and the workload's exit still has to be.
Three problems in the relay, all of them found in review.
Handing an envelope over waited for room. That loop also carries stop,
shutdown and the workload's exit, so a workload slow to read its own mail — or
one that never connected — could hold the lifecycle layer still behind it. It
never waits now: an envelope that will not fit is dropped, which costs nothing,
because what crosses this layer is re-sent when it changes.
Envelopes were queued for a workload that was not there. The queue filled with
copies that would be stale by the time anyone connected, and filling it was
what stalled the session. Nothing is held while the socket has nobody on it.
A frame had no maximum length. The workload can write for as long as it likes
without ever sending a newline, and the process assembling that is the one the
kernel has been told not to kill, so the memory it takes comes out of
everything else in the guest. Past 64 KiB the connection is dropped and the
relay waits for the next one; the failure says how long the frame got and
nothing about what was in it.
Also, a tag or a mount point with a nul byte in it was quietly turned into an
empty string, so an unmountable descriptor arrived later as a mount failure
about something else, after the mount point had already been created. It is
refused by name now, before anything is created.
The relay's tests grew a harness that waits for the connection to be carried
before sending anything down it, because dropping what arrives with nobody
connected made "connected" something a test has to establish rather than
assume.
Builds on the previous change, which had PID 1, the channel and the trait but
mounted nothing.
The shares are mounted now: a tag names an export, the descriptor names where
it lands, and every share goes on nosuid and nodev whether or not it is
writable — a share is data handed to the guest, and no descriptor has a way to
ask for a setuid binary or a device node in one. Mounting needs privileges a
test does not have, so the arguments and flags are derived by a function the
tests can assert, which is where the read-only decision lives.
Progress is reported in two messages rather than one. A share that did not
mount and a command that did not run are not the same incident, and each
carries the reason the operating system gave and the path it happened on: a
permission error on a named directory can be acted on, where "the share did not
mount" cannot.
The second layer is relayed and never read. Bytes arrive on the channel in an
envelope, cross a unix socket to the workload, and come back the same way. The
body is a string rather than nested JSON on purpose: a document this component
can index into is a document it can grow to depend on, and then the layer is no
longer opaque and the boundary it exists to draw is gone. An envelope is never
logged — not the body, not truncated, not at debug level — and the channel name
with a byte count is the whole of what may be said about one. The type's Debug
is written by hand for the same reason, because a derived one puts the body one
careless format string away from a log line.
A write to a channel nobody is reading now ends the session the same way a
closed read does. A caller that has stopped listening has also stopped being
able to say stop, which is one situation and was two outcomes.
The guest listens on the relay socket and the workload dials in, which is the
convention the other guest sockets already use and removes the startup ordering
problem: a workload that is not running yet has simply not connected yet.
A pid is only a name for a process until that process is reaped; after that
the kernel may hand the same number to something else. The handle kept the
number, so a stop or a kill issued during shutdown — which every session
outcome reaches — could land on a process nobody meant, and the one aimed at
the workload would have been a SIGKILL.
The registry now clears the flag as it delivers the exit, in the same call, and
nothing signals a pid whose flag is down. Waiting for the workload on the way
out takes the same answer: already reaped is already gone.
There is a window left, between the reap and the delivery, and closing it
entirely needs a handle the kernel keeps rather than a number. Recorded rather
than papered over.
Three problems in the shutdown and reaping paths, all of them found in review.
Reaping and waiting cannot be two mechanisms. `waitpid(-1, ...)` collects any
child, so the reaper and a caller waiting on its own child race for the same
status, and whichever loses gets nothing — losing the workload's exit, which is
the one thing this component exists to report. The reaper is now the only
waiter and hands each exit to whoever asked for that pid. Registering interest
holds the same lock the delivery takes, so a child that exits before its caller
is registered is still delivered rather than dropped; there is a test that
fails without that.
Killing the workload killed everything. `kill(-1, SIGKILL)` is every process
init may signal, so a workload that overstayed its grace period took the
guest's services with it, before the ordered stop the shutdown promises them
had even started. It signals the one pid now.
The shutdown had no workload to stop. It built a fresh handle with no pid, so
the graceful stop was a no-op and the workload only died in the sweep that
follows — which is exactly the order this was written to avoid. The handle the
session used is now the handle the shutdown uses, and waiting for the workload
waits for that pid rather than for any child to leave.
A microVM has no init unless something is it, and three of the jobs belong to
nothing else in the guest: reaping whatever the workload orphans, turning a
signal into an ordered shutdown, and being the guest end of the one channel
out.
None of it knows what it is running. The guest dials out on a fixed vsock port,
says its protocol version first, is handed one boot descriptor — a command
line, shares, output geometry, and what an exit means — and carries that out.
There is no code path that branches on which workload started, which is the
property the component exists to keep.
It reports and does not supervise. When the workload ends, the exit goes up the
channel and the session is over; `on_exit` says what the exit means, and
starting something again is a decision for the end that can see whether
restarting is repair or a loop. A signalled workload is reported as signalled
with no exit code, because reporting 0 for a killed process makes a kill look
like a clean run.
Two seams keep this testable without a VM, which is the reason for both of
them. Reaping runs against real forked children, with the subreaper bit making
a test process inherit orphans the way PID 1 does. The channel is generic over
the byte stream, so the exchange is driven over an in-memory pipe — the
transport contributes nothing to the protocol beyond ordering and framing.
The lifecycle types live in nesprotocol behind a feature, off by default: both
ends of the channel read one definition and cannot drift from it silently,
while the media components keep building without serde.
Mounting shares is not implemented in this build. The descriptor's mounts are
refused rather than ignored — a workload started without the shares it was
promised fails later, somewhere else, for a reason nobody can see from here.
Wanjohi
DatCaptainHorseandClaude Opus 5.5
KAAL1
