f74de9beb8
Four findings from review, all of them real. The relay's directory was mounted on the tree a session's shares live in. A fresh tmpfs there hides every directory the image prepared underneath it: the install, the user state, the work directory, and the mount point the log share is attached to from fstab. A box would have come up with a socket and without any of the places its workload looks for its files, and the exact-path check could not notice, because what fstab mounts is a directory inside that tree rather than the tree itself. It moves to /run, which is where a runtime socket belongs, is a tmpfs already, and has nothing else mounted inside it. It was also owned by this process and closed to everyone else, which stopped the workload traversing it to reach the relay at all. The directory is now readable and searchable, and still writable by nothing but this process, which is what makes the socket in it unreplaceable; the socket itself is what the workload is allowed to connect to. The permission belongs on the socket rather than on the path. The address served to a reader was built once at startup and served forever, so a reader that polls for a better one could only ever get the first. An endpoint does not know all of its own addresses when it binds: the first is the one that works on the same network and fails from anywhere else. It is now rebuilt per read, which is what makes polling for it worth doing. And the address was taken from whoever held a path in a directory the workload can write. Workload code could unlink the socket a service was listening on, bind its own, and every read afterwards would hand the client an address of its choosing -- a session given to somebody else rather than a session that fails. The peer's credentials are now checked before a byte is read, from the kernel rather than from anything the peer says about itself, and an address served by the workload's own user is refused and said loudly. That check is only worth something while the workload has a user of its own, so the image grows one. Two users, and they must stay two: one runs the services that ship in the image, the other is who a workload runs as. Sharing one does not weaken the check, it makes every session fail it. A workload running as root is every user at once and cannot be told apart from anything; the check stands down there and says so at boot instead, because refusing root would refuse whatever legitimately serves the address as well. Also bumps tinyvec by a patch release. It does not build on this toolchain -- `vec` resolves to the module and not the macro -- which made every crate that depends on an endpoint, including this one, unbuildable. Pre-existing and nothing to do with this change; the lockfile said the same version before it.
nesinit
PID 1 inside a box.
A microVM has no init unless something is it. Three of the jobs are nobody else's, and this is all of them:
- Reaping. A process whose parent dies is reparented to PID 1. Without a reaper, every orphan the workload leaves behind holds a pid and a slot in the process table until the guest is gone.
- Ordered shutdown. The workload stops first and alone, then everything else, then the disks are flushed and the machine is powered off. An init that returns leaves a guest running with nothing in it.
- The guest end of the control channel. One vsock connection out, carrying what to run in and what happened back.
It does not know what it is running. It is handed a command line, a set of shares and what an exit means; there is no code path here that branches on which workload it started, and there is not meant to be.
The channel
The guest dials out on a fixed vsock port and speaks first:
guest → { "type": "ready", "protocol_version": 2 }
guest ← { "type": "boot", "exec": {...}, "mounts": [...], "geometry": {...}, "on_exit": {...} }
guest → { "type": "mounted" }
guest → { "type": "started" }
guest → { "type": "workload_exited", "exit_code": 0 }
Newline-delimited JSON. Dialling out rather than being connected to is worth keeping for two reasons: the listener is up before the VM starts, so nothing races a booting kernel and nothing has to retry, and the connection establishing is itself the liveness signal — without it the far end needs a timeout to tell a slow boot from a dead one.
The version goes out before anything is read, so a peer that cannot talk to this build refuses it before handing over a descriptor rather than failing later on a field that turned out to be missing.
The types are in nesprotocol::lifecycle,
behind the lifecycle feature, so both ends of the channel read one definition
and neither can drift from it silently.
mounted / mount_failed stay separate from started / start_failed
because the two want different things looked at: a share that did not appear
and a command that did not run are not the same incident. A failure carries the
reason in the words the operating system used, and the path it happened on — a
permission error on a named directory can be acted on, where "the share did not
mount" cannot.
Two layers, one channel
The channel carries a lifecycle layer, above, and a payload layer that nesinit relays and never reads:
{ "type": "payload", "channel": "<name>", "body": "<opaque string>" }
Both directions. Inside the guest an envelope crosses a unix socket at
/nestri/payload.sock, which the guest listens on and the workload dials into.
That socket is a mechanism and expected to change; the envelope is the boundary
and is not.
Nothing is held for a workload that is not on the relay, and nothing waits on one that is slow to read. An envelope that arrives with nobody connected is dropped, and so is one that arrives faster than the workload reads: what crosses this layer is re-sent when it changes, so a queued copy is a stale copy — and the queue that would hold it is on the same loop that carries stop, shutdown and the workload's exit, none of which may wait behind it.
A frame is capped at 64 KiB. The workload is on the other end of that socket and can write for as long as it likes without ever sending a newline; the process assembling it is the one the kernel has been told not to kill, so an unbounded buffer there comes out of everything else in the guest. Past the cap the connection is dropped and the relay waits for the next one.
body is a string rather than nested JSON, deliberately. A document nesinit
can index into is a document nesinit can grow to depend on, and then the layer
is no longer opaque and the boundary it exists to draw is gone.
An envelope is never logged. Not the body, not truncated, not at debug
level. The channel name and the byte count are the whole of what may be said
about one — what crosses here includes credentials meant for the workload and
nothing else. Payload's Debug is written by hand for the same reason: a
derived one puts the body one careless {:?} away from a log line.
The shares
Each mounts entry is a tag, a path to put it at, and whether it is read-only.
The tag names an export and is never a path on the other side of the channel,
so the guest learns nothing about the filesystem it is handed a piece of.
Choosing where a share lands is the descriptor's job, not the guest's:
deciding that means knowing what the workload expects to find there, which is
exactly the knowledge a workload-independent init does not have.
Every share is mounted nosuid and nodev, whether or not it is writable. A
share is data handed to the guest, and no descriptor has a way to ask for a
setuid binary or a device node in one.
uid and gid in exec are load-bearing rather than hygiene. Whoever writes
the descriptor also exported the writable share, so the two have to agree; when
they do not, the first write is refused and the failure surfaces here as a
permission error with a path, instead of as a workload that misbehaves much
later for no visible reason.
It reports; it does not supervise
When the workload ends, the exit goes up the channel and the session is over.
on_exit says what that exit means — whether it ends the session — and
nothing here restarts anything. Starting something again is a decision for the
end that can see whether restarting is repair or a loop.
A signalled workload is reported as signalled, with no exit code. Reporting
0 for a killed process would make a kill look like a clean run.
What is not here yet
geometry is carried and parsed but nothing consumes it: nesinit does not
start the guest's own services yet. ticket exists as a message with no
producer wired to it.
Testing
cargo test -p nesinit
No VM required, and that is the point of the seams. Reaping is tested against
real forked children — PR_SET_CHILD_SUBREAPER makes a test process inherit
orphans the same way PID 1 does. The channel is tested over an in-memory pipe,
because the transport contributes nothing to the protocol beyond ordering and
framing. The relay is tested over a real unix socket. Mounting needs
privileges a test does not have, so what is asserted is the arguments and flags
the mount is given, which is where the read-only and nosuid decisions live.