mirror of
https://github.com/nestriness/nestri.git
synced 2026-09-19 09:15:19 +03:00
0e94620808
## What was missing `neshub` serves the session's address on a socket, and its own flag has always said how that address gets out: > *"neshub listens; nesinit dials and carries the ticket to the host, because > the person who needs it is outside this VM and stdout here is a log file > inside one."* Nothing dialled it. `grep -rn ticket apps/nesinit/src` returned nothing at all, so the address never left the guest — and the one lifecycle message for it, `Ticket`, had no sender. This adds the carrier. ## Three decisions worth reading **Polled, not read once.** An address is not a value, it is the best answer so far: an endpoint discovers more ways to reach it after it binds — a local one immediately, a relayed one seconds later. Reading once means whoever asked first decides, and the first answer is the one that works on a local network and fails from anywhere else. Only a *changed* answer is forwarded, so an unchanged one costs nothing. **Dials, does not listen.** The opposite of the payload relay next door, and deliberately so. There the guest listens because the workload starts later; here the server is the long-lived one. Dialling also makes "not bound yet" an error to retry rather than a connection to wait for without knowing whether it is coming — which is the ordinary case at boot, since this starts before the server does. **The address is never logged.** It is a capability to reach the session, and a log inside the guest is the one place it has no reason to be. The log line says whether it is the first one and nothing else. ## Failing first The carrier's tests fail against unmodified code by not compiling: `ticket.rs` does not exist and `session::run` takes three arguments. Said plainly rather than manufactured. The behavioural gap is better shown as the `grep` above — nothing in the guest ever sent a `Ticket`, so the message had one end. `nesinit`: **35 tests passing**, up from 27. Four on the carrier itself (an address arrives; a better one replaces it; a socket that is not there yet is waited out rather than failed; an empty answer is not an address) and two on the session (an address reaches the caller as `Ticket`; a carrier that stops does not end the session). ## Verified in a real guest Built static for musl, run as PID 1 in a real microVM under a real VMM with a real vsock. It dialled out, completed the handshake at version 2, took a boot descriptor, started its workload, read the address that workload published and sent it up the channel — twice, the second time because a better one appeared. The caller saw `nestri:local-only` and then `nestri:with-relays`. Guest boot to init was **310 ms**. ## Review round (f74de9b, on top of two fixes from a hardware run) `18864b9` and `00a2bda` came from running this on real hardware and are the reason it works at all: nothing was mounting `/proc` or anywhere writable, so the relay and the address socket both failed with `EROFS` and the session was reported as a workload that ran and published nothing. And one look at the socket had a size cap but no time cap, so a peer that accepted and then said nothing stopped the search for a better address permanently. Four review findings on top, all real: | finding | outcome | |---|---| | a tmpfs over the share tree hid the install, user, work and log-share directories the image prepares | moved to `/run`, where a runtime socket belongs; a test asserts the share tree is never mounted over, and another that nothing is mounted before the mount containing it | | the relay directory's mode stopped the workload reaching the socket | the directory is searchable and writable only by init — which is what makes the socket unreplaceable — and the socket itself is what the workload may connect to | | the address was built once and served forever, so polling could only return the first one | rebuilt per read from the endpoint, which is what makes polling worth doing | | the workload could replace the socket and publish an address of its choosing | `SO_PEERCRED` before a byte is read; an address from the workload's user is refused and logged loudly. **Fails closed**, so the host's default workload uid moved off the services' uid and the image grew a second user — see the thread | **These two PRs are now order-dependent.** This one merged alone, against a host still defaulting the workload to the services' uid, refuses every legitimate address. Also bumps `tinyvec` by a patch release. It does not build on this toolchain — `vec` resolves to the module, not the macro — which made every crate depending on an endpoint unbuildable, `neshub` included. Pre-existing: the lockfile named the same version before this branch. It is why `neshub` could be built and tested here at all. **40 tests** in `nesinit` (was 27), 3 in `neshub`, fmt clean, and no clippy warning in any file this branch touches. ## What this does not verify - **The separated-uid pairing has not been run on hardware.** The rig has no init system, so the only process it can start is the workload — which means the address producer *is* the workload and the two cannot be given different uids in it. The refusal and the root stand-down are both verified on a real VM; the legitimate combination is covered by a unit test over a real socket and real kernel credentials, and by nothing else. - **The image was not built.** The second user is one `useradd` in a Dockerfile that needs a base image this machine does not have. - **The real address server was not in the loop.** The socket was served by a stand-in written for the test, which speaks the same one-line protocol. The real one has never had anything read from it, and its behaviour when it learns a new address — whether it re-serves the newer one on the next dial, which is the entire premise of polling — is **asserted, not measured**. - **No media.** Nothing rendered or encoded; the address pointed at nothing. - **Nobody connected to it.** The address left the guest. It was never dialled. - **Not the real guest image.** A 3 MB rootfs built for this, with no init system, no compositor and no Mesa. The image build does not ship this binary at all — its `default` runlevel still registers services on the assumption that something else starts the payload. - **The relay was down for the whole run**, because the test root was read-only and `/nestri` could not be created. That is non-fatal by design and the session ran anyway, but it means the payload layer and this carrier have not been exercised in the same boot. - **One address, one guest, one session.** Nothing concurrent. <!-- greptile_comment --> <h3>Greptile Summary</h3> This PR carries changing session tickets from `neshub` through `nesinit` to the host and establishes writable runtime filesystems for guest sockets. - Rebuilds tickets from the endpoint on every IPC connection so newly learned addresses can propagate. - Polls the ticket socket and forwards changed tickets over the lifecycle channel. - Moves the payload socket into a dedicated `/run/nestri` runtime directory. - Adds workload/service UID separation infrastructure and peer-credential checks. - Fixes the previously reported stale-ticket and filesystem-mount failures. <h3>Confidence Score: 3/5</h3> The PR is not yet safe to merge because the unresolved ticket-source impersonation issue remains for root workloads and for deployments that do not select the new workload UID. The stale-ticket issue and both filesystem findings are fixed. However, the unresolved ticket-source finding is only partially addressed: `Untrusted::refuse` deliberately disables peer rejection when the descriptor selects UID 0, allowing a root workload to replace the `/tmp` socket and supply an attacker-chosen ticket. The new `nesplay` account also does not enforce separation because the workload UID still comes from an external descriptor producer; if that producer continues using UID 1000, the carrier rejects the legitimate UID-1000 `neshub` peer as well. **Files Needing Attention:** apps/nesinit/src/ticket.rs, build/Dockerfile <h3>Important Files Changed</h3> | Filename | Overview | |----------|----------| | apps/nesinit/src/ticket.rs | Adds ticket polling and peer-credential rejection, but the protection is disabled for root workloads and depends on external UID separation. | | apps/nesinit/src/session.rs | Integrates ticket forwarding into the lifecycle loop and records the descriptor-provided workload UID before startup. | | apps/nesinit/src/filesystems.rs | Establishes early mounts without hiding the session share tree and makes the payload socket directory traversable. | | apps/nesinit/src/payload.rs | Relocates the payload socket under `/run/nestri` and grants workloads permission to connect. | | apps/neshub/src/ipc_listener.rs | Rebuilds the served ticket per connection so later endpoint addresses are included. | | build/Dockerfile | Adds a distinct `nesplay` account, although the external boot-descriptor producer must select that UID for separation to hold. | <h3>Sequence Diagram</h3> ```mermaid sequenceDiagram participant Hub as neshub participant Socket as Ticket Unix socket participant Init as nesinit carrier participant Session as nesinit session participant Host as Host controller loop Every polling interval Init->>Socket: Connect Socket->>Hub: Accept connection Hub->>Hub: Rebuild ticket from endpoint.addr() Hub-->>Init: Current ticket Init->>Init: Check peer UID and compare with prior ticket alt Trusted and changed Init->>Session: Queue ticket Session->>Host: GuestToHost::Ticket end end ``` <sub>Reviews (3): Last reviewed commit: ["fix(nesinit): do not mount over the shar..."](f74de9beb8) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=60934382)</sub> <!-- /greptile_comment -->