0e94620808
## What was missing `neshub` serves the session's address on a socket, and its own flag has always said how that address gets out: > *"neshub listens; nesinit dials and carries the ticket to the host, because > the person who needs it is outside this VM and stdout here is a log file > inside one."* Nothing dialled it. `grep -rn ticket apps/nesinit/src` returned nothing at all, so the address never left the guest — and the one lifecycle message for it, `Ticket`, had no sender. This adds the carrier. ## Three decisions worth reading **Polled, not read once.** An address is not a value, it is the best answer so far: an endpoint discovers more ways to reach it after it binds — a local one immediately, a relayed one seconds later. Reading once means whoever asked first decides, and the first answer is the one that works on a local network and fails from anywhere else. Only a *changed* answer is forwarded, so an unchanged one costs nothing. **Dials, does not listen.** The opposite of the payload relay next door, and deliberately so. There the guest listens because the workload starts later; here the server is the long-lived one. Dialling also makes "not bound yet" an error to retry rather than a connection to wait for without knowing whether it is coming — which is the ordinary case at boot, since this starts before the server does. **The address is never logged.** It is a capability to reach the session, and a log inside the guest is the one place it has no reason to be. The log line says whether it is the first one and nothing else. ## Failing first The carrier's tests fail against unmodified code by not compiling: `ticket.rs` does not exist and `session::run` takes three arguments. Said plainly rather than manufactured. The behavioural gap is better shown as the `grep` above — nothing in the guest ever sent a `Ticket`, so the message had one end. `nesinit`: **35 tests passing**, up from 27. Four on the carrier itself (an address arrives; a better one replaces it; a socket that is not there yet is waited out rather than failed; an empty answer is not an address) and two on the session (an address reaches the caller as `Ticket`; a carrier that stops does not end the session). ## Verified in a real guest Built static for musl, run as PID 1 in a real microVM under a real VMM with a real vsock. It dialled out, completed the handshake at version 2, took a boot descriptor, started its workload, read the address that workload published and sent it up the channel — twice, the second time because a better one appeared. The caller saw `nestri:local-only` and then `nestri:with-relays`. Guest boot to init was **310 ms**. ## Review round (f74de9b, on top of two fixes from a hardware run) `18864b9` and `00a2bda` came from running this on real hardware and are the reason it works at all: nothing was mounting `/proc` or anywhere writable, so the relay and the address socket both failed with `EROFS` and the session was reported as a workload that ran and published nothing. And one look at the socket had a size cap but no time cap, so a peer that accepted and then said nothing stopped the search for a better address permanently. Four review findings on top, all real: | finding | outcome | |---|---| | a tmpfs over the share tree hid the install, user, work and log-share directories the image prepares | moved to `/run`, where a runtime socket belongs; a test asserts the share tree is never mounted over, and another that nothing is mounted before the mount containing it | | the relay directory's mode stopped the workload reaching the socket | the directory is searchable and writable only by init — which is what makes the socket unreplaceable — and the socket itself is what the workload may connect to | | the address was built once and served forever, so polling could only return the first one | rebuilt per read from the endpoint, which is what makes polling worth doing | | the workload could replace the socket and publish an address of its choosing | `SO_PEERCRED` before a byte is read; an address from the workload's user is refused and logged loudly. **Fails closed**, so the host's default workload uid moved off the services' uid and the image grew a second user — see the thread | **These two PRs are now order-dependent.** This one merged alone, against a host still defaulting the workload to the services' uid, refuses every legitimate address. Also bumps `tinyvec` by a patch release. It does not build on this toolchain — `vec` resolves to the module, not the macro — which made every crate depending on an endpoint unbuildable, `neshub` included. Pre-existing: the lockfile named the same version before this branch. It is why `neshub` could be built and tested here at all. **40 tests** in `nesinit` (was 27), 3 in `neshub`, fmt clean, and no clippy warning in any file this branch touches. ## What this does not verify - **The separated-uid pairing has not been run on hardware.** The rig has no init system, so the only process it can start is the workload — which means the address producer *is* the workload and the two cannot be given different uids in it. The refusal and the root stand-down are both verified on a real VM; the legitimate combination is covered by a unit test over a real socket and real kernel credentials, and by nothing else. - **The image was not built.** The second user is one `useradd` in a Dockerfile that needs a base image this machine does not have. - **The real address server was not in the loop.** The socket was served by a stand-in written for the test, which speaks the same one-line protocol. The real one has never had anything read from it, and its behaviour when it learns a new address — whether it re-serves the newer one on the next dial, which is the entire premise of polling — is **asserted, not measured**. - **No media.** Nothing rendered or encoded; the address pointed at nothing. - **Nobody connected to it.** The address left the guest. It was never dialled. - **Not the real guest image.** A 3 MB rootfs built for this, with no init system, no compositor and no Mesa. The image build does not ship this binary at all — its `default` runlevel still registers services on the assumption that something else starts the payload. - **The relay was down for the whole run**, because the test root was read-only and `/nestri` could not be created. That is non-fatal by design and the session ran anyway, but it means the payload layer and this carrier have not been exercised in the same boot. - **One address, one guest, one session.** Nothing concurrent. <!-- greptile_comment --> <h3>Greptile Summary</h3> This PR carries changing session tickets from `neshub` through `nesinit` to the host and establishes writable runtime filesystems for guest sockets. - Rebuilds tickets from the endpoint on every IPC connection so newly learned addresses can propagate. - Polls the ticket socket and forwards changed tickets over the lifecycle channel. - Moves the payload socket into a dedicated `/run/nestri` runtime directory. - Adds workload/service UID separation infrastructure and peer-credential checks. - Fixes the previously reported stale-ticket and filesystem-mount failures. <h3>Confidence Score: 3/5</h3> The PR is not yet safe to merge because the unresolved ticket-source impersonation issue remains for root workloads and for deployments that do not select the new workload UID. The stale-ticket issue and both filesystem findings are fixed. However, the unresolved ticket-source finding is only partially addressed: `Untrusted::refuse` deliberately disables peer rejection when the descriptor selects UID 0, allowing a root workload to replace the `/tmp` socket and supply an attacker-chosen ticket. The new `nesplay` account also does not enforce separation because the workload UID still comes from an external descriptor producer; if that producer continues using UID 1000, the carrier rejects the legitimate UID-1000 `neshub` peer as well. **Files Needing Attention:** apps/nesinit/src/ticket.rs, build/Dockerfile <h3>Important Files Changed</h3> | Filename | Overview | |----------|----------| | apps/nesinit/src/ticket.rs | Adds ticket polling and peer-credential rejection, but the protection is disabled for root workloads and depends on external UID separation. | | apps/nesinit/src/session.rs | Integrates ticket forwarding into the lifecycle loop and records the descriptor-provided workload UID before startup. | | apps/nesinit/src/filesystems.rs | Establishes early mounts without hiding the session share tree and makes the payload socket directory traversable. | | apps/nesinit/src/payload.rs | Relocates the payload socket under `/run/nestri` and grants workloads permission to connect. | | apps/neshub/src/ipc_listener.rs | Rebuilds the served ticket per connection so later endpoint addresses are included. | | build/Dockerfile | Adds a distinct `nesplay` account, although the external boot-descriptor producer must select that UID for separation to hold. | <h3>Sequence Diagram</h3> ```mermaid sequenceDiagram participant Hub as neshub participant Socket as Ticket Unix socket participant Init as nesinit carrier participant Session as nesinit session participant Host as Host controller loop Every polling interval Init->>Socket: Connect Socket->>Hub: Accept connection Hub->>Hub: Rebuild ticket from endpoint.addr() Hub-->>Init: Current ticket Init->>Init: Check peer UID and compare with prior ticket alt Trusted and changed Init->>Session: Queue ticket Session->>Host: GuestToHost::Ticket end end ``` <sub>Reviews (3): Last reviewed commit: ["fix(nesinit): do not mount over the shar..."](f74de9beb8) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=60934382)</sub> <!-- /greptile_comment -->
Run your games on a GPU you don't own — or one you do. Nestri puts an interactive workload in a hardware-accelerated virtual machine and streams it to you over QUIC, at a latency that lets you play rather than watch.
Note
This repository is mid-rewrite, and the documentation is behind the code. The guest-side components arrived recently and their docs are thin. Nothing here is stable yet: expect directories to move and interfaces to change. Proper documentation is on the way — issues and questions are welcome in the meantime, and are genuinely useful for deciding what to write first.
Try it now — nesdoctor
One thing here is finished and runs on its own machine, today:
# Linux and macOS
curl -fsSL https://doctor.nestri.io/install.sh | sh
# Windows
powershell -c "irm https://doctor.nestri.io/install.ps1 | iex"
It tells you whether your machine could host games for other people, and
measures the number that actually decides whether streaming a game feels
right — not your download speed, but how much latency your connection adds
when it is busy. A 500 Mbps uplink that queues for 300 ms under load cannot
carry a game; a 25 Mbps one with fq_codel can. Almost nobody has seen their
own figure.
upstream 35 Mbps
latency, idle floor 56 ms
latency, loaded 185 ms
added under load +129 ms grade F
presentation path x11 · bspwm
eDP-1 1920x1200 @ 60 Hz, 8-bit
Vulkan decode h264, h265
It also reads your display out of its EDID — resolution, refresh, colour depth, HDR transfer functions, BT.2020, chroma — and what your hardware can decode. Those decide what is worth sending over the wire, and we would otherwise be guessing from one panel in one room.
It does not stream a game. It is the piece that has to exist before
anything else can, and most machines will come back CLIENT — which is a real
answer, not a failure.
Downloads one binary, verifies its checksum, runs it, deletes it. Installs
nothing, needs no administrator rights, touches no system directory. Nothing is
uploaded: it prints a link, lists exactly what the link contains, and opens it
only if you press Enter. The scripts those URLs serve are
apps/nesdoctor/install/ in this repository, so you
can read them before you run them.
Source and the full story: apps/nesdoctor.
What is here
Two halves that meet over the network and share very little else, plus one thing that runs on your own machine.
The control plane — TypeScript
apps/api |
The public REST API. Identity, teams, machines, games, pairing. |
apps/auth |
A self-hosted OpenAuth issuer — Steam and SSH-key login. |
packages/core |
The domain: every table, every operation, no HTTP. |
packages/auth |
Shared auth types and subjects. |
Postgres for state. Both run on Cloudflare Workers today and as ordinary
containers wherever you like — one handler each, no infrastructure-as-code, and
a Dockerfile in each app. See docs/deploy.md and
docs/dns.md.
The guest — Rust, inside the box
These run inside a virtual machine, beside the game. None of them talk to the control plane.
apps/nescope |
A headless Wayland compositor for one fullscreen client. A lighter answer to the same problem gamescope solves. |
apps/nescapture |
A Vulkan implicit layer. It captures frames from inside the workload's own process and encodes them on the GPU that drew them — no copy out to the CPU and back. |
apps/neswire |
Audio capture and transport. |
apps/neshub |
One connection out of the box. Muxes video, audio, cursor and input into a single QUIC stream to the client. |
crates/nesprotocol |
The wire types they all share, so no two ends can drift apart silently. |
On your own machine — Rust
apps/nesdoctor |
Whether a machine can host a box, and what its connection and display can really do. The first executable form of our host requirements — until it existed, a host was qualified by a human reading a table. Four dependencies; everything that could be done with the standard library is. |
The hypervisor the guest components run under is nesbox,
a separate repository: a micro-VM with a real GPU in it, using virtio-gpu native
context rather than passthrough, so one card can host several boxes at once.
Why a virtual machine
A container shares the host kernel, which makes strong isolation hard and a GPU harder. A micro-VM boots in about as long, isolates properly, and — with native context — gets close to bare-metal graphics. That choice is what makes "many sandboxes, one GPU" possible instead of one tenant per card.
Getting started
bun install
cp .env.example .env # compose reads every credential from here
docker compose up postgres # the database
bun run db:migrate # schema
bun dev # control plane, local Cloudflare runtime
docker compose up --build # or: the whole control plane as containers
cargo build --workspace # guest components
cargo test --workspace
The guest components expect a Linux host with a Wayland-capable GPU stack, and are not much use on their own yet — they are pieces of a box, and the thing that assembles a box is not open yet.
nesdoctor is the exception and needs none of that:
cargo run --release -p nesdoctor
Status
Working: nesdoctor — released, and the only part a stranger can operate
today. The API, auth, the domain model, and the guest components listed above.
Not here yet: the box lifecycle, storage, the edge, and the client. Some of that will open as it is written; some is deliberately closed. What decides which is whether it handles your data — that half is open on principle — or decides our capacity, which is the part we sell.
Contributing
Early, and the ground moves. The two most useful things you can do right now
cost a minute each: run nesdoctor and send the result, because we have
almost no idea what the machines on the other end of this look like; and tell
us where the documentation failed you. Conventional commits; explain why in
the body.