mirror of
https://github.com/nestriness/nestri.git
synced 2026-09-20 01:35:19 +03:00
736c0013e9
PID 1 inside a box. A microVM has no init unless something is it, and
three of
the jobs belong to nothing else in the guest: reaping whatever the
workload
orphans, turning a signal into an ordered shutdown, and holding the
guest end
of the one channel out.
- `reap.rs` — reaping, plus the subreaper bit and taking init out of the
OOM
killer's reach.
- `shutdown.rs` — the order, behind a trait so it can be asserted
without a VM:
workload first and alone, then everything else, then flush, then power
off.
- `session.rs` — the exchange. The guest speaks first with its protocol
version, is handed one boot descriptor, reports, and stops. Generic over
the
byte stream, so the whole protocol is testable over an in-memory pipe.
- `workload.rs` — one trait the descriptor drops into, a real process
behind
it, and a double.
- `nesprotocol::lifecycle` — the types, behind a feature that is off by
default so the media components keep building without serde.
It reports and does not supervise: when the workload ends, the exit goes
up the
channel and the session is over. Nothing here restarts anything.
**Mounting is not implemented in this PR.** The descriptor's `mounts`
are
refused rather than ignored, and the next PR in the stack implements
them.
### On the OOM killer
`refuse_oom_kill()` writes `-1000` to this process's own
`oom_score_adj`, and
that part is init's job: everything else in the guest exiting is a
message up
the channel, whereas init exiting takes the channel with it, and the far
end
sees a box that stopped answering for no stated reason.
The rest is the image's job and cannot be done from here. Making the
workload
the *preferred* victim means scoring processes this component did not
start, so
the image has to leave the workload's score at or above the default and
must
not lower it for the guest's own services either.
## The tests, failing first
Reaping, with `reap_exited()` returning nothing:
```
running 3 tests
test a_child_that_exits_is_reaped_with_its_code ... FAILED
test a_child_that_is_killed_is_reaped_as_signalled ... FAILED
test an_orphan_is_reaped_by_whoever_inherits_it ... FAILED
failures:
---- a_child_that_exits_is_reaped_with_its_code stdout ----
thread 'a_child_that_exits_is_reaped_with_its_code' (304365) panicked at apps/nesinit/tests/reaping.rs:52:32:
the child was left a zombie
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- a_child_that_is_killed_is_reaped_as_signalled stdout ----
thread 'a_child_that_is_killed_is_reaped_as_signalled' (304366) panicked at apps/nesinit/tests/reaping.rs:66:32:
the child was left a zombie
---- an_orphan_is_reaped_by_whoever_inherits_it stdout ----
thread 'an_orphan_is_reaped_by_whoever_inherits_it' (304367) panicked at apps/nesinit/tests/reaping.rs:98:5:
the child was left a zombie
failures:
a_child_that_exits_is_reaped_with_its_code
a_child_that_is_killed_is_reaped_as_signalled
an_orphan_is_reaped_by_whoever_inherits_it
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.02s
```
Shutdown, with one signal to everything instead of an order:
```
running 3 tests
test shutdown::tests::a_workload_that_leaves_in_time_is_not_killed ... ok
test shutdown::tests::a_workload_that_overstays_its_grace_is_killed_and_shutdown_still_finishes ... FAILED
test shutdown::tests::the_workload_stops_before_anything_else_and_the_disks_flush_before_power ... FAILED
failures:
---- shutdown::tests::a_workload_that_overstays_its_grace_is_killed_and_shutdown_still_finishes stdout ----
thread 'shutdown::tests::a_workload_that_overstays_its_grace_is_killed_and_shutdown_still_finishes' (304568) panicked at apps/nesinit/src/shutdown.rs:105:79:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- shutdown::tests::the_workload_stops_before_anything_else_and_the_disks_flush_before_power stdout ----
thread 'shutdown::tests::the_workload_stops_before_anything_else_and_the_disks_flush_before_power' (304569) panicked at apps/nesinit/src/shutdown.rs:80:9:
assertion `left == right` failed
left: ["signal_rest", "kill_rest", "flush_disks", "power_off"]
right: ["signal_workload", "await_workload", "signal_rest", "kill_rest", "flush_disks", "power_off"]
failures:
shutdown::tests::a_workload_that_overstays_its_grace_is_killed_and_shutdown_still_finishes
shutdown::tests::the_workload_stops_before_anything_else_and_the_disks_flush_before_power
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 8 filtered out; finished in 0.00s
error: test failed, to rerun pass `-p nesinit --lib`
```
The handshake, with the version left where it was before the bump:
```
running 1 test
test session::tests::the_guest_speaks_first_and_says_its_version ... FAILED
failures:
---- session::tests::the_guest_speaks_first_and_says_its_version stdout ----
thread 'session::tests::the_guest_speaks_first_and_says_its_version' (304772) panicked at apps/nesinit/src/session.rs:180:9:
assertion `left == right` failed
left: Ready { protocol_version: 1 }
right: Ready { protocol_version: 2 }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
session::tests::the_guest_speaks_first_and_says_its_version
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 10 filtered out; finished in 0.00s
error: test failed, to rerun pass `-p nesinit --lib`
```
And the property most likely to be quietly regressed later — a
supervisor loop
that restarts what it started, instead of reporting:
```
running 1 test
test session::tests::an_exit_is_reported_and_the_workload_is_not_started_again ... FAILED
failures:
---- session::tests::an_exit_is_reported_and_the_workload_is_not_started_again stdout ----
thread 'session::tests::an_exit_is_reported_and_the_workload_is_not_started_again' (304968) panicked at apps/nesinit/src/session.rs:237:9:
assertion `left == right` failed: an exit is reported, never restarted
left: 2
right: 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
session::tests::an_exit_is_reported_and_the_workload_is_not_started_again
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 10 filtered out; finished in 0.00s
error: test failed, to rerun pass `-p nesinit --lib`
```
All green after: 11 unit tests, 3 against real forked children, plus 3
new in
`nesprotocol`.
## What this does not verify
- **Nothing has run in a VM, and nothing has run as PID 1.** Orphan
reparenting is exercised with `PR_SET_CHILD_SUBREAPER` in a test
process,
which is the same kernel mechanism but not the same privilege.
- **No real vsock connection has been made.** The protocol is tested
over an
in-memory pipe; the dial itself — the address, the port, a listener that
is
not there — is unexercised, and the deliberate no-retry behaviour has
never
met a refused connection.
- **The real shutdown is untested.** The order is asserted through a
double;
`kill(-1)`, `sync` and the power-off call themselves need a guest, and a
test
process must not make them.
- **The OOM write is unchecked.** It warns and continues where there is
no
procfs, and nothing here confirms the kernel honoured the score.
- **Dropping to `uid`/`gid` before exec is unexercised** — it needs
privileges
a test does not have, so the `pre_exec` path has run in no test.
- **No number here is measured.** Nothing in this PR claims a timing, a
rate
or a count from hardware.
- **A version mismatch is not refused by this end.** The guest announces
its
version in its first line and the far end compares; as the layer stands
there
is nothing for the guest to compare against, so "both ends refuse on
mismatch" is only half-implementable. Worth settling in the channel's
specification before either end grows a second version, and I would
rather
raise it than invent a message for it here.
## Since review
`cb8f37a` — three fixes, all from the review, described in its message.
The one
worth naming here is that the reaper is now the only thing in the
component
that calls `wait`: it hands each exit to whoever asked for that pid, and
registering interest holds the same lock the delivery takes, so an exit
that
happens before its caller is registered is delivered rather than
dropped.
There is a test for exactly that, and it fails without the lock:
```
running 1 test
test an_exit_that_happens_before_the_caller_is_registered_is_not_lost ... FAILED
failures:
---- an_exit_that_happens_before_the_caller_is_registered_is_not_lost stdout ----
thread 'an_exit_that_happens_before_the_caller_is_registered_is_not_lost' (321463) panicked at apps/nesinit/tests/reaping.rs:171:9:
the exit was dropped on the way through
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
an_exit_that_happens_before_the_caller_is_registered_is_not_lost
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 4 filtered out; finished in 5.20s
error: test failed, to rerun pass `-p nesinit --test reaping`
```
Also, and still not verified: the ordered shutdown's real calls now
include
signalling and waiting for one pid rather than every child, and none of
that
has run in a guest either.
`071241f` — a pid stops being the workload's the moment it is reaped, so
a stop
or a kill during shutdown can no longer land on whatever the kernel gave
that
number to next. One window is left, between the reap and the delivery,
and
closing it needs a handle the kernel keeps rather than a number — noted
below
rather than papered over.
- **A pid is still a number here.** Between a reap and the exit being
handed
on, a freed pid is briefly treated as the workload's. Nothing has hit
that
window, and nothing can until a guest recycles pids under load; a
`pidfd`
would remove the class of bug rather than narrow it.
<!-- greptile_comment -->
<h3>Greptile Summary</h3>
This PR adds `nesinit`, a guest PID 1 implementation that reaps orphaned
children, exchanges lifecycle messages over vsock, launches one
workload, and performs ordered shutdown.
- Adds a centralized child reaper and waiter registry.
- Adds workload launch, identity changes, exit reporting, and explicit
rejection of unsupported mounts.
- Adds the lifecycle protocol behind an optional `nesprotocol` feature.
- Adds ordered workload-first shutdown and associated tests.
- Since the previous review, tracks whether the watched workload PID is
still live to reduce stale-PID signaling risk.
<h3>Confidence Score: 5/5</h3>
The PR appears safe to merge based on the accepted findings and the
resolved state of all previous review threads.
No new actionable finding remains after excluding the stale-PID race as
a duplicate of a manually resolved previous thread and confirming that
the stale state left by the shutdown-only wait path is not subsequently
used to signal a workload PID. Previous thread PRRC_kwDOLnCyk87q1Eew was
manually resolved without explanation.
<h3>Important Files Changed</h3>
| Filename | Overview |
|----------|----------|
| apps/nesinit/src/reap.rs | Adds centralized child reaping, waiter
registration, exit delivery, and shared workload-running state. |
| apps/nesinit/src/workload.rs | Implements workload launch and
signaling through a watched PID whose delivered exit marks it inactive.
|
| apps/nesinit/src/main.rs | Wires the reaper, vsock session, workload
handle, and real ordered-shutdown operations together. |
| apps/nesinit/src/session.rs | Implements the versioned one-descriptor
lifecycle exchange and reports one workload exit without restarting it.
|
| apps/nesinit/src/shutdown.rs | Encodes and tests workload-first
shutdown followed by guest services, disk flush, and power-off. |
| crates/nesprotocol/src/lifecycle.rs | Defines the feature-gated
lifecycle protocol types shared by the guest and host. |
<h3>Flowchart</h3>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[nesinit starts] --> B[Become subreaper and adjust OOM score]
B --> C[Connect to host over vsock]
C --> D[Send protocol version]
D --> E[Receive boot descriptor]
E --> F[Start workload and register PID]
F --> G{Session event}
G -->|Workload exits| H[Reaper delivers exit]
H --> I[Report workload exit]
G -->|Stop or channel closes| J[Signal workload]
G -->|Shutdown or session ends| K[Ordered shutdown]
I --> K
J --> K
K --> L[Stop workload]
L --> M[Stop remaining processes]
M --> N[Sync disks]
N --> O[Power off]
```
<sub>Reviews (3): Last reviewed commit: ["fix(nesinit): a pid stops
being the
work..."](071241f944)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=60585100)</sub>
<!-- /greptile_comment -->
Run your games on a GPU you don't own — or one you do. Nestri puts an interactive workload in a hardware-accelerated virtual machine and streams it to you over QUIC, at a latency that lets you play rather than watch.
Note
This repository is mid-rewrite, and the documentation is behind the code. The guest-side components arrived recently and their docs are thin. Nothing here is stable yet: expect directories to move and interfaces to change. Proper documentation is on the way — issues and questions are welcome in the meantime, and are genuinely useful for deciding what to write first.
Try it now — nesdoctor
One thing here is finished and runs on its own machine, today:
# Linux and macOS
curl -fsSL https://doctor.nestri.io/install.sh | sh
# Windows
powershell -c "irm https://doctor.nestri.io/install.ps1 | iex"
It tells you whether your machine could host games for other people, and
measures the number that actually decides whether streaming a game feels
right — not your download speed, but how much latency your connection adds
when it is busy. A 500 Mbps uplink that queues for 300 ms under load cannot
carry a game; a 25 Mbps one with fq_codel can. Almost nobody has seen their
own figure.
upstream 35 Mbps
latency, idle floor 56 ms
latency, loaded 185 ms
added under load +129 ms grade F
presentation path x11 · bspwm
eDP-1 1920x1200 @ 60 Hz, 8-bit
Vulkan decode h264, h265
It also reads your display out of its EDID — resolution, refresh, colour depth, HDR transfer functions, BT.2020, chroma — and what your hardware can decode. Those decide what is worth sending over the wire, and we would otherwise be guessing from one panel in one room.
It does not stream a game. It is the piece that has to exist before
anything else can, and most machines will come back CLIENT — which is a real
answer, not a failure.
Downloads one binary, verifies its checksum, runs it, deletes it. Installs
nothing, needs no administrator rights, touches no system directory. Nothing is
uploaded: it prints a link, lists exactly what the link contains, and opens it
only if you press Enter. The scripts those URLs serve are
apps/nesdoctor/install/ in this repository, so you
can read them before you run them.
Source and the full story: apps/nesdoctor.
What is here
Two halves that meet over the network and share very little else, plus one thing that runs on your own machine.
The control plane — TypeScript, on Cloudflare Workers
apps/api |
The public REST API. Identity, teams, machines, games, pairing. |
apps/auth |
A self-hosted OpenAuth issuer — Steam and SSH-key login. |
packages/core |
The domain: every table, every operation, no HTTP. |
packages/auth |
Shared auth types and subjects. |
Postgres for state, Alchemy for infrastructure. See
docs/alchemy.md.
The guest — Rust, inside the box
These run inside a virtual machine, beside the game. None of them talk to the control plane.
apps/nescope |
A headless Wayland compositor for one fullscreen client. A lighter answer to the same problem gamescope solves. |
apps/nescapture |
A Vulkan implicit layer. It captures frames from inside the workload's own process and encodes them on the GPU that drew them — no copy out to the CPU and back. |
apps/neswire |
Audio capture and transport. |
apps/neshub |
One connection out of the box. Muxes video, audio, cursor and input into a single QUIC stream to the client. |
crates/nesprotocol |
The wire types they all share, so no two ends can drift apart silently. |
On your own machine — Rust
apps/nesdoctor |
Whether a machine can host a box, and what its connection and display can really do. The first executable form of our host requirements — until it existed, a host was qualified by a human reading a table. Four dependencies; everything that could be done with the standard library is. |
The hypervisor the guest components run under is nesbox,
a separate repository: a micro-VM with a real GPU in it, using virtio-gpu native
context rather than passthrough, so one card can host several boxes at once.
Why a virtual machine
A container shares the host kernel, which makes strong isolation hard and a GPU harder. A micro-VM boots in about as long, isolates properly, and — with native context — gets close to bare-metal graphics. That choice is what makes "many sandboxes, one GPU" possible instead of one tenant per card.
Getting started
bun install
bun dev # control plane, local Cloudflare runtime
cargo build --workspace # guest components
cargo test --workspace
The guest components expect a Linux host with a Wayland-capable GPU stack, and are not much use on their own yet — they are pieces of a box, and the thing that assembles a box is not open yet.
nesdoctor is the exception and needs none of that:
cargo run --release -p nesdoctor
Status
Working: nesdoctor — released, and the only part a stranger can operate
today. The API, auth, the domain model, and the guest components listed above.
Not here yet: the box lifecycle, storage, the edge, and the client. Some of that will open as it is written; some is deliberately closed. What decides which is whether it handles your data — that half is open on principle — or decides our capacity, which is the part we sell.
Contributing
Early, and the ground moves. The two most useful things you can do right now
cost a minute each: run nesdoctor and send the result, because we have
almost no idea what the machines on the other end of this look like; and tell
us where the documentation failed you. Conventional commits; explain why in
the body.