Wanjohi
f889516ef4
The box half of phase 1 of window streams: two games in one box, each its own stream. Wire formats are nesprotocol's `window` module (#421). ## What each app does now **nescope** - Gives every toplevel an id on first map: 0 for the first window, never reused, none past 65535. - Tells the hub, on the existing input socket, when a window opens, changes title or size, and closes, in nesprotocol's window encoding. It also sends a box-internal `MSG_WINDOW_PID` saying which process owns the window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id comes from that process's `NESTRI_LAUNCH_ID`. - Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and `INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means window 0. There is still one output: a window's size applies while that window has focus and is remembered otherwise. On every input connection, input resets to window 0. - Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they describe changes. **nescapture** - Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in the reserved byte). An older reader never sees it. - Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared `/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there still reaches the newest encoder, as before. The shared socket was the only resource two capturing processes fought over: the second one took it, and commands went only to that one. The video and stats sockets are send-only datagram sockets and are safe to share. Capture slots and devices are per process. **neshub** - Joins frames to windows by pid. An untagged frame is window 0. A pid with no window falls back to the newest window of its launch, then to window 0 while no other window is open. A pid whose window closed belongs to nobody. - A connection that never subscribes gets the box's primary picture in the old encoding. The primary picture is the frames of the first encoder to draw, until it has been silent for a second. A subscribed connection gets only its window's frames, windowed for any window but 0, with per-window seq. PyroWave stays window 0 only. - `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the writer's seq and resync gate in line with the frames, then requests a keyframe. The first subscribe opens `STREAM_WINDOWS` on the video connection, starting with `OPENED` (and the icon, if known) for every open window. Later lifecycle events go to every subscribed connection. - Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window changes, and the hub does no merging across tags. A gamepad message moves focus only when its window differs from the current one and no other client sent keyboard or pointer input in the last second. Focus tags move keyboard and pointer focus only: nescope neither raises the window nor resizes the output for one. The output changes size only on an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it, last writer wins with 500 ms hysteresis between windows. A client's `INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except for window 0. - A cursor update goes to the clients subscribed to the window it describes. Clients that never subscribed get every update, as before. - When nescope reconnects (a new compositor), the hub sends `CLOSED` for every window and every subscription goes dead until the client subscribes again. - `DatagramSender` sizes fragments with `max_fragment_payload` and writes headers with `write_video_datagram_header`. - Commands go to every known encoder on its own socket, or to the shared socket while none is known (as today). An encoder that appears later is sent the commands so far (the latest of each message type) and then a keyframe request. The first encoder is not sent them again. - A keyframe on a client's loss or subscribe goes only to that client's window's encoder (the primary's, for a client that never subscribed). - Known limit: bitrate is box-wide. One controller sets one target for every window's encoder, so one client's congestion sets every window's rate. Per-window rate is follow-up work. - An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with the same id. nescope no longer exits when its first game exits; nesinit owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs as another uid): nescapture sends its launch id. ## Shipping nescope and neshub ship together in the guest image and must not ship apart. ## Compatibility One game with an old client sends the same bytes as before. The client never subscribes, so it gets the primary picture, which is that game, as `DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows stream. Cursor updates reach it as before. Focus tags appear only if the game's window is not 0, and window 0 resizes use the old form. Mixed versions also work: - An old nescapture is window 0. - An old nescope never sends window messages. Every frame is then window 0, or the primary picture. - An old hub sends no tags, which nescope reads as window 0, and it still reaches nescapture through the shared socket. ## Tests `cargo test` on nesprotocol (122), nescope (17+5), nescapture (161), neshub (74), all passing. New tests: - The window table: joining pids to windows, the launch fallback, retired pids, the primary picture, the snapshot. - Focus tagging and resize rewriting. - The command fan-out. - nescope's id registry (first window 0, ids never reused, refused past 65535) and per-window focus and sizes. - The IPC pid trailer. - Two QUIC end-to-end tests: - Two tagged encoders: an unsubscribed client sees only the first game in the old encoding. After subscribing to window 1 it sees only the second game, windowed, with seq starting again, and a windows stream that starts with both `OPENED` events. - An untagged encoder gives the old stream. Not run on a box: no neslet, nestri-host or box was touched.
Run your games on a GPU you don't own — or one you do. Nestri puts an interactive workload in a hardware-accelerated virtual machine and streams it to you over QUIC, at a latency that lets you play rather than watch.
Note
This repository is mid-rewrite, and the documentation is behind the code. The guest-side components arrived recently and their docs are thin. Nothing here is stable yet: expect directories to move and interfaces to change. Proper documentation is on the way — issues and questions are welcome in the meantime, and are genuinely useful for deciding what to write first.
Try it now — nesdoctor
One thing here is finished and runs on its own machine, today:
# Linux and macOS
curl -fsSL https://doctor.nestri.io/install.sh | sh
# Windows
powershell -c "irm https://doctor.nestri.io/install.ps1 | iex"
It tells you whether your machine could host games for other people, and
measures the number that actually decides whether streaming a game feels
right — not your download speed, but how much latency your connection adds
when it is busy. A 500 Mbps uplink that queues for 300 ms under load cannot
carry a game; a 25 Mbps one with fq_codel can. Almost nobody has seen their
own figure.
upstream 35 Mbps
latency, idle floor 56 ms
latency, loaded 185 ms
added under load +129 ms grade F
presentation path x11 · bspwm
eDP-1 1920x1200 @ 60 Hz, 8-bit
Vulkan decode h264, h265
It also reads your display out of its EDID — resolution, refresh, colour depth, HDR transfer functions, BT.2020, chroma — and what your hardware can decode. Those decide what is worth sending over the wire, and we would otherwise be guessing from one panel in one room.
It does not stream a game. It is the piece that has to exist before
anything else can, and most machines will come back CLIENT — which is a real
answer, not a failure.
Downloads one binary, verifies its checksum, runs it, deletes it. Installs
nothing, needs no administrator rights, touches no system directory. Nothing is
uploaded: it prints a link, lists exactly what the link contains, and opens it
only if you press Enter. The scripts those URLs serve are
apps/nesdoctor/install/ in this repository, so you
can read them before you run them.
Source and the full story: apps/nesdoctor.
What is here
Two halves that meet over the network and share very little else, plus one thing that runs on your own machine.
The control plane — TypeScript
apps/api |
The public REST API. Identity, teams, machines, games, pairing. |
apps/auth |
A self-hosted OpenAuth issuer — Steam and SSH-key login. |
packages/core |
The domain: every table, every operation, no HTTP. |
packages/auth |
Shared auth types and subjects. |
Postgres for state. Both run on Cloudflare Workers today and as ordinary
containers wherever you like — one handler each, no infrastructure-as-code, and
a Dockerfile in each app. See docs/deploy.md and
docs/dns.md.
The guest — Rust, inside the box
These run inside a virtual machine, beside the game. None of them talk to the control plane.
apps/nescope |
A headless Wayland compositor for one fullscreen client. A lighter answer to the same problem gamescope solves. |
apps/nescapture |
A Vulkan implicit layer. It captures frames from inside the workload's own process and encodes them on the GPU that drew them — no copy out to the CPU and back. |
apps/neswire |
Audio capture and transport. |
apps/neshub |
One connection out of the box. Muxes video, audio, cursor and input into a single QUIC stream to the client. |
crates/nesprotocol |
The wire types they all share, so no two ends can drift apart silently. |
On your own machine — Rust
apps/nesdoctor |
Whether a machine can host a box, and what its connection and display can really do. The first executable form of our host requirements — until it existed, a host was qualified by a human reading a table. Four dependencies; everything that could be done with the standard library is. |
The hypervisor the guest components run under is nesbox,
a separate repository: a micro-VM with a real GPU in it, using virtio-gpu native
context rather than passthrough, so one card can host several boxes at once.
Why a virtual machine
A container shares the host kernel, which makes strong isolation hard and a GPU harder. A micro-VM boots in about as long, isolates properly, and — with native context — gets close to bare-metal graphics. That choice is what makes "many sandboxes, one GPU" possible instead of one tenant per card.
Getting started
bun install
cp .env.example .env # compose reads every credential from here
docker compose up postgres # the database
bun run db:migrate # schema
bun dev # control plane, local Cloudflare runtime
docker compose up --build # or: the whole control plane as containers
cargo build --workspace # guest components
cargo test --workspace
The guest components expect a Linux host with a Wayland-capable GPU stack, and are not much use on their own yet — they are pieces of a box, and the thing that assembles a box is not open yet.
nesdoctor is the exception and needs none of that:
cargo run --release -p nesdoctor
Hosting a box: what the machine needs
Run nesdoctor on the machine first; it checks the disk, network and display
this list cannot.
- Linux on the host itself. Nestri spawns micro-VMs, so it needs
/dev/kvm. It does not run in Docker: a container cannot give it that. - Distro: Arch is what we recommend. Ubuntu works.
- Filesystem: ext4 is fine.
- GPU: NVIDIA is supported.
Unraid (experimental, untested)
Unraid is a poor host directly (its root filesystem lives in RAM and is rebuilt every boot), but it is a good hypervisor. A route nobody has run yet:
- Create an Arch or Ubuntu VM in Unraid's VM manager.
- Pass the GPU through to that VM.
- Enable nested virtualisation on the Unraid host (
kvm_intel nested=1orkvm_amd nested=1) and set the VM's CPU mode to host-passthrough, so the VM has its own/dev/kvm. - Run
nesdoctorinside the VM, then install as usual.
Performance and driver behaviour under nested KVM are unmeasured. If you try it, tell us where it works or breaks.
Status
Working: nesdoctor — released, and the only part a stranger can operate
today. The API, auth, the domain model, and the guest components listed above.
Not here yet: the box lifecycle, storage, the edge, and the client. Some of that will open as it is written; some is deliberately closed. What decides which is whether it handles your data — that half is open on principle — or decides our capacity, which is the part we sell.
Contributing
Early, and the ground moves. The two most useful things you can do right now
cost a minute each: run nesdoctor and send the result, because we have
almost no idea what the machines on the other end of this look like; and tell
us where the documentation failed you. Conventional commits; explain why in
the body.