Most desktop applications and games speak PulseAudio, and the box served only PipeWire, so they played no sound. pipewire-pulse now runs beside PipeWire, on a socket the workload can reach, and the workload is told its address. A 32-bit launcher helper also has a 32-bit overlay preloaded into it that links libGL.so.1; libglvnd's dispatcher alone satisfies that, with Mesa assumed rather than installed.
A 64-bit launcher can still start its games through a 32-bit helper process. With no 32-bit loader in the image that helper cannot exec, and every launch exits immediately with no message. Only lib32-glibc is added; the check that keeps every other lib32 package out stays.
- **nescope:** smithay spawned Xwayland with an empty environment, so
Xwayland never saw the GL driver selection the box sets
(`MESA_LOADER_DRIVER_OVERRIDE`, `GALLIUM_DRIVER`,
`__GLX_VENDOR_LIBRARY_NAME`). glamor then found no driver for the card
(`virtio_gpu: driver missing`), fell back to software, and the server
offered no GLX — any X11 client asking for a GLX visual failed
(`glXChooseVisual failed`). Those three variables are now passed
through.
- **Image:** `chromium` and `xdg-utils`, so applications that open
sign-in pages, OAuth flows or links have somewhere to send them.
Verified by hand in a box: with the nescope change, an X11 client that
previously died at `glXChooseVisual` started and drew.
Applications open sign-in pages, OAuth flows and links through xdg-open, and with no browser installed every one of those dead-ends. Chromium is in the distribution's own repositories.
POST /box matched the fleet model by exact string, but nesktop sends the
name it shows (vendor stripped: "Radeon RX 9060 XT" for the host's "AMD
Radeon RX 9060 XT"), so every fleet room was refused with "No ... is
free right now". Models now match with or without the vendor in
placement and in the checkout's gpu.
A listed box's `gpu` was its host's first card; it is now the card the
box is pinned to (slot, then PCI id, then first card).
Tests: two new cases in apps/api/test/rooms.test.ts fail without the
change; rooms, fleet and billing-checkout-gpu suites pass (40) against
the local test Postgres.
POST /box matched the fleet's model by exact string, but GET /location lists
the same models to clients and nesktop shows them without the vendor
("Radeon RX 9060 XT"), so every fleet room it asked for was refused as none
free. Models now match with or without the vendor, in placement and in the
checkout's GPU too.
A listed box's gpu was its host's first card, so a second room on a
multi-GPU host was named after the first. It is the card the box is pinned
to.
POST /billing/checkout accepts an optional fleet `gpu`, validated
against the fleet's models and passed to the provider as checkout
metadata. When the subscription becomes active and paid, the webhook
creates a fleet room on that model for the team owner through the locked
Box.createPlaced path. If the model is taken it uses the closest free
one (same vendor first, then nearest VRAM) and records the substitution;
if nothing is free it creates nothing and logs it. A new checkout_room
table keyed on the subscription id makes redeliveries harmless. Tests
cover metadata passthrough, room creation, substitution, redelivery and
an unknown gpu. API suite 230 pass, core 256 pass on local Postgres.
Step 1 of decision 0088 in the API.
- `box.gpu_pci_id` (nullable, migration 0029)
- `GET /location`: own GPUs per card; fleet GPUs grouped by model with
available / entry / allowed
- `POST /box`: `{on:"own",machineId,pciId}` or `{on:"fleet",model}`; old
bodies still work
- Free: 1 cloud room, entry models; Pro: 3, any model; own unlimited.
Refusals are 403 with code `room_limit` or `gpu_needs_pro`, message
"Upgrade to create a room on <model>"
- Entry models: `FLEET_ENTRY_GPUS` (comma separated substrings), default
V710, RTX A2000, RTX 3060
- Weekend trial and the three windows removed from enforcement and from
GET /billing (plan + rooms used/allowed). Polar and webhook untouched
- Idle limit: `Box.stopIdle`, run on each host job poll, stops fleet
rooms with nothing live for 3 h
Core 256 pass, API 217 pass (local Postgres).
Create a room under a per-user advisory lock (limit, card and insert in one
transaction) and book cards under per-machine locks. A free plan with no model
is placed on an entry card only. Boxes with no card hold their machine's first
card; the new migration backfills single-GPU fleet boxes and adds
box.gpu_slot. Machine.Gpu accepts slot and renderNode; session.start carries
gpuSlot and gpuPciId. A stopped room re-checks its card when a run is
requested.
A box records the GPU it is pinned to. GET /location lists your own
GPUs and the fleet's by model. POST /box takes { on: own, machineId,
pciId } or { on: fleet, model }; free holds 1 cloud room on entry
models, Pro 3 on any, own rooms are unlimited, and refusals are typed
(room_limit, gpu_needs_pro). The weekend trial and the usage windows no
longer decide anything and GET /billing reports plan and room counts.
Cloud rooms with nothing live for 3 hours are stopped when their host
polls for work.
Promotes dev to prod: #450 billing portal 404 when a team has no Polar
customer; #448, #449 per-window pause and resume (neshub). Merge commit,
not squash.
GET /billing/portal threw a raw Polar error (500) for a team that never
checked out. Polar's 404/422 for an unknown external customer now
becomes 404 "This team has no billing yet". Tests added.
Fixes the failure of `nesprobe scenario --pause` on NestriTest (534
units in the 10 s after the pause, no pause line in the journal).
## Cause
`ClientSession::encoder_pid` is how the hub knows which encoder to pause
for a client. It was written in only two places:
`ClientSession::subscribe` (from `WindowTable::encoder_of`, the encoder
that has already drawn that window) and, on a primary switch, for
clients that never subscribed. On a real box the order is: the window
opens, the client subscribes to it (the log shows `client subscribes to
window 0` at 21:29:51 and again at 21:29:59.14), and only then does the
game's encoder draw its first frame. At subscribe time `encoder_of` is
`None`, so the pid was stored as 0 and never updated. `MSG_WINDOW_PAUSE`
hit `pid_or_all(..) == None` and was dropped at debug level (hence
nothing in the journal). Keyframe requests were not affected because for
pid 0 they broadcast to every encoder, which hid the problem.
Checked against the other two questions:
- (a) nesprobe uses the subscribe path (window 0 subscribe, same control
stream), and the pause message is sent on it.
- (c) The hub-to-encoder socket is fine for Proton: `nescapture` in pid
238 listens on `/tmp/nescapture-cmd-238.sock` and receives the IDR
requests the hub sends.
## Fix
`broadcast_video` now stores the frame's encoder pid into every session
that route() shows the frame to, so a client's encoder follows the
frames it is sent, whenever it subscribed. This also makes its keyframe
requests targeted instead of broadcast.
Also added info-level logs: hub `pause:`/`resume:` with client and pid
or "no encoder known"; hub `sent pause|resume to <socket>`; nescapture
`window pause|resume received (was ...)`.
## Tests
- New e2e test
`a_client_that_subscribed_before_the_first_frame_learns_its_encoder`:
subscribes before any frame, then draws; fails without the fix (pid
stays 0, verified by disabling the store), passes with it.
- neshub 89 pass, nescapture 162 pass (nestripc-1).
- Not re-run on a box.
The session's encoder pid was set only at subscribe time (from the encoder
that had drawn the window) and on a primary switch for clients that never
subscribed. A client that subscribed before the game drew its first frame,
which is the real order, kept pid 0, so its pause was ignored. Also logs
pause and resume at info in the hub and in nescapture.
Implements the box side of decision 0087 (items 1 and 2): a per-window
Pause / Resume.
## Plan
A correction to the brief: `nesprotocol` lives in this repo (nestri),
not in nescore. nescore and nesktop consume it by git rev. So the
protocol and the box side land together here, then nescore gets the
client call and a pin move, then nesktop gets the probe scenario and its
pin move. Also, the video encoder is `nescapture` (a Vulkan layer in the
game's process), not `neswire` (audio), and the compositor `nescope`
needs no change: capture happens in the game's present hook.
**Messages** (control stream, client to hub, empty payload, like
`MSG_IDR_REQUEST`):
- `MSG_WINDOW_PAUSE = 0x18`
- `MSG_WINDOW_RESUME = 0x19`
**Window id flow.** No id in the message. The hub already knows which
window a client is subscribed to (`MSG_WINDOW_SUBSCRIBE`) and the pid of
that window's encoder (`subscribe.2`, the same one keyframe requests
use). So:
1. client sends pause/resume on the control stream;
2. `neshub` `run_control_reader` turns it into `MSG_WINDOW_PAUSE/RESUME`
targeted at that encoder's own command socket
(`capture_cmd::window_pause/resume`, via `to_encoder`). Targeted
commands are not recorded in the replay history, so a later encoder is
not born paused;
3. `nescapture`'s command listener sets a `paused` flag; `try_capture`
in the present hook returns early while it is set (before the frame
gate, before any blit, slot or encode). The game's process is untouched;
4. resume clears the flag and sets `idr_requested`, so the first frame
after it is a keyframe. The hub also sets `awaiting_keyframe` for that
client, as for an IDR request.
A pause from a client whose encoder the hub does not know yet is
ignored, because "all encoders" is the wrong target for a pause.
Old hub, new client: logs an unknown control message. Old nescapture
ignores the unknown byte.
## Tested
- `cargo test -p nesprotocol -p neshub`: 122 + 79 pass (new: type bytes
distinct, including 0x17 subscribe; pause/resume target one encoder and
are not replayed).
- `cargo test -p nescapture`: 162 pass (new: pause stops, resume
requests a keyframe). Built on nestripc-1 with `SLANGC` pointed at a
copy in a container layer, because the box has no slangc.
## Not verified
- No real game or GPU run. That the present hook really stops blitting
while paused, and that a keyframe lands within 1 s of resume, is proven
only by unit tests here; the nesprobe `--pause` scenario in the nesktop
PR is what checks it end to end, and it has not been run against a box.
- A game that presents nothing after resume (a fully still scene with no
presents) would delay the keyframe until its next present, same as any
existing IDR request.
## Update after review (78a8bc8)
- A pause is per session. `neshub/src/pauses.rs` keeps the set of
sessions pausing each encoder; the encoder is paused while the set is
non-empty. Resume goes only to the pid that session paused and never
broadcasts (a resume from a session that paused nothing starts nothing).
- A session releases its pause on drop (`ClientSession::drop`), on
resubscribe, and when the primary picture moves (`restart_picture`).
Keyed by the never-reused gamepad session number, so a reconnecting
client cannot release or inherit another session's pause.
- Tests (neshub 86 pass): two sessions on one encoder; session that
never paused resumes nothing; drop while paused (unit and through
`remove_session` on the e2e rig); resubscribe while paused releases the
old encoder only; pausing another encoder moves the pause.
Dropping a session aborts its tasks, but a control reader mid-poll on another
thread could still pause after the release and leave the encoder paused
forever. The closed flag is set under the book's lock; pause checks it under
the same lock.
The hub keeps the set of sessions pausing each encoder. An encoder is paused
while the set is not empty; a resume goes only to the encoder that session
paused; a session that drops, resubscribes or sees the primary picture move
releases its pause.
MSG_WINDOW_PAUSE and MSG_WINDOW_RESUME on the control stream. The hub sends
them to the encoder of the window the client is shown; nescapture stops
capturing in the present hook while paused, and resume requests a keyframe.
dev → prod. Needs neslet 0.2.23, which is published.
- fix(install): run `onboard --check` before stopping a running agent,
and restart the agent if onboarding fails after the stop (#446)
- fix(install): an upgrade keeps the existing box store and GPU
selection and doesn't prompt
- feat(install): `--gpus` passes through
- chore(install): default host agent 0.2.23
Fixes the installer half of nescore#92 and #445.
- On an upgrade, `nestri-host onboard --check` (nescore#93, no lock, no
changes) runs before the agent is stopped. If it fails, the agent is
left running and the installer exits.
- If onboarding fails after the agent was stopped, the installer starts
the agent again before exiting.
- A host with /etc/nestri-host/env keeps its NESLET_STORE (or
NESLET_BOX_STORE) and NESLET_GPUS and is not asked where box images go.
Only a first install is prompted. NESTRI_BOX_STORE and --gpus still
override.
- `--gpus SLOTS` (or `--gpus=SLOTS`, or NESLET_GPUS) is accepted after
the token, survives the sudo re-exec, and reaches the check and
onboarding.
- DEFAULT_VERSION is 0.2.23.
**Merge only after neslet 0.2.23 (nescore#93) is released**: the
installer calls `--check` and `--gpus`, which older binaries lack, and
pins that version. Verified with `sh -n` and by hand-checking the
argument errors and env-file parsing; not run end to end on a host.
On a host already running the agent, run preflight first and stop the agent only if it passes, so a failing check no longer leaves the machine offline. An existing host keeps the box store from its environment file instead of being asked, since the guessed default moved a store and hid its games. --gpus is accepted after the token and reaches onboarding.
dev → prod.
- fix(nescope): a window can't ask for more than the output has, and
clamped requests are logged (#438)
- fix(nescope): a hub that closes the input socket is no longer a panic;
nescope ends if the hub stays gone (#440)
- fix(nescope, neswire): default log filter is info; dropped audio
frames log once per burst (#441)
nescope used `EnvFilter::from_default_env()`, whose default is ERROR, so
its info/warn lines never reached the host journal. nesinit only
forwards RUST_LOG if it has it itself.
- nescope: fall back to `info` when RUST_LOG is unset or unparsable
(same as neshub, nesinit, nesgamepad); debug stays opt-in. 4 unit tests
on the filter helper.
- neswire: same ERROR default, same fix.
- Not changed: nescapture (library inside the game process, env_logger).
Tested on nestripc-1: `cargo test -p nescope log_filter` 4 passed,
`cargo check -p neswire` ok.
Note: no 'configure_request clamped' warn exists on this base; once it
lands at warn level it is covered by the info default.
A stalled encoder drops about fifty frames a second, and with warn now
visible by default each drop would be its own journal line. Count them
and log the first and every 250th.
EnvFilter::from_default_env defaults to ERROR, so nescope's info and warn
lines never reached the host journal. Fall back to info like neshub,
nesinit and nesgamepad; align neswire, which had the same default.
Found in a passing multi-launch probe on NestriTest
(nesprobe-1792327018): at shutdown nescope logged `panicked at
apps/nescope/src/main.rs:639:10: Event loop error: ... IPC socket
closed`.
Cause: `InputIpcSource::process_events` returned EOF as `Err`; calloop
passes that out of `run()` into the `.expect("Event loop error")`. At
box shutdown nesinit does `kill(-1, SIGTERM)`, so neshub can exit before
nescope reads its own SIGTERM, and the panic skips the cleanup after the
loop (socket removal, child kill, the clean-exit log).
Change: on EOF, deliver the frames already read, warn, and return
`PostAction::Remove` so the loop carries on and the signal or auto-exit
path ends it. Read loop split into `fill()` so it is unit-tested with a
socket pair.
Tested on nestripc-1 (CARGO_TARGET_DIR=~/target-fix-nescope-ipc-exit):
`cargo test -p nescope`, 3 new tests pass.
The input source now removes itself on EOF or reset and raises a flag; the
main loop exits 2 s later unless a signal already ended it, so a hub crash
mid-run still ends the compositor (and the box via CompositorDied) instead of
holding a GPU with no input. Also corrects the comment that promised a
reconnect that does not exist.
The input source returned UnixStream EOF as an error, which calloop passes
out of run() into the expect at the end of main. At box shutdown init
signals every process at once, so neshub can go before nescope reads its
SIGTERM: nescope panicked and skipped its cleanup. Deliver what was read,
remove the source and warn instead.
Phase 0 of decision 0086.
Cause (inferred from the code, not observed live): the client's window
size resizes the box's output, but a Proton game under XWayland
re-asserts its launch size with a ConfigureRequest, which
configure_request honoured. The window ends up larger than the output;
the pointer lives on the output (clamp_cursor), so the cut-off part of
the window is unreachable. The client sends relative motion only, so no
client-side mapping is involved.
Fix: configure_request clamps a non-override-redirect X11 window's
requested size to the current output size (and keeps it inside the
output). The output keeps following the client. Pure fn clamp_to_output,
unit-tested incl. width-only, height-only and multiple windows.
Risk: a game that ignores the configure and keeps drawing at its own
size will still be cropped. Wayland toplevels are unchanged (they
already get the output size).
Live proof is still pending: pointer reach on a real Proton window after
a resize has not been exercised.
Adds an optional `daemon` to the boot descriptor (decision 0085).
nesinit starts it after the compositor, as the given uid, announces it
in /run/nestri/daemon before the box is ready, lists it among the box's
services, and on its exit reports ServiceDied by name and ends the box
(as for the compositor). Optional both ways: an old descriptor reads the
same; an old guest refuses a new descriptor and the host already reports
that as a stale image. Tests: protocol round trip, started after the
compositor, start failure refuses the box, exit reported.
Dev to prod: GET /machine/jobs?wait=&seen= long-poll (#435), nesinit
multi-launch fixes (#432, #433), launch/launch-stop wakes. Hosts that
don't send `wait` see no change. **Merge with a merge commit, not
squash.**
Cuts host job pickup from the heartbeat interval to under a second.
`GET /machine/jobs?wait=<s>` (capped at 25) answers at once when there
is work, else holds until woken or the wait is up, then answers empty.
Without `wait` nothing changes.
Wake (in-process, keyed by machine id, fired after commit through
`Database.effect`):
- run requested: `Session.create`
- stop asked for on a starting or live run: `stopRun`
- launch added: `addLaunch`; launch stop asked: `stopLaunch`
- run ended or failed (frees a slot for a queued run):
`compareAndSetState`
Not woken, covered by the 5 s recheck: trial and weekend-window expiry,
a heartbeat that raises slots.
Multi-process: `redis` is in `apps/api/package.json` but nothing imports
it and nothing configures a URL, so no pub/sub path here. Each held call
re-reads every 5 s, so a wake that lands in another process costs a few
seconds. Add a channel when a second process exists.
Worker runtime: answers at once (no hold). Bun: `idleTimeout: 60` added,
since the default 10 s would cut the held call.
Tests: `apps/api/test/jobs-wait.test.ts` (8, on the local test Postgres)
plus the full API suite, 216 pass. Pairs with the neslet PR in nescore
(`feat/jobs-long-poll`); an old neslet is unaffected and a new neslet
against an old API falls back to its interval.
A host found new work on its next poll, 8 to 21 seconds on average. With wait (at most 25 s) a call that finds nothing is held until a run is requested, a stop is asked for, a launch is added or stopped, or a run ends and frees a slot, then answers. A held call also looks again every 5 s so a wake in another process costs seconds, not the job. Workers answer at once. Bun's idle timeout is raised so the held call is not cut.
Stacked on #432 (fix/nesinit-loop-held); its commits show here until it
merges.
Stopping a launch in a two-launch box left that launch's game running:
`stop_launch` signalled only the launch's top process, so the children
(compositor wrapper, wine, the game) survived and the window never
closed. Killing the whole group naively would have taken the box's one
compositor, which the other launch is drawing on.
- Each launch runs under `setsid`. A stop signals the whole session
(SIGTERM, SIGKILL after 5 s). Members are found from /proc by session
id, so children that moved to a group of their own are still reached.
- A launch's exit is delivered only when its session is empty. If the
top process leaves by itself with leftovers, they are ended first.
- The compositor is a box-level process: the boot descriptor carries an
optional `Compositor` (geometry, rate, uid, env) and nesinit starts it
before the box reports ready, waiting for its socket. Absent, nothing
changes. A failure to start it refuses the box like a failed service.
- SIGKILL on the way down also takes each launch's group.
Tests: real-process tests for stop reaching children, a child ignoring
SIGTERM being killed after the grace, a top process leaving with
leftovers, and one launch's stop leaving another's group alone; session
tests for the compositor start and refusal; descriptor round trip. All
of nesinit and nesprotocol pass on nestripc-1.
Host side: nestrilabs/nescore (fix/launch-group-nescore). A host that
sends `compositor` needs this guest image first, because the descriptor
refuses unknown fields.
Two bugs found running 0085 on NestriTest (2026-10-17):
- **Loop devices released before mount.** `loop_attach` configured
`/dev/loopN` with `LO_FLAGS_AUTOCLEAR` and dropped its fd before
mounting, so the kernel detached it and the writable layer's attach got
the same device. The build image mount then read ext4: `erofs (device
loop0): cannot find valid erofs superblock` → EINVAL. The device fd is
now held until the mount is made.
- **A box refused new launches after one ended.** A terminal launch
exiting beside another marked the box ending and refused every new
launch, so re-opening a closed game failed with "the box is ending". Per
0085 closing one window is not the person done: the box still takes
launches and ends when its last one goes.
Verified live on NestriTest: Everhood mounted (loop0 erofs, loop1 ext4)
beside Dome Keeper. nesinit tests: 124 pass.
Wine setsid's its processes, so stopping a launch by session left wineserver,
explorer.exe and the game running with their window open. Each launch is now
put in cgroup /sys/fs/cgroup/nestri-launch-<id> before exec; stop is SIGTERM to
its members, then cgroup.kill after the grace, and the launch ends when the
cgroup is empty. Without cgroup v2 the members are the session plus every
process whose environment carries NESTRI_LAUNCH_ID. cgroup2 is mounted at
/sys/fs/cgroup early in boot.
Stopping a launch signalled only its top process, so the compositor wrapper,
wine and the game survived and the window stayed open. Each launch now runs
under setsid; a stop signals the whole session (TERM, then KILL after 5s) and
a launch counts as ended only once its session is empty.
The compositor moves out of the first launch into the box: the boot
descriptor carries an optional Compositor (geometry, rate, uid, env) and
nesinit starts it before the box reports ready, so every launch joins it.
Dev plus a merge that records prod's squash commits as already in dev,
so this PR has no conflicts. Merge it with **Create a merge commit**,
not squash.
All of dev to prod. Prod's squash copies are merged in with dev's
version kept, so the tree is identical to dev.
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #422, from the nesktop #3 review. POST /session/{id}/launch
for a game that already has an open launch (requested or running, no
stop requested) in the run now answers that launch with 200 instead of
inserting a second one. The check runs under the run's row lock that
addLaunch already takes, so two concurrent adds resolve to one launch.
OpenAPI documents the 200.
Tested: apps/api 208 pass against the local test Postgres, including a
new test adding the same game twice concurrently.
Follow-up to #423 from its re-review. The dgram writer now raises
awaiting_keyframe when it reaches the Resubscribe marker, in queue
order, and restart_picture sets the flag before queuing the marker.
Before this, an old keyframe still queued ahead of a primary switch
could clear the flag, letting the new picture's first delta out as the
first frame an old client saw after the switch. Needs a writer stalled
for at least 1 s to trigger.
Tested: cargo test -p neshub on nestripc-1, 78 passed.
Runtime half of the window streams contract (joins 1 and 6).
- nesinit runs any number of launches side by side, keyed by launch id.
Each process gets NESTRI_LAUNCH_ID
(nesprotocol::lifecycle::LAUNCH_ID_ENV), set after the caller's
environment so it cannot be overridden. Stop and restart reach one
launch only. A second launch under an id that is already running is
refused.
- A terminal launch ends the box only when no other launch is running,
so the box stops when its last launch exits. A box with one launch
behaves as before.
- New lifecycle messages, additive, no CONTROL_VERSION bump: host to
guest `mount {id, images, overlays}`; guest to host `launch_mounted
{id}` / `launch_mount_failed {id, reason}`. A failed mount does not take
the box down.
- A mount source that is a regular file (in a virtiofs share) is
attached to a loop device first (LOOP_CONFIGURE, autoclear), because
nesbox cannot hot-add block devices to a running VM. The guest kernel
fragment gains CONFIG_BLK_DEV_LOOP=y, so this needs a new guest
kernel/image to work end to end.
Tests: cargo test -p nesinit (117 passed) and the nesprotocol lifecycle
tests on nestripc-1. New session tests cover two launches side by side,
refusing a duplicate id, stopping one launch, the box ending with its
last launch, and mount success/failure/before boot. Not run in a real
box.
Companion: nestrilabs/nescore feat/window-streams-runtime (neslet
launch.start/launch.stop, nesteam --join-compositor).
Content layout and trust:
- Game and compat images come from a read-only virtiofs share
`launch-images` (mounted read-only at
/nestri/launch-images/<launchId>/build.erofs and compat.erofs). The
per-launch writable layer is in the read-write share `launches`
(/nestri/launches/<launchId>/upper.ext4). Which shares exist is decided
host-side in the boot descriptor; nesinit mounts what it is told.
- The share's only guest-writable files are the per-launch upper.ext4
files.
- Mount sources are opened with openat2(RESOLVE_NO_SYMLINKS): a link at
any component is refused, and only block devices and regular files are
accepted. Loop attach and mounts run on the blocking pool, so other
launches' exits and stops are not stalled.
- CONFIG_BLK_DEV_LOOP must ship in the guest kernel together with this
nesinit (build/kernel/nestri.fragment has it); without it a mount into a
running box fails with a launch_mount_failed.
The box half of phase 1 of window streams: two games in one box, each
its own stream. Wire formats are nesprotocol's `window` module (#421).
## What each app does now
**nescope**
- Gives every toplevel an id on first map: 0 for the first window, never
reused, none past 65535.
- Tells the hub, on the existing input socket, when a window opens,
changes title or size, and closes, in nesprotocol's window encoding. It
also sends a box-internal `MSG_WINDOW_PID` saying which process owns the
window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id
comes from that process's `NESTRI_LAUNCH_ID`.
- Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and
`INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means
window 0. There is still one output: a window's size applies while that
window has focus and is remembered otherwise. On every input connection,
input resets to window 0.
- Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they
describe changes.
**nescapture**
- Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in
the reserved byte). An older reader never sees it.
- Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared
`/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there
still reaches the newest encoder, as before. The shared socket was the
only resource two capturing processes fought over: the second one took
it, and commands went only to that one. The video and stats sockets are
send-only datagram sockets and are safe to share. Capture slots and
devices are per process.
**neshub**
- Joins frames to windows by pid. An untagged frame is window 0. A pid
with no window falls back to the newest window of its launch, then to
window 0 while no other window is open. A pid whose window closed
belongs to nobody.
- A connection that never subscribes gets the box's primary picture in
the old encoding. The primary picture is the frames of the first encoder
to draw, until it has been silent for a second. A subscribed connection
gets only its window's frames, windowed for any window but 0, with
per-window seq. PyroWave stays window 0 only.
- `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the
writer's seq and resync gate in line with the frames, then requests a
keyframe. The first subscribe opens `STREAM_WINDOWS` on the video
connection, starting with `OPENED` (and the icon, if known) for every
open window. Later lifecycle events go to every subscribed connection.
- Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window
changes, and the hub does no merging across tags. A gamepad message
moves focus only when its window differs from the current one and no
other client sent keyboard or pointer input in the last second. Focus
tags move keyboard and pointer focus only: nescope neither raises the
window nor resizes the output for one. The output changes size only on
an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it,
last writer wins with 500 ms hysteresis between windows. A client's
`INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except
for window 0.
- A cursor update goes to the clients subscribed to the window it
describes. Clients that never subscribed get every update, as before.
- When nescope reconnects (a new compositor), the hub sends `CLOSED` for
every window and every subscription goes dead until the client
subscribes again.
- `DatagramSender` sizes fragments with `max_fragment_payload` and
writes headers with `write_video_datagram_header`.
- Commands go to every known encoder on its own socket, or to the shared
socket while none is known (as today). An encoder that appears later is
sent the commands so far (the latest of each message type) and then a
keyframe request. The first encoder is not sent them again.
- A keyframe on a client's loss or subscribe goes only to that client's
window's encoder (the primary's, for a client that never subscribed).
- Known limit: bitrate is box-wide. One controller sets one target for
every window's encoder, so one client's congestion sets every window's
rate. Per-window rate is follow-up work.
- An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with
the same id. nescope no longer exits when its first game exits; nesinit
owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs
as another uid): nescapture sends its launch id.
## Shipping
nescope and neshub ship together in the guest image and must not ship
apart.
## Compatibility
One game with an old client sends the same bytes as before. The client
never subscribes, so it gets the primary picture, which is that game, as
`DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows
stream. Cursor updates reach it as before. Focus tags appear only if the
game's window is not 0, and window 0 resizes use the old form. Mixed
versions also work:
- An old nescapture is window 0.
- An old nescope never sends window messages. Every frame is then window
0, or the primary picture.
- An old hub sends no tags, which nescope reads as window 0, and it
still reaches nescapture through the shared socket.
## Tests
`cargo test` on nesprotocol (122), nescope (17+5), nescapture (161),
neshub (74), all passing. New tests:
- The window table: joining pids to windows, the launch fallback,
retired pids, the primary picture, the snapshot.
- Focus tagging and resize rewriting.
- The command fan-out.
- nescope's id registry (first window 0, ids never reused, refused past
65535) and per-window focus and sizes.
- The IPC pid trailer.
- Two QUIC end-to-end tests:
- Two tagged encoders: an unsubscribed client sees only the first game
in the old encoding. After subscribing to window 1 it sees only the
second game, windowed, with seq starting again, and a windows stream
that starts with both `OPENED` events.
- An untagged encoder gives the old stream.
Not run on a box: no neslet, nestri-host or box was touched.
Phase 1 of decision 0085, the api piece (see .nestri contract
window-streams).
A box run (session) now holds launches: one game or app each. Billing,
allowance and the free weekend still count sessions; nothing about
limits changes. session_box_active_unique stays.
- Migration 0027: launch table (id lch_..., session_id, game_id, state
requested|running|ended|failed, first, time_started, time_stopped,
time_stop_requested, end_reason, error_message). Backfills a first
launch for every run that has not stopped.
- POST /session: same request; the response data gains launches (the
first launch).
- POST /session/{id}/launch {gameId}: 201 with the launch. Same library
check as POST /session. 409 unless the run is starting or live and not
being stopped.
- POST /session/{id}/launch/{launchId}/stop: marks the launch; host gets
launch.stop until it reports it over. A launch in a run no host has
claimed ends at once.
- POST /session/{id}/launch/{launchId}/state (host, run's claimToken):
running | ended | failed, optional endReason/errorMessage.
- When the last launch is over, the run is stopped through the same path
as POST /session/{id}/stop. A run ending ends every launch in it.
- GET /session/{id} gains launches.
- GET /machine/jobs: session.start gains launchId (the first launch);
new kinds launch.start and launch.stop. Agents skip unknown kinds, so
old hosts behave as before.
Tests: apps/api 203 pass (new test/launch.test.ts), packages/core 256
pass, against the local test Postgres.
Fleet placement picked the online machine with the fewest boxes,
whatever its capacity. A box never moves, so a two-slot host got as many
people as an eight-slot one and queued while the bigger one idled.
Now it picks the fewest boxes per slot (the `slots` each host reports in
its heartbeat). A host that reports none counts as one slot, so today's
fleet behaves as before.
- New test: a 2-slot and an 8-slot machine, five people, four land on
the bigger one.
- api 194, core 256 pass.
Protocol for several windows in one box, each its own stream,
ref(d-0085). Design in `crates/nesprotocol/src/window.rs` module docs.
- Video: window 0 is byte-identical to today (DGRAM_VIDEO / MSG_DATA).
Other windows use DGRAM_VIDEO_WINDOW (4, 17-byte header with u16 window
id after sent_us) and MSG_WINDOW_DATA (0x01, u16 window id after seq).
- Lifecycle (hub -> client, new uni STREAM_WINDOWS = 8, length-prefixed
frames): OPENED 0x20, TITLE 0x21, ICON 0x22 (RGBA), SIZE 0x23, CLOSED
0x24.
- MSG_WINDOW_SUBSCRIBE 0x17 on the control stream; never sent means
window 0. Old hubs log it as unknown and continue.
- Box-internal input: INPUT_WINDOW_OUTPUT_SIZE (5) and
INPUT_WINDOW_FOCUS (6) tag resize and input with a window; client
INPUT_OUTPUT_SIZE unchanged.
- Cursor unchanged: per subscription.
nescope gains a no-op match arm for the two new DecodedInput variants.
Tests: cargo test -p nesprotocol --all-features, 140 passed. cargo check
--workspace (excluding nespyro/nescapture, slangc missing on the build
box) clean.
A new input event, INPUT_OUTPUT_SIZE [4][w u16][h u16], rides the input
path to nescope, which resizes its output and every window on it. The
size is clamped to the launch size, so the box's encode budget holds,
and kept even for 4:2:0. apply_resize existed but nothing called it.
Old hubs stop a batch at an unknown type, so clients send it alone.
The installer no longer needs a token copied from the dashboard. Run
bare, the host agent shows a link, a code and a QR; approving it in a
browser registers the machine.
**Issuer**
- New device client `installer`. Its access token lasts 15 minutes and
it gets no refresh token: a refresh lifetime of 0 now means none is
stored or returned.
- `clientTTL(clientID)` sets lifetimes per client and wins over what a
sign-in asks for. Desktop is unchanged.
- `/device/authorize` allows 10 requests per address per 10 minutes.
Past that it answers 429 `slow_down` with `Retry-After`.
- The confirm page for the installer reads "Add a computer as a Nestri
host?". The wording is fixed per client id.
**API**
- An `installer` token is accepted only on `POST
/machine/install-token`. Everything else answers 403
`insufficient_permissions`.
- Install tokens: at most 5 per person per hour, and at most 3 unspent
at once (429 `too_many_requests`).
- A team may own at most 10 machines. This is checked at mint and at
redeem under a row lock (429 `quota_exceeded`). A refused redeem leaves
the token unspent. Organisation hosts are not counted.
**install.sh**
- The token is optional. With none, `NESTRI_INSTALL_TOKEN` is left unset
and onboarding signs in.
- **Do not release this file until `DEFAULT_VERSION` points at a host
agent that can sign in.** Older agents need a token.
Tests: packages/core 256, apps/api 193, packages/auth 74, apps/auth 28,
all passing.
- **Sessions (#412):** `end_reason` and `time_claimed`, with migration
`0026_session_end_reason` (two nullable columns, readable by the release
it replaces). `POST /session/:id/state` takes an optional `endReason`,
and a run stopped before it went live can now go from `starting` to
`ended`.
- **Installer (#411):**
- `bash <(curl -fsSL https://api.nestri.io/install.sh) -u` updates a
host; `-h` prints help; a plain token installs.
- When the script is not root, it asks sudo itself.
- The installed host agent version is unchanged (0.2.18).
**Ships before host agent 0.2.19**, which sends `endReason` and the new
transition.
Cut from prod with dev's files, so squash histories cannot conflict; the
tree is identical to dev.
```
bash <(curl -fsSL https://api.nestri.io/install.sh) -u # update this host
bash <(curl -fsSL https://api.nestri.io/install.sh) <install-token> # make this machine a host
bash <(curl -fsSL https://api.nestri.io/install.sh) -h # help
```
Piping still works: `curl -fsSL … | sh -s -- -u`.
**`-u` / `--update`.** No token and no questions. On a registered
machine it downloads and checksum-verifies the current host agent, swaps
it in and restarts the service. The agent then pulls whatever its
release changed (a raised image layout re-pulls the guest, jail and
Proton). It refuses:
- on a machine that was never installed, before downloading anything;
- while a box is running, since restarting the agent would end that
session;
- on an unknown option.
On a machine that already has the target version, it says so and exits
0.
**Root, asked once.** The host agent runs as root, so when the script is
started as anyone else it asks sudo once, before it changes anything.
Neither a pipe nor `bash <(…)` leaves a file sudo could run again, so it
fetches the same script to a private temp file, runs that as root with
the same mode and token, and removes the copy as soon as root has it
open. Only the settings somebody actually set (`NESTRI_HOST_VERSION`,
`NESTRI_BIN_DIR`, `NESTRI_BOX_STORE`) are carried across.
**Tested.**
- **On an NVIDIA host:** `--update` took it 0.2.17 → 0.2.18, verified
and restarted. A second run said "already 0.2.18. Nothing to do".
- **On the fleet host, run as an ordinary user** with `bash <(curl …)
-u`:
- it asked for sudo, the root copy ran and said "Nothing to do", and no
temp copy was left behind;
- `-h` prints usage, and an unknown flag is refused.
- **In a clean container:** `-u` refuses with "not a Nestri host yet",
and a run with no token asks for one.
Until now every finished run said only `ended` or `failed`. A user stop,
a trial window running out, an idle timeout, a host restart and a game
quitting were indistinguishable afterwards.
**Schema.** Migration `0026_session_end_reason` adds two nullable
columns to `session`:
- `end_reason`: one of `user`, `window`, `hours`, `idle`, `left_queue`,
`game_exited`, `launch_failed`, `host_stopped`, `host_restart`.
- `time_claimed`: when a host took the run, set on `requested →
starting`.
**API.**
- `POST /session/:id/state` takes an optional `endReason`. With `ended`
or `failed` it is stored. With `starting` or `live` it is a 400 and the
run does not move.
- Leaving it out of a terminal report stores null, so host agents that
predate this keep working.
- `endReason` and `timeClaimed` are returned on every session response.
- The API writes the reasons it decides itself: `user` when a run nobody
has claimed is stopped, and `left_queue` when a waiting run is abandoned
(the `Left the queue` message is kept).
**New transition, `starting → ended`.** A run stopped while its game is
still downloading has failed at nothing, and never reached `live`, which
is where billing starts. Until now the only way out of `starting` was
`failed`.
**Release order: this goes first.** The host agent that sends
`endReason` is a separate change. This route's body schema is strict, so
without this deployed the current API answers that agent with a 400.
**Tests.**
- API: 185 pass, 5 of them new.
- Core: 41 session tests pass, 3 of them new.
- The new tests fail when the implementation is broken on purpose.
- No new type errors.
- **Guest image (#405):** carries `libvpx`, `speex`, `libtheora`,
`libvdpau` and `jsoncpp`, which Proton's GStreamer plugins link. Without
them, `libgstlibav` and `libgstspeex` failed to load in every box on
every Proton shipped so far. Verified: every library both published
Protons need is now provided.
- **Billing (#406):** every Polar webhook logs its type, the team it
resolved to, the plan and status, whether it changed anything, and the
subscription and customer ids. Logging only.
Cut from prod rather than opened from dev: after squash merges the two
histories conflict, though the files differ only in these four. This
branch's tree is identical to dev's.
After merge, the guest image is published from this prod commit. Host
agent 0.2.18 then makes hosts pull it.
**Problem.** A billing webhook that changes nothing answers `200`
exactly like one that worked.
- A subscription made before checkouts carried the team id (`db43068`)
never matches a team.
- When one was cancelled, the API logged `a paid subscription matched no
team` with only the event type. Nothing said whose subscription it was,
so nobody could act on it.
- Deliveries that did apply were not logged at all.
**Changes.**
- `Polar.receive` returns the provider's `subscriptionId` and
`customerId`, in both camelCase and snake_case, as the rest of the
payload is read.
- For a `subscription.*` event, the event's own id is the
subscription's.
- For anything else, such as `order.refunded`, it is the
`subscription_id` field.
- The webhook route logs one line per delivery: `billing webhook {type,
team, standing, applied, subscription, customer}`.
- The unmatched-paid warning carries those ids and says what to do: find
the customer in the provider dashboard and set the team's plan by hand.
- A delivery naming a team that no longer exists is logged as an error,
not silently applied to nothing.
No behaviour change: the same deliveries apply to the same teams. This
is logging only.
**Tests.**
- `bun test src/billing/polar.test.ts`: 17 pass. One new test covers the
ids in both spellings, on a subscription event and on a non-subscription
one.
- `tsc` has no errors in the changed files. The existing errors are in
`utils/hook.ts` and `utils/validator.ts`, from Zod 4.
**Problem.** Proton's GStreamer plugins link `libvpx`, `speex`,
`libtheora`, `libvdpau` and `jsoncpp`, and neither Proton nor the guest
image carries them. Boxes log:
```
Failed to load plugin '.../gstreamer-1.0/libgstlibav.so': libvpx.so.12: cannot open shared object file
Failed to load plugin '.../gstreamer-1.0/libgstspeex.so': libspeex.so.1: cannot open shared object file
```
Wine's media stack then has no decoders, so a game whose videos go
through it plays them as nothing.
**Scope.**
- Not new. The stock `11.0-20261005` Proton and the patched `-n1` build
need exactly the same set.
- The fleet logged it on 2026-10-10 and again on 2026-10-15.
**Fix.** These five packages are added to the guest's runtime package
list.
- They were found by listing every `NEEDED` library of each published
Proton against this image; they were the whole gap.
- They track the sonames Proton was built against, as the new comment
says.
**Verification.** In progress: the image is building, and the library
check is rerun against it for both Protons.
New installs get host agent 0.2.17. It is published at
/install/host/0.2.17/; the binary is checksum-verified and reports
`neslet 0.2.17`. 0.2.17 raises LAYOUT to 6, so hosts rebuild their guest
from the current image (OpenGL frontends, nescope answering every
frame).
Dev still said host agent 0.2.13 and desktop 0.1.0, while prod serves
0.2.16 and 0.1.2. The next dev→prod merge would have sent new installs
back three host releases. This brings dev to prod's numbers. No other
change: the rest of this branch is already on dev.
Guest image fixes from #401, plus the dev changes that had not reached
prod yet.
- **OpenGL games render** (Dome Keeper, confirmed on an AMD host). The
guest's Mesa gains its EGL/GLX frontends, and `vm.overcommit_memory=1`
stops a large reservation being refused before it is used.
- **Chromium / NW.js games get past their first frame** (Sheepy).
nescope answers every commit, and paces Xwayland windows too.
- **Proton patch step.** `build/patches/proton/` and the `-nN` tag.
Nothing uses it until hosts are pointed at the new tag.
- Also included: the pixelforge update already on dev. It is in the
image tested today.
**Versions kept, not downgraded.** Dev still had host agent 0.2.13 and
desktop 0.1.0. The last commit restores prod's 0.2.16 and 0.1.2, so this
merge changes no install versions.
**After merge.** `make push` from this prod commit publishes the guest
image. The host agent release that raises LAYOUT to 6 makes hosts pull
it.
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Games that showed a black screen with sound now render. Found by testing
on an AMD host: none of these were GPU-vendor specific.
**OpenGL games (Dome Keeper, Godot): fixed and confirmed.**
- `c017462`: the guest's Mesa had no EGL/GLX frontends, so a GL game had
nothing to load and never created a device. The image now ships them,
with glvnd's vendor file, so GL reaches zink and the capture layer sees
the frames.
- `5f2ccdf`: `vm.overcommit_memory=1`. The kernel default refused Dome
Keeper's 9 GiB reservation on an 8 GiB box before a byte was used.
**Chromium / NW.js games (Sheepy) stalled after one frame: fixed.**
- `a73c0c4`: nescope answers every commit with frame callbacks and
feedback, not only commits from a surface it maps to a window.
Chromium's GPU process draws for a window another process owns. Wine's
Wayland driver gives that surface no role, so its FIFO present waited
forever in the driver.
- `23ba9b5`: a window is matched by its surface, not its xdg toplevel,
so Xwayland windows are paced too.
**Proton could not create non-ASCII file names: patch added, build
pending.**
- `6770684`: Steam starts games with `LC_ALL=C`, and Wine reads that as
ASCII for every Unix file name. NW.js stops unpacking its app at
`media/• drum loop….webm` and shows an empty window.
- The patch reads an ASCII locale as UTF-8. Proton now takes patches
from `build/patches/proton/<submodule>/`, and a patched image is tagged
`-nN` (`11.0-20261005-n1`).
- Nothing points at that tag yet; hosts move to it in a separate change,
after it is built and tested.
Not covered: Metro 2033 Redux's report looks like the frame-callback
stall, but it was not retested.
Steam starts games with LC_ALL=C, and Wine reads the C locale as ASCII
for every Unix file name it makes, so a name like '• drum loop.webm'
could not be created at all. Sheepy, an NW.js game, unpacks its app from
a zip at start, stopped at that file, and showed NW.js's empty window.
Proton is now patched before it builds (patches/proton/<submodule>/), and
the first patch reads an ASCII locale as UTF-8. The image name gains -nN
for our patch level, so a patched build never takes the plain tag's name.
A Vulkan client presenting FIFO waits in the driver for its frame
callback. Chromium's GPU process draws for a window another process
owns, which Wine's Wayland driver gives a surface with no role, so no
window here matched it and the game presented one frame and waited
forever: Sheepy's black screen. Nothing here paces against a display,
so every commit now gets its frame callbacks and feedback.
Since frame callbacks moved onto the commit path, the commit handler
found a window by its xdg toplevel. An Xwayland window has none, so no
Proton game got a frame callback or presentation feedback. Xwayland then
completes a flip only on its one-second fallback timer, and a game
presenting FIFO stalls in the driver after its first frame.
The kernel's default overcommit policy refuses any single allocation
larger than RAM plus swap. A box has no swap, so the ceiling is the
box's own size, and an engine or driver that reserves address space it
never touches is refused before it uses any of it. Dome Keeper asked
for 9 GiB on an 8 GiB box and never started.
Set vm.overcommit_memory=1 alongside vm.max_map_count, so a request is
granted and the OOM killer acts on what is actually used. In a box that
is the game: init has already exempted itself.
Mesa was built with zink as its only Gallium driver and every frontend disabled, so a game rendering with OpenGL found no GL and showed a black screen. EGL, GLX, GBM and GLES2 are enabled, with the x11 platform for Xwayland clients.
The real change. #399 merged by mistake without it: a rewritten copy of
the #398 merge commit, same tree, so nothing in prod changed. New hosts
install 0.2.16 of the host agent: onboarding retries a failed phase, and
finds steamclient.dll where Valve's client package now keeps it.
A customer the payment provider already knew under the same email kept
their old customer record, without our team id, so their paid
subscription was acknowledged and applied to no team. Checkout now
carries the team in its metadata and the webhook falls back to it; a
paid delivery that matches no team is logged.
Wanjohi
