Promotes dev to prod: #450 billing portal 404 when a team has no Polar
customer; #448, #449 per-window pause and resume (neshub). Merge commit,
not squash.
GET /billing/portal threw a raw Polar error (500) for a team that never
checked out. Polar's 404/422 for an unknown external customer now
becomes 404 "This team has no billing yet". Tests added.
Fixes the failure of `nesprobe scenario --pause` on NestriTest (534
units in the 10 s after the pause, no pause line in the journal).
## Cause
`ClientSession::encoder_pid` is how the hub knows which encoder to pause
for a client. It was written in only two places:
`ClientSession::subscribe` (from `WindowTable::encoder_of`, the encoder
that has already drawn that window) and, on a primary switch, for
clients that never subscribed. On a real box the order is: the window
opens, the client subscribes to it (the log shows `client subscribes to
window 0` at 21:29:51 and again at 21:29:59.14), and only then does the
game's encoder draw its first frame. At subscribe time `encoder_of` is
`None`, so the pid was stored as 0 and never updated. `MSG_WINDOW_PAUSE`
hit `pid_or_all(..) == None` and was dropped at debug level (hence
nothing in the journal). Keyframe requests were not affected because for
pid 0 they broadcast to every encoder, which hid the problem.
Checked against the other two questions:
- (a) nesprobe uses the subscribe path (window 0 subscribe, same control
stream), and the pause message is sent on it.
- (c) The hub-to-encoder socket is fine for Proton: `nescapture` in pid
238 listens on `/tmp/nescapture-cmd-238.sock` and receives the IDR
requests the hub sends.
## Fix
`broadcast_video` now stores the frame's encoder pid into every session
that route() shows the frame to, so a client's encoder follows the
frames it is sent, whenever it subscribed. This also makes its keyframe
requests targeted instead of broadcast.
Also added info-level logs: hub `pause:`/`resume:` with client and pid
or "no encoder known"; hub `sent pause|resume to <socket>`; nescapture
`window pause|resume received (was ...)`.
## Tests
- New e2e test
`a_client_that_subscribed_before_the_first_frame_learns_its_encoder`:
subscribes before any frame, then draws; fails without the fix (pid
stays 0, verified by disabling the store), passes with it.
- neshub 89 pass, nescapture 162 pass (nestripc-1).
- Not re-run on a box.
The session's encoder pid was set only at subscribe time (from the encoder
that had drawn the window) and on a primary switch for clients that never
subscribed. A client that subscribed before the game drew its first frame,
which is the real order, kept pid 0, so its pause was ignored. Also logs
pause and resume at info in the hub and in nescapture.
Implements the box side of decision 0087 (items 1 and 2): a per-window
Pause / Resume.
## Plan
A correction to the brief: `nesprotocol` lives in this repo (nestri),
not in nescore. nescore and nesktop consume it by git rev. So the
protocol and the box side land together here, then nescore gets the
client call and a pin move, then nesktop gets the probe scenario and its
pin move. Also, the video encoder is `nescapture` (a Vulkan layer in the
game's process), not `neswire` (audio), and the compositor `nescope`
needs no change: capture happens in the game's present hook.
**Messages** (control stream, client to hub, empty payload, like
`MSG_IDR_REQUEST`):
- `MSG_WINDOW_PAUSE = 0x18`
- `MSG_WINDOW_RESUME = 0x19`
**Window id flow.** No id in the message. The hub already knows which
window a client is subscribed to (`MSG_WINDOW_SUBSCRIBE`) and the pid of
that window's encoder (`subscribe.2`, the same one keyframe requests
use). So:
1. client sends pause/resume on the control stream;
2. `neshub` `run_control_reader` turns it into `MSG_WINDOW_PAUSE/RESUME`
targeted at that encoder's own command socket
(`capture_cmd::window_pause/resume`, via `to_encoder`). Targeted
commands are not recorded in the replay history, so a later encoder is
not born paused;
3. `nescapture`'s command listener sets a `paused` flag; `try_capture`
in the present hook returns early while it is set (before the frame
gate, before any blit, slot or encode). The game's process is untouched;
4. resume clears the flag and sets `idr_requested`, so the first frame
after it is a keyframe. The hub also sets `awaiting_keyframe` for that
client, as for an IDR request.
A pause from a client whose encoder the hub does not know yet is
ignored, because "all encoders" is the wrong target for a pause.
Old hub, new client: logs an unknown control message. Old nescapture
ignores the unknown byte.
## Tested
- `cargo test -p nesprotocol -p neshub`: 122 + 79 pass (new: type bytes
distinct, including 0x17 subscribe; pause/resume target one encoder and
are not replayed).
- `cargo test -p nescapture`: 162 pass (new: pause stops, resume
requests a keyframe). Built on nestripc-1 with `SLANGC` pointed at a
copy in a container layer, because the box has no slangc.
## Not verified
- No real game or GPU run. That the present hook really stops blitting
while paused, and that a keyframe lands within 1 s of resume, is proven
only by unit tests here; the nesprobe `--pause` scenario in the nesktop
PR is what checks it end to end, and it has not been run against a box.
- A game that presents nothing after resume (a fully still scene with no
presents) would delay the keyframe until its next present, same as any
existing IDR request.
## Update after review (78a8bc8)
- A pause is per session. `neshub/src/pauses.rs` keeps the set of
sessions pausing each encoder; the encoder is paused while the set is
non-empty. Resume goes only to the pid that session paused and never
broadcasts (a resume from a session that paused nothing starts nothing).
- A session releases its pause on drop (`ClientSession::drop`), on
resubscribe, and when the primary picture moves (`restart_picture`).
Keyed by the never-reused gamepad session number, so a reconnecting
client cannot release or inherit another session's pause.
- Tests (neshub 86 pass): two sessions on one encoder; session that
never paused resumes nothing; drop while paused (unit and through
`remove_session` on the e2e rig); resubscribe while paused releases the
old encoder only; pausing another encoder moves the pause.
Dropping a session aborts its tasks, but a control reader mid-poll on another
thread could still pause after the release and leave the encoder paused
forever. The closed flag is set under the book's lock; pause checks it under
the same lock.
The hub keeps the set of sessions pausing each encoder. An encoder is paused
while the set is not empty; a resume goes only to the encoder that session
paused; a session that drops, resubscribes or sees the primary picture move
releases its pause.
MSG_WINDOW_PAUSE and MSG_WINDOW_RESUME on the control stream. The hub sends
them to the encoder of the window the client is shown; nescapture stops
capturing in the present hook while paused, and resume requests a keyframe.
dev → prod. Needs neslet 0.2.23, which is published.
- fix(install): run `onboard --check` before stopping a running agent,
and restart the agent if onboarding fails after the stop (#446)
- fix(install): an upgrade keeps the existing box store and GPU
selection and doesn't prompt
- feat(install): `--gpus` passes through
- chore(install): default host agent 0.2.23
Fixes the installer half of nescore#92 and #445.
- On an upgrade, `nestri-host onboard --check` (nescore#93, no lock, no
changes) runs before the agent is stopped. If it fails, the agent is
left running and the installer exits.
- If onboarding fails after the agent was stopped, the installer starts
the agent again before exiting.
- A host with /etc/nestri-host/env keeps its NESLET_STORE (or
NESLET_BOX_STORE) and NESLET_GPUS and is not asked where box images go.
Only a first install is prompted. NESTRI_BOX_STORE and --gpus still
override.
- `--gpus SLOTS` (or `--gpus=SLOTS`, or NESLET_GPUS) is accepted after
the token, survives the sudo re-exec, and reaches the check and
onboarding.
- DEFAULT_VERSION is 0.2.23.
**Merge only after neslet 0.2.23 (nescore#93) is released**: the
installer calls `--check` and `--gpus`, which older binaries lack, and
pins that version. Verified with `sh -n` and by hand-checking the
argument errors and env-file parsing; not run end to end on a host.
On a host already running the agent, run preflight first and stop the agent only if it passes, so a failing check no longer leaves the machine offline. An existing host keeps the box store from its environment file instead of being asked, since the guessed default moved a store and hid its games. --gpus is accepted after the token and reaches onboarding.
dev → prod.
- fix(nescope): a window can't ask for more than the output has, and
clamped requests are logged (#438)
- fix(nescope): a hub that closes the input socket is no longer a panic;
nescope ends if the hub stays gone (#440)
- fix(nescope, neswire): default log filter is info; dropped audio
frames log once per burst (#441)
nescope used `EnvFilter::from_default_env()`, whose default is ERROR, so
its info/warn lines never reached the host journal. nesinit only
forwards RUST_LOG if it has it itself.
- nescope: fall back to `info` when RUST_LOG is unset or unparsable
(same as neshub, nesinit, nesgamepad); debug stays opt-in. 4 unit tests
on the filter helper.
- neswire: same ERROR default, same fix.
- Not changed: nescapture (library inside the game process, env_logger).
Tested on nestripc-1: `cargo test -p nescope log_filter` 4 passed,
`cargo check -p neswire` ok.
Note: no 'configure_request clamped' warn exists on this base; once it
lands at warn level it is covered by the info default.
A stalled encoder drops about fifty frames a second, and with warn now
visible by default each drop would be its own journal line. Count them
and log the first and every 250th.
EnvFilter::from_default_env defaults to ERROR, so nescope's info and warn
lines never reached the host journal. Fall back to info like neshub,
nesinit and nesgamepad; align neswire, which had the same default.
Found in a passing multi-launch probe on NestriTest
(nesprobe-1792327018): at shutdown nescope logged `panicked at
apps/nescope/src/main.rs:639:10: Event loop error: ... IPC socket
closed`.
Cause: `InputIpcSource::process_events` returned EOF as `Err`; calloop
passes that out of `run()` into the `.expect("Event loop error")`. At
box shutdown nesinit does `kill(-1, SIGTERM)`, so neshub can exit before
nescope reads its own SIGTERM, and the panic skips the cleanup after the
loop (socket removal, child kill, the clean-exit log).
Change: on EOF, deliver the frames already read, warn, and return
`PostAction::Remove` so the loop carries on and the signal or auto-exit
path ends it. Read loop split into `fill()` so it is unit-tested with a
socket pair.
Tested on nestripc-1 (CARGO_TARGET_DIR=~/target-fix-nescope-ipc-exit):
`cargo test -p nescope`, 3 new tests pass.
The input source now removes itself on EOF or reset and raises a flag; the
main loop exits 2 s later unless a signal already ended it, so a hub crash
mid-run still ends the compositor (and the box via CompositorDied) instead of
holding a GPU with no input. Also corrects the comment that promised a
reconnect that does not exist.
The input source returned UnixStream EOF as an error, which calloop passes
out of run() into the expect at the end of main. At box shutdown init
signals every process at once, so neshub can go before nescope reads its
SIGTERM: nescope panicked and skipped its cleanup. Deliver what was read,
remove the source and warn instead.
Phase 0 of decision 0086.
Cause (inferred from the code, not observed live): the client's window
size resizes the box's output, but a Proton game under XWayland
re-asserts its launch size with a ConfigureRequest, which
configure_request honoured. The window ends up larger than the output;
the pointer lives on the output (clamp_cursor), so the cut-off part of
the window is unreachable. The client sends relative motion only, so no
client-side mapping is involved.
Fix: configure_request clamps a non-override-redirect X11 window's
requested size to the current output size (and keeps it inside the
output). The output keeps following the client. Pure fn clamp_to_output,
unit-tested incl. width-only, height-only and multiple windows.
Risk: a game that ignores the configure and keeps drawing at its own
size will still be cropped. Wayland toplevels are unchanged (they
already get the output size).
Live proof is still pending: pointer reach on a real Proton window after
a resize has not been exercised.
Adds an optional `daemon` to the boot descriptor (decision 0085).
nesinit starts it after the compositor, as the given uid, announces it
in /run/nestri/daemon before the box is ready, lists it among the box's
services, and on its exit reports ServiceDied by name and ends the box
(as for the compositor). Optional both ways: an old descriptor reads the
same; an old guest refuses a new descriptor and the host already reports
that as a stale image. Tests: protocol round trip, started after the
compositor, start failure refuses the box, exit reported.
Dev to prod: GET /machine/jobs?wait=&seen= long-poll (#435), nesinit
multi-launch fixes (#432, #433), launch/launch-stop wakes. Hosts that
don't send `wait` see no change. **Merge with a merge commit, not
squash.**
Cuts host job pickup from the heartbeat interval to under a second.
`GET /machine/jobs?wait=<s>` (capped at 25) answers at once when there
is work, else holds until woken or the wait is up, then answers empty.
Without `wait` nothing changes.
Wake (in-process, keyed by machine id, fired after commit through
`Database.effect`):
- run requested: `Session.create`
- stop asked for on a starting or live run: `stopRun`
- launch added: `addLaunch`; launch stop asked: `stopLaunch`
- run ended or failed (frees a slot for a queued run):
`compareAndSetState`
Not woken, covered by the 5 s recheck: trial and weekend-window expiry,
a heartbeat that raises slots.
Multi-process: `redis` is in `apps/api/package.json` but nothing imports
it and nothing configures a URL, so no pub/sub path here. Each held call
re-reads every 5 s, so a wake that lands in another process costs a few
seconds. Add a channel when a second process exists.
Worker runtime: answers at once (no hold). Bun: `idleTimeout: 60` added,
since the default 10 s would cut the held call.
Tests: `apps/api/test/jobs-wait.test.ts` (8, on the local test Postgres)
plus the full API suite, 216 pass. Pairs with the neslet PR in nescore
(`feat/jobs-long-poll`); an old neslet is unaffected and a new neslet
against an old API falls back to its interval.
A host found new work on its next poll, 8 to 21 seconds on average. With wait (at most 25 s) a call that finds nothing is held until a run is requested, a stop is asked for, a launch is added or stopped, or a run ends and frees a slot, then answers. A held call also looks again every 5 s so a wake in another process costs seconds, not the job. Workers answer at once. Bun's idle timeout is raised so the held call is not cut.
Stacked on #432 (fix/nesinit-loop-held); its commits show here until it
merges.
Stopping a launch in a two-launch box left that launch's game running:
`stop_launch` signalled only the launch's top process, so the children
(compositor wrapper, wine, the game) survived and the window never
closed. Killing the whole group naively would have taken the box's one
compositor, which the other launch is drawing on.
- Each launch runs under `setsid`. A stop signals the whole session
(SIGTERM, SIGKILL after 5 s). Members are found from /proc by session
id, so children that moved to a group of their own are still reached.
- A launch's exit is delivered only when its session is empty. If the
top process leaves by itself with leftovers, they are ended first.
- The compositor is a box-level process: the boot descriptor carries an
optional `Compositor` (geometry, rate, uid, env) and nesinit starts it
before the box reports ready, waiting for its socket. Absent, nothing
changes. A failure to start it refuses the box like a failed service.
- SIGKILL on the way down also takes each launch's group.
Tests: real-process tests for stop reaching children, a child ignoring
SIGTERM being killed after the grace, a top process leaving with
leftovers, and one launch's stop leaving another's group alone; session
tests for the compositor start and refusal; descriptor round trip. All
of nesinit and nesprotocol pass on nestripc-1.
Host side: nestrilabs/nescore (fix/launch-group-nescore). A host that
sends `compositor` needs this guest image first, because the descriptor
refuses unknown fields.
Two bugs found running 0085 on NestriTest (2026-10-17):
- **Loop devices released before mount.** `loop_attach` configured
`/dev/loopN` with `LO_FLAGS_AUTOCLEAR` and dropped its fd before
mounting, so the kernel detached it and the writable layer's attach got
the same device. The build image mount then read ext4: `erofs (device
loop0): cannot find valid erofs superblock` → EINVAL. The device fd is
now held until the mount is made.
- **A box refused new launches after one ended.** A terminal launch
exiting beside another marked the box ending and refused every new
launch, so re-opening a closed game failed with "the box is ending". Per
0085 closing one window is not the person done: the box still takes
launches and ends when its last one goes.
Verified live on NestriTest: Everhood mounted (loop0 erofs, loop1 ext4)
beside Dome Keeper. nesinit tests: 124 pass.
Wine setsid's its processes, so stopping a launch by session left wineserver,
explorer.exe and the game running with their window open. Each launch is now
put in cgroup /sys/fs/cgroup/nestri-launch-<id> before exec; stop is SIGTERM to
its members, then cgroup.kill after the grace, and the launch ends when the
cgroup is empty. Without cgroup v2 the members are the session plus every
process whose environment carries NESTRI_LAUNCH_ID. cgroup2 is mounted at
/sys/fs/cgroup early in boot.
Stopping a launch signalled only its top process, so the compositor wrapper,
wine and the game survived and the window stayed open. Each launch now runs
under setsid; a stop signals the whole session (TERM, then KILL after 5s) and
a launch counts as ended only once its session is empty.
The compositor moves out of the first launch into the box: the boot
descriptor carries an optional Compositor (geometry, rate, uid, env) and
nesinit starts it before the box reports ready, so every launch joins it.
Dev plus a merge that records prod's squash commits as already in dev,
so this PR has no conflicts. Merge it with **Create a merge commit**,
not squash.
All of dev to prod. Prod's squash copies are merged in with dev's
version kept, so the tree is identical to dev.
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #422, from the nesktop #3 review. POST /session/{id}/launch
for a game that already has an open launch (requested or running, no
stop requested) in the run now answers that launch with 200 instead of
inserting a second one. The check runs under the run's row lock that
addLaunch already takes, so two concurrent adds resolve to one launch.
OpenAPI documents the 200.
Tested: apps/api 208 pass against the local test Postgres, including a
new test adding the same game twice concurrently.
Wanjohi
