Follow-up to #422, from the nesktop #3 review. POST /session/{id}/launch
for a game that already has an open launch (requested or running, no
stop requested) in the run now answers that launch with 200 instead of
inserting a second one. The check runs under the run's row lock that
addLaunch already takes, so two concurrent adds resolve to one launch.
OpenAPI documents the 200.
Tested: apps/api 208 pass against the local test Postgres, including a
new test adding the same game twice concurrently.
Follow-up to #423 from its re-review. The dgram writer now raises
awaiting_keyframe when it reaches the Resubscribe marker, in queue
order, and restart_picture sets the flag before queuing the marker.
Before this, an old keyframe still queued ahead of a primary switch
could clear the flag, letting the new picture's first delta out as the
first frame an old client saw after the switch. Needs a writer stalled
for at least 1 s to trigger.
Tested: cargo test -p neshub on nestripc-1, 78 passed.
Runtime half of the window streams contract (joins 1 and 6).
- nesinit runs any number of launches side by side, keyed by launch id.
Each process gets NESTRI_LAUNCH_ID
(nesprotocol::lifecycle::LAUNCH_ID_ENV), set after the caller's
environment so it cannot be overridden. Stop and restart reach one
launch only. A second launch under an id that is already running is
refused.
- A terminal launch ends the box only when no other launch is running,
so the box stops when its last launch exits. A box with one launch
behaves as before.
- New lifecycle messages, additive, no CONTROL_VERSION bump: host to
guest `mount {id, images, overlays}`; guest to host `launch_mounted
{id}` / `launch_mount_failed {id, reason}`. A failed mount does not take
the box down.
- A mount source that is a regular file (in a virtiofs share) is
attached to a loop device first (LOOP_CONFIGURE, autoclear), because
nesbox cannot hot-add block devices to a running VM. The guest kernel
fragment gains CONFIG_BLK_DEV_LOOP=y, so this needs a new guest
kernel/image to work end to end.
Tests: cargo test -p nesinit (117 passed) and the nesprotocol lifecycle
tests on nestripc-1. New session tests cover two launches side by side,
refusing a duplicate id, stopping one launch, the box ending with its
last launch, and mount success/failure/before boot. Not run in a real
box.
Companion: nestrilabs/nescore feat/window-streams-runtime (neslet
launch.start/launch.stop, nesteam --join-compositor).
Content layout and trust:
- Game and compat images come from a read-only virtiofs share
`launch-images` (mounted read-only at
/nestri/launch-images/<launchId>/build.erofs and compat.erofs). The
per-launch writable layer is in the read-write share `launches`
(/nestri/launches/<launchId>/upper.ext4). Which shares exist is decided
host-side in the boot descriptor; nesinit mounts what it is told.
- The share's only guest-writable files are the per-launch upper.ext4
files.
- Mount sources are opened with openat2(RESOLVE_NO_SYMLINKS): a link at
any component is refused, and only block devices and regular files are
accepted. Loop attach and mounts run on the blocking pool, so other
launches' exits and stops are not stalled.
- CONFIG_BLK_DEV_LOOP must ship in the guest kernel together with this
nesinit (build/kernel/nestri.fragment has it); without it a mount into a
running box fails with a launch_mount_failed.
The box half of phase 1 of window streams: two games in one box, each
its own stream. Wire formats are nesprotocol's `window` module (#421).
## What each app does now
**nescope**
- Gives every toplevel an id on first map: 0 for the first window, never
reused, none past 65535.
- Tells the hub, on the existing input socket, when a window opens,
changes title or size, and closes, in nesprotocol's window encoding. It
also sends a box-internal `MSG_WINDOW_PID` saying which process owns the
window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id
comes from that process's `NESTRI_LAUNCH_ID`.
- Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and
`INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means
window 0. There is still one output: a window's size applies while that
window has focus and is remembered otherwise. On every input connection,
input resets to window 0.
- Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they
describe changes.
**nescapture**
- Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in
the reserved byte). An older reader never sees it.
- Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared
`/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there
still reaches the newest encoder, as before. The shared socket was the
only resource two capturing processes fought over: the second one took
it, and commands went only to that one. The video and stats sockets are
send-only datagram sockets and are safe to share. Capture slots and
devices are per process.
**neshub**
- Joins frames to windows by pid. An untagged frame is window 0. A pid
with no window falls back to the newest window of its launch, then to
window 0 while no other window is open. A pid whose window closed
belongs to nobody.
- A connection that never subscribes gets the box's primary picture in
the old encoding. The primary picture is the frames of the first encoder
to draw, until it has been silent for a second. A subscribed connection
gets only its window's frames, windowed for any window but 0, with
per-window seq. PyroWave stays window 0 only.
- `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the
writer's seq and resync gate in line with the frames, then requests a
keyframe. The first subscribe opens `STREAM_WINDOWS` on the video
connection, starting with `OPENED` (and the icon, if known) for every
open window. Later lifecycle events go to every subscribed connection.
- Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window
changes, and the hub does no merging across tags. A gamepad message
moves focus only when its window differs from the current one and no
other client sent keyboard or pointer input in the last second. Focus
tags move keyboard and pointer focus only: nescope neither raises the
window nor resizes the output for one. The output changes size only on
an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it,
last writer wins with 500 ms hysteresis between windows. A client's
`INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except
for window 0.
- A cursor update goes to the clients subscribed to the window it
describes. Clients that never subscribed get every update, as before.
- When nescope reconnects (a new compositor), the hub sends `CLOSED` for
every window and every subscription goes dead until the client
subscribes again.
- `DatagramSender` sizes fragments with `max_fragment_payload` and
writes headers with `write_video_datagram_header`.
- Commands go to every known encoder on its own socket, or to the shared
socket while none is known (as today). An encoder that appears later is
sent the commands so far (the latest of each message type) and then a
keyframe request. The first encoder is not sent them again.
- A keyframe on a client's loss or subscribe goes only to that client's
window's encoder (the primary's, for a client that never subscribed).
- Known limit: bitrate is box-wide. One controller sets one target for
every window's encoder, so one client's congestion sets every window's
rate. Per-window rate is follow-up work.
- An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with
the same id. nescope no longer exits when its first game exits; nesinit
owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs
as another uid): nescapture sends its launch id.
## Shipping
nescope and neshub ship together in the guest image and must not ship
apart.
## Compatibility
One game with an old client sends the same bytes as before. The client
never subscribes, so it gets the primary picture, which is that game, as
`DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows
stream. Cursor updates reach it as before. Focus tags appear only if the
game's window is not 0, and window 0 resizes use the old form. Mixed
versions also work:
- An old nescapture is window 0.
- An old nescope never sends window messages. Every frame is then window
0, or the primary picture.
- An old hub sends no tags, which nescope reads as window 0, and it
still reaches nescapture through the shared socket.
## Tests
`cargo test` on nesprotocol (122), nescope (17+5), nescapture (161),
neshub (74), all passing. New tests:
- The window table: joining pids to windows, the launch fallback,
retired pids, the primary picture, the snapshot.
- Focus tagging and resize rewriting.
- The command fan-out.
- nescope's id registry (first window 0, ids never reused, refused past
65535) and per-window focus and sizes.
- The IPC pid trailer.
- Two QUIC end-to-end tests:
- Two tagged encoders: an unsubscribed client sees only the first game
in the old encoding. After subscribing to window 1 it sees only the
second game, windowed, with seq starting again, and a windows stream
that starts with both `OPENED` events.
- An untagged encoder gives the old stream.
Not run on a box: no neslet, nestri-host or box was touched.
Phase 1 of decision 0085, the api piece (see .nestri contract
window-streams).
A box run (session) now holds launches: one game or app each. Billing,
allowance and the free weekend still count sessions; nothing about
limits changes. session_box_active_unique stays.
- Migration 0027: launch table (id lch_..., session_id, game_id, state
requested|running|ended|failed, first, time_started, time_stopped,
time_stop_requested, end_reason, error_message). Backfills a first
launch for every run that has not stopped.
- POST /session: same request; the response data gains launches (the
first launch).
- POST /session/{id}/launch {gameId}: 201 with the launch. Same library
check as POST /session. 409 unless the run is starting or live and not
being stopped.
- POST /session/{id}/launch/{launchId}/stop: marks the launch; host gets
launch.stop until it reports it over. A launch in a run no host has
claimed ends at once.
- POST /session/{id}/launch/{launchId}/state (host, run's claimToken):
running | ended | failed, optional endReason/errorMessage.
- When the last launch is over, the run is stopped through the same path
as POST /session/{id}/stop. A run ending ends every launch in it.
- GET /session/{id} gains launches.
- GET /machine/jobs: session.start gains launchId (the first launch);
new kinds launch.start and launch.stop. Agents skip unknown kinds, so
old hosts behave as before.
Tests: apps/api 203 pass (new test/launch.test.ts), packages/core 256
pass, against the local test Postgres.
Fleet placement picked the online machine with the fewest boxes,
whatever its capacity. A box never moves, so a two-slot host got as many
people as an eight-slot one and queued while the bigger one idled.
Now it picks the fewest boxes per slot (the `slots` each host reports in
its heartbeat). A host that reports none counts as one slot, so today's
fleet behaves as before.
- New test: a 2-slot and an 8-slot machine, five people, four land on
the bigger one.
- api 194, core 256 pass.
Protocol for several windows in one box, each its own stream,
ref(d-0085). Design in `crates/nesprotocol/src/window.rs` module docs.
- Video: window 0 is byte-identical to today (DGRAM_VIDEO / MSG_DATA).
Other windows use DGRAM_VIDEO_WINDOW (4, 17-byte header with u16 window
id after sent_us) and MSG_WINDOW_DATA (0x01, u16 window id after seq).
- Lifecycle (hub -> client, new uni STREAM_WINDOWS = 8, length-prefixed
frames): OPENED 0x20, TITLE 0x21, ICON 0x22 (RGBA), SIZE 0x23, CLOSED
0x24.
- MSG_WINDOW_SUBSCRIBE 0x17 on the control stream; never sent means
window 0. Old hubs log it as unknown and continue.
- Box-internal input: INPUT_WINDOW_OUTPUT_SIZE (5) and
INPUT_WINDOW_FOCUS (6) tag resize and input with a window; client
INPUT_OUTPUT_SIZE unchanged.
- Cursor unchanged: per subscription.
nescope gains a no-op match arm for the two new DecodedInput variants.
Tests: cargo test -p nesprotocol --all-features, 140 passed. cargo check
--workspace (excluding nespyro/nescapture, slangc missing on the build
box) clean.
A new input event, INPUT_OUTPUT_SIZE [4][w u16][h u16], rides the input
path to nescope, which resizes its output and every window on it. The
size is clamped to the launch size, so the box's encode budget holds,
and kept even for 4:2:0. apply_resize existed but nothing called it.
Old hubs stop a batch at an unknown type, so clients send it alone.
The installer no longer needs a token copied from the dashboard. Run
bare, the host agent shows a link, a code and a QR; approving it in a
browser registers the machine.
**Issuer**
- New device client `installer`. Its access token lasts 15 minutes and
it gets no refresh token: a refresh lifetime of 0 now means none is
stored or returned.
- `clientTTL(clientID)` sets lifetimes per client and wins over what a
sign-in asks for. Desktop is unchanged.
- `/device/authorize` allows 10 requests per address per 10 minutes.
Past that it answers 429 `slow_down` with `Retry-After`.
- The confirm page for the installer reads "Add a computer as a Nestri
host?". The wording is fixed per client id.
**API**
- An `installer` token is accepted only on `POST
/machine/install-token`. Everything else answers 403
`insufficient_permissions`.
- Install tokens: at most 5 per person per hour, and at most 3 unspent
at once (429 `too_many_requests`).
- A team may own at most 10 machines. This is checked at mint and at
redeem under a row lock (429 `quota_exceeded`). A refused redeem leaves
the token unspent. Organisation hosts are not counted.
**install.sh**
- The token is optional. With none, `NESTRI_INSTALL_TOKEN` is left unset
and onboarding signs in.
- **Do not release this file until `DEFAULT_VERSION` points at a host
agent that can sign in.** Older agents need a token.
Tests: packages/core 256, apps/api 193, packages/auth 74, apps/auth 28,
all passing.
```
bash <(curl -fsSL https://api.nestri.io/install.sh) -u # update this host
bash <(curl -fsSL https://api.nestri.io/install.sh) <install-token> # make this machine a host
bash <(curl -fsSL https://api.nestri.io/install.sh) -h # help
```
Piping still works: `curl -fsSL … | sh -s -- -u`.
**`-u` / `--update`.** No token and no questions. On a registered
machine it downloads and checksum-verifies the current host agent, swaps
it in and restarts the service. The agent then pulls whatever its
release changed (a raised image layout re-pulls the guest, jail and
Proton). It refuses:
- on a machine that was never installed, before downloading anything;
- while a box is running, since restarting the agent would end that
session;
- on an unknown option.
On a machine that already has the target version, it says so and exits
0.
**Root, asked once.** The host agent runs as root, so when the script is
started as anyone else it asks sudo once, before it changes anything.
Neither a pipe nor `bash <(…)` leaves a file sudo could run again, so it
fetches the same script to a private temp file, runs that as root with
the same mode and token, and removes the copy as soon as root has it
open. Only the settings somebody actually set (`NESTRI_HOST_VERSION`,
`NESTRI_BIN_DIR`, `NESTRI_BOX_STORE`) are carried across.
**Tested.**
- **On an NVIDIA host:** `--update` took it 0.2.17 → 0.2.18, verified
and restarted. A second run said "already 0.2.18. Nothing to do".
- **On the fleet host, run as an ordinary user** with `bash <(curl …)
-u`:
- it asked for sudo, the root copy ran and said "Nothing to do", and no
temp copy was left behind;
- `-h` prints usage, and an unknown flag is refused.
- **In a clean container:** `-u` refuses with "not a Nestri host yet",
and a run with no token asks for one.
Until now every finished run said only `ended` or `failed`. A user stop,
a trial window running out, an idle timeout, a host restart and a game
quitting were indistinguishable afterwards.
**Schema.** Migration `0026_session_end_reason` adds two nullable
columns to `session`:
- `end_reason`: one of `user`, `window`, `hours`, `idle`, `left_queue`,
`game_exited`, `launch_failed`, `host_stopped`, `host_restart`.
- `time_claimed`: when a host took the run, set on `requested →
starting`.
**API.**
- `POST /session/:id/state` takes an optional `endReason`. With `ended`
or `failed` it is stored. With `starting` or `live` it is a 400 and the
run does not move.
- Leaving it out of a terminal report stores null, so host agents that
predate this keep working.
- `endReason` and `timeClaimed` are returned on every session response.
- The API writes the reasons it decides itself: `user` when a run nobody
has claimed is stopped, and `left_queue` when a waiting run is abandoned
(the `Left the queue` message is kept).
**New transition, `starting → ended`.** A run stopped while its game is
still downloading has failed at nothing, and never reached `live`, which
is where billing starts. Until now the only way out of `starting` was
`failed`.
**Release order: this goes first.** The host agent that sends
`endReason` is a separate change. This route's body schema is strict, so
without this deployed the current API answers that agent with a 400.
**Tests.**
- API: 185 pass, 5 of them new.
- Core: 41 session tests pass, 3 of them new.
- The new tests fail when the implementation is broken on purpose.
- No new type errors.
**Problem.** A billing webhook that changes nothing answers `200`
exactly like one that worked.
- A subscription made before checkouts carried the team id (`db43068`)
never matches a team.
- When one was cancelled, the API logged `a paid subscription matched no
team` with only the event type. Nothing said whose subscription it was,
so nobody could act on it.
- Deliveries that did apply were not logged at all.
**Changes.**
- `Polar.receive` returns the provider's `subscriptionId` and
`customerId`, in both camelCase and snake_case, as the rest of the
payload is read.
- For a `subscription.*` event, the event's own id is the
subscription's.
- For anything else, such as `order.refunded`, it is the
`subscription_id` field.
- The webhook route logs one line per delivery: `billing webhook {type,
team, standing, applied, subscription, customer}`.
- The unmatched-paid warning carries those ids and says what to do: find
the customer in the provider dashboard and set the team's plan by hand.
- A delivery naming a team that no longer exists is logged as an error,
not silently applied to nothing.
No behaviour change: the same deliveries apply to the same teams. This
is logging only.
**Tests.**
- `bun test src/billing/polar.test.ts`: 17 pass. One new test covers the
ids in both spellings, on a subscription event and on a non-subscription
one.
- `tsc` has no errors in the changed files. The existing errors are in
`utils/hook.ts` and `utils/validator.ts`, from Zod 4.
**Problem.** Proton's GStreamer plugins link `libvpx`, `speex`,
`libtheora`, `libvdpau` and `jsoncpp`, and neither Proton nor the guest
image carries them. Boxes log:
```
Failed to load plugin '.../gstreamer-1.0/libgstlibav.so': libvpx.so.12: cannot open shared object file
Failed to load plugin '.../gstreamer-1.0/libgstspeex.so': libspeex.so.1: cannot open shared object file
```
Wine's media stack then has no decoders, so a game whose videos go
through it plays them as nothing.
**Scope.**
- Not new. The stock `11.0-20261005` Proton and the patched `-n1` build
need exactly the same set.
- The fleet logged it on 2026-10-10 and again on 2026-10-15.
**Fix.** These five packages are added to the guest's runtime package
list.
- They were found by listing every `NEEDED` library of each published
Proton against this image; they were the whole gap.
- They track the sonames Proton was built against, as the new comment
says.
**Verification.** In progress: the image is building, and the library
check is rerun against it for both Protons.
Dev still said host agent 0.2.13 and desktop 0.1.0, while prod serves
0.2.16 and 0.1.2. The next dev→prod merge would have sent new installs
back three host releases. This brings dev to prod's numbers. No other
change: the rest of this branch is already on dev.
Games that showed a black screen with sound now render. Found by testing
on an AMD host: none of these were GPU-vendor specific.
**OpenGL games (Dome Keeper, Godot): fixed and confirmed.**
- `c017462`: the guest's Mesa had no EGL/GLX frontends, so a GL game had
nothing to load and never created a device. The image now ships them,
with glvnd's vendor file, so GL reaches zink and the capture layer sees
the frames.
- `5f2ccdf`: `vm.overcommit_memory=1`. The kernel default refused Dome
Keeper's 9 GiB reservation on an 8 GiB box before a byte was used.
**Chromium / NW.js games (Sheepy) stalled after one frame: fixed.**
- `a73c0c4`: nescope answers every commit with frame callbacks and
feedback, not only commits from a surface it maps to a window.
Chromium's GPU process draws for a window another process owns. Wine's
Wayland driver gives that surface no role, so its FIFO present waited
forever in the driver.
- `23ba9b5`: a window is matched by its surface, not its xdg toplevel,
so Xwayland windows are paced too.
**Proton could not create non-ASCII file names: patch added, build
pending.**
- `6770684`: Steam starts games with `LC_ALL=C`, and Wine reads that as
ASCII for every Unix file name. NW.js stops unpacking its app at
`media/• drum loop….webm` and shows an empty window.
- The patch reads an ASCII locale as UTF-8. Proton now takes patches
from `build/patches/proton/<submodule>/`, and a patched image is tagged
`-nN` (`11.0-20261005-n1`).
- Nothing points at that tag yet; hosts move to it in a separate change,
after it is built and tested.
Not covered: Metro 2033 Redux's report looks like the frame-callback
stall, but it was not retested.
Steam starts games with LC_ALL=C, and Wine reads the C locale as ASCII
for every Unix file name it makes, so a name like '• drum loop.webm'
could not be created at all. Sheepy, an NW.js game, unpacks its app from
a zip at start, stopped at that file, and showed NW.js's empty window.
Proton is now patched before it builds (patches/proton/<submodule>/), and
the first patch reads an ASCII locale as UTF-8. The image name gains -nN
for our patch level, so a patched build never takes the plain tag's name.
A Vulkan client presenting FIFO waits in the driver for its frame
callback. Chromium's GPU process draws for a window another process
owns, which Wine's Wayland driver gives a surface with no role, so no
window here matched it and the game presented one frame and waited
forever: Sheepy's black screen. Nothing here paces against a display,
so every commit now gets its frame callbacks and feedback.
Since frame callbacks moved onto the commit path, the commit handler
found a window by its xdg toplevel. An Xwayland window has none, so no
Proton game got a frame callback or presentation feedback. Xwayland then
completes a flip only on its one-second fallback timer, and a game
presenting FIFO stalls in the driver after its first frame.
The kernel's default overcommit policy refuses any single allocation
larger than RAM plus swap. A box has no swap, so the ceiling is the
box's own size, and an engine or driver that reserves address space it
never touches is refused before it uses any of it. Dome Keeper asked
for 9 GiB on an 8 GiB box and never started.
Set vm.overcommit_memory=1 alongside vm.max_map_count, so a request is
granted and the OOM killer acts on what is actually used. In a box that
is the game: init has already exempted itself.
Mesa was built with zink as its only Gallium driver and every frontend disabled, so a game rendering with OpenGL found no GL and showed a black screen. EGL, GLX, GBM and GLES2 are enabled, with the x11 platform for Xwayland clients.
The provider reuses a customer it already has under the same email and
leaves our team id off it, so the subscription arrived with no team,
was acknowledged, and changed nothing: a paid plan never applied. The
checkout now carries the team in its metadata, which the subscription
keeps, and the webhook reads it there when the customer has none. A
paid delivery that still matches no team is logged instead of dropped
in silence.
A sign-in on a person's own host recorded the enrolment but never
linked the Steam account, so their first launch asked them to link it
again with nothing in the app to do it from. The QR sign-in is the same
proof on their host as on ours.
/download/windows and /download/linux redirect to a one-minute signed URL
for the current desktop app, from the same private bucket as the host
agent, so a download page can link to them once and every download still
passes through the API. Which version they hand out is a constant bumped
by a commit after a release is published, as the host installer's is.
/desktop.sh installs it on Linux for the calling user: one binary in
~/.local/bin and a launcher entry, no root.
A host holds only so many runs at once, and until now every waiting run
was handed to it regardless; the ones it could not hold failed there.
The host now says on its heartbeat how many runs it holds, and is handed
only as many waiting runs as it has free. The rest wait in line: paid
runs before free-weekend runs, then first come first served. Reading a
waiting run returns its place in line, and says when a paid run that
asked later has gone ahead, so a free player is never reordered quietly.
A waiting run nobody has asked after for a minute was abandoned, and
leaves the line rather than holding up everyone behind it. A host that
does not report its slots is handed every run, as before.
The desktop app gives a new cloud user a fleet box and then opens that
machine's Steam sign-in on its hostname; GET /box now says which hostname,
and which box is the fleet one.
The codec the clients agree on goes to the encoder over a socket that does
not exist until the game draws, and a client can connect first. The command
was dropped, the encoder started on AV1, and a client without AV1 sat on a
splash with sound and no picture. Commands now wait for the socket, and a
frame in a codec no client decodes sends the agreed set again, which also
covers a launcher handing over to a game with an encoder of its own.
xs is a box with no stream, sm 720p, md 1080p, lg 1440p and xl 4K. Each
resolution keeps its price, so md is now 1x. New boxes default to md, and
existing boxes move up one size so none of them drops a resolution.
POST /session takes an optional tier. The host starts the run at that
size, the allowance is checked against it, and the run is billed at it.
A run asked for without one runs at its box's size, as before.
The size belongs to the run rather than the box: a box is a game's
install on a machine, and the size is chosen each time it is played, so
playing bigger never means a second box and a second install.
Asking for a run on a busy box is refused, but the refusal does not say
what holds the box. A client that just lost its connection needs to know
whether the run in the way is the game it asked for, which it can rejoin,
or another one it should offer to stop. GET /session answers that.
GET /library picked each game's most recent download row from any host,
so one person's library could show another person's machine id and
progress. It now reads only hosts the caller's boxes are on.
Boxes and library downloads gain `gpu`, the host's first reported card
model, so a client names a place by hardware a person recognises rather
than by a machine id. Null until a host reports one.
GET /machine/:id/downloads lists every game the machine is downloading,
has ready or failed to download, with progress and each game's name.
The library already carries a download per game, but it is the most
recent one across every host, which is not what a person watching their
own machine set up wants to see.
Owner-scoped in the query: a machine that is not yours answers 404, the
same as one that does not exist.
A host cannot tell a run somebody is playing from one nobody has looked
at in an hour, so it cannot end the second without guessing.
The hub now answers the address socket with a second line,
`viewers <n>`, counting clients that are streaming. Init reads it
alongside the address and sends a new `viewers` message to the host
when the count changes. A reader that takes only the first line gets
the address exactly as before, a server that never writes the second
line is still read, and a host that does not know the message skips it,
so either end can ship first.
POST /session/:id/stop. A run no host has claimed ends at once; one that
is starting or live is marked with time_stop_requested and its host is
handed a session.stop with reason `user` on every poll until it reports
the run ended. The host already ends a run the same way whatever the
reason, so no host change is needed.
Asking twice keeps the first time asked, and asking of a finished run
changes nothing. Somebody else's run answers 404, and a host cannot
stop a run as its owner.
The host report gains gpus: each card's model, VRAM in MiB when the
driver says, and the PCI id it was named by. Kept on the machine (a jsonb
column, written only when it changes) and returned by /machine. What a
person calls the machine, and the first measured part of the capacity a
scheduler will place by.
A sync is the whole of what the account owns, but it only ever upserted,
so a revoked licence or a library from a previous Steam account stayed
listed forever. A sync that fully lands now soft-deletes the rest; an
empty list prunes nothing, and a game owned again comes back.
The report schema pinned each box's tier to our size enum, but hosts name
a box's shape by resolution (1080p). Every report naming a box was a 400,
so stop reasons never reached the run. Placement already stores the size;
the reported label is free text and is not stored.
- **Installer:** piped into `sudo`, installs to `/usr/local/bin`, and
hands onboarding `/var/lib/nestri`. Root lets the host agent jail each
box under its own uid and give guests a network. A host that ran the
agent as the sudo user has that state copied over (never moved), so it
keeps its machine identity and its downloads. Pins host agent 0.2.4.
- **Base image:** `vhost-user-nvgpu` and `nvgpu-userspace`, static
(musl), from the same `NVGPU_REF` as the guest driver and Vulkan layer,
under `/boot/host`. The host agent takes them out at unpack, as it does
the kernel, and forwards the GPU when it finds the NVIDIA driver loaded.
Merge after the host agent 0.2.4 release: the installer pins it.
A `kernel-build` stage runs `kernel-build.sh` and puts the result at
`/boot/vmlinux`; host onboarding takes it out beside the root image, so
kernel and image (and their shared NVGPU_REF) always ship as one pair.
`make push` publishes `:latest` and a commit tag from the build box.
Built on the build box: `/boot/vmlinux` is 24 MB, and
`vmlinux.nvgpu-rev` is beside it.
A QR sign-in on Nestri's own hardware proves the Steam account, so `POST
/machine/enrolment` from a machine in `FLEET_ORGANISATION_ID` also links
it to the person. Hosts somebody brought never link (a machine vouching
for itself could attach any Steam account to anyone enrolled on it). A
Steam account already linked elsewhere stays there; the report is
recorded either way.
Tests: three new in `enrolment.test.ts` (fleet links and is idempotent,
own host does not link, taken account is left alone). 164 pass.
POST /steam/link took a Steam id from the request body and linked it. A
Steam id is public, so anyone could claim any account, its library, and
the one free weekend that is keyed on it.
Linking is now POST /steam/link/start, which returns a Steam sign-in URL
tied to a one-time, ten-minute request, and GET /steam/link/callback,
where the browser comes back. The callback accepts the id only if Steam
confirms the assertion, the assertion was issued for exactly this
callback, and the id is read from Steam's claimed identity URL in the
one shape Steam issues it. The request is spent on use.
An install token can now name an organisation instead of a team, and the
host it registers belongs to that organisation's fleet. Only members of
the organisation may mint one, since fleet hardware is rented to other
people. A token for both is refused, by the route and by a check
constraint, as it already is for a machine.
A team with no plan may run on the fleet from Friday 00:00 to Monday
00:00, Helsinki time, for two hours of live play per weekend. One trial
per team, per email address and per Steam account; the claim outlives
the account, so closing it and signing up again is not a second trial.
Once five teams pay, the trial closes for everyone.
A trial run is marked on the session. Its time is counted from the run's
own live interval and kept out of the plan's allowance. Unlike an
allowance, the trial ends a run: the host's work poll now carries a
session.stop job for any trial run past its hours or its window, which
an agent that does not know the kind already skips.
A fleet machine belongs to an organisation, which is not something that
pays, so fleet runs were neither checked nor metered. The payer is now
the box owner's personal team, and the meter that starts and stops with
a run asks the same function the start check does, so the team checked
and the team billed cannot differ.
Fleet machines refused everyone, because nothing could yet say who had
paid for them. A run on one can only exist once it passed the start
check, so a starting or live run of your own box on that machine is now
what grants access, and it ends with the run. Holding a box is not
enough: a box costs nothing to hold.
POST /box places a box once, at creation. On your own host it keeps the
existing rule (exactly one host, or refuse). On the fleet it picks an
online machine owned by the fleet organisation, the one holding the
fewest boxes. Each person holds at most one fleet box. Which organisation
is the fleet is deploy configuration, so no deployment rents hardware it
was not told to.
No machine online is a 429, since the same request succeeds once one is
back.
On an NVIDIA host, NVIDIA's Vulkan driver in the guest lists ray
tracing, optical flow, binary import and CUDA kernel launch, then fails
`vkCreateDevice` with `VK_ERROR_INITIALIZATION_FAILED` for each of them.
An engine that enables what it is offered does not start. virtio-nvgpu's
guest layer (`vklayer/`, nestrilabs/virtio-nvgpu#5) hides those
extensions. It acts only on an NVIDIA device in a guest that has the
virtio-nvgpu driver, so on AMD and Intel boxes it loads and does
nothing.
- A new `nvgpu-layer-build` stage builds the layer from virtio-nvgpu at
`NVGPU_REF`, the same setting the kernel build uses for the guest
driver. The commit is recorded in `/usr/share/nestri/virtio-nvgpu.rev`.
- The layer and its manifest go in `/usr/lib` and
`/usr/share/vulkan/implicit_layer.d`. The missing-library check covers
the layer.
**Merge after nestrilabs/virtio-nvgpu#5.** The default `NVGPU_REF=dev`
has no `vklayer/` until that PR lands, so `make build` fails at this
stage before then.
Built on nestripc-1 with `NVGPU_REF=feat/v0.2-compute` (virtio-nvgpu
112eac4) and pushed:
- `ghcr.io/nestrilabs/nestri/base:latest` sha256:152a4485…
- `ghcr.io/nestrilabs/nesbox/jail:latest` sha256:df813051…, rebuilt on
that base from nesbox dev 6d37597, which includes the vhost-user
forwarding
Measured on box2 (RTX 3060, 595.104.02) with virtio-nvgpu's vkrt probe:
with the layer on, a plain device is created and none of the seven
extensions are offered. With it off, all seven are offered and fail with
-3.
On an NVIDIA host the guest's Vulkan driver offers ray tracing, optical
flow and a few others it cannot create a device with, and an engine that
enables them does not start. virtio-nvgpu's layer hides them; it acts
only on a forwarded NVIDIA device, so AMD and Intel boxes are unaffected.
Built from virtio-nvgpu at NVGPU_REF, the ref the kernel build already
takes the guest driver from, and recorded in the image.
Polishes the sign-in screens and adds a way to see all of them locally.
**Screens**
- Code fields are drawn as one box per character (device code shows a
dash between halves). One real input sits under the boxes, so typing,
paste and autofill work normally, and with script off it falls back to
the plain field.
- A wrong answer or a refused character shakes the field and rings it
red.
- Focus is white instead of the brand colour; the button stays dim until
the form can be sent.
- One alert at a time: an error replaces "Code sent to …" instead of
joining it.
- The device confirm screen says "A device" instead of the client's
self-chosen name, with the Deny warning moved under the buttons.
- A device code typed with its dash (`BCDF-3467`) is no longer cut to
`BCDF-346`.
**Preview**
`bun run preview` in `apps/auth` serves all 18 screen states on :1338,
at desktop and phone sizes, reloading on save. No database or mail
needed.
To make that possible the issuer's screens moved out of the route
handlers into `packages/auth/src/ui/issuer.ts`, unchanged.
Tests: `bun test` in `packages/auth` (67 pass).
A code field silently dropped a character it does not accept, so typing
a letter into the emailed code, which is digits only, looked exactly
like a field that does not work. It now shakes the boxes and rings them
red, the same answer a wrong code gets. Dashes and spaces are still
taken quietly, since they are how a code is written.
The shake an error page plays on load now rings the field red too,
fading back to its usual border once it settles, on every field
including the email address. Reduced motion keeps the red and drops
the movement, since the red is the message.
The boxes were one input each, with script moving focus between them.
That worked in a clean browser and failed in an ordinary one: on the
emailed code, whose first box is marked as a one-time-code field, typing
and pasting did nothing, while the device code, which is not marked,
worked. A password manager or autofill prompt hooking that field and a
script moving focus between six fields were working against each other.
It also left gaps: pasting into the third box with the second one empty
produced a code with a character missing.
Now the real input is stretched invisibly over the boxes and is the only
thing typed into, pasted into or autofilled; the boxes only display what
it holds, with the next one ringed and showing a caret. With nothing
moving focus there is nothing to fight, and a code cannot have a gap.
The browser's own validation message works again too, since the field
it points at is where the boxes are.
The emailed code and the device code are now typed into a row of boxes,
with a dash drawn between the device code's two halves. Typing moves to
the next box, backspace on an empty box goes back, the arrow keys move,
and a pasted or autofilled code, dashes and all, is spread across them.
It is an enhancement of the field and not a replacement for it. The
server still sends the single plain input, which works on its own with
script off; a short inline script draws the boxes over it and keeps it
in step. The real input is still the one that submits and the one whose
validity dims the button, so nothing else had to learn about boxes.
Pressing the button early moves to the first empty box, since the
browser's own message would point at a field nobody can see.
A device code is shown as BCDF-3467 and will be typed or pasted that
way. The field allowed exactly 8 characters, so a correct code was cut
to BCDF-346 before the server, which ignores separators, ever saw it.
A grouped code's field now has room for its separators and a pattern
that allows a dash or a space between groups, and the device code is
declared as two groups of four.
Every error arrives as a new page, so an animation that runs on load
runs exactly when one does, with no script to trigger it. It plays once,
only on a form showing an error alert, and not at all for anyone who
has asked for reduced motion.
The confirm screen named the client asking to sign in. That name is
chosen by whoever started the request, so it let a stranger's request
introduce itself as anything at all; and to the person reading, it is
just a device. It now says "A device is asking to sign in", and a test
checks that a client's self-chosen name never appears.
The warning about choosing Deny moves under the two buttons, as small
print in the same style as the terms line on the email screen, so the
question and its two answers read first.
bun --watch watches each imported file, and many editors save by
writing a new file and renaming it over the old one. The watch does not
follow the rename, so the first such save reloaded the preview and every
later one was silently ignored.
The preview now runs under a small supervisor that watches the source
directories and restarts the server, which sees a save however it was
made. It also stops the server when it is stopped itself, so the port
is not left held.
A wrong code showed "Invalid code" above "Code sent to ...", two answers
to one question in two colours. An error now replaces the confirmation
instead of joining it.
A focused field was ringed in the brand orange, which reads as an error
and competes with the wordmark. Focus is now grey while the value is
incomplete and white once it would pass, so the field says it is done.
The submit button is dimmed while any field in its form is invalid. It
uses the browser's own validity, so there is still no script, and the
code field gains a pattern so a partly typed code counts as invalid.
The button stays clickable, so pressing it early still gets the
browser's message saying what is missing.
Alerts were a bordered, tinted box with an icon, above the fields. They
are now a line of small text under the fields: red when something went
wrong, green when something just happened, no box and no icon.
The sign-in screens no longer carry the "One place for all the ways you
play" heading. Screens that pass their own heading (device approve,
deny and the rest) keep it.
bun run preview (in apps/auth) serves a gallery of all 18 states the
sign-in flow can show, at desktop and phone sizes, each one also
viewable full size. It needs no database or mail: each state is the
real screen value, drawn by the real renderer with the real theme.
bun --watch restarts the server on any change to the renderer, the
stylesheet or the copy, and every page polls for the restart and
reloads itself, so styling is edit, save, look.
The issuer's screens were written inline in its route handlers, so the
only way to look at one was to reach the condition that produces it: an
expired cookie, a refused grant, ten wrong guesses. They now live in
ui/issuer.ts and the handlers call them, with no change to copy, status
or markup.
The app's theme moves to its own module for the same reason: it can be
used without importing the issuer and its database stores.
POST /team makes another team with the caller as owner. The website now
serves a team at nestri.io/<slug>, so Team.Slug refuses its top-level
pages (home, new, pricing, ...) and a personal team whose generated slug
would be one gets a suffix instead.
GET /team, PATCH /team/:id (owner or admin; a taken slug is a 409 on
slug), GET /team/:id/members, GET /box. A team you are not on is a 404
everywhere, so ids cannot be probed. Team.Name and Team.Slug are exported
so a client checks the same rule the API does.
Lets a person put a machine on their team by pasting one command from
the dashboard, instead of copying a user session onto the machine.
## What changes
- **Install tokens** (`packages/core/src/machine/install-token.*`,
migration `0017`). Issued for one team, spent by their first use, expire
after an hour, stored only as a SHA-256 digest. Redeeming is one
conditional `UPDATE` inside the registration transaction: concurrent
redemptions of one token register exactly one machine, and a failed
registration leaves the token unspent.
- **`POST /machine/install-token`** (session): mints a token for a team
the caller belongs to, with the same team resolution and membership rule
as `/register`.
- **`POST /machine/install`** (public): spends a token and returns the
same id, slug and one-time secret as `/register`. Unknown, expired and
used tokens are refused identically.
- **`GET /install.sh`**: a POSIX script embedded at build time. It
checks the platform, asks where box images should live, downloads the
host agent at a pinned version, verifies `SHA256SUMS`, installs to
`~/.local/bin` for the calling user (never root), and hands over with
the token in the environment rather than argv.
- **`GET /install/:component/:version/:asset`**: redirects to a
one-minute signed URL on a private S3-compatible bucket, so every
download passes through a route that can be logged or turned off. The
SigV4 signer uses Web Crypto and is tested against AWS's published
example.
## Configuration
New optional settings: `RELEASES_BUCKET`, `RELEASES_ENDPOINT`,
`RELEASES_REGION`, `RELEASES_ACCESS_KEY_ID`,
`RELEASES_SECRET_ACCESS_KEY`. Without them, downloads return `503` and
everything else is unaffected. Scope the key to reads on that one
bucket.
## Tested
- Core: single use, a five-way race registering exactly one machine,
expired and unknown tokens refused alike.
- Signer: matches the AWS example signature.
- End to end on a real Linux host against a local stack: install script,
signed download from an S3 store that verifies SigV4 (a tampered
signature is refused), registration, and the agent coming online.
GET /install.sh serves a POSIX script, embedded in the API at build time
so the script and the routes that redeem its token ship together. It
checks the platform, asks where box images should live, downloads the
host agent at a pinned version, verifies it against SHA256SUMS, installs
it for the calling user and hands over with the token in the environment
rather than argv.
GET /install/:component/:version/:asset redirects to a one-minute signed
URL on a private S3-compatible bucket, so every download passes through a
route that can be logged or switched off. The SigV4 signer is written
against Web Crypto and checked against AWS's published example.
POST /machine/install-token mints a token for a team the caller belongs
to, with the same team resolution and membership rule as /register.
POST /machine/install needs no session: it spends the token and returns
the same id, slug and one-time secret as registering directly, refusing
unknown, expired and used tokens identically.
Installing on a host should be a command a person pastes, not a user
session copied onto a machine. A token is issued for a team, spent by its
first use, expires after an hour and is stored only as a digest, because
it travels in that command and so lands in shell history.
Redeeming is one conditional update inside the registration transaction:
concurrent redemptions of one token register exactly one machine, and a
registration that fails leaves the token unspent.
"No suitable memory type for DMA-BUF import" names neither mask, so it cannot
tell a driver that refused the descriptor outright (fd_type_bits=0) from one
whose memory types simply do not overlap the image's — which are different
bugs in different components. The three numbers were already computed for a
debug! two lines above, and a debug line only fires at a level nobody runs in
the field; this one did not print at all while a neighbouring crate's did.
Now the failure carries image_type_bits, fd_type_bits, their intersection, and
the size, format, modifier and stride it was importing. On a virtualised NVIDIA
node this turned three unknowns into one measurement and pointed straight at
the kernel side.
The comment explaining the `cmd /C start` bug spelled out the cut submit
link in full. This repository is public and is pull-mirrored onto forges
that crawlers index far more eagerly than GitHub does, so GPTBot found it
there and followed it -- twice. The worker classed both as truncated
submissions, correctly, and raised an alert apiece for a bug fixed in
0.2.1.
Describe the cut URL instead of writing it, and say why, so the next
person documenting a bad link does not paste a working one.
Deliveries arrived intact, verified correctly, and applied to nobody.
Checking the signature here rather than through the SDK means the body is
exactly what was sent, and what is sent is snake_case. The SDK's parser renames
fields to camelCase on the way through, so the reader written against it looked
for `externalId` in a payload that says `external_id` — found nothing, decided
the delivery was about a customer we did not create, and acknowledged it.
That is the worst shape a bug of this kind can take. Every visible signal was
healthy: a 200 back to the provider, no retries, no errors, and a plan that
silently never changed.
Both spellings are now read, so neither path can regress the other.
Registering a team assumed creating a subscription would create the customer it
names. It does not — the customer has to exist first, and creating one fails if
the address is already taken, which it often is: a checkout taken before the
team existed leaves one behind, and so does making one by hand.
So the lookup is now in three steps. A customer already carrying this team id is
used. Otherwise one is found by address and adopted. Otherwise one is made.
**A customer carrying a different team's id is left alone.** Taking it would
move where that subscription is billed, and the team that lost it would go quiet
rather than fail — which is the kind of thing found a month later in a revenue
figure that does not add up.
A team whose owner has no address yet gets no customer, reported rather than
guessed around: an invented address makes a customer nobody can be reached at,
and the next call fixes it once there is a real one.
Checked against the provider rather than only in tests: all four paths — new,
repeated, taken by another team, and no address — behave as written.
Every delivery was refused as a signature mismatch, and nothing in the error
said why.
The provider signs one of two ways. A `whsec_` prefix means Standard Webhooks,
where the secret is a base64 key the verifier decodes; anything else is the
older scheme, where the secret is used as its own raw bytes. Which one a secret
belongs to is decided by when it was created, and every secret created now is
the new one.
The SDK's helper only implements the older scheme — it base64-encodes whatever
it is handed, so a Standard Webhooks secret becomes the literal bytes of the
string including its prefix, and every signature then fails against a key
derived quite differently.
The scheme is now read off the secret rather than configured, so rotating one
cannot put the two out of step. The verifier for the new scheme is the same
library the SDK uses underneath; nothing here hand-rolls crypto.
Tested with a real signature rather than only a rejection. A mismatch is easy to
assert by accident, and a test that only proved bad input is refused would have
passed against the broken version too.
Also teaches the product script to check shape and not just name: a one-time
product where a subscription is wanted cannot be subscribed to at all, and
reporting it as already-there hands back an id that fails at its first use.
Three comments named a file in the internal repo: a contract, a design
guide, and a user-research document with section numbers. A filename plus
a section tells a reader what to ask for, which is the escape the marker
convention exists to prevent.
Each comment now states the requirement on its own terms. The render-node
requirement carries a decision marker; the other two have no decision
behind them, so they cite nothing.
Nine commits, in the order they are easiest to read.
## The shared operator secret is gone
`x-nestri-admin-token` had no caller left — the device pairing it
existed for is
on hold, and nothing in any tree sent it. What remained was a key that
bypassed
authentication entirely and was required to boot.
Every route behind it had a better answer. The two Steam sync routes
took a
`userId` **in the body**, so one secret could write into anybody's
library; they
now authenticate as the host holding that person's Steam sign-in, and
the claim
is checked against the enrolment record. Download-state narrows to hosts
alone.
Creating a game by hand is deleted (syncing already upserts the
catalogue), as is
reading the waitlist — every address on it belongs to someone who has
not agreed
to anything, and answering it over HTTP made that list something a
leaked key
could drain.
Nothing in the API now accepts a credential standing for more than one
caller.
## An organisation owns hardware
Two kinds of machine were modelled as one. A host somebody brings is
theirs,
reached through a team, and should die with their account. A host bought
to serve
other people's workloads is none of those things, and had to be
registered under
an employee's personal team — where their account going away took it
with them.
Ownership is now an either/or, enforced by a check constraint rather
than a
convention: both null is a host nothing can bill, and both set is two
answers to
"whose is this?". Membership of an organisation is derived from a
verified email
domain rather than stored, so signing in with a personal address still
gets an
ordinary personal account.
Not a billing subject. A team pays for what it uses either way.
## Burn, and the three windows
The unit is one second of a reference session, so an allowance is
measured in
time and a bar prints the stored number rather than converting into it.
Counters are stored beside the time they began, and a total whose stamp
has
rolled outside its window reads as zero — so a window clears without
anything
running. No schedule to misfire, no race between a reset and a write.
Two rules on the allowances are enforced rather than remembered: an
allowance
must exceed its own window, or one uninterrupted session hits a wall;
and each
longer one must sit under what the shorter already permits, or it never
binds and
is decoration.
Burn is recorded as segments at one rate, because a run's rate does not
survive
its own lifetime. On our own hardware a bigger tier costs more; on a
caller's own
card it does not, since there is no share of a card of ours being spent.
The gate is at the one moment it may speak — before a run starts, never
again. A
limit refuses the next run and never interrupts one going.
## Payment
Checkout, portal, and a webhook. No price, no currency and no card
detail is
stored: a subscription's existence and its state are the whole of what
crosses
back. Free is a real subscription too, created outright since nothing a
month
needs no payment, so an upgrade changes a subscription rather than
inventing a
customer.
The webhook is the only route no session protects. A signature over the
raw body
stands in for one, checked before the body is parsed, and with no secret
configured it refuses everything.
## Breaking
- `x-nestri-admin-token` is no longer accepted; `ADMIN_SHARED_SECRET` is
no longer read
- `POST /games`, `GET /waitlist` and the `/pairing-code` routes are gone
- `POST /games/sync` and `POST /library/sync` now need host credentials
and take `userId` in the body
- `POST /steam/link` no longer accepts `userId`; `POST
/games/download-state` no longer accepts `hostId`
- `POST /session` responds `{ data, billing }` rather than `{ data }`
## Checks
342 tests pass. Typecheck unchanged from before the branch — the two
pre-existing
errors in `utils/hook.ts` and `utils/validator.ts` are untouched.
Billing is inert until configured: unset `POLAR_*` means every team is
free, no
checkout starts, and the webhook refuses every delivery.
Every team now exists with the payment provider, free ones included. An upgrade
then changes a subscription rather than inventing a customer, and there is one
question to ask about anybody instead of two.
A subscription at nothing a month needs no payment, so it is created outright
rather than by sending somebody through a checkout to pay zero.
It runs after the team rows are committed and cannot affect them. Signing up is
not allowed to depend on a third party being reachable, so this cannot fail the
call and does not retry — a team that misses it is free, which is what it would
have been anyway, and the next call puts it right because the operation is
idempotent.
This broke the webhook mapping, which read the plan off the event type. A free
subscription announces itself with the same `subscription.created` a paid one
does, so every new signup would have landed on the paid allowance. The plan now
comes from the product, and a product we do not sell is left alone rather than
guessed at — somebody selling something else through the same account must not
be able to change what a team may run by doing so.
The external id stays the team. It is the billing subject, and keying on the
user would collapse somebody with two teams into one customer with no way to
say which subscription belonged to which.
Wanjohi