Commit Graph
546 Commits
Author SHA1 Message Date
Wanjohi 9269ee5d51 fix(neshub): an encoder that missed the codec choice is told again
The codec the clients agree on goes to the encoder over a socket that does
not exist until the game draws, and a client can connect first. The command
was dropped, the encoder started on AV1, and a client without AV1 sat on a
splash with sound and no picture. Commands now wait for the socket, and a
frame in a codec no client decodes sends the agreed set again, which also
covers a launcher handing over to a game with an encoder of its own.
2026-10-10 15:34:23 +03:00
Wanjohi ac713e62ae feat(billing): sizes move up one; 1080p is the reference and the default
xs is a box with no stream, sm 720p, md 1080p, lg 1440p and xl 4K. Each
resolution keeps its price, so md is now 1x. New boxes default to md, and
existing boxes move up one size so none of them drops a resolution.
2026-10-10 15:18:23 +03:00
Wanjohi 0f38898fd9 fix(session): commit the migration that adds a run's size 2026-10-10 15:13:54 +03:00
Wanjohi 8608506682 feat(session): a run picks its own size
POST /session takes an optional tier. The host starts the run at that
size, the allowance is checked against it, and the run is billed at it.
A run asked for without one runs at its box's size, as before.

The size belongs to the run rather than the box: a box is a game's
install on a machine, and the size is chosen each time it is played, so
playing bigger never means a second box and a second install.
2026-10-10 14:03:13 +03:00
Wanjohi 05a3fee87a feat(session): list your runs that have not stopped
Asking for a run on a busy box is refused, but the refusal does not say
what holds the box. A client that just lost its connection needs to know
whether the run in the way is the game it asked for, which it can rejoin,
or another one it should offer to stop. GET /session answers that.
2026-10-10 11:19:20 +03:00
Wanjohi 152337296d fix(library): downloads only from your boxes' hosts, named by their GPU
GET /library picked each game's most recent download row from any host,
so one person's library could show another person's machine id and
progress. It now reads only hosts the caller's boxes are on.

Boxes and library downloads gain `gpu`, the host's first reported card
model, so a client names a place by hardware a person recognises rather
than by a machine id. Null until a host reports one.
2026-10-10 10:03:49 +03:00
Wanjohi 77595ecb65 chore(install): host agent 0.2.11 2026-10-09 13:29:58 +03:00
Wanjohi 8054e62bd9 feat(machine): let the owner read what their machine has downloaded
GET /machine/:id/downloads lists every game the machine is downloading,
has ready or failed to download, with progress and each game's name.
The library already carries a download per game, but it is the most
recent one across every host, which is not what a person watching their
own machine set up wants to see.

Owner-scoped in the query: a machine that is not yours answers 404, the
same as one that does not exist.
2026-10-09 12:50:44 +03:00
Wanjohi 7670988ece feat(lifecycle): tell the host how many clients are watching
A host cannot tell a run somebody is playing from one nobody has looked
at in an hour, so it cannot end the second without guessing.

The hub now answers the address socket with a second line,
`viewers <n>`, counting clients that are streaming. Init reads it
alongside the address and sends a new `viewers` message to the host
when the count changes. A reader that takes only the first line gets
the address exactly as before, a server that never writes the second
line is still read, and a host that does not know the message skips it,
so either end can ship first.
2026-10-09 12:33:43 +03:00
Wanjohi f0bfe82d58 feat(session): let the owner end a run
POST /session/:id/stop. A run no host has claimed ends at once; one that
is starting or live is marked with time_stop_requested and its host is
handed a session.stop with reason `user` on every poll until it reports
the run ended. The host already ends a run the same way whatever the
reason, so no host change is needed.

Asking twice keeps the first time asked, and asking of a finished run
changes nothing. Somebody else's run answers 404, and a host cannot
stop a run as its owner.
2026-10-09 12:27:09 +03:00
Wanjohi 2ca3229d57 chore(install): host agent 0.2.10 2026-10-09 11:56:43 +03:00
Wanjohi 3fbcf459f2 machine: a host reports its GPUs, and the machine keeps them
The host report gains gpus: each card's model, VRAM in MiB when the
driver says, and the PCI id it was named by. Kept on the machine (a jsonb
column, written only when it changes) and returned by /machine. What a
person calls the machine, and the first measured part of the capacity a
scheduler will place by.
2026-10-09 11:37:40 +03:00
Wanjohi 3975de5027 install: host agent 0.2.9 2026-10-09 10:33:19 +03:00
DatCaptainHorseandClaude Opus 5.5 4054bb20d9 build: mesa patch keeping the GE rings off the discardable path
An evicted discardable BO has no GTT fallback, so under VRAM pressure a
box could lose its GE rings and fail every graphics submission until
VRAM freed up. Seen live: Control failed 127k submissions after a
neighbour inside its share evicted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:35:17 +03:00
Wanjohi ba9191825c library: a sync drops the games the account no longer owns
A sync is the whole of what the account owns, but it only ever upserted,
so a revoked licence or a library from a previous Steam account stayed
listed forever. A sync that fully lands now soft-deletes the rest; an
empty list prunes nothing, and a game owned again comes back.
2026-10-08 12:19:05 +03:00
Wanjohi 4445f4836e api: accept the host's own tier label in a report
The report schema pinned each box's tier to our size enum, but hosts name
a box's shape by resolution (1080p). Every report naming a box was a 400,
so stop reasons never reached the run. Placement already stores the size;
the reported label is free text and is not stored.
2026-10-08 12:14:46 +03:00
Wanjohi 56e64d4ba8 install: host agent 0.2.8 2026-10-08 11:10:59 +03:00
Wanjohi a939defd5e install: host agent 0.2.7 2026-10-08 10:29:09 +03:00
Wanjohi 2fb59f4dd8 install: host agent 0.2.6 2026-10-08 09:58:43 +03:00
Wanjohi d89d73c263 install: host agent 0.2.5 (#368)
Pins the installer to host agent 0.2.5.
2026-10-08 06:37:58 +00:00
Wanjohi 514212bbbe Host agent runs as root; NVIDIA host programs ship in the base image (#366)
- **Installer:** piped into `sudo`, installs to `/usr/local/bin`, and
hands onboarding `/var/lib/nestri`. Root lets the host agent jail each
box under its own uid and give guests a network. A host that ran the
agent as the sudo user has that state copied over (never moved), so it
keeps its machine identity and its downloads. Pins host agent 0.2.4.
- **Base image:** `vhost-user-nvgpu` and `nvgpu-userspace`, static
(musl), from the same `NVGPU_REF` as the guest driver and Vulkan layer,
under `/boot/host`. The host agent takes them out at unpack, as it does
the kernel, and forwards the GPU when it finds the NVIDIA driver loaded.

Merge after the host agent 0.2.4 release: the installer pins it.
2026-10-07 19:59:29 +00:00
Wanjohi 7481150e97 fix(install): stop a running host agent before upgrading it 2026-10-07 19:09:01 +03:00
Wanjohi b0a42b4c37 fix(install): fetch the glibc host build, which is what releases publish from 0.2.3 2026-10-07 19:05:23 +03:00
Wanjohi 8520ddf6c5 install.sh: install host agent 0.2.3 2026-10-07 18:48:46 +03:00
Wanjohi 4e4dd8896f build: ship the guest kernel inside the image, publish from the build box (#361)
A `kernel-build` stage runs `kernel-build.sh` and puts the result at
`/boot/vmlinux`; host onboarding takes it out beside the root image, so
kernel and image (and their shared NVGPU_REF) always ship as one pair.
`make push` publishes `:latest` and a commit tag from the build box.

Built on the build box: `/boot/vmlinux` is 24 MB, and
`vmlinux.nvgpu-rev` is beside it.
2026-10-07 15:43:31 +00:00
Wanjohi 5bf520c464 Name the run's owner on session.start (#362)
Adds `userId` (the box owner) to `session.start` so a host knows whose
Steam sign-in plays the run. API tests: 161 pass.
2026-10-07 15:43:25 +00:00
Wanjohi 27dc33f9ab Link the Steam account when a fleet machine reports a sign-in (#360)
A QR sign-in on Nestri's own hardware proves the Steam account, so `POST
/machine/enrolment` from a machine in `FLEET_ORGANISATION_ID` also links
it to the person. Hosts somebody brought never link (a machine vouching
for itself could attach any Steam account to anyone enrolled on it). A
Steam account already linked elsewhere stays there; the report is
recorded either way.

Tests: three new in `enrolment.test.ts` (fleet links and is idempotent,
own host does not link, taken account is left alone). 164 pass.
2026-10-07 15:43:20 +00:00
DatCaptainHorseandClaude Opus 5.5 d0b4a6ce76 fix(audio): run the guest graph at 240 frames, and account for missed cycles
The guest's audio stream ran about 3.5% short of realtime: 193 packets a
second where 200 were due, with no network loss at all. The client plays
at its own 48 kHz, so its buffer drained and underran every few seconds,
idle or loaded.

The graph was running 128-frame (2.67 ms) cycles. The 256 floor for VMs
matched on `cpu.vm.name`, which PipeWire reads from DMI and nesbox does
not provide, so it never applied, and neswire's 240-frame request was
rounded down to a power of two. A busy guest missed ~20 of those cycles a
second (pw-top ERR on neswire), and each missed cycle's audio never
reached neswire while the graph clock moved on.

- PipeWire is fixed at 240 frames, power-of-two rounding off: one cycle,
  one 5 ms Opus packet, twice the headroom of before.
- neswire reads the graph position each cycle and fills exactly the
  frames a missed cycle skipped, as silence, so the stream stays at
  48 kHz whatever the scheduler does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 12:55:08 +03:00
DatCaptainHorseandClaude Opus 5.5 471d08d20f fix(neshub): pick a codec every connected client can decode
Each client states what it decodes when it connects, and the hub passed
each statement to the encoder as it arrived. The newest client decided, so
a client that only decodes H.264 lost its picture as soon as one that
decodes AV1 or H.265 joined.

The hub now keeps every client's statement and gives the encoder their
intersection, recomputed when a client joins, when one leaves, and when the
controller returns to automatic. A room settles on the best codec all of
its clients share, and moves up again once the client holding it back has
left. Manual mode still leaves the encoder alone; the change is applied
when the mode goes back to automatic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 12:55:08 +03:00
Wanjohi 89fc7592e6 fix(install): install host agent 0.2.2 2026-10-07 11:03:56 +03:00
Wanjohi 543fd3f361 fix(steam): link a Steam account only through Steam's own sign-in
POST /steam/link took a Steam id from the request body and linked it. A
Steam id is public, so anyone could claim any account, its library, and
the one free weekend that is keyed on it.

Linking is now POST /steam/link/start, which returns a Steam sign-in URL
tied to a one-time, ten-minute request, and GET /steam/link/callback,
where the browser comes back. The callback accepts the id only if Steam
confirms the assertion, the assertion was issued for exactly this
callback, and the id is read from Steam's claimed identity URL in the
one shape Steam issues it. The request is spent on use.
2026-10-07 10:52:19 +03:00
Wanjohi 69ad5d09bb fix(install): install host agent 0.2.1, which ends a run the control plane stops 2026-10-07 10:25:02 +03:00
Wanjohi 6fee7f86e6 Merge branch 'feat/fleet-trial' into dev 2026-10-07 09:43:44 +03:00
Wanjohi d92bc13ed9 feat(machine): an install command that adds a host to the fleet
An install token can now name an organisation instead of a team, and the
host it registers belongs to that organisation's fleet. Only members of
the organisation may mint one, since fleet hardware is rented to other
people. A token for both is refused, by the route and by a check
constraint, as it already is for a machine.
2026-10-07 09:42:51 +03:00
Wanjohi 47f8ff6f61 feat(trial): a free weekend on rented GPUs, for teams without a plan
A team with no plan may run on the fleet from Friday 00:00 to Monday
00:00, Helsinki time, for two hours of live play per weekend. One trial
per team, per email address and per Steam account; the claim outlives
the account, so closing it and signing up again is not a second trial.
Once five teams pay, the trial closes for everyone.

A trial run is marked on the session. Its time is counted from the run's
own live interval and kept out of the plan's allowance. Unlike an
allowance, the trial ends a run: the host's work poll now carries a
session.stop job for any trial run past its hours or its window, which
an agent that does not know the kind already skips.
2026-10-07 09:30:28 +03:00
Wanjohi 2da95deb36 feat(billing): bill a run on rented hardware to the box owner's team
A fleet machine belongs to an organisation, which is not something that
pays, so fleet runs were neither checked nor metered. The payer is now
the box owner's personal team, and the meter that starts and stops with
a run asks the same function the start check does, so the team checked
and the team billed cannot differ.
2026-10-07 09:25:32 +03:00
Wanjohi 08bbb22a1c feat(machine): rented hardware is open during a run you asked for
Fleet machines refused everyone, because nothing could yet say who had
paid for them. A run on one can only exist once it passed the start
check, so a starting or live run of your own box on that machine is now
what grants access, and it ends with the run. Holding a box is not
enough: a box costs nothing to hold.
2026-10-07 09:24:18 +03:00
Wanjohi f82666254e feat(box): create a box, on your own host or on rented GPUs
POST /box places a box once, at creation. On your own host it keeps the
existing rule (exactly one host, or refuse). On the fleet it picks an
online machine owned by the fleet organisation, the one holding the
fewest boxes. Each person holds at most one fleet box. Which organisation
is the fleet is deploy configuration, so no deployment rents hardware it
was not told to.

No machine online is a 429, since the same request succeeds once one is
back.
2026-10-07 09:23:12 +03:00
Wanjohi 9d7239a280 Merge branch 'feat/team-and-box-reads' into dev 2026-10-07 08:49:07 +03:00
DatCaptainHorse 459f787360 feat: proton-cachyos returns 2026-10-05 16:25:33 +03:00
DatCaptainHorseandClaude Opus 5.5 10fdeddc70 feat(neshub): iroh 1.3, and send each frame as one datagram batch
send_many_datagrams queues a frame's fragments under one lock hold and
one driver wake instead of one per fragment. It also rejects the whole
frame on TooLarge, so a path that shrinks mid-frame no longer leaves half
a frame on the wire for the client to time out. Eviction behaviour is
unchanged. A refused PyroWave frame now keeps its sequence number, since
nothing went out under it. pyro_bench mirrors both ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:57:53 +03:00
DatCaptainHorse 030dcc5a8c feat: RT for audio 2026-10-05 14:19:11 +03:00
Wanjohi 8c61cb5cbb feat(build): put virtio-nvgpu's Vulkan layer into the guest rootfs (#356)
On an NVIDIA host, NVIDIA's Vulkan driver in the guest lists ray
tracing, optical flow, binary import and CUDA kernel launch, then fails
`vkCreateDevice` with `VK_ERROR_INITIALIZATION_FAILED` for each of them.
An engine that enables what it is offered does not start. virtio-nvgpu's
guest layer (`vklayer/`, nestrilabs/virtio-nvgpu#5) hides those
extensions. It acts only on an NVIDIA device in a guest that has the
virtio-nvgpu driver, so on AMD and Intel boxes it loads and does
nothing.

- A new `nvgpu-layer-build` stage builds the layer from virtio-nvgpu at
`NVGPU_REF`, the same setting the kernel build uses for the guest
driver. The commit is recorded in `/usr/share/nestri/virtio-nvgpu.rev`.
- The layer and its manifest go in `/usr/lib` and
`/usr/share/vulkan/implicit_layer.d`. The missing-library check covers
the layer.

**Merge after nestrilabs/virtio-nvgpu#5.** The default `NVGPU_REF=dev`
has no `vklayer/` until that PR lands, so `make build` fails at this
stage before then.

Built on nestripc-1 with `NVGPU_REF=feat/v0.2-compute` (virtio-nvgpu
112eac4) and pushed:
- `ghcr.io/nestrilabs/nestri/base:latest` sha256:152a4485…
- `ghcr.io/nestrilabs/nesbox/jail:latest` sha256:df813051…, rebuilt on
that base from nesbox dev 6d37597, which includes the vhost-user
forwarding

Measured on box2 (RTX 3060, 595.104.02) with virtio-nvgpu's vkrt probe:
with the layer on, a plain device is created and none of the seven
extensions are offered. With it off, all seven are offered and fail with
-3.
2026-10-04 02:30:10 +00:00
DatCaptainHorse ef32d5f394 fix: Remove proton leftovers 2026-10-03 20:47:21 +03:00
Wanjohi 01d6f2302d build: put virtio-nvgpu's Vulkan layer into the guest rootfs
On an NVIDIA host the guest's Vulkan driver offers ray tracing, optical
flow and a few others it cannot create a device with, and an engine that
enables them does not start. virtio-nvgpu's layer hides them; it acts
only on a forwarded NVIDIA device, so AMD and Intel boxes are unaffected.

Built from virtio-nvgpu at NVGPU_REF, the ref the kernel build already
takes the guest driver from, and recorded in the image.
2026-10-02 22:43:03 +03:00
DatCaptainHorse 1aa20cffbb smaller rootfs 2026-10-02 17:09:15 +03:00
DatCaptainHorseandClaude Sonnet 5.5 86249001db feat: let a frame reader hand back what it read past a frame
The end that reads the first frame of an attach stream to learn whom the
connection is for then forwards the rest untouched, and what arrived behind that
frame in the same read belongs to the stream.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:09:15 +03:00
DatCaptainHorseandClaude Sonnet 5.5 d0ec87a139 feat: attach a shell to a running box
The host can now ask a box for a shell on a pseudo-terminal, the way a container
runtime attaches to a container. `HostToGuest::Attach` starts the process, and
the guest then dials the host on a vsock port of its own (7001) and carries the
terminal's bytes over a small framed stream: data both ways, a resize, a hangup,
and the exit status last. A screenful of output on the JSON-lines control channel
would hold every other message in the box behind it, which is why it is not
there. A failure to start is the exception, and is said on the control channel
because nothing has connected to the data port yet.

A shell is not a launch: it never ends the box, and the box ending ends it. It
is started through the same reaper as everything else, so it cannot become the
zombie that exists to be prevented. With a terminal on the other end the PTY is
ordinary; without one it is put in raw mode so a command run for its output gets
its bytes back unrewritten. Closing the stream hangs the shell up, process group
and all.

The frame codec is a pure bytes-in, frames-out module in nesprotocol with a
payload cap and no skipping of unknown tags, since a length read from a guest is
read across a trust boundary. No version bump: adding a message does not need
one, and a guest that predates it simply never dials.

The runtime stage gains strace, lsof and less, so a root shell in a box is not
one that cannot trace a process.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:09:15 +03:00
DatCaptainHorseandClaude Sonnet 5.5 e5447a91e4 fix: raise the map-count and open-file limits a Windows game needs
There is no service manager in a box, so nothing applies the distribution's
sysctl configuration, and the guest ran on the kernel's own defaults. Two of
them are too low for a modern engine under Proton:

- vm.max_map_count defaults to 65530. A large Unreal Engine 5 title maps far
  more regions than that, and the failure is a null allocation deep inside the
  engine or Wine rather than a message about mappings. It is set to 2147483642,
  the value SteamOS and the distributions that package Proton use. It is read
  back after the write, because an accepted write that did not take reports
  success while the box is still unconfigured.
- RLIMIT_NOFILE is raised on init, which every process in the box inherits. A
  child can lower a limit but cannot raise its hard one, so the unprivileged
  workload could never get above what it started with. 524288 is the hard limit
  Proton's documentation asks for. It is only ever raised.

Both are best effort and each failure says what it costs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:09:15 +03:00
DatCaptainHorseandClaude Sonnet 5.5 71801261a4 fix: get Rust from rustup in the builder stage
The distribution's rust package is built against the distribution's LLVM, and
when the two repositories this image draws from disagree about llvm-libs, rustc
itself fails to start with a symbol-lookup error, before any of our code is
involved and in every stage that builds Rust. rustup's toolchain links nothing
of the distribution's, so `rustup default stable` replaces the package.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:09:15 +03:00