netris-nestri/build/README.md

# build/ — the guest rootfs

Builds a bootable Arch image for the box's virtio-blk root: Mesa (virtio-gpu
native context) plus the six open guest components —
[`nesinit`](../apps/nesinit), [`nescope`](../apps/nescope),
[`neshub`](../apps/neshub), [`neswire`](../apps/neswire),
[`nescapture`](../apps/nescapture), [`nesgamepad`](../apps/nesgamepad) — laid out
the way [borealis](https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/main/project-borealis)
lays out its `build/`: one big multi-stage `Containerfile`, `--target` picks the
flavor, `etc/` holds the files that get overlaid onto the image verbatim.

```
build/
├── Containerfile       everything, in stages: mesa-build, nestri-build,
│                       os-base, runtime, runtime_prod, runtime_debug
├── etc/                overlaid onto the image's /etc as-is
├── scripts/
│   └── mkimage.sh      docker export → raw ext4, for nesbox's virtio-blk
├── Makefile
└── output/             `make image` writes here (gitignored)
```

```sh
make build          # docker build --target runtime_prod   → ghcr.io/nestrilabs/nestri/base:latest
make build-debug    # docker build --target runtime_debug  → ghcr.io/nestrilabs/nestri/base:debug
make image          # + pack into output/rootfs.ext4
make image-debug    # + pack into output/rootfs-debug.ext4

make proton-image   # build Proton from source — hours
make proton-push    # build it and publish it
```

## Design notes

Three things worth knowing about how this is put together:

1. **No privileged host chroot.** A bare `chroot` into a hand-extracted
   rootfs needs `/proc`, `/sys`, `/dev` bind-mounted in first — they don't
   exist inside a chroot target until something puts them there. `os-base`
   here is `FROM archlinux:base` directly, with `pacman -S` as plain `RUN`
   steps — a Docker build step already runs inside a real container with its
   own `/proc`, `/sys`, `/dev`, so that whole bind-mount mechanism has nothing
   to do.

2. **No host-side ownership bug to guard against.** `COPY --from=` runs as
   root inside the build with no host user in the loop, so there's no
   invoking-user uid getting stamped onto `/`, `/usr/bin`, or anywhere else
   a build step touches — a failure mode some overlay approaches need an
   explicit sanity check for doesn't exist here to check for.

3. **One `cargo build --release --workspace`, not one stage per binary.**
   `nescope`, `neshub`, `neswire`, `nescapture` and `nesgamepad` share one Cargo workspace
   and one `Cargo.lock` — a BuildKit cache mount on `target/` gives cargo's
   own incremental compiler per-crate isolation without needing a separate
   Docker stage (and a separate full rebuild of `nesprotocol`) per binary.

**What is deliberately not here: Valve's `steamclient.so`.**
`nestri/CLAUDE.md` is explicit — *"Nothing closed may enter this repo. Not
source, not a dependency, not a directory that 'looked convenient'."* That one
is closed, so whatever layers it on top is a build outside this repo — not
something this repo names, links to, or depends on.

**Proton is not in the image.** It used to be, at a cost of about 1.4 GB, and
it is attached to a box at run time now: a read-only EROFS image of a Proton
tree, mounted at `/nestri/compat`, chosen per session by whatever starts the
box. That is what makes trying another Proton (a newer build of ours, or someone
else's) a matter of attaching a different image rather than rebuilding this one.

What stays here is everything Proton needs around itself: the interpreter for its
entry point and every library its programs and Wine's core link. The
`compat_check` stage in the `Containerfile` is what keeps that true. It copies a
Proton tree in (the published one by default) and fails the build when a library
is missing or the entry point has no interpreter, then leaves a one-line marker
in the image naming the Proton it was checked against
(`/usr/share/nestri/compat-verified`). Check another one with
`make PROTON_IMAGE=<ref> build`. The check cannot be skipped as an unused
stage, because the leaf stages copy that marker out of it.

The image holds no Proton path, so a box started without one fails saying where
it looked instead of quietly using something stale.

What the published Proton is: **proton-cachyos built with `--enable-wow64`**, as
a published image rather than rebuilt here, because it takes hours and moves only when its
own tag does. wow64 runs 32-bit Windows code inside a 64-bit unix process, so a
box needs no lib32 glibc, no second Mesa for i686, and no second capture layer
for 32-bit titles to be captured. Upstream Wine defaults to it from Proton 11,
so a released build needs no more than ours does.

**umu-launcher is in the image, built here.** `umu-run` is how a title that is
not Steam's is launched, so the game fixes Proton carries apply outside Steam.
The distribution's package is not used: it declares a long list of `lib32-*`
libraries and desktop packages for the Steam Linux Runtime container, which this
image never runs. The `umu-build` stage builds a pinned tag (`UMU_TAG`) instead,
against the distribution's own `urllib3` and `pyzstd` (the copies it can vendor
are built from source with pinned build tooling, which a rolling distribution
outgrows), and the image adds `python-xlib`, `python-urllib3` and `python-pyzstd`.

Two checks keep it honest: `umu-run --help` must actually run, because a missing
Python module otherwise shows up as a game that never starts, and no `lib32-`
package may be installed. umu would also like to fetch a runtime container for a
Proton whose manifest asks for one; the host side shows it the compat layer with
that request removed, so there is nothing for it to fetch.

`runtime_prod` from this Containerfile — tagged
`ghcr.io/nestrilabs/nestri/base:latest` — is a complete, bootable guest image,
and also the shared foundation other builds start from: nesbox's jail image
(see `nesbox/build/`) extracts **Mesa** from it so the guest and host sides of
the virtio-gpu native-context protocol never drift apart. Only Mesa —
`virglrenderer` is the host half of that protocol and nesbox builds its own,
patched, from `nesbox/patches/`; nothing in this image carries it.

## The guest kernel

```sh
make kernel                          # clone if needed, configure, verify, build → output/vmlinux
make KERNEL_SRC=~/src/linux kernel   # build an existing tree instead
make kernel-clean                    # drop the tree and the image
```

CachyOS's fork (`KERNEL_REF` in the `Makefile`), taken for its scheduler
patches, not its config: theirs is a desktop build with thousands of modules,
and this guest has `CONFIG_MODULES` off and no `/lib/modules` at all.

- **`kernel/nestri.fragment` is the source of truth**, and says why each entry
  is there. It is merged onto the tree's `.config`, resolved with
  `olddefconfig`, and then **checked**: any entry that did not survive fails the
  build. `merge_config.sh` and `olddefconfig` both drop options quietly, and
  the worst of these fails as perfectly healthy, perfectly silent audio.
- **`kernel/base.config` is only a seed** for a tree with no `.config`, so a
  fresh clone does not start from `defconfig`'s enormous driver set. Change the
  fragment, not the seed and not a tree's `.config`.
- **`vmlinux`, not `bzImage`.** The guest is loaded as a raw ELF with no
  bootloader in the path. It is ~16 MB unstripped, which costs nothing at run
  time: only the loadable segments are mapped.
- **Built with clang, thin-LTO.** `kernel-build.sh` passes `LLVM=1` to every
  step, config included, because `HAS_LTO_CLANG` is decided when Kconfig runs
  and gcc would make `olddefconfig` drop `LTO_CLANG_THIN` silently. `clang`,
  `ld.lld` and the `llvm-*` binutils must be on `PATH`; the build says which is
  missing before it starts.
- **`-march=x86-64-v3`** goes in through `KCFLAGS`. It is safe in a kernel:
  the kernel's own `-mno-sse -mno-avx …` masks every vector extension off
  whatever the flag order, leaving v3's integer ISA.
- The tree is off the pinned ref (a bisect, a local patch)? The build warns
  and builds what is there rather than checking the ref out over your work.

### NVIDIA: the forwarding driver

```sh
make kernel                          # fetches virtio-nvgpu's dev branch and builds its driver in
make NVGPU_REF=<tag-or-commit> kernel
```

On an NVIDIA host the guest has no GPU of its own. It gets a virtio device
that carries the NVIDIA driver's ioctls to the host, and it runs NVIDIA's own
user-mode libraries against it. The guest half of that is a kernel driver from
[virtio-nvgpu](https://github.com/nestrilabs/virtio-nvgpu), which every build
fetches into `output/virtio-nvgpu` and builds into the kernel as
`CONFIG_VIRTIO_GPU_NV`. It has to be built in, because this kernel cannot load
modules. The driver's parameters therefore go on the command line as
`virtio_gpu_nv.<name>=`.

- **`NVGPU_REF` is a branch by default**, so a build takes the driver as it is
  that day. The commit that went in is written to `output/vmlinux.nvgpu-rev`,
  because a branch name does not tell you which driver a given `vmlinux`
  contains. If the fetch fails, the build uses the checkout it already has and
  warns that it is doing so.
- An AMD or Intel host is unaffected. The driver binds only to the forwarding
  device, and nothing offers that device to those guests.
- **The rootfs carries virtio-nvgpu's Vulkan layer**, built from the same
  `NVGPU_REF`. NVIDIA's driver in a guest offers ray tracing and a few other
  extensions it cannot then create a device with; the layer hides them. It
  acts only on an NVIDIA device in a guest with the forwarding driver, so AMD
  and Intel boxes are unaffected. Build the kernel and the image from the same
  ref, or the layer describes a different driver than the one booted.
- The NVIDIA libraries are **not** part of the image. They must be the same
  build as the host's kernel module, so the host shares its own copy with the
  guest at run time.

### Experimental: the Infinity scheduler

```sh
make KERNEL_INFINITY=1 kernel        # → output/vmlinux-infinity
```

Applies [infinity-sched](https://github.com/galpt/infinity-sched-new)'s
series (GPL-2), which reworks the fair, RT and DRM schedulers for latency
under load. It is pinned by commit (`INFINITY_REV`), and the series directory
comes from `KERNEL_REF`, since upstream publishes one per CachyOS release.

- **Its own tree and its own image.** It builds in `output/kernel-infinity`,
  so the stock kernel is never patched and switching between the two needs no
  revert.
- **All or nothing, zero fuzz.** The whole series is checked against the
  stacked result before any of it is applied. The applied commit is recorded in
  the tree; to move `INFINITY_REV`, start over with `make kernel-clean`.
- **Only the CPU half does anything here.** virtio-gpu does not use the DRM
  scheduler and `CONFIG_DRM_SCHED` is not built, so the GPU patch is compiled
  out. It is applied anyway because upstream says a partial series
  misbehaves.
- Upstream's `/sys/kernel/debug/infinity_*` counters need `CONFIG_DEBUG_FS`,
  which this kernel does not have.

## Two packages that look droppable and are not

`llvm-libs` is 164 MB, the largest single thing in the image after Proton, and
`lm_sensors` is only there because something links `libsensors`. Both look like
leftovers of a Mesa configuration that has since been trimmed, and both have
been checked rather than reasoned about: **`libgbm` links them**, and the
compositor needs GBM. Trimming the Mesa build does not reach them.

`lm_sensors` in particular was found the hard way. It used to arrive as a
dependency of the distribution's Mesa package, and dropping that package took
it away — leaving our own Mesa unable to resolve `libsensors.so.5`. Nothing in
a package list says that; the check below is what said it.

## Proton has its own cadence, and its own Containerfile

`make build` neither builds Proton nor ships it. Building it takes hours and it
changes only when its tag moves, so it is one image published once and attached
to boxes at run time. `Containerfile.proton` is that build, and it lives here so
the published tag stays reproducible from this tree rather than from somebody's
laptop.

```sh
make proton-image                              # the current tag
make PROTON_TAG=cachyos-11.1-20261115-native proton-image
make proton-push                               # build it and publish it
```

**`PROTON_TAG` is the only thing to change.** The published version is derived
from it in the `Makefile` rather than written a second time, because the two
are the same number in two spellings — and an image whose name does not say
which Proton is inside it is worse than no image.

A tag bump also means checking `GECKO_VER`, `MONO_VER` and `XALIA_VER` in
`Containerfile.proton`: they are the runtimes the tag's own `Makefile.in` names,
fetched in the slow layer so that a failure later does not repeat the fetch.

The result is `FROM scratch` over the tree, so the image's root *is* the tree,
under `usr/share/steam/compatibilitytools.d/proton-cachyos`, and there is
nothing in it to run.

Its **context is `build/`**, not the repository root the guest build uses. All
it needs is the two scripts beside it, and `Containerfile.proton.containerignore`
keeps `output/` out of that context — a build context is copied before the
first instruction runs, so without it every Proton build would begin by moving
the last rootfs image, or kernel tree, it produced.

Two things in the recipe are worth knowing before changing it:

- **Fetch and build are separate layers on purpose.** The submodule checkout
  runs well past ten minutes, and a build that fails on a flag or a missing
  tool must not pay for that again. Keep anything that can fail *fast* in
  `proton-build.sh`.
- **`widl` is built by hand from the mingw-w64 release.** Without it autoconf
  quietly sets `HAVE_WIDL` to false, vkd3d's public headers are never
  generated, and the build dies an hour later on a missing header. Arch ships
  `widl` only inside `wine`, which wants multilib — which is the thing
  `--enable-wow64` exists to avoid.

## There is no init system in here, and that is the design

`nesinit` is PID 1. The image carries **no service manager, no init scripts,
no `udev` and no systemd** — `systemd-libs` stays, because `dbus-daemon` and
`wireplumber` link `libsystemd.so.0`, but nothing in the image can be PID 1
except `nesinit`, and the build fails if anything that could be turns up.

That is why this is plain Arch. The image used to be Artix, chosen for OpenRC,
and every cost of that choice — no `eudev`, no `agetty-openrc`, `udev` being
systemd's anyway, a runlevel edit not stopping a service another one still
needs — was paid for an init system that is no longer here.

**What replaced fourteen `rc-update` lines and nine init scripts:**

| was | now |
|---|---|
| `devfs`, `dmesg`, `udev`, `udev-trigger` | `devtmpfs` makes the nodes; init sets the two modes that matter. The compositor takes input through Wayland and opens nothing `udev` provides. Controllers are the one thing a game finds through `udev`, and `nesgamepad` announces the ones it creates itself |
| `guest-net`, `hostname`, `xdg-runtime`, `cgroups` | init, before it dials out |
| `dbus`, `dbus-session`, `pipewire`, `wireplumber`, `neshub`, `neswire`, `nesgamepad` | a table compiled into `nesinit` |
| `nescope` in the `default` runlevel | **not a service.** It wraps the workload and is started by a launch, with that launch's geometry, and dies with it |
| `agetty` on `hvc0` | nothing. See below |
| `/etc/fstab` | init's own mounts, and shares named in the boot descriptor |

**A box is launched into, not booted into something.** Init mounts what the
descriptor names, brings the table up, says it is ready, and then takes
commands — so an image on its own runs nothing at all, which is the point:
this image is payload-independent and there is no payload in it.

### Getting into a guest that will not boot

`init=/bin/bash` on the kernel command line. Nothing in the image offers a
login prompt — there is no getty in either flavour — and that is cheaper than
carrying one: `nesinit` is an ordinary program, so from that shell you can run
it by hand and watch it fail. `make build-debug` adds `vulkaninfo` and friends
and gives root a password for `su`; it does not add a console.

**A shell is not a booted box, and the difference bites immediately.** Nothing
the init does has happened: the root is read-only, `/run` and `/tmp` are still
directories on it rather than tmpfs, `/run/user/1000` is unwritable, and the
hostname is `(none)` rather than `nesbox` — which is the quickest way to tell
the two states apart. A compositor started in that shell fails on its own
socket, and the error names the runtime directory rather than the cause.

So run `nesinit` first. It mounts, prepares the directories, brings the
services up, then fails to reach a control channel that is not there and
exits — **leaving everything it prepared behind**, which is exactly what makes
the hand-run useful. Then start what you came to debug.

It is safe to run outside a box, and that took fixing: the shutdown path signals
every process it may signal and then powers the machine off, which is right for
PID 1 of a box and catastrophic anywhere else. Both steps are refused when it is
not PID 1, and it says so rather than doing it quietly.

If you would rather not run it at all, the two mounts it does that a compositor
needs are:

```sh
mount -t tmpfs -o mode=1777,size=64m tmpfs /tmp
mount -t tmpfs -o mode=755,size=32m  tmpfs /run
mkdir -p /run/user/1000 && chown 1000:1000 /run/user/1000 && chmod 0700 /run/user/1000
```

Making `/run/user/1000` writable in the *image* does not help, and is worth
saying because it is the obvious first thing to try: before the init runs, the
directory is on a read-only root, so its ownership is not what stops a write;
after the init runs, a fresh tmpfs is mounted over `/run` and the image's copy
of the directory is hidden underneath it.

The one thing this does not reach is a failure *before* the shell. If that
happens the evidence is on `console=hvc0` and nowhere else.

### Two build-time checks worth knowing about

Both exist because the failure they catch is invisible at runtime rather than
loud, which is the same reason the old build checked its `conf.d` files:

- **No hook may point at a program that is not in the image.** Removing
  systemd removes the script `dbus-reload.hook` calls, and a leftover hook
  produces `error: command failed to execute correctly` on every future pacman
  transaction — indistinguishable, in a log, from something that matters.
- **Nothing `nesinit` will look for may be missing.** Its service table is
  compiled in, so an absent `dbus-daemon` is not a build error by itself; it
  is a service that does not come up in a box somebody is waiting on.

## Network defaults

`nesinit` reads `nestri.ip=`/`nestri.gw=` off the kernel command line,
falling back to `172.30.0.2/24` via `172.30.0.1` if neither is set — the
host's own default tap addressing. Keep these in step if that changes on the
host side. A box started with no network device at all is a valid box and
boots without one.

The address is a per-boot parameter rather than an image setting because the
alternative makes every box built from this image the same host on the
network, and two of them collide the moment they run together. Same reasoning
for `/etc/machine-id`, which is a symlink into a tmpfs that init fills at
boot — the previous image baked one in, so every box built from it was the
same machine to anything that asked.

This is also the one thing in the image that keeps `iproute2` installed:
init runs `ip` rather than talking netlink, which is a hundred lines of
`unsafe` saved for an interface configured once.