netris-nestri/build

build/ — the guest rootfs

Builds a bootable Arch image for the box's virtio-blk root: Mesa (virtio-gpu native context) plus the six open guest components — nesinit, nescope, neshub, neswire, nescapture, nesgamepad — laid out the way borealis lays out its build/: one big multi-stage Containerfile, --target picks the flavor, etc/ holds the files that get overlaid onto the image verbatim.

build/
├── Containerfile       everything, in stages: mesa-build, nestri-build,
│                       os-base, runtime, runtime_prod, runtime_debug
├── etc/                overlaid onto the image's /etc as-is
├── scripts/
│   └── mkimage.sh      docker export → raw ext4, for nesbox's virtio-blk
├── Makefile
└── output/             `make image` writes here (gitignored)

make build          # docker build --target runtime_prod   → ghcr.io/nestrilabs/nestri/base:latest
make build-debug    # docker build --target runtime_debug  → ghcr.io/nestrilabs/nestri/base:debug
make image          # + pack into output/rootfs.ext4
make image-debug    # + pack into output/rootfs-debug.ext4

make proton-image   # build Proton from source — hours
make proton-push    # build it and publish it

Design notes

Three things worth knowing about how this is put together:

1. No privileged host chroot. A bare chroot into a hand-extracted rootfs needs /proc, /sys, /dev bind-mounted in first — they don't exist inside a chroot target until something puts them there. os-base here is FROM archlinux:base directly, with pacman -S as plain RUN steps — a Docker build step already runs inside a real container with its own /proc, /sys, /dev, so that whole bind-mount mechanism has nothing to do.

2. No host-side ownership bug to guard against. COPY --from= runs as root inside the build with no host user in the loop, so there's no invoking-user uid getting stamped onto /, /usr/bin, or anywhere else a build step touches — a failure mode some overlay approaches need an explicit sanity check for doesn't exist here to check for.

3. One cargo build --release --workspace, not one stage per binary. nescope, neshub, neswire, nescapture and nesgamepad share one Cargo workspace and one Cargo.lock — a BuildKit cache mount on target/ gives cargo's own incremental compiler per-crate isolation without needing a separate Docker stage (and a separate full rebuild of nesprotocol) per binary.

What is deliberately not here: Valve's steamclient.so. nestri/CLAUDE.md is explicit — "Nothing closed may enter this repo. Not source, not a dependency, not a directory that 'looked convenient'." That one is closed, so whatever layers it on top is a build outside this repo — not something this repo names, links to, or depends on.

Proton is not in the image. It used to be, at a cost of about 1.4 GB, and it is attached to a box at run time now: a read-only EROFS image of a Proton tree, mounted at /nestri/compat, chosen per session by whatever starts the box. That is what makes trying another Proton (a newer build of ours, or someone else's) a matter of attaching a different image rather than rebuilding this one.

What stays here is everything Proton needs around itself: the interpreter for its entry point and every library its programs and Wine's core link. The compat_check stage in the Containerfile is what keeps that true. It copies a Proton tree in (the published one by default) and fails the build when a library is missing or the entry point has no interpreter, then leaves a one-line marker in the image naming the Proton it was checked against (/usr/share/nestri/compat-verified). Check another one with make PROTON_IMAGE=<ref> build. The check cannot be skipped as an unused stage, because the leaf stages copy that marker out of it.

The image holds no Proton path, so a box started without one fails saying where it looked instead of quietly using something stale.

What the published Proton is: proton-cachyos built with --enable-wow64, as a published image rather than rebuilt here, because it takes hours and moves only when its own tag does. wow64 runs 32-bit Windows code inside a 64-bit unix process, so a box needs no lib32 glibc, no second Mesa for i686, and no second capture layer for 32-bit titles to be captured. Upstream Wine defaults to it from Proton 11, so a released build needs no more than ours does.

umu-launcher is in the image, built here. umu-run is how a title that is not Steam's is launched, so the game fixes Proton carries apply outside Steam. The distribution's package is not used: it declares a long list of lib32-* libraries and desktop packages for the Steam Linux Runtime container, which this image never runs. The umu-build stage builds a pinned tag (UMU_TAG) instead, against the distribution's own urllib3 and pyzstd (the copies it can vendor are built from source with pinned build tooling, which a rolling distribution outgrows), and the image adds python-xlib, python-urllib3 and python-pyzstd.

Two checks keep it honest: umu-run --help must actually run, because a missing Python module otherwise shows up as a game that never starts, and no lib32- package may be installed. umu would also like to fetch a runtime container for a Proton whose manifest asks for one; the host side shows it the compat layer with that request removed, so there is nothing for it to fetch.

runtime_prod from this Containerfile — tagged ghcr.io/nestrilabs/nestri/base:latest — is a complete, bootable guest image, and also the shared foundation other builds start from: nesbox's jail image (see nesbox/build/) extracts Mesa from it so the guest and host sides of the virtio-gpu native-context protocol never drift apart. Only Mesa — virglrenderer is the host half of that protocol and nesbox builds its own, patched, from nesbox/patches/; nothing in this image carries it.

The guest kernel

make kernel                          # clone if needed, configure, verify, build → output/vmlinux
make KERNEL_SRC=~/src/linux kernel   # build an existing tree instead
make kernel-clean                    # drop the tree and the image

CachyOS's fork (KERNEL_REF in the Makefile), taken for its scheduler patches, not its config: theirs is a desktop build with thousands of modules, and this guest has CONFIG_MODULES off and no /lib/modules at all.

• kernel/nestri.fragment is the source of truth, and says why each entry is there. It is merged onto the tree's .config, resolved with olddefconfig, and then checked: any entry that did not survive fails the build. merge_config.sh and olddefconfig both drop options quietly, and the worst of these fails as perfectly healthy, perfectly silent audio.
• kernel/base.config is only a seed for a tree with no .config, so a fresh clone does not start from defconfig's enormous driver set. Change the fragment, not the seed and not a tree's .config.
• vmlinux, not bzImage. The guest is loaded as a raw ELF with no bootloader in the path. It is ~16 MB unstripped, which costs nothing at run time: only the loadable segments are mapped.
• Built with clang, thin-LTO. kernel-build.sh passes LLVM=1 to every step, config included, because HAS_LTO_CLANG is decided when Kconfig runs and gcc would make olddefconfig drop LTO_CLANG_THIN silently. clang, ld.lld and the llvm-* binutils must be on PATH; the build says which is missing before it starts.
• -march=x86-64-v3 goes in through KCFLAGS. It is safe in a kernel: the kernel's own -mno-sse -mno-avx … masks every vector extension off whatever the flag order, leaving v3's integer ISA.
• The tree is off the pinned ref (a bisect, a local patch)? The build warns and builds what is there rather than checking the ref out over your work.

NVIDIA: the forwarding driver

make kernel                          # fetches virtio-nvgpu's dev branch and builds its driver in
make NVGPU_REF=<tag-or-commit> kernel

On an NVIDIA host the guest has no GPU of its own. It gets a virtio device that carries the NVIDIA driver's ioctls to the host, and it runs NVIDIA's own user-mode libraries against it. The guest half of that is a kernel driver from virtio-nvgpu, which every build fetches into output/virtio-nvgpu and builds into the kernel as CONFIG_VIRTIO_GPU_NV. It has to be built in, because this kernel cannot load modules. The driver's parameters therefore go on the command line as virtio_gpu_nv.<name>=.

• NVGPU_REF is a branch by default, so a build takes the driver as it is that day. The commit that went in is written to output/vmlinux.nvgpu-rev, because a branch name does not tell you which driver a given vmlinux contains. If the fetch fails, the build uses the checkout it already has and warns that it is doing so.
• An AMD or Intel host is unaffected. The driver binds only to the forwarding device, and nothing offers that device to those guests.
• The rootfs carries virtio-nvgpu's Vulkan layer, built from the same NVGPU_REF. NVIDIA's driver in a guest offers ray tracing and a few other extensions it cannot then create a device with; the layer hides them. It acts only on an NVIDIA device in a guest with the forwarding driver, so AMD and Intel boxes are unaffected. Build the kernel and the image from the same ref, or the layer describes a different driver than the one booted.
• The NVIDIA libraries are not part of the image. They must be the same build as the host's kernel module, so the host shares its own copy with the guest at run time.

Experimental: the Infinity scheduler

make KERNEL_INFINITY=1 kernel        # → output/vmlinux-infinity

Applies infinity-sched's series (GPL-2), which reworks the fair, RT and DRM schedulers for latency under load. It is pinned by commit (INFINITY_REV), and the series directory comes from KERNEL_REF, since upstream publishes one per CachyOS release.

• Its own tree and its own image. It builds in output/kernel-infinity, so the stock kernel is never patched and switching between the two needs no revert.
• All or nothing, zero fuzz. The whole series is checked against the stacked result before any of it is applied. The applied commit is recorded in the tree; to move INFINITY_REV, start over with make kernel-clean.
• Only the CPU half does anything here. virtio-gpu does not use the DRM scheduler and CONFIG_DRM_SCHED is not built, so the GPU patch is compiled out. It is applied anyway because upstream says a partial series misbehaves.
• Upstream's /sys/kernel/debug/infinity_* counters need CONFIG_DEBUG_FS, which this kernel does not have.

Two packages that look droppable and are not

llvm-libs is 164 MB, the largest single thing in the image after Proton, and lm_sensors is only there because something links libsensors. Both look like leftovers of a Mesa configuration that has since been trimmed, and both have been checked rather than reasoned about: libgbm links them, and the compositor needs GBM. Trimming the Mesa build does not reach them.

lm_sensors in particular was found the hard way. It used to arrive as a dependency of the distribution's Mesa package, and dropping that package took it away — leaving our own Mesa unable to resolve libsensors.so.5. Nothing in a package list says that; the check below is what said it.

Proton has its own cadence, and its own Containerfile

make build neither builds Proton nor ships it. Building it takes hours and it changes only when its tag moves, so it is one image published once and attached to boxes at run time. Containerfile.proton is that build, and it lives here so the published tag stays reproducible from this tree rather than from somebody's laptop.

make proton-image                              # the current tag
make PROTON_TAG=cachyos-11.1-20261115-native proton-image
make proton-push                               # build it and publish it

PROTON_TAG is the only thing to change. The published version is derived from it in the Makefile rather than written a second time, because the two are the same number in two spellings — and an image whose name does not say which Proton is inside it is worse than no image.

A tag bump also means checking GECKO_VER, MONO_VER and XALIA_VER in Containerfile.proton: they are the runtimes the tag's own Makefile.in names, fetched in the slow layer so that a failure later does not repeat the fetch.

The result is FROM scratch over the tree, so the image's root is the tree, under usr/share/steam/compatibilitytools.d/proton-cachyos, and there is nothing in it to run.

Its context is build/, not the repository root the guest build uses. All it needs is the two scripts beside it, and Containerfile.proton.containerignore keeps output/ out of that context — a build context is copied before the first instruction runs, so without it every Proton build would begin by moving the last rootfs image, or kernel tree, it produced.

Two things in the recipe are worth knowing before changing it:

• Fetch and build are separate layers on purpose. The submodule checkout runs well past ten minutes, and a build that fails on a flag or a missing tool must not pay for that again. Keep anything that can fail fast in proton-build.sh.
• widl is built by hand from the mingw-w64 release. Without it autoconf quietly sets HAVE_WIDL to false, vkd3d's public headers are never generated, and the build dies an hour later on a missing header. Arch ships widl only inside wine, which wants multilib — which is the thing --enable-wow64 exists to avoid.

There is no init system in here, and that is the design

nesinit is PID 1. The image carries no service manager, no init scripts, no udev and no systemd — systemd-libs stays, because dbus-daemon and wireplumber link libsystemd.so.0, but nothing in the image can be PID 1 except nesinit, and the build fails if anything that could be turns up.

That is why this is plain Arch. The image used to be Artix, chosen for OpenRC, and every cost of that choice — no eudev, no agetty-openrc, udev being systemd's anyway, a runlevel edit not stopping a service another one still needs — was paid for an init system that is no longer here.

What replaced fourteen rc-update lines and nine init scripts:

was	now
devfs, dmesg, udev, udev-trigger	devtmpfs makes the nodes; init sets the two modes that matter. The compositor takes input through Wayland and opens nothing udev provides. Controllers are the one thing a game finds through udev, and nesgamepad announces the ones it creates itself
guest-net, hostname, xdg-runtime, cgroups	init, before it dials out
dbus, dbus-session, pipewire, wireplumber, neshub, neswire, nesgamepad	a table compiled into nesinit
nescope in the default runlevel	not a service. It wraps the workload and is started by a launch, with that launch's geometry, and dies with it
agetty on hvc0	nothing. See below
/etc/fstab	init's own mounts, and shares named in the boot descriptor

A box is launched into, not booted into something. Init mounts what the descriptor names, brings the table up, says it is ready, and then takes commands — so an image on its own runs nothing at all, which is the point: this image is payload-independent and there is no payload in it.

Getting into a guest that will not boot

init=/bin/bash on the kernel command line. Nothing in the image offers a login prompt — there is no getty in either flavour — and that is cheaper than carrying one: nesinit is an ordinary program, so from that shell you can run it by hand and watch it fail. make build-debug adds vulkaninfo and friends and gives root a password for su; it does not add a console.

A shell is not a booted box, and the difference bites immediately. Nothing the init does has happened: the root is read-only, /run and /tmp are still directories on it rather than tmpfs, /run/user/1000 is unwritable, and the hostname is (none) rather than nesbox — which is the quickest way to tell the two states apart. A compositor started in that shell fails on its own socket, and the error names the runtime directory rather than the cause.

So run nesinit first. It mounts, prepares the directories, brings the services up, then fails to reach a control channel that is not there and exits — leaving everything it prepared behind, which is exactly what makes the hand-run useful. Then start what you came to debug.

It is safe to run outside a box, and that took fixing: the shutdown path signals every process it may signal and then powers the machine off, which is right for PID 1 of a box and catastrophic anywhere else. Both steps are refused when it is not PID 1, and it says so rather than doing it quietly.

If you would rather not run it at all, the two mounts it does that a compositor needs are:

mount -t tmpfs -o mode=1777,size=64m tmpfs /tmp
mount -t tmpfs -o mode=755,size=32m  tmpfs /run
mkdir -p /run/user/1000 && chown 1000:1000 /run/user/1000 && chmod 0700 /run/user/1000

Making /run/user/1000 writable in the image does not help, and is worth saying because it is the obvious first thing to try: before the init runs, the directory is on a read-only root, so its ownership is not what stops a write; after the init runs, a fresh tmpfs is mounted over /run and the image's copy of the directory is hidden underneath it.

The one thing this does not reach is a failure before the shell. If that happens the evidence is on console=hvc0 and nowhere else.

Two build-time checks worth knowing about

Both exist because the failure they catch is invisible at runtime rather than loud, which is the same reason the old build checked its conf.d files:

• No hook may point at a program that is not in the image. Removing systemd removes the script dbus-reload.hook calls, and a leftover hook produces error: command failed to execute correctly on every future pacman transaction — indistinguishable, in a log, from something that matters.
• Nothing nesinit will look for may be missing. Its service table is compiled in, so an absent dbus-daemon is not a build error by itself; it is a service that does not come up in a box somebody is waiting on.

Network defaults

nesinit reads nestri.ip=/nestri.gw= off the kernel command line, falling back to 172.30.0.2/24 via 172.30.0.1 if neither is set — the host's own default tap addressing. Keep these in step if that changes on the host side. A box started with no network device at all is a valid box and boots without one.

The address is a per-boot parameter rather than an image setting because the alternative makes every box built from this image the same host on the network, and two of them collide the moment they run together. Same reasoning for /etc/machine-id, which is a symlink into a tmpfs that init fills at boot — the previous image baked one in, so every box built from it was the same machine to anything that asked.

This is also the one thing in the image that keeps iproute2 installed: init runs ip rather than talking netlink, which is a hundred lines of unsafe saved for an interface configured once.