200bc9c75f
Stacked on #313, which this depends on.
Found while checking whether the 10-bit HDR path actually works now that
a
client can obtain an HDR swapchain (see #314). **It does** — verified
end to
end rather than from the format list: a client requesting `A2B10G10R10`
+
`HDR10_ST2084` produces
```
pix_fmt=yuv420p10le color_range=pc
color_space=bt2020nc color_transfer=smpte2084 color_primaries=bt2020
```
which is a correctly tagged HDR10 stream, and the first pixel-level HDR
check
here with 10-bit rather than 8-bit input. Three ways it could have gone
wrong
instead.
## Unrecognised formats defaulted to BGRA
`vk_format_to_input_format` returned `BGRA` for anything it did not
know, which
reads a packed 10-bit or FP16 buffer as eight-bit channels. It now
returns
`None`, and the encode loop drops those frames with one log line per
format.
A stalled stream is a complaint. A stream at full frame rate carrying
nonsense
is not, and that is the failure this area keeps producing.
## Bit depth and input format had drifted apart
They were two separate matches on the same `VkFormat`. `A2R10G10B10`
counted as
ten-bit in one and had no entry in the other, so it fell back to
eight-bit
BGRA — the encoder configured for ten bits while the converter read
eight.
Depth now derives from the input format, so that disagreement is
unrepresentable. `A2R10G10B10` stays unmapped deliberately: a WSI layer
offers
it as one of its HDR pairs and the compositor dmabuf list advertises it,
but
the converter has no red-first 10-bit input, so there is nothing correct
to map
it to.
## The CPU fallback could not read either HDR format
It read four bytes per pixel for every format and encoded eight-bit
regardless, so a packed 10-bit buffer became garbage and an FP16 one was
half
an image of misread floats. It now refuses what it cannot read.
## Recorded, not fixed: the colour space we see is not always the one
requested
A FIXME at the point the value is read. A WSI layer rewrites
`imageColorSpace`
to `SRGB_NONLINEAR` before calling down — deliberately, since it carries
the
real colour space to the compositor out of band. We sit below it, so we
read
the rewrite. Measured, all three lines from one run:
```
[Gamescope WSI] ... colorspace: VK_COLOR_SPACE_HDR10_ST2084_EXT
swapchain created — format=A2B10G10R10 colorspace=SRGB_NONLINEAR
(re)init encoder: H265 Yuv420 Ten Bt709 → P010
```
Ten-bit right, BT.709 wrong: PQ samples encoded and tagged as SDR. The
same
client *without* the layer gives `Ten Bt2020` and an smpte2084 stream,
so this
is specific to the layer path — which is the path Proton titles take.
The fix cannot be local; the true colour space only exists in the
compositor,
which does receive it, so it needs a channel from there. Layer ordering
is not
a fix — we do not control it, and the non-layer path still needs the
Vulkan
value. Left out of this PR as a design change rather than a bug fix.
## Verification
- 4 new tests, 12 total, all passing.
- No new clippy warnings (diffed against the base branch).
- 10-bit HDR path: unchanged, still `Ten Bt2020` / smpte2084.
- SDR path: `verify-chain.sh` passes, 835 frames, 8-bit BGRA, brightness
agreement 0.57.
<!-- greptile_comment -->
<h3>Greptile Summary</h3>
This PR makes Vulkan format handling fail safely instead of interpreting
unsupported swapchain buffers as BGRA.
- Maps supported Vulkan formats to explicit converter inputs and derives
bit depth from that mapping.
- Drops unsupported GPU frames with rate-limited logging.
- Rejects unsupported HDR formats in the eight-bit CPU fallback.
- Documents the color-space limitation caused by rewritten WSI metadata.
- Adds tests covering unsupported, eight-bit, 10-bit, and FP16 formats.
<h3>Confidence Score: 5/5</h3>
The PR appears safe to merge, with no new actionable issues introduced
since the previous review.
The changes since the previous review are empty, the sole previous
finding was manually resolved after Greptile conceded it based on the
stacked PR dependency, and the full PR introduces no confirmed rule
violations or remaining correctness failures.
<h3>Important Files Changed</h3>
| Filename | Overview |
|----------|----------|
| apps/nescapture/src/encode.rs | Replaces unsafe BGRA fallback behavior
with explicit format validation, consistent bit-depth derivation,
guarded CPU fallback, and focused tests. |
| apps/nescapture/src/swapchain.rs | Documents the known WSI color-space
rewrite limitation at the point where swapchain metadata is recorded. |
<h3>Flowchart</h3>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Captured Vulkan frame] --> B{Known converter input?}
B -->|No| C[Log format change and drop frame]
B -->|Yes| D[Derive input format and bit depth]
D --> E{DMA-BUF path available?}
E -->|Yes| F[GPU color conversion and encoding]
E -->|No| G{Eight-bit RGBA or BGRA?}
G -->|Yes| H[CPU conversion and encoding]
G -->|No| I[Return recoverable error]
```
<sub>Reviews (3): Last reviewed commit: ["docs(nescapture): the
colour-space note
..."](7b05908e0a)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=60377562)</sub>
**Context used:**
- Knowledge Base — [Vulkan capture
layer](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/capture-layer.md)
<!-- /greptile_comment -->
Run your games on a GPU you don't own — or one you do. Nestri puts an interactive workload in a hardware-accelerated virtual machine and streams it to you over QUIC, at a latency that lets you play rather than watch.
Note
This repository is mid-rewrite, and the documentation is behind the code. The guest-side components arrived recently and their docs are thin. Nothing here is stable yet: expect directories to move and interfaces to change. Proper documentation is on the way — issues and questions are welcome in the meantime, and are genuinely useful for deciding what to write first.
Try it now — nesdoctor
One thing here is finished and runs on its own machine, today:
# Linux and macOS
curl -fsSL https://doctor.nestri.io/install.sh | sh
# Windows
powershell -c "irm https://doctor.nestri.io/install.ps1 | iex"
It tells you whether your machine could host games for other people, and
measures the number that actually decides whether streaming a game feels
right — not your download speed, but how much latency your connection adds
when it is busy. A 500 Mbps uplink that queues for 300 ms under load cannot
carry a game; a 25 Mbps one with fq_codel can. Almost nobody has seen their
own figure.
upstream 35 Mbps
latency, idle floor 56 ms
latency, loaded 185 ms
added under load +129 ms grade F
presentation path x11 · bspwm
eDP-1 1920x1200 @ 60 Hz, 8-bit
Vulkan decode h264, h265
It also reads your display out of its EDID — resolution, refresh, colour depth, HDR transfer functions, BT.2020, chroma — and what your hardware can decode. Those decide what is worth sending over the wire, and we would otherwise be guessing from one panel in one room.
It does not stream a game. It is the piece that has to exist before
anything else can, and most machines will come back CLIENT — which is a real
answer, not a failure.
Downloads one binary, verifies its checksum, runs it, deletes it. Installs
nothing, needs no administrator rights, touches no system directory. Nothing is
uploaded: it prints a link, lists exactly what the link contains, and opens it
only if you press Enter. The scripts those URLs serve are
apps/nesdoctor/install/ in this repository, so you
can read them before you run them.
Source and the full story: apps/nesdoctor.
What is here
Two halves that meet over the network and share very little else, plus one thing that runs on your own machine.
The control plane — TypeScript, on Cloudflare Workers
apps/api |
The public REST API. Identity, teams, machines, games, pairing. |
apps/auth |
A self-hosted OpenAuth issuer — Steam and SSH-key login. |
packages/core |
The domain: every table, every operation, no HTTP. |
packages/auth |
Shared auth types and subjects. |
Postgres for state, Alchemy for infrastructure. See
docs/alchemy.md.
The guest — Rust, inside the box
These run inside a virtual machine, beside the game. None of them talk to the control plane.
apps/nescope |
A headless Wayland compositor for one fullscreen client. A lighter answer to the same problem gamescope solves. |
apps/nescapture |
A Vulkan implicit layer. It captures frames from inside the workload's own process and encodes them on the GPU that drew them — no copy out to the CPU and back. |
apps/neswire |
Audio capture and transport. |
apps/neshub |
One connection out of the box. Muxes video, audio, cursor and input into a single QUIC stream to the client. |
crates/nesprotocol |
The wire types they all share, so no two ends can drift apart silently. |
On your own machine — Rust
apps/nesdoctor |
Whether a machine can host a box, and what its connection and display can really do. The first executable form of our host requirements — until it existed, a host was qualified by a human reading a table. Four dependencies; everything that could be done with the standard library is. |
The hypervisor the guest components run under is nesbox,
a separate repository: a micro-VM with a real GPU in it, using virtio-gpu native
context rather than passthrough, so one card can host several boxes at once.
Why a virtual machine
A container shares the host kernel, which makes strong isolation hard and a GPU harder. A micro-VM boots in about as long, isolates properly, and — with native context — gets close to bare-metal graphics. That choice is what makes "many sandboxes, one GPU" possible instead of one tenant per card.
Getting started
bun install
bun dev # control plane, local Cloudflare runtime
cargo build --workspace # guest components
cargo test --workspace
The guest components expect a Linux host with a Wayland-capable GPU stack, and are not much use on their own yet — they are pieces of a box, and the thing that assembles a box is not open yet.
nesdoctor is the exception and needs none of that:
cargo run --release -p nesdoctor
Status
Working: nesdoctor — released, and the only part a stranger can operate
today. The API, auth, the domain model, and the guest components listed above.
Not here yet: the box lifecycle, storage, the edge, and the client. Some of that will open as it is written; some is deliberately closed. What decides which is whether it handles your data — that half is open on principle — or decides our capacity, which is the part we sell.
Contributing
Early, and the ground moves. The two most useful things you can do right now
cost a minute each: run nesdoctor and send the result, because we have
almost no idea what the machines on the other end of this look like; and tell
us where the documentation failed you. Conventional commits; explain why in
the body.