Wanjohi
f889516ef4
The box half of phase 1 of window streams: two games in one box, each its own stream. Wire formats are nesprotocol's `window` module (#421). ## What each app does now **nescope** - Gives every toplevel an id on first map: 0 for the first window, never reused, none past 65535. - Tells the hub, on the existing input socket, when a window opens, changes title or size, and closes, in nesprotocol's window encoding. It also sends a box-internal `MSG_WINDOW_PID` saying which process owns the window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id comes from that process's `NESTRI_LAUNCH_ID`. - Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and `INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means window 0. There is still one output: a window's size applies while that window has focus and is remembered otherwise. On every input connection, input resets to window 0. - Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they describe changes. **nescapture** - Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in the reserved byte). An older reader never sees it. - Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared `/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there still reaches the newest encoder, as before. The shared socket was the only resource two capturing processes fought over: the second one took it, and commands went only to that one. The video and stats sockets are send-only datagram sockets and are safe to share. Capture slots and devices are per process. **neshub** - Joins frames to windows by pid. An untagged frame is window 0. A pid with no window falls back to the newest window of its launch, then to window 0 while no other window is open. A pid whose window closed belongs to nobody. - A connection that never subscribes gets the box's primary picture in the old encoding. The primary picture is the frames of the first encoder to draw, until it has been silent for a second. A subscribed connection gets only its window's frames, windowed for any window but 0, with per-window seq. PyroWave stays window 0 only. - `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the writer's seq and resync gate in line with the frames, then requests a keyframe. The first subscribe opens `STREAM_WINDOWS` on the video connection, starting with `OPENED` (and the icon, if known) for every open window. Later lifecycle events go to every subscribed connection. - Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window changes, and the hub does no merging across tags. A gamepad message moves focus only when its window differs from the current one and no other client sent keyboard or pointer input in the last second. Focus tags move keyboard and pointer focus only: nescope neither raises the window nor resizes the output for one. The output changes size only on an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it, last writer wins with 500 ms hysteresis between windows. A client's `INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except for window 0. - A cursor update goes to the clients subscribed to the window it describes. Clients that never subscribed get every update, as before. - When nescope reconnects (a new compositor), the hub sends `CLOSED` for every window and every subscription goes dead until the client subscribes again. - `DatagramSender` sizes fragments with `max_fragment_payload` and writes headers with `write_video_datagram_header`. - Commands go to every known encoder on its own socket, or to the shared socket while none is known (as today). An encoder that appears later is sent the commands so far (the latest of each message type) and then a keyframe request. The first encoder is not sent them again. - A keyframe on a client's loss or subscribe goes only to that client's window's encoder (the primary's, for a client that never subscribed). - Known limit: bitrate is box-wide. One controller sets one target for every window's encoder, so one client's congestion sets every window's rate. Per-window rate is follow-up work. - An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with the same id. nescope no longer exits when its first game exits; nesinit owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs as another uid): nescapture sends its launch id. ## Shipping nescope and neshub ship together in the guest image and must not ship apart. ## Compatibility One game with an old client sends the same bytes as before. The client never subscribes, so it gets the primary picture, which is that game, as `DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows stream. Cursor updates reach it as before. Focus tags appear only if the game's window is not 0, and window 0 resizes use the old form. Mixed versions also work: - An old nescapture is window 0. - An old nescope never sends window messages. Every frame is then window 0, or the primary picture. - An old hub sends no tags, which nescope reads as window 0, and it still reaches nescapture through the shared socket. ## Tests `cargo test` on nesprotocol (122), nescope (17+5), nescapture (161), neshub (74), all passing. New tests: - The window table: joining pids to windows, the launch fallback, retired pids, the primary picture, the snapshot. - Focus tagging and resize rewriting. - The command fan-out. - nescope's id registry (first window 0, ids never reused, refused past 65535) and per-window focus and sizes. - The IPC pid trailer. - Two QUIC end-to-end tests: - Two tagged encoders: an unsubscribed client sees only the first game in the old encoding. After subscribing to window 1 it sees only the second game, windowed, with seq starting again, and a windows stream that starts with both `OPENED` events. - An untagged encoder gives the old stream. Not run on a box: no neslet, nestri-host or box was touched.
nescope
A lightweight headless Wayland compositor designed for game capture scenarios. nescope creates a complete compositor environment for games while delegating frame capture to an external Vulkan interception layer.
Overview
nescope is built on top of Smithay, the Rust Wayland compositor framework. Unlike traditional compositors that manage window rendering and buffer scanout, nescope operates in headless mode—it provides the Wayland/X11 environment games expect but never allocates GBM buffers or performs rendering.
┌─────────────────┐
│ nescope │ ← Headless Wayland compositor
├─────────────────┤
│ Virtual Output │ ← 1920x1080 @ 60Hz (configurable)
│ XWayland │ ← X11 compatibility layer
│ DMA-BUF (v4) │ ← Required for XWayland DRI3
│ wp_color_mgmt │ ← HDR signaling
│ gamescope WSI │ ← Vulkan bypass protocol
└────────┬────────┘
│
▼
┌─────────────────┐
│ Game │ ← Render target
└────────┬────────┘
│ frame capture
▼
┌─────────────────┐
│ nescapture │ ← Vulkan layer, inside the game's own process
└─────────────────┘
Architecture
Core Components
| Module | Purpose |
|---|---|
main.rs |
CLI entry point, event loop, process management |
state.rs |
NescopeState — central compositor state, Wayland globals |
handlers.rs |
Smithay protocol handlers (CompositorHandler, XdgShellHandler) |
xwm.rs |
X11 window management |
focus.rs |
Keyboard focus routing for X11 windows |
input.rs |
Input decoding and injection via calloop::channel |
input_ipc.rs |
The client side of neshub's input socket |
libinput_backend.rs |
Real input devices, when there are any |
hdr.rs |
HDR / colour management protocol handlers |
gpu_readback.rs |
Reading a dmabuf back to the CPU, for screenshots |
screenshot_ipc.rs |
Serving captures to whoever is listening |
screenshot_wire.rs |
The screenshot frame format |
protocols/ |
Generated gamescope swapchain bindings |
bin/nescope-shot.rs |
Ask a running nescope what is on screen |
Design Decisions
-
Headless Operation: nescope never allocates GBM buffers. The DMA-BUF global exists solely so XWayland can initialize DRI3/GBM/glamor.
-
Frame Callbacks: Driven by an internal calloop timer at the configured FPS, not by a real scanout. This keeps the game's render loop alive.
-
Input via Channel: Input events arrive through a
calloop::channel::Sender<InputEvent>rather than being decoded from a proxy connection. This allows external code (streaming servers, test harnesses) to inject input. -
Process Subreaper: nescope registers as a subreaper (
PR_SET_CHILD_SUBREAPER) so orphaned descendants — a launcher that execs the real client and exits, say — are reparented to it rather than to PID 1, and can be cleaned up reliably. -
Two shapes, one binary: given a command after
--, nescope launches it and exits when it and its windows are gone. Given none, it comes up as a plain compositor and waits for something to connect. The second is what a real session needs, where the processes that draw are started by something else entirely.
Building
cargo build --release
Usage
nescope [OPTIONS] [-- <command> [args...]]
Options:
--width <N> Output width [default: 1920]
--height <N> Output height [default: 1080]
--fps <N> Virtual refresh rate [default: 60]
--hdr Enable HDR protocols
--socket <NAME> Wayland socket name [default: nescope-0]
--input-ipc <PATH> neshub's input socket [default: /tmp/nestri-input.sock]
--screenshot-ipc <PATH> Serve screenshots here [default: off]
--render-device <PATH> GPU to pin the workload to, e.g. /dev/dri/renderD128
--x-display <N> XWayland display number [default: 1]
The command is optional; see design decision 5 above.
Environment Variables
| Variable | Effect |
|---|---|
DISPLAY |
XWayland display (:N) |
NESCOPE_WIDTH |
Override --width |
NESCOPE_HEIGHT |
Override --height |
NESCOPE_FPS |
Override --fps |
NESCOPE_HDR |
Enable HDR |
NESCOPE_SOCKET |
Override socket name |
NESCOPE_INPUT_IPC |
Override --input-ipc |
NESCOPE_SCREENSHOT_IPC |
Override --screenshot-ipc |
NESCOPE_RENDER_DEVICE |
Override --render-device |
NESCOPE_X_DISPLAY |
Override --x-display |
RUST_LOG |
Tracing filter (e.g., nescope=debug) |
Example
# Wrap one program at 1440p with HDR
nescope --width 2560 --height 1440 --hdr -- %command%
# Plain compositor mode: come up and wait for clients
nescope --x-display 1
# Debug logging
RUST_LOG=nescope=debug nescope -- %command%
Seeing what is on screen
nescope is headless, so a black stream, a window that never mapped and a client
rendering fine all look identical from outside. nescope-shot is the listener
side of the screenshot socket:
# start this first — nescope dials out
nescope-shot --socket /tmp/nestri-screenshot.sock --watch --out shot.ppm
nescope --screenshot-ipc /tmp/nestri-screenshot.sock -- <program>
HDR Support
nescope supports two HDR signaling paths:
-
wp_color_management_v1 — Standard Wayland staging protocol used by Wine/Proton/SDL2 when requesting HDR via Vulkan color-space extensions (disabled by default).
-
gamescope_swapchain_factory_v2 — Valve's private protocol used by the gamescope WSI Vulkan layer. This is the primary path for Steam games using
PROTON_ENABLE_NVAPI.
When --hdr is enabled, nescope:
- Advertises both protocol globals
- Sets
ENABLE_GAMESCOPE_WSI=1andGAMESCOPE_WAYLAND_DISPLAY=$(socket_path_here)on the X11 root window - Tracks per-surface color space for external capture layer retrieval
Shutdown Behavior
- First SIGINT/SIGTERM: Sets an atomic flag; the event loop exits cleanly, killing all child process groups.
- Second signal: Falls through to OS default handler (hard kill).
nescope waits 5 seconds after the last mapped window disappears before exiting, allowing save-game/cleanup processes to complete.
Dependencies
- smithay — Wayland compositor framework (
wayland_frontend,xwayland,backend_drm,desktop,renderer_pixman) - wayland-client — Connects to host compositor (if needed)
- calloop — Event loop
- tracing — Structured logging
- clap — CLI parsing
- x11rb — X11 atom management
What nescope does not do
Frames reach the client through nescapture, which captures
inside the workload's process, and neshub, which muxes and
sends. Input arrives the same way in reverse. nescope provides the display
those components need and injects what they hand it; it never touches the
network.
Licence
Apache 2.0. See LICENSE.