Wanjohi
f889516ef4
The box half of phase 1 of window streams: two games in one box, each its own stream. Wire formats are nesprotocol's `window` module (#421). ## What each app does now **nescope** - Gives every toplevel an id on first map: 0 for the first window, never reused, none past 65535. - Tells the hub, on the existing input socket, when a window opens, changes title or size, and closes, in nesprotocol's window encoding. It also sends a box-internal `MSG_WINDOW_PID` saying which process owns the window: the Wayland client, or `_NET_WM_PID` for Xwayland. The launch id comes from that process's `NESTRI_LAUNCH_ID`. - Handles `INPUT_WINDOW_FOCUS` by raising and focusing that window, and `INPUT_WINDOW_OUTPUT_SIZE` per window. The old `INPUT_OUTPUT_SIZE` means window 0. There is still one output: a window's size applies while that window has focus and is remembered otherwise. On every input connection, input resets to window 0. - Before cursor updates, sends `MSG_CURSOR_WINDOW` when the window they describe changes. **nescapture** - Tags each video IPC frame with its pid in a trailer (`IPC_FLAG_PID` in the reserved byte). An older reader never sees it. - Listens on `/tmp/nescapture-cmd-<pid>.sock`. It still binds the shared `/tmp/nescapture-cmd.sock` and relays from it, so whatever sends there still reaches the newest encoder, as before. The shared socket was the only resource two capturing processes fought over: the second one took it, and commands went only to that one. The video and stats sockets are send-only datagram sockets and are safe to share. Capture slots and devices are per process. **neshub** - Joins frames to windows by pid. An untagged frame is window 0. A pid with no window falls back to the newest window of its launch, then to window 0 while no other window is open. A pid whose window closed belongs to nobody. - A connection that never subscribes gets the box's primary picture in the old encoding. The primary picture is the frames of the first encoder to draw, until it has been silent for a second. A subscribed connection gets only its window's frames, windowed for any window but 0, with per-window seq. PyroWave stays window 0 only. - `MSG_WINDOW_SUBSCRIBE` switches the connection first, then resets the writer's seq and resync gate in line with the frames, then requests a keyframe. The first subscribe opens `STREAM_WINDOWS` on the video connection, starting with `OPENED` (and the icon, if known) for every open window. Later lifecycle events go to every subscribed connection. - Input to nescope carries `INPUT_WINDOW_FOCUS` only when the window changes, and the hub does no merging across tags. A gamepad message moves focus only when its window differs from the current one and no other client sent keyboard or pointer input in the last second. Focus tags move keyboard and pointer focus only: nescope neither raises the window nor resizes the output for one. The output changes size only on an explicit `INPUT_WINDOW_OUTPUT_SIZE` from the window that owns it, last writer wins with 500 ms hysteresis between windows. A client's `INPUT_OUTPUT_SIZE` is rewritten to `INPUT_WINDOW_OUTPUT_SIZE`, except for window 0. - A cursor update goes to the clients subscribed to the window it describes. Clients that never subscribed get every update, as before. - When nescope reconnects (a new compositor), the hub sends `CLOSED` for every window and every subscription goes dead until the client subscribes again. - `DatagramSender` sizes fragments with `max_fragment_payload` and writes headers with `write_video_datagram_header`. - Commands go to every known encoder on its own socket, or to the shared socket while none is known (as today). An encoder that appears later is sent the commands so far (the latest of each message type) and then a keyframe request. The first encoder is not sent them again. - A keyframe on a client's loss or subscribe goes only to that client's window's encoder (the primary's, for a client that never subscribed). - Known limit: bitrate is box-wide. One controller sets one target for every window's encoder, so one client's congestion sets every window's rate. Per-window rate is follow-up work. - An X11 unmap is `CLOSED` for the hub, and a re-map is `OPENED` with the same id. nescope no longer exits when its first game exits; nesinit owns box power-off. The hub never reads `/proc/<pid>/environ` (it runs as another uid): nescapture sends its launch id. ## Shipping nescope and neshub ship together in the guest image and must not ship apart. ## Compatibility One game with an old client sends the same bytes as before. The client never subscribes, so it gets the primary picture, which is that game, as `DGRAM_VIDEO`/`MSG_DATA` with an unchanged seq. It gets no windows stream. Cursor updates reach it as before. Focus tags appear only if the game's window is not 0, and window 0 resizes use the old form. Mixed versions also work: - An old nescapture is window 0. - An old nescope never sends window messages. Every frame is then window 0, or the primary picture. - An old hub sends no tags, which nescope reads as window 0, and it still reaches nescapture through the shared socket. ## Tests `cargo test` on nesprotocol (122), nescope (17+5), nescapture (161), neshub (74), all passing. New tests: - The window table: joining pids to windows, the launch fallback, retired pids, the primary picture, the snapshot. - Focus tagging and resize rewriting. - The command fan-out. - nescope's id registry (first window 0, ids never reused, refused past 65535) and per-window focus and sizes. - The IPC pid trailer. - Two QUIC end-to-end tests: - Two tagged encoders: an unsubscribed client sees only the first game in the old encoding. After subscribing to window 1 it sees only the second game, windowed, with seq starting again, and a windows stream that starts with both `OPENED` events. - An untagged encoder gives the old stream. Not run on a box: no neslet, nestri-host or box was touched.
nescapture
A Vulkan implicit layer that captures frames from inside the workload's own
process, encodes them with Vulkan Video on the GPU that drew them, and
sends them to neshub over a Unix socket — no copy out to the
CPU and back.
Being a layer rather than a screen-scraper is the whole point: the frame is already on the GPU when we get it, and it never leaves.
Where it sits
Game process
│ Vulkan calls
▼
┌──────────────────────────────────────────────┐
│ nescapture implicit layer │
│ │
│ vkCreateShaderModule → SHA-256 hash │
│ vkCreateGraphicsPipelines → track hashes │
│ vkCmdBindPipeline → detect HUD │
│ vkQueuePresentKHR → capture+encode │
└──────────────────────────────────────────────┘
│ GPU blit on the game's queue, signals a timeline semaphore
▼
ring slot (an image on the game's own device)
│ read in place, once the blit's point is reached
▼
ColorConverter (GPU compute shader, the encoder's own queue)
│ BGRA/RGB10/FP16 → NV12/P010/YUV444
▼
Encoder (Vulkan Video: H.264 / H.265 / AV1)
│ Annex-B packets
▼
Unix datagram → neshub → the client
The encoder runs on the game's own VkDevice. The layer creates that device
with what the encoder needs: the extensions and feature bits it asks for, and
queues of its own wherever a queue family has one to spare, since a VkQueue
may not be submitted to from two threads at once. Where no family has room,
the game's queue is created internally synchronized and shared. Every step of
a frame is ordered on the GPU; nothing waits on the CPU.
Where the conversion is only the YUV matrix and the device has
VK_VALVE_video_encode_rgb_conversion, the encoder takes the RGB frame and
converts it itself, and the converter is not built. That path runs at limited
range, since the one driver offering it writes limited range whatever it is
asked; everything else is full range.
When the game's device cannot host the encoder -- an instance the loader
cannot raise to Vulkan 1.1, no queue the encoder could safely use, a driver
refusing the additions, or NESCAPTURE_SHARED_DEVICE=0 -- the encoder gets a
device of its own, and each frame is read back on the CPU and uploaded there.
That works everywhere and costs a copy each way.
Sockets
| path | direction | carries |
|---|---|---|
/tmp/nestri-video.sock |
nescapture → neshub | encoded frames |
/tmp/nestri-stats.sock |
nescapture → neshub | capture fps, encode ms, drops |
/tmp/nescapture-cmd.sock |
neshub → nescapture | IDR requests, encode settings |
nescapture binds the command socket and connects to the other two. The stats
path is derived from NESCAPTURE_IPC_PATH's directory, so moving the video
socket moves both.
Quick start
# 1. Build
cargo build --release -p nescapture
# 2. Install the layer manifest
sudo cp apps/nescapture/manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
# Point the manifest's `library_path` at target/release/libnescapture_layer.so
# 3. Run something that draws
export NESCAPTURE_ENABLE=1
export NESCAPTURE_CODEC=h265
export NESCAPTURE_BITRATE=10000
export RUST_LOG=info
./my-vulkan-app
The layer is inert unless NESCAPTURE_ENABLE=1. That is deliberate — an
implicit layer is loaded into every Vulkan process on the system.
Environment variables
| Variable | Default | Description |
|---|---|---|
NESCAPTURE_ENABLE |
(unset) | Set to 1 to activate the layer. Nothing happens otherwise |
NESCAPTURE_IPC_PATH |
/tmp/nestri-video.sock |
Where to send encoded frames |
NESCAPTURE_CODEC |
best available | h264, h265 or av1; probes if unset |
NESCAPTURE_FORMAT |
yuv420 |
yuv420 or yuv444 |
NESCAPTURE_DEPTH |
auto | 8 or 10; inferred from the swapchain VkFormat if unset |
NESCAPTURE_RC |
(inferred) | cqp, cbr or vbr. Unset infers cqp when NESCAPTURE_QP is set, cbr otherwise |
NESCAPTURE_BITRATE |
10000 |
Target bitrate in kbps, under cbr and vbr |
NESCAPTURE_BITRATE_MAX |
1.5x the target | VBR ceiling in kbps. Ignored outside vbr |
NESCAPTURE_QP |
(unset) | Constant QP, under cqp |
NESCAPTURE_FPS |
60 |
Target frame rate |
NESCAPTURE_IDR_INTERVAL |
4 |
Force an IDR every N seconds |
NESCAPTURE_INTRA_REFRESH |
(off) | Set to 1 to replace periodic key frames with an intra refresh cycle |
NESCAPTURE_INTRA_REFRESH_QP_DELTA |
-4 |
QP shift inside the refresh band; negative spends bits on it |
NESCAPTURE_INTRA_REFRESH_SHAPE |
auto | rows, columns or partitions; the driver chooses if unset |
NESCAPTURE_TUNE |
(unset) | highquality, lowlatency, ultralowlatency, lossless |
NESCAPTURE_SHARED_DEVICE |
(on) | Set to 0 to leave the game's device as the game asked for it, and encode on a device of the encoder's own with CPU readback |
NESCAPTURE_RGB_ENCODE |
(on) | Set to 0 to always convert with the shader, even where the encoder could convert RGB itself |
NESCAPTURE_CONFIG |
(unset) | Path to the per-app shader-hash TOML |
NESCAPTURE_GAME_NAME |
exe basename | Override app identification for that config |
NESCAPTURE_DISCOVER |
(unset) | Set to 1 to log every draw, for finding HUD shaders |
RUST_LOG |
error |
Standard env_logger filter, e.g. nescapture_layer=debug |
Everything else is decided at runtime: the client asks neshub for a codec or
bitrate change and it arrives on the command socket, so the encoder is
reconfigured without a restart.
Intra refresh
Instead of a key frame every few seconds, each picture codes one slice of the image as intra, so after a full cycle every part has been refreshed. The same cost, paid evenly, with no picture much larger than any other — which is what a link with a latency budget wants, since a key frame is the largest frame there is.
The cycle length is not configurable, deliberately. It is bounded by how many refresh regions the picture actually has, and that depends on the codec's block size: at 1080p an H.265 picture is 17 CTB rows tall where an H.264 one is 68 macroblock rows, so the same duration is comfortable for one codec and impossible for the other. The encoder knows the codec, the resolution and what the device allows, and derives it from the key frame interval it replaces.
The refresh here spreads cost only; it does not make the cycle a recovery point. Doing that would restrict prediction on every picture — expensive, and what turns the refreshed band into a visible discontinuity — to buy a guarantee this stream gets more cheaply from the client asking for an IDR.
The band is coded intra every cycle, so it carries none of the refinement its
neighbours have accumulated and reads as a strip of lower quality sweeping
across the picture. NESCAPTURE_INTRA_REFRESH_QP_DELTA spends bits back into
it, out of the rest of the frame. Measured at 1080p with ColorVideoVDP, -4
recovers a fifth of what intra refresh costs and the encoded size does not
grow — but the best value depends on the content, and too large a shift
starves the rest of the frame faster than too small a one helps. Devices that
cannot express a negative delta, or whose refresh regions follow the slice
layout rather than a block sweep, decline it and say so.
NESCAPTURE_INTRA_REFRESH_SHAPE stays configurable because the device cannot
answer it: whether a horizontal or vertical sweep looks better depends on how
the content moves. It also changes how many regions there are — a 1080p
picture in 64×64 blocks is 17 rows but 30 columns, so columns allows a
longer cycle and thus less intra per picture.
Rate control
cbr holds every frame to the same size, which is what a link with a fixed
budget wants. vbr holds the same average while letting a frame that needs
it spend up to the ceiling — a scene change is coded rather than smeared, at
the cost of a burst the path has to absorb. cqp holds quality constant and
lets the bitrate go wherever the content takes it, which is a recording
setting rather than a streaming one.
The command socket carries a target bitrate but has no way to name a mode
beyond CBR and constant QP. A target arriving while the encode is vbr is
therefore applied as a target, leaving the mode and the ceiling alone — so the
ceiling asked for at launch survives a session, and a congestion controller
adjusts underneath it. Retargeting costs no rebuild and no key frame.
Per-app shader-hash config
HUD detection needs to know which pipelines draw the HUD, and that is
per-application. Run once with NESCAPTURE_DISCOVER=1 to log every shader,
then pick out the HUD pipelines by the [SUSPECT] marker — blend on, depth
off, six vertices or fewer.
# ~/.config/nescapture/apps.toml
[game."MyApp.exe"]
hud_fragment_shaders = ["0xaabbccddeeff0011"]
hud_vertex_shaders = ["0x1a2b3c4d5e6f7890"]
skip_fragment_shaders = ["0x1122334455667788"]
export NESCAPTURE_CONFIG=~/.config/nescapture/apps.toml
export NESCAPTURE_GAME_NAME=MyApp.exe
Modules
src/
├── lib.rs entry points, dispatch routing
├── dispatch.rs Vulkan function-pointer types and tables
├── state.rs global DashMaps, DeviceState, CbState
├── instance.rs vkCreateInstance / vkDestroyInstance
├── device.rs vkCreateDevice / vkDestroyDevice
├── shader.rs SPIR-V hashing
├── pipeline.rs graphics pipeline tracking
├── framebuffer.rs image view and framebuffer tracking
├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
├── swapchain.rs vkCreateSwapchainKHR, image enumeration
├── capture.rs GPU blit into the capture ring, CPU readback fallback
├── present.rs vkQueuePresentKHR, encode dispatch
├── encode.rs pixelforge pipeline, codec probing, IPC send
├── shared.rs creating the game's device for the encoder to share
├── config.rs per-app TOML shader-hash config
└── discovery.rs draw-call logging for shader discovery
Dependencies
| Crate | Purpose |
|---|---|
pixelforge |
Vulkan Video hardware encode |
ash |
Vulkan bindings |
nesprotocol |
The IPC frame format neshub reads |
sha2, bytemuck |
SPIR-V fingerprinting |
dashmap, once_cell |
Lock-free concurrent state |
serde, toml |
Per-app shader config |
ash and pixelforge are pinned to git revisions — both track Vulkan Video
support that has not landed in a release.
Licence
Apache 2.0. See LICENSE.