diff --git a/apps/nescapture/README.md b/apps/nescapture/README.md index ed3cf96f..13a88441 100644 --- a/apps/nescapture/README.md +++ b/apps/nescapture/README.md @@ -1,209 +1,176 @@ # nescapture -A Vulkan implicit layer that captures frames from a running game, encodes them -with **Vulkan Video** hardware acceleration (H.264 / H.265 / AV1), packetizes with -Reed-Solomon FEC, and streams over RTP/UDP to a Moonlight-compatible client — -all with **zero CPU copies** — fully GPU from game rendering to RTP output. +A Vulkan implicit layer that captures frames from inside the workload's own +process, encodes them with **Vulkan Video** on the GPU that drew them, and +sends them to [`neshub`](../neshub) over a Unix socket — no copy out to the +CPU and back. + +Being a layer rather than a screen-scraper is the whole point: the frame is +already on the GPU when we get it, and it never leaves. --- -## Architecture +## Where it sits ``` Game process │ Vulkan calls ▼ -┌───────────────────────────────────────────────┐ -│ nescapture Vulkan implicit layer │ -│ │ -│ vkCreateShaderModule → SHA-256 hash │ -│ vkCreateGraphicsPipelines → track hashes │ -│ vkCmdBindPipeline → detect HUD shaders │ -│ vkCmdEndRenderPass/Rendering → inject copy │ -│ vkQueuePresentKHR → capture + encode │ -└───────────────────────────────────────────────┘ - │ GPU blit (same device) +┌──────────────────────────────────────────────┐ +│ nescapture implicit layer │ +│ │ +│ vkCreateShaderModule → SHA-256 hash │ +│ vkCreateGraphicsPipelines → track hashes │ +│ vkCmdBindPipeline → detect HUD │ +│ vkQueuePresentKHR → capture+encode │ +└──────────────────────────────────────────────┘ + │ GPU blit, same device ▼ final_image (DMA-BUF exportable) │ get_dmabuf_fd(final_memory) ▼ -DmaBufImporter (pixelforge VkDevice) +DmaBufImporter (pixelforge VkDevice) │ import_or_reuse() → vk::Image ▼ ColorConverter (GPU compute shader) │ BGRA/RGB10/FP16 → NV12/P010/YUV444 ▼ -Encoder (Vulkan Video, hardware H.264/H.265) - │ encode() +Encoder (Vulkan Video: H.264 / H.265 / AV1) + │ Annex-B packets ▼ -EncodedPacket (Annex-B) - │ - ▼ -Packetizer (Moonlight wire format, Reed-Solomon FEC) - │ UDP datagrams - ▼ -Moonlight client (or any RTP/UDP receiver) - -CPU fallback: only when DMA-BUF export unavailable (rare driver config) +Unix datagram → neshub → the client ``` +CPU fallback exists only for driver configurations without DMA-BUF external +memory export. + +--- + +## Sockets + +| path | direction | carries | +| --- | --- | --- | +| `/tmp/nestri-video.sock` | nescapture → neshub | encoded frames | +| `/tmp/nestri-stats.sock` | nescapture → neshub | capture fps, encode ms, drops | +| `/tmp/nescapture-cmd.sock` | neshub → nescapture | IDR requests, encode settings | + +nescapture binds the command socket and connects to the other two. The stats +path is derived from `NESCAPTURE_IPC_PATH`'s directory, so moving the video +socket moves both. + --- ## Quick start ```bash # 1. Build -cargo build --release +cargo build --release -p nescapture # 2. Install the layer manifest -sudo cp manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/ -# Edit the manifest's `library_path` to point at target/release/libnescapture.so +sudo cp apps/nescapture/manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/ +# Point the manifest's `library_path` at target/release/libnescapture_layer.so -# 3. Configure and launch a game +# 3. Run something that draws export NESCAPTURE_ENABLE=1 -export NESCAPTURE_RTP_HOST=192.168.1.50 # Moonlight / receiver IP -export NESCAPTURE_RTP_PORT=47998 # default -export NESCAPTURE_CODEC=h265 # h264 | h265 | av1 (auto-probes if unset) -export NESCAPTURE_BITRATE=10000 # kbps (CBR; ignored if NESCAPTURE_QP is set) -export NESCAPTURE_FPS=60 -export RUST_LOG=info # or NESCAPTURE_LOG=debug +export NESCAPTURE_CODEC=h265 +export NESCAPTURE_BITRATE=10000 +export RUST_LOG=info -wine MyGame.exe # or native Vulkan game +./my-vulkan-app ``` +The layer is inert unless `NESCAPTURE_ENABLE=1`. That is deliberate — an +implicit layer is loaded into *every* Vulkan process on the system. + --- ## Environment variables -| Variable | Default | Description | -| ------------------------- | -------------- | ------------------------------------------------------ | -| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer | -| `NESCAPTURE_RTP_HOST` | _(required)_ | Destination IP / hostname for RTP stream | -| `NESCAPTURE_RTP_PORT` | `47998` | Destination UDP port | -| `NESCAPTURE_CODEC` | auto | `h264`, `h265` or `av1` — falls back to h264 if unsupported | -| `NESCAPTURE_BITRATE` | `10000` | CBR target bitrate in kbps | -| `NESCAPTURE_QP` | _(unset)_ | If set, use CQP with this quality level instead of CBR | -| `NESCAPTURE_FPS` | `60` | Target frame rate | -| `NESCAPTURE_IDR_INTERVAL` | `120` | Force an IDR keyframe every N frames | -| `NESCAPTURE_FEC_PCT` | `20` | Reed-Solomon FEC percentage | -| `NESCAPTURE_MIN_FEC` | `2` | Minimum FEC packets per block | -| `NESCAPTURE_PACKET_SIZE` | `1392` | Max UDP payload size (bytes) | -| `NESCAPTURE_CTRL_PORT` | `47999` | UDP port for the control stream | -| `NESCAPTURE_CAPTURE_HUDLESS` | _(unset)_ | Set to `1` to also capture HUDless frames | -| `NESCAPTURE_CONFIG` | _(unset)_ | Path to per-game shader-hash TOML config | -| `NESCAPTURE_GAME_NAME` | (exe basename) | Override game identification | -| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to enable discovery mode (logs all draws) | -| `NESCAPTURE_LOG` | `info` | Log level (`error`, `warn`, `info`, `debug`, `trace`) | -| `NESCAPTURE_RTP_FORMAT` | `moonlight` | `standard` for RFC 6184/7798 RTP (GStreamer/FFmpeg compatible), `moonlight` for Moonlight wire format with FEC | +| Variable | Default | Description | +| --- | --- | --- | +| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer. Nothing happens otherwise | +| `NESCAPTURE_IPC_PATH` | `/tmp/nestri-video.sock` | Where to send encoded frames | +| `NESCAPTURE_CODEC` | best available | `h264`, `h265` or `av1`; probes if unset | +| `NESCAPTURE_FORMAT` | `yuv420` | `yuv420` or `yuv444` | +| `NESCAPTURE_DEPTH` | auto | `8` or `10`; inferred from the swapchain `VkFormat` if unset | +| `NESCAPTURE_BITRATE` | `10000` | CBR target in kbps. Ignored when `NESCAPTURE_QP` is set | +| `NESCAPTURE_QP` | _(unset)_ | Constant QP instead of CBR | +| `NESCAPTURE_FPS` | `60` | Target frame rate | +| `NESCAPTURE_IDR_INTERVAL` | `4` | Force an IDR every N **seconds** | +| `NESCAPTURE_TUNE` | _(unset)_ | `highquality`, `lowlatency`, `ultralowlatency`, `lossless` | +| `NESCAPTURE_CONFIG` | _(unset)_ | Path to the per-app shader-hash TOML | +| `NESCAPTURE_GAME_NAME` | exe basename | Override app identification for that config | +| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to log every draw, for finding HUD shaders | +| `RUST_LOG` | `error` | Standard `env_logger` filter, e.g. `nescapture_layer=debug` | + +Everything else is decided at runtime: the client asks `neshub` for a codec or +bitrate change and it arrives on the command socket, so the encoder is +reconfigured without a restart. --- -## Control stream +## Per-app shader-hash config -Send single-byte UDP datagrams to `NESCAPTURE_CTRL_PORT` (default 47999): - -| Byte | Command | -| ------ | -------------------- | -| `0x01` | Start streaming | -| `0x02` | Stop streaming | -| `0x03` | Request IDR keyframe | - -Example with netcat: - -```bash -# Request IDR -printf '\x03' | nc -u -q1 localhost 47999 -``` - -The control handle wiring in `present.rs` is currently a commented-out stub. -See `control.rs` for the full wiring instructions. - ---- - -## Per-game shader-hash config - -HUD shader detection requires a per-game config. Run the game once with -`NESCAPTURE_DISCOVER=1` to log all shaders, then identify HUD pipelines by the -`[SUSPECT]` marker (blend=true, depth=false, ≤6 vertices). +HUD detection needs to know which pipelines draw the HUD, and that is +per-application. Run once with `NESCAPTURE_DISCOVER=1` to log every shader, +then pick out the HUD pipelines by the `[SUSPECT]` marker — blend on, depth +off, six vertices or fewer. ```toml -# ~/.config/nescapture/games.toml -[game."GameName.exe"] +# ~/.config/nescapture/apps.toml +[game."MyApp.exe"] hud_fragment_shaders = ["0xaabbccddeeff0011"] hud_vertex_shaders = ["0x1a2b3c4d5e6f7890"] skip_fragment_shaders = ["0x1122334455667788"] ``` ```bash -export NESCAPTURE_CONFIG=~/.config/nescapture/games.toml -export NESCAPTURE_GAME_NAME=GameName.exe +export NESCAPTURE_CONFIG=~/.config/nescapture/apps.toml +export NESCAPTURE_GAME_NAME=MyApp.exe +``` + +--- + +## Modules + +``` +src/ +├── lib.rs entry points, dispatch routing +├── dispatch.rs Vulkan function-pointer types and tables +├── state.rs global DashMaps, DeviceState, CbState +├── instance.rs vkCreateInstance / vkDestroyInstance +├── device.rs vkCreateDevice / vkDestroyDevice +├── shader.rs SPIR-V hashing +├── pipeline.rs graphics pipeline tracking +├── framebuffer.rs image view and framebuffer tracking +├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass +├── swapchain.rs vkCreateSwapchainKHR, image enumeration +├── capture.rs GPU blit to the capture image, DMA-BUF export +├── present.rs vkQueuePresentKHR, encode dispatch +├── encode.rs pixelforge pipeline, codec probing, IPC send +├── dmabuf_import.rs cross-device zero-copy import +├── config.rs per-app TOML shader-hash config +└── discovery.rs draw-call logging for shader discovery ``` --- ## Dependencies -| Crate | Purpose | -| ---------------------- | -------------------------------------------- | -| `pixelforge` | Vulkan Video hardware encode (H.264 / H.265) | -| `ash` | Vulkan bindings | -| `reed-solomon-erasure` | FEC for RTP packetizer | -| `sha2` + `bytemuck` | SPIR-V shader fingerprinting | -| `dashmap` | Lock-free concurrent state maps | -| `serde` + `toml` | Per-game shader config | +| Crate | Purpose | +| --- | --- | +| `pixelforge` | Vulkan Video hardware encode | +| `ash` | Vulkan bindings | +| `nesprotocol` | The IPC frame format `neshub` reads | +| `sha2`, `bytemuck` | SPIR-V fingerprinting | +| `dashmap`, `once_cell` | Lock-free concurrent state | +| `serde`, `toml` | Per-app shader config | + +`ash` and `pixelforge` are pinned to git revisions — both track Vulkan Video +support that has not landed in a release. --- -## Zero-copy GPU pipeline +## Licence -The full GPU path is implemented — no CPU color conversion: - -1. `final_image` allocated with `DMA_BUF_EXT` external memory (`capture.rs`). -2. `get_dmabuf_fd()` exports the DMA-BUF fd (`capture.rs`). -3. `DmaBufImporter::import_or_reuse()` imports the fd as a `vk::Image` (`dmabuf_import.rs`). -4. `ColorConverter::convert()` runs a GPU compute shader: - BGRA/RGBA/RGB10/FP16 → NV12/P010/YUV444 (`encode.rs`). -5. `Encoder::encode()` encodes from the converter's output directly. -6. RTP packetizer + UDP send. - -CPU fallback exists only for rare driver configurations that don't support -DMA-BUF external memory export. - ---- - -## File structure - -``` -nescapture/ -├── Cargo.toml -├── README.md -├── ARCHITECTURE.md -├── manifest/ -│ └── VK_LAYER_nescapture.json -└── src/ - ├── lib.rs — entry points, dispatch routing - ├── dispatch.rs — Vulkan function-pointer types + tables - ├── state.rs — global DashMaps, DeviceState, CbState - ├── instance.rs — vkCreateInstance / vkDestroyInstance - ├── device.rs — vkCreateDevice / vkDestroyDevice - ├── shader.rs — SPIR-V hashing - ├── pipeline.rs — graphics pipeline tracking - ├── framebuffer.rs — image view / framebuffer tracking - ├── commands.rs — vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass - ├── swapchain.rs — vkCreateSwapchainKHR, image enumeration - ├── capture.rs — GPU blit to capture image, DMA-BUF export - ├── present.rs — vkQueuePresentKHR, encode dispatch - ├── encode.rs — pixelforge pipeline, codec probing, RTP send - ├── dmabuf_import.rs — DmaBufImporter (cross-device zero-copy import) - ├── packetizer.rs — Moonlight RTP packetizer (Reed-Solomon FEC) - ├── shard_batch.rs — zero-alloc shard buffer - ├── control.rs — UDP control stream (IDR / start / stop) - ├── config.rs — per-game TOML shader-hash config - └── discovery.rs — draw-call logging for shader discovery -``` - ---- - -## License - -TBD +Apache 2.0. See [LICENSE](../../LICENSE). diff --git a/apps/nescope/README.md b/apps/nescope/README.md index 226507ae..42ccbe60 100644 --- a/apps/nescope/README.md +++ b/apps/nescope/README.md @@ -24,7 +24,7 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com │ frame capture ▼ ┌─────────────────┐ -│ vkcapture layer │ ← External Vulkan interception +│ nescapture │ ← Vulkan layer, inside the game's own process └─────────────────┘ ``` @@ -32,15 +32,22 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com ### Core Components -| Module | Purpose | -| ------------------ | -------------------------------------------------------------------------------- | -| `main.rs` | CLI entry point, event loop, process management | -| `state.rs` | `NescopeState` — central compositor state, Wayland globals | -| `handlers.rs` | Smithay protocol handlers (`CompositorHandler`, `XdgShellHandler`, `XwmHandler`) | -| `input.rs` | Programmatic input injection via `calloop::channel` | -| `hdr.rs` | HDR/color management protocol handlers | -| `focus.rs` | Keyboard focus routing for X11 windows | -| `protocols/mod.rs` | Generated gamescope swapchain bindings | +| Module | Purpose | +| -------------------- | -------------------------------------------------------------------------------- | +| `main.rs` | CLI entry point, event loop, process management | +| `state.rs` | `NescopeState` — central compositor state, Wayland globals | +| `handlers.rs` | Smithay protocol handlers (`CompositorHandler`, `XdgShellHandler`) | +| `xwm.rs` | X11 window management | +| `focus.rs` | Keyboard focus routing for X11 windows | +| `input.rs` | Input decoding and injection via `calloop::channel` | +| `input_ipc.rs` | The client side of `neshub`'s input socket | +| `libinput_backend.rs`| Real input devices, when there are any | +| `hdr.rs` | HDR / colour management protocol handlers | +| `gpu_readback.rs` | Reading a dmabuf back to the CPU, for screenshots | +| `screenshot_ipc.rs` | Serving captures to whoever is listening | +| `screenshot_wire.rs` | The screenshot frame format | +| `protocols/` | Generated gamescope swapchain bindings | +| `bin/nescope-shot.rs`| Ask a running nescope what is on screen | ### Design Decisions @@ -50,7 +57,9 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com 3. **Input via Channel**: Input events arrive through a `calloop::channel::Sender` rather than being decoded from a proxy connection. This allows external code (streaming servers, test harnesses) to inject input. -4. **Process Subreaper**: nescope registers as a subreaper (`PR_SET_CHILD_SUBREAPER`) so orphaned descendants (e.g., Steam launcher → game client) are reparented to it, enabling reliable cleanup. +4. **Process Subreaper**: nescope registers as a subreaper (`PR_SET_CHILD_SUBREAPER`) so orphaned descendants — a launcher that execs the real client and exits, say — are reparented to it rather than to PID 1, and can be cleaned up reliably. + +5. **Two shapes, one binary**: given a command after `--`, nescope launches it and exits when it and its windows are gone. Given none, it comes up as a plain compositor and waits for something to connect. The second is what a real session needs, where the processes that draw are started by something else entirely. ## Building @@ -61,16 +70,22 @@ cargo build --release ## Usage ```text -nescope [OPTIONS] -- [args...] +nescope [OPTIONS] [-- [args...]] Options: - --width Output width [default: 1920] - --height Output height [default: 1080] - --fps Virtual refresh rate [default: 60] - --hdr Enable HDR protocols - --socket Wayland socket name [default: nescope-0] + --width Output width [default: 1920] + --height Output height [default: 1080] + --fps Virtual refresh rate [default: 60] + --hdr Enable HDR protocols + --socket Wayland socket name [default: nescope-0] + --input-ipc neshub's input socket [default: /tmp/nestri-input.sock] + --screenshot-ipc Serve screenshots here [default: off] + --render-device GPU to pin the workload to, e.g. /dev/dri/renderD128 + --x-display XWayland display number [default: 1] ``` +The command is optional; see design decision 5 above. + ### Environment Variables | Variable | Effect | @@ -81,18 +96,37 @@ Options: | `NESCOPE_FPS` | Override `--fps` | | `NESCOPE_HDR` | Enable HDR | | `NESCOPE_SOCKET` | Override socket name | +| `NESCOPE_INPUT_IPC` | Override `--input-ipc` | +| `NESCOPE_SCREENSHOT_IPC` | Override `--screenshot-ipc` | +| `NESCOPE_RENDER_DEVICE` | Override `--render-device` | +| `NESCOPE_X_DISPLAY` | Override `--x-display` | | `RUST_LOG` | Tracing filter (e.g., `nescope=debug`) | ### Example ```sh -# Launch a game at 1440p with HDR +# Wrap one program at 1440p with HDR nescope --width 2560 --height 1440 --hdr -- %command% -# 1080p with debug logging +# Plain compositor mode: come up and wait for clients +nescope --x-display 1 + +# Debug logging RUST_LOG=nescope=debug nescope -- %command% ``` +### Seeing what is on screen + +nescope is headless, so a black stream, a window that never mapped and a client +rendering fine all look identical from outside. `nescope-shot` is the listener +side of the screenshot socket: + +```sh +# start this first — nescope dials out +nescope-shot --socket /tmp/nestri-screenshot.sock --watch --out shot.ppm +nescope --screenshot-ipc /tmp/nestri-screenshot.sock -- +``` + ## HDR Support nescope supports two HDR signaling paths: @@ -123,12 +157,14 @@ nescope waits 5 seconds after the last mapped window disappears before exiting, - **clap** — CLI parsing - **x11rb** — X11 atom management -## License +## What nescope does not do -See project repository. +Frames reach the client through [`nescapture`](../nescapture), which captures +inside the workload's process, and [`neshub`](../neshub), which muxes and +sends. Input arrives the same way in reverse. nescope provides the display +those components need and injects what they hand it; it never touches the +network. -## TODO +## Licence -- [ ] Handle sending frames thru to the client -- [ ] Handle packetizing and sharding -- [ ] Handle mouse and kb input from the client +Apache 2.0. See [LICENSE](../../LICENSE). diff --git a/apps/neswire/README.md b/apps/neswire/README.md index b4c86b89..3ad536c0 100644 --- a/apps/neswire/README.md +++ b/apps/neswire/README.md @@ -1,36 +1,81 @@ -## neswire +# neswire -A small custom PipeWire sink for cloud gaming audio capture. +A PipeWire sink that Opus-encodes guest audio and sends it to +[`neshub`](../neshub). -Currently for debugging uses RTP to send Opus (with FEC enabled by default) over to target address. +Audio in the guest has no speakers to reach, so neswire registers itself as an +output device and takes what anything plays into it. From the application's +side it is an ordinary sink. +--- -### Testing +## Running it -#### Mono/Stereo - -Launch gstreamer pipeline to receive audio as so (will save incoming RTP audio into test.mkv): ```bash -gst-launch-1.0 udpsrc port=12345 caps="application/x-rtp,media=audio,encoding-name=OPUS,clock-rate=48000,payload=111" ! rtpopusdepay2 ! opusdec ! matroskamux ! filesink location=test.mkv sync=false +cargo run --release --bin neswire ``` -Then run neswire like so for example: +Then select the **neswire** sink as the system output, or point an application +at it. + +| Flag | Env | Default | Description | +| --- | --- | --- | --- | +| `--ipc-path` | `NESWIRE_IPC_PATH` | `/tmp/nestri-audio.sock` | Where to send Opus packets | +| `--channels` | `NESWIRE_CHANNELS` | `2` | `2` stereo, `6` for 5.1, `8` for 7.1 | +| `--packet-duration-ms` | `NESWIRE_PACKET_DURATION_MS` | `5` | Opus frame size in ms | +| `--bitrate-per-channel` | `NESWIRE_BITRATE_PER_CHANNEL` | `64` | kbps per channel | + +`RUST_LOG` takes a standard tracing filter, e.g. `RUST_LOG=neswire=debug`. + +Sample rate is fixed at 48 kHz, which is what Opus wants and what every +consumer device runs at anyway. + +--- + +## Testing without a hub + +neswire's only output is a Unix datagram socket that `neshub` binds, so running +it on a desktop means it has nothing to talk to — it retries `connect` forever +and there is no way to see what it would have sent. `hub-stub` binds that +socket and reports what arrives: + ```bash -cargo run --release --bin neswire -- --rtp-addr 127.0.0.1:12345 +# terminal 1 +cargo run --bin hub-stub + +# terminal 2 +cargo run --bin neswire ``` -#### Surround +It **decodes** the Opus rather than counting bytes, and that distinction +matters more than it looks. Opus codes digital silence in about two bytes a +packet, so a sink that is receiving nothing but zeros still produces a steady +~3 kbps and looks correct on every meter downstream. Peak amplitude is what +tells a working sink from a silent one. -Launch gstreamer pipeline to receive audio as so (will save incoming RTP audio into test_multi.mkv): ```bash -gst-launch-1.0 udpsrc port=12345 caps='application/x-rtp,media=audio,encoding-name=MULTIOPUS,clock-rate=48000,payload=111,encoding-params=(string)8,num_streams=(string)5,coupled_streams=(string)3,channel_mapping=(string)"0,6,1,2,3,4,5,7"' ! rtpopusdepay2 ! opusdec ! matroskamux ! filesink location=test.mkv sync=false +cargo run --bin hub-stub -- --channels 8 # match neswire's channel count ``` -Then run neswire like so for example: -```bash -cargo run --release --bin neswire -- --rtp-addr 127.0.0.1:12345 --channels 8 +`hub-stub` mirrors `ipc_listener::run_audio_listener` in `neshub`: same bind, +same `0o666`, same stream-type and codec checks. Where the two disagree, +`neshub` is right and the stub should be corrected. + +--- + +## Layout + +``` +src/ +├── main.rs CLI, wiring, the sink→encoder→IPC chain +├── sink.rs the PipeWire sink itself +├── encoder.rs Opus encode, including the multichannel mappings +└── bin/ + └── hub-stub.rs ``` +--- -For both afterwards, set the neswire sink as audio output source in your system (or specify it as output in some app/game), - then play some audio, stop gst pipeline and sink, listen to results after. +## Licence + +Apache 2.0. See [LICENSE](../../LICENSE).