github-mirrors/netris-nestri
1
0
Fork 0
mirror of https://github.com/nestriness/nestri.git synced 2026-09-19 09:15:19 +03:00
Code Issues Packages Projects Releases Wiki Activity

docs: the guest READMEs described code that is gone

Browse Source 
All three came across with their standalone repos and none had been
touched since the transport changed.

nescapture claimed to packetize with Reed-Solomon FEC and stream RTP/UDP
to a Moonlight client. It sends encoded frames to neshub over a Unix
socket. packetizer.rs, control.rs and shard_batch.rs do not exist, and
neither does ARCHITECTURE.md. Nine documented environment variables are
not read by anything -- NESCAPTURE_RTP_HOST was listed as *required* --
and five that are read were undocumented, including the one that says
where frames go. Someone following that quick start would have set a
required variable that does nothing and got no output.

Documented the three sockets, since nescapture binds one and connects to
two and that was written down nowhere.

neswire documented --rtp-addr and --channels against a gstreamer pipeline.
It has --ipc-path, --channels, --packet-duration-ms and
--bitrate-per-channel, and hub-stub exists precisely so it can be tested
without a hub. Kept the reason hub-stub decodes rather than counts bytes:
Opus codes silence at ~3 kbps, so a dead sink looks alive on a meter.

nescope was mostly right. It called the capture layer "vkcapture", listed
seven of fifteen modules and five of nine flags, and its TODO list was
three items that neshub and nescapture now do. Added compositor mode,
which is the shape a real session uses and was undocumented.

All three licence lines were "TBD" or "See project repository".
This commit is contained in:
Wanjohi
2026-08-26 19:09:53 +03:00
parent c103e1257f
commit 6ea241c910
3 changed files with 239 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

263
apps/nescapture/README.md
View File

@@ -1,209 +1,176 @@
# nescapture
A Vulkan implicit layer that captures frames from a running game, encodes them
with **Vulkan Video** hardware acceleration (H.264 / H.265 / AV1), packetizes with
Reed-Solomon FEC, and streams over RTP/UDP to a Moonlight-compatible client —
all with **zero CPU copies** — fully GPU from game rendering to RTP output.
A Vulkan implicit layer that captures frames from inside the workload's own
process, encodes them with **Vulkan Video** on the GPU that drew them, and
sends them to [`neshub`](../neshub) over a Unix socket — no copy out to the
CPU and back.
Being a layer rather than a screen-scraper is the whole point: the frame is
already on the GPU when we get it, and it never leaves.
---
## Architecture
## Where it sits
```
Game process
│ Vulkan calls
▼
┌───────────────────────────────────────────────┐
│ nescapture Vulkan implicit layer │
│ │
│ vkCreateShaderModule → SHA-256 hash │
│ vkCreateGraphicsPipelines → track hashes │
│ vkCmdBindPipeline → detect HUD shaders │
│ vkCmdEndRenderPass/Rendering → inject copy │
│ vkQueuePresentKHR → capture + encode │
└───────────────────────────────────────────────┘
│ GPU blit (same device)
┌──────────────────────────────────────────────┐
│ nescapture implicit layer │
│ │
│ vkCreateShaderModule → SHA-256 hash │
│ vkCreateGraphicsPipelines → track hashes │
│ vkCmdBindPipeline → detect HUD │
│ vkQueuePresentKHR → capture+encode │
└──────────────────────────────────────────────┘
│ GPU blit, same device
▼
final_image (DMA-BUF exportable)
│ get_dmabuf_fd(final_memory)
▼
DmaBufImporter (pixelforge VkDevice)
DmaBufImporter (pixelforge VkDevice)
│ import_or_reuse() → vk::Image
▼
ColorConverter (GPU compute shader)
│ BGRA/RGB10/FP16 → NV12/P010/YUV444
▼
Encoder (Vulkan Video, hardware H.264/H.265)
│ encode()
Encoder (Vulkan Video: H.264 / H.265 / AV1)
│ Annex-B packets
▼
EncodedPacket (Annex-B)
│
▼
Packetizer (Moonlight wire format, Reed-Solomon FEC)
│ UDP datagrams
▼
Moonlight client (or any RTP/UDP receiver)
CPU fallback: only when DMA-BUF export unavailable (rare driver config)
Unix datagram → neshub → the client
```
CPU fallback exists only for driver configurations without DMA-BUF external
memory export.
---
## Sockets
| path | direction | carries |
| --- | --- | --- |
| `/tmp/nestri-video.sock` | nescapture → neshub | encoded frames |
| `/tmp/nestri-stats.sock` | nescapture → neshub | capture fps, encode ms, drops |
| `/tmp/nescapture-cmd.sock` | neshub → nescapture | IDR requests, encode settings |
nescapture binds the command socket and connects to the other two. The stats
path is derived from `NESCAPTURE_IPC_PATH`'s directory, so moving the video
socket moves both.
---
## Quick start
```bash
# 1. Build
cargo build --release
cargo build --release -p nescapture
# 2. Install the layer manifest
sudo cp manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
# Edit the manifest's `library_path` to point at target/release/libnescapture.so
sudo cp apps/nescapture/manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
# Point the manifest's `library_path` at target/release/libnescapture_layer.so
# 3. Configure and launch a game
# 3. Run something that draws
export NESCAPTURE_ENABLE=1
export NESCAPTURE_RTP_HOST=192.168.1.50 # Moonlight / receiver IP
export NESCAPTURE_RTP_PORT=47998 # default
export NESCAPTURE_CODEC=h265 # h264 | h265 | av1 (auto-probes if unset)
export NESCAPTURE_BITRATE=10000 # kbps (CBR; ignored if NESCAPTURE_QP is set)
export NESCAPTURE_FPS=60
export RUST_LOG=info # or NESCAPTURE_LOG=debug
export NESCAPTURE_CODEC=h265
export NESCAPTURE_BITRATE=10000
export RUST_LOG=info
wine MyGame.exe # or native Vulkan game
./my-vulkan-app
```
The layer is inert unless `NESCAPTURE_ENABLE=1`. That is deliberate — an
implicit layer is loaded into *every* Vulkan process on the system.
---
## Environment variables
| Variable | Default | Description |
| ------------------------- | -------------- | ------------------------------------------------------ |
| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer |
| `NESCAPTURE_RTP_HOST` | _(required)_ | Destination IP / hostname for RTP stream |
| `NESCAPTURE_RTP_PORT` | `47998` | Destination UDP port |
| `NESCAPTURE_CODEC` | auto | `h264`, `h265` or `av1` — falls back to h264 if unsupported |
| `NESCAPTURE_BITRATE` | `10000` | CBR target bitrate in kbps |
| `NESCAPTURE_QP` | _(unset)_ | If set, use CQP with this quality level instead of CBR |
| `NESCAPTURE_FPS` | `60` | Target frame rate |
| `NESCAPTURE_IDR_INTERVAL` | `120` | Force an IDR keyframe every N frames |
| `NESCAPTURE_FEC_PCT` | `20` | Reed-Solomon FEC percentage |
| `NESCAPTURE_MIN_FEC` | `2` | Minimum FEC packets per block |
| `NESCAPTURE_PACKET_SIZE` | `1392` | Max UDP payload size (bytes) |
| `NESCAPTURE_CTRL_PORT` | `47999` | UDP port for the control stream |
| `NESCAPTURE_CAPTURE_HUDLESS` | _(unset)_ | Set to `1` to also capture HUDless frames |
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to per-game shader-hash TOML config |
| `NESCAPTURE_GAME_NAME` | (exe basename) | Override game identification |
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to enable discovery mode (logs all draws) |
| `NESCAPTURE_LOG` | `info` | Log level (`error`, `warn`, `info`, `debug`, `trace`) |
| `NESCAPTURE_RTP_FORMAT` | `moonlight` | `standard` for RFC 6184/7798 RTP (GStreamer/FFmpeg compatible), `moonlight` for Moonlight wire format with FEC |
| Variable | Default | Description |
| --- | --- | --- |
| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer. Nothing happens otherwise |
| `NESCAPTURE_IPC_PATH` | `/tmp/nestri-video.sock` | Where to send encoded frames |
| `NESCAPTURE_CODEC` | best available | `h264`, `h265` or `av1`; probes if unset |
| `NESCAPTURE_FORMAT` | `yuv420` | `yuv420` or `yuv444` |
| `NESCAPTURE_DEPTH` | auto | `8` or `10`; inferred from the swapchain `VkFormat` if unset |
| `NESCAPTURE_BITRATE` | `10000` | CBR target in kbps. Ignored when `NESCAPTURE_QP` is set |
| `NESCAPTURE_QP` | _(unset)_ | Constant QP instead of CBR |
| `NESCAPTURE_FPS` | `60` | Target frame rate |
| `NESCAPTURE_IDR_INTERVAL` | `4` | Force an IDR every N **seconds** |
| `NESCAPTURE_TUNE` | _(unset)_ | `highquality`, `lowlatency`, `ultralowlatency`, `lossless` |
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to the per-app shader-hash TOML |
| `NESCAPTURE_GAME_NAME` | exe basename | Override app identification for that config |
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to log every draw, for finding HUD shaders |
| `RUST_LOG` | `error` | Standard `env_logger` filter, e.g. `nescapture_layer=debug` |
Everything else is decided at runtime: the client asks `neshub` for a codec or
bitrate change and it arrives on the command socket, so the encoder is
reconfigured without a restart.
---
## Control stream
## Per-app shader-hash config
Send single-byte UDP datagrams to `NESCAPTURE_CTRL_PORT` (default 47999):
| Byte | Command |
| ------ | -------------------- |
| `0x01` | Start streaming |
| `0x02` | Stop streaming |
| `0x03` | Request IDR keyframe |
Example with netcat:
```bash
# Request IDR
printf '\x03' | nc -u -q1 localhost 47999
```
The control handle wiring in `present.rs` is currently a commented-out stub.
See `control.rs` for the full wiring instructions.
---
## Per-game shader-hash config
HUD shader detection requires a per-game config. Run the game once with
`NESCAPTURE_DISCOVER=1` to log all shaders, then identify HUD pipelines by the
`[SUSPECT]` marker (blend=true, depth=false, ≤6 vertices).
HUD detection needs to know which pipelines draw the HUD, and that is
per-application. Run once with `NESCAPTURE_DISCOVER=1` to log every shader,
then pick out the HUD pipelines by the `[SUSPECT]` marker — blend on, depth
off, six vertices or fewer.
```toml
# ~/.config/nescapture/games.toml
[game."GameName.exe"]
# ~/.config/nescapture/apps.toml
[game."MyApp.exe"]
hud_fragment_shaders = ["0xaabbccddeeff0011"]
hud_vertex_shaders = ["0x1a2b3c4d5e6f7890"]
skip_fragment_shaders = ["0x1122334455667788"]
```
```bash
export NESCAPTURE_CONFIG=~/.config/nescapture/games.toml
export NESCAPTURE_GAME_NAME=GameName.exe
export NESCAPTURE_CONFIG=~/.config/nescapture/apps.toml
export NESCAPTURE_GAME_NAME=MyApp.exe
```
---
## Modules
```
src/
├── lib.rs entry points, dispatch routing
├── dispatch.rs Vulkan function-pointer types and tables
├── state.rs global DashMaps, DeviceState, CbState
├── instance.rs vkCreateInstance / vkDestroyInstance
├── device.rs vkCreateDevice / vkDestroyDevice
├── shader.rs SPIR-V hashing
├── pipeline.rs graphics pipeline tracking
├── framebuffer.rs image view and framebuffer tracking
├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
├── swapchain.rs vkCreateSwapchainKHR, image enumeration
├── capture.rs GPU blit to the capture image, DMA-BUF export
├── present.rs vkQueuePresentKHR, encode dispatch
├── encode.rs pixelforge pipeline, codec probing, IPC send
├── dmabuf_import.rs cross-device zero-copy import
├── config.rs per-app TOML shader-hash config
└── discovery.rs draw-call logging for shader discovery
```
---
## Dependencies
| Crate | Purpose |
| ---------------------- | -------------------------------------------- |
| `pixelforge` | Vulkan Video hardware encode (H.264 / H.265) |
| `ash` | Vulkan bindings |
| `reed-solomon-erasure` | FEC for RTP packetizer |
| `sha2` + `bytemuck` | SPIR-V shader fingerprinting |
| `dashmap` | Lock-free concurrent state maps |
| `serde` + `toml` | Per-game shader config |
| Crate | Purpose |
| --- | --- |
| `pixelforge` | Vulkan Video hardware encode |
| `ash` | Vulkan bindings |
| `nesprotocol` | The IPC frame format `neshub` reads |
| `sha2`, `bytemuck` | SPIR-V fingerprinting |
| `dashmap`, `once_cell` | Lock-free concurrent state |
| `serde`, `toml` | Per-app shader config |
`ash` and `pixelforge` are pinned to git revisions — both track Vulkan Video
support that has not landed in a release.
---
## Zero-copy GPU pipeline
## Licence
The full GPU path is implemented — no CPU color conversion:
1. `final_image` allocated with `DMA_BUF_EXT` external memory (`capture.rs`).
2. `get_dmabuf_fd()` exports the DMA-BUF fd (`capture.rs`).
3. `DmaBufImporter::import_or_reuse()` imports the fd as a `vk::Image` (`dmabuf_import.rs`).
4. `ColorConverter::convert()` runs a GPU compute shader:
BGRA/RGBA/RGB10/FP16 → NV12/P010/YUV444 (`encode.rs`).
5. `Encoder::encode()` encodes from the converter's output directly.
6. RTP packetizer + UDP send.
CPU fallback exists only for rare driver configurations that don't support
DMA-BUF external memory export.
---
## File structure
```
nescapture/
├── Cargo.toml
├── README.md
├── ARCHITECTURE.md
├── manifest/
│ └── VK_LAYER_nescapture.json
└── src/
├── lib.rs — entry points, dispatch routing
├── dispatch.rs — Vulkan function-pointer types + tables
├── state.rs — global DashMaps, DeviceState, CbState
├── instance.rs — vkCreateInstance / vkDestroyInstance
├── device.rs — vkCreateDevice / vkDestroyDevice
├── shader.rs — SPIR-V hashing
├── pipeline.rs — graphics pipeline tracking
├── framebuffer.rs — image view / framebuffer tracking
├── commands.rs — vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
├── swapchain.rs — vkCreateSwapchainKHR, image enumeration
├── capture.rs — GPU blit to capture image, DMA-BUF export
├── present.rs — vkQueuePresentKHR, encode dispatch
├── encode.rs — pixelforge pipeline, codec probing, RTP send
├── dmabuf_import.rs — DmaBufImporter (cross-device zero-copy import)
├── packetizer.rs — Moonlight RTP packetizer (Reed-Solomon FEC)
├── shard_batch.rs — zero-alloc shard buffer
├── control.rs — UDP control stream (IDR / start / stop)
├── config.rs — per-game TOML shader-hash config
└── discovery.rs — draw-call logging for shader discovery
```
---
## License
TBD
Apache 2.0. See [LICENSE](../../LICENSE).