mirror of
https://github.com/nestriness/nestri.git
synced 2026-09-27 04:52:25 +03:00
0811f57f1a
Fixes: #335 Still a work-in-progress. --------- Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Wanjohi <elviswanjohi47@gmail.com>
248 lines
11 KiB
Markdown
248 lines
11 KiB
Markdown
# nescapture
|
||
|
||
A Vulkan implicit layer that captures frames from inside the workload's own
|
||
process, encodes them with **Vulkan Video** on the GPU that drew them, and
|
||
sends them to [`neshub`](../neshub) over a Unix socket — no copy out to the
|
||
CPU and back.
|
||
|
||
Being a layer rather than a screen-scraper is the whole point: the frame is
|
||
already on the GPU when we get it, and it never leaves.
|
||
|
||
---
|
||
|
||
## Where it sits
|
||
|
||
```
|
||
Game process
|
||
│ Vulkan calls
|
||
▼
|
||
┌──────────────────────────────────────────────┐
|
||
│ nescapture implicit layer │
|
||
│ │
|
||
│ vkCreateShaderModule → SHA-256 hash │
|
||
│ vkCreateGraphicsPipelines → track hashes │
|
||
│ vkCmdBindPipeline → detect HUD │
|
||
│ vkQueuePresentKHR → capture+encode │
|
||
└──────────────────────────────────────────────┘
|
||
│ GPU blit on the game's queue, signals a timeline semaphore
|
||
▼
|
||
ring slot (an image on the game's own device)
|
||
│ read in place, once the blit's point is reached
|
||
▼
|
||
ColorConverter (GPU compute shader, the encoder's own queue)
|
||
│ BGRA/RGB10/FP16 → NV12/P010/YUV444
|
||
▼
|
||
Encoder (Vulkan Video: H.264 / H.265 / AV1)
|
||
│ Annex-B packets
|
||
▼
|
||
Unix datagram → neshub → the client
|
||
```
|
||
|
||
The encoder runs on the game's own `VkDevice`. The layer creates that device
|
||
with what the encoder needs: the extensions and feature bits it asks for, and
|
||
queues of its own wherever a queue family has one to spare, since a `VkQueue`
|
||
may not be submitted to from two threads at once. Where no family has room,
|
||
the game's queue is created internally synchronized and shared. Every step of
|
||
a frame is ordered on the GPU; nothing waits on the CPU.
|
||
|
||
Where the conversion is only the YUV matrix and the device has
|
||
`VK_VALVE_video_encode_rgb_conversion`, the encoder takes the RGB frame and
|
||
converts it itself, and the converter is not built. That path runs at limited
|
||
range, since the one driver offering it writes limited range whatever it is
|
||
asked; everything else is full range.
|
||
|
||
When the game's device cannot host the encoder -- an instance the loader
|
||
cannot raise to Vulkan 1.1, no queue the encoder could safely use, a driver
|
||
refusing the additions, or `NESCAPTURE_SHARED_DEVICE=0` -- the encoder gets a
|
||
device of its own, and each frame is read back on the CPU and uploaded there.
|
||
That works everywhere and costs a copy each way.
|
||
|
||
---
|
||
|
||
## Sockets
|
||
|
||
| path | direction | carries |
|
||
| --- | --- | --- |
|
||
| `/tmp/nestri-video.sock` | nescapture → neshub | encoded frames |
|
||
| `/tmp/nestri-stats.sock` | nescapture → neshub | capture fps, encode ms, drops |
|
||
| `/tmp/nescapture-cmd.sock` | neshub → nescapture | IDR requests, encode settings |
|
||
|
||
nescapture binds the command socket and connects to the other two. The stats
|
||
path is derived from `NESCAPTURE_IPC_PATH`'s directory, so moving the video
|
||
socket moves both.
|
||
|
||
---
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
# 1. Build
|
||
cargo build --release -p nescapture
|
||
|
||
# 2. Install the layer manifest
|
||
sudo cp apps/nescapture/manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
|
||
# Point the manifest's `library_path` at target/release/libnescapture_layer.so
|
||
|
||
# 3. Run something that draws
|
||
export NESCAPTURE_ENABLE=1
|
||
export NESCAPTURE_CODEC=h265
|
||
export NESCAPTURE_BITRATE=10000
|
||
export RUST_LOG=info
|
||
|
||
./my-vulkan-app
|
||
```
|
||
|
||
The layer is inert unless `NESCAPTURE_ENABLE=1`. That is deliberate — an
|
||
implicit layer is loaded into *every* Vulkan process on the system.
|
||
|
||
---
|
||
|
||
## Environment variables
|
||
|
||
| Variable | Default | Description |
|
||
| --- | --- | --- |
|
||
| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer. Nothing happens otherwise |
|
||
| `NESCAPTURE_IPC_PATH` | `/tmp/nestri-video.sock` | Where to send encoded frames |
|
||
| `NESCAPTURE_CODEC` | best available | `h264`, `h265` or `av1`; probes if unset |
|
||
| `NESCAPTURE_FORMAT` | `yuv420` | `yuv420` or `yuv444` |
|
||
| `NESCAPTURE_DEPTH` | auto | `8` or `10`; inferred from the swapchain `VkFormat` if unset |
|
||
| `NESCAPTURE_RC` | _(inferred)_ | `cqp`, `cbr` or `vbr`. Unset infers `cqp` when `NESCAPTURE_QP` is set, `cbr` otherwise |
|
||
| `NESCAPTURE_BITRATE` | `10000` | Target bitrate in kbps, under `cbr` and `vbr` |
|
||
| `NESCAPTURE_BITRATE_MAX` | 1.5x the target | VBR ceiling in kbps. Ignored outside `vbr` |
|
||
| `NESCAPTURE_QP` | _(unset)_ | Constant QP, under `cqp` |
|
||
| `NESCAPTURE_FPS` | `60` | Target frame rate |
|
||
| `NESCAPTURE_IDR_INTERVAL` | `4` | Force an IDR every N **seconds** |
|
||
| `NESCAPTURE_INTRA_REFRESH` | _(off)_ | Set to `1` to replace periodic key frames with an intra refresh cycle |
|
||
| `NESCAPTURE_INTRA_REFRESH_QP_DELTA` | `-4` | QP shift inside the refresh band; negative spends bits on it |
|
||
| `NESCAPTURE_INTRA_REFRESH_SHAPE` | auto | `rows`, `columns` or `partitions`; the driver chooses if unset |
|
||
| `NESCAPTURE_TUNE` | _(unset)_ | `highquality`, `lowlatency`, `ultralowlatency`, `lossless` |
|
||
| `NESCAPTURE_SHARED_DEVICE` | _(on)_ | Set to `0` to leave the game's device as the game asked for it, and encode on a device of the encoder's own with CPU readback |
|
||
| `NESCAPTURE_RGB_ENCODE` | _(on)_ | Set to `0` to always convert with the shader, even where the encoder could convert RGB itself |
|
||
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to the per-app shader-hash TOML |
|
||
| `NESCAPTURE_GAME_NAME` | exe basename | Override app identification for that config |
|
||
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to log every draw, for finding HUD shaders |
|
||
| `RUST_LOG` | `error` | Standard `env_logger` filter, e.g. `nescapture_layer=debug` |
|
||
|
||
Everything else is decided at runtime: the client asks `neshub` for a codec or
|
||
bitrate change and it arrives on the command socket, so the encoder is
|
||
reconfigured without a restart.
|
||
|
||
### Intra refresh
|
||
|
||
Instead of a key frame every few seconds, each picture codes one slice of the
|
||
image as intra, so after a full cycle every part has been refreshed. The same
|
||
cost, paid evenly, with no picture much larger than any other — which is what
|
||
a link with a latency budget wants, since a key frame is the largest frame
|
||
there is.
|
||
|
||
**The cycle length is not configurable, deliberately.** It is bounded by how
|
||
many refresh regions the picture actually has, and that depends on the codec's
|
||
block size: at 1080p an H.265 picture is 17 CTB rows tall where an H.264 one is
|
||
68 macroblock rows, so the same duration is comfortable for one codec and
|
||
impossible for the other. The encoder knows the codec, the resolution and what
|
||
the device allows, and derives it from the key frame interval it replaces.
|
||
|
||
The refresh here spreads cost only; it does not make the cycle a recovery
|
||
point. Doing that would restrict prediction on every picture — expensive, and
|
||
what turns the refreshed band into a visible discontinuity — to buy a
|
||
guarantee this stream gets more cheaply from the client asking for an IDR.
|
||
|
||
The band is coded intra every cycle, so it carries none of the refinement its
|
||
neighbours have accumulated and reads as a strip of lower quality sweeping
|
||
across the picture. `NESCAPTURE_INTRA_REFRESH_QP_DELTA` spends bits back into
|
||
it, out of the rest of the frame. Measured at 1080p with ColorVideoVDP, `-4`
|
||
recovers a fifth of what intra refresh costs and the encoded size does not
|
||
grow — but the best value depends on the content, and too large a shift
|
||
starves the rest of the frame faster than too small a one helps. Devices that
|
||
cannot express a negative delta, or whose refresh regions follow the slice
|
||
layout rather than a block sweep, decline it and say so.
|
||
|
||
`NESCAPTURE_INTRA_REFRESH_SHAPE` stays configurable because the device cannot
|
||
answer it: whether a horizontal or vertical sweep looks better depends on how
|
||
the content moves. It also changes how many regions there are — a 1080p
|
||
picture in 64×64 blocks is 17 rows but 30 columns, so `columns` allows a
|
||
longer cycle and thus less intra per picture.
|
||
|
||
### Rate control
|
||
|
||
`cbr` holds every frame to the same size, which is what a link with a fixed
|
||
budget wants. `vbr` holds the same *average* while letting a frame that needs
|
||
it spend up to the ceiling — a scene change is coded rather than smeared, at
|
||
the cost of a burst the path has to absorb. `cqp` holds quality constant and
|
||
lets the bitrate go wherever the content takes it, which is a recording
|
||
setting rather than a streaming one.
|
||
|
||
The command socket carries a target bitrate but has no way to name a mode
|
||
beyond CBR and constant QP. A target arriving while the encode is `vbr` is
|
||
therefore applied as a target, leaving the mode and the ceiling alone — so the
|
||
ceiling asked for at launch survives a session, and a congestion controller
|
||
adjusts underneath it. Retargeting costs no rebuild and no key frame.
|
||
|
||
---
|
||
|
||
## Per-app shader-hash config
|
||
|
||
HUD detection needs to know which pipelines draw the HUD, and that is
|
||
per-application. Run once with `NESCAPTURE_DISCOVER=1` to log every shader,
|
||
then pick out the HUD pipelines by the `[SUSPECT]` marker — blend on, depth
|
||
off, six vertices or fewer.
|
||
|
||
```toml
|
||
# ~/.config/nescapture/apps.toml
|
||
[game."MyApp.exe"]
|
||
hud_fragment_shaders = ["0xaabbccddeeff0011"]
|
||
hud_vertex_shaders = ["0x1a2b3c4d5e6f7890"]
|
||
skip_fragment_shaders = ["0x1122334455667788"]
|
||
```
|
||
|
||
```bash
|
||
export NESCAPTURE_CONFIG=~/.config/nescapture/apps.toml
|
||
export NESCAPTURE_GAME_NAME=MyApp.exe
|
||
```
|
||
|
||
---
|
||
|
||
## Modules
|
||
|
||
```
|
||
src/
|
||
├── lib.rs entry points, dispatch routing
|
||
├── dispatch.rs Vulkan function-pointer types and tables
|
||
├── state.rs global DashMaps, DeviceState, CbState
|
||
├── instance.rs vkCreateInstance / vkDestroyInstance
|
||
├── device.rs vkCreateDevice / vkDestroyDevice
|
||
├── shader.rs SPIR-V hashing
|
||
├── pipeline.rs graphics pipeline tracking
|
||
├── framebuffer.rs image view and framebuffer tracking
|
||
├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
|
||
├── swapchain.rs vkCreateSwapchainKHR, image enumeration
|
||
├── capture.rs GPU blit into the capture ring, CPU readback fallback
|
||
├── present.rs vkQueuePresentKHR, encode dispatch
|
||
├── encode.rs pixelforge pipeline, codec probing, IPC send
|
||
├── shared.rs creating the game's device for the encoder to share
|
||
├── config.rs per-app TOML shader-hash config
|
||
└── discovery.rs draw-call logging for shader discovery
|
||
```
|
||
|
||
---
|
||
|
||
## Dependencies
|
||
|
||
| Crate | Purpose |
|
||
| --- | --- |
|
||
| `pixelforge` | Vulkan Video hardware encode |
|
||
| `ash` | Vulkan bindings |
|
||
| `nesprotocol` | The IPC frame format `neshub` reads |
|
||
| `sha2`, `bytemuck` | SPIR-V fingerprinting |
|
||
| `dashmap`, `once_cell` | Lock-free concurrent state |
|
||
| `serde`, `toml` | Per-app shader config |
|
||
|
||
`ash` and `pixelforge` are pinned to git revisions — both track Vulkan Video
|
||
support that has not landed in a release.
|
||
|
||
---
|
||
|
||
## Licence
|
||
|
||
Apache 2.0. See [LICENSE](../../LICENSE).
|