github-mirrors/netris-nestri
1
0
Fork 0
mirror of https://github.com/nestriness/nestri.git synced 2026-09-27 21:12:25 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat: media bitrate control, HDR (#346)

Browse Source 
Fixes: #335 

Still a work-in-progress.

---------

Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Wanjohi <elviswanjohi47@gmail.com>
This commit is contained in:
Kristian Ollikainen
2026-09-25 12:13:34 +03:00
committed by GitHub
co-authored by DatCaptainHorse Claude Opus 5 Wanjohi
parent 1c721962f4
commit 0811f57f1a
64 changed files with 15151 additions and 2702 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/nescapture/README.md
+84 -13
View File
@@ -24,15 +24,12 @@ Game process
│ vkCmdBindPipeline → detect HUD │
│ vkQueuePresentKHR → capture+encode │
└──────────────────────────────────────────────┘
│ GPU blit, same device
│ GPU blit on the game's queue, signals a timeline semaphore
▼
final_image (DMA-BUF exportable)
│ get_dmabuf_fd(final_memory)
ring slot (an image on the game's own device)
│ read in place, once the blit's point is reached
▼
DmaBufImporter (pixelforge VkDevice)
│ import_or_reuse() → vk::Image
▼
ColorConverter (GPU compute shader)
ColorConverter (GPU compute shader, the encoder's own queue)
│ BGRA/RGB10/FP16 → NV12/P010/YUV444
▼
Encoder (Vulkan Video: H.264 / H.265 / AV1)
@@ -41,8 +38,24 @@ Encoder (Vulkan Video: H.264 / H.265 / AV1)
Unix datagram → neshub → the client
```
CPU fallback exists only for driver configurations without DMA-BUF external
memory export.
The encoder runs on the game's own `VkDevice`. The layer creates that device
with what the encoder needs: the extensions and feature bits it asks for, and
queues of its own wherever a queue family has one to spare, since a `VkQueue`
may not be submitted to from two threads at once. Where no family has room,
the game's queue is created internally synchronized and shared. Every step of
a frame is ordered on the GPU; nothing waits on the CPU.
Where the conversion is only the YUV matrix and the device has
`VK_VALVE_video_encode_rgb_conversion`, the encoder takes the RGB frame and
converts it itself, and the converter is not built. That path runs at limited
range, since the one driver offering it writes limited range whatever it is
asked; everything else is full range.
When the game's device cannot host the encoder -- an instance the loader
cannot raise to Vulkan 1.1, no queue the encoder could safely use, a driver
refusing the additions, or `NESCAPTURE_SHARED_DEVICE=0` -- the encoder gets a
device of its own, and each frame is read back on the CPU and uploaded there.
That works everywhere and costs a copy each way.
---
@@ -93,11 +106,18 @@ implicit layer is loaded into *every* Vulkan process on the system.
| `NESCAPTURE_CODEC` | best available | `h264`, `h265` or `av1`; probes if unset |
| `NESCAPTURE_FORMAT` | `yuv420` | `yuv420` or `yuv444` |
| `NESCAPTURE_DEPTH` | auto | `8` or `10`; inferred from the swapchain `VkFormat` if unset |
| `NESCAPTURE_BITRATE` | `10000` | CBR target in kbps. Ignored when `NESCAPTURE_QP` is set |
| `NESCAPTURE_QP` | _(unset)_ | Constant QP instead of CBR |
| `NESCAPTURE_RC` | _(inferred)_ | `cqp`, `cbr` or `vbr`. Unset infers `cqp` when `NESCAPTURE_QP` is set, `cbr` otherwise |
| `NESCAPTURE_BITRATE` | `10000` | Target bitrate in kbps, under `cbr` and `vbr` |
| `NESCAPTURE_BITRATE_MAX` | 1.5x the target | VBR ceiling in kbps. Ignored outside `vbr` |
| `NESCAPTURE_QP` | _(unset)_ | Constant QP, under `cqp` |
| `NESCAPTURE_FPS` | `60` | Target frame rate |
| `NESCAPTURE_IDR_INTERVAL` | `4` | Force an IDR every N **seconds** |
| `NESCAPTURE_INTRA_REFRESH` | _(off)_ | Set to `1` to replace periodic key frames with an intra refresh cycle |
| `NESCAPTURE_INTRA_REFRESH_QP_DELTA` | `-4` | QP shift inside the refresh band; negative spends bits on it |
| `NESCAPTURE_INTRA_REFRESH_SHAPE` | auto | `rows`, `columns` or `partitions`; the driver chooses if unset |
| `NESCAPTURE_TUNE` | _(unset)_ | `highquality`, `lowlatency`, `ultralowlatency`, `lossless` |
| `NESCAPTURE_SHARED_DEVICE` | _(on)_ | Set to `0` to leave the game's device as the game asked for it, and encode on a device of the encoder's own with CPU readback |
| `NESCAPTURE_RGB_ENCODE` | _(on)_ | Set to `0` to always convert with the shader, even where the encoder could convert RGB itself |
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to the per-app shader-hash TOML |
| `NESCAPTURE_GAME_NAME` | exe basename | Override app identification for that config |
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to log every draw, for finding HUD shaders |
@@ -107,6 +127,57 @@ Everything else is decided at runtime: the client asks `neshub` for a codec or
bitrate change and it arrives on the command socket, so the encoder is
reconfigured without a restart.
### Intra refresh
Instead of a key frame every few seconds, each picture codes one slice of the
image as intra, so after a full cycle every part has been refreshed. The same
cost, paid evenly, with no picture much larger than any other — which is what
a link with a latency budget wants, since a key frame is the largest frame
there is.
**The cycle length is not configurable, deliberately.** It is bounded by how
many refresh regions the picture actually has, and that depends on the codec's
block size: at 1080p an H.265 picture is 17 CTB rows tall where an H.264 one is
68 macroblock rows, so the same duration is comfortable for one codec and
impossible for the other. The encoder knows the codec, the resolution and what
the device allows, and derives it from the key frame interval it replaces.
The refresh here spreads cost only; it does not make the cycle a recovery
point. Doing that would restrict prediction on every picture — expensive, and
what turns the refreshed band into a visible discontinuity — to buy a
guarantee this stream gets more cheaply from the client asking for an IDR.
The band is coded intra every cycle, so it carries none of the refinement its
neighbours have accumulated and reads as a strip of lower quality sweeping
across the picture. `NESCAPTURE_INTRA_REFRESH_QP_DELTA` spends bits back into
it, out of the rest of the frame. Measured at 1080p with ColorVideoVDP, `-4`
recovers a fifth of what intra refresh costs and the encoded size does not
grow — but the best value depends on the content, and too large a shift
starves the rest of the frame faster than too small a one helps. Devices that
cannot express a negative delta, or whose refresh regions follow the slice
layout rather than a block sweep, decline it and say so.
`NESCAPTURE_INTRA_REFRESH_SHAPE` stays configurable because the device cannot
answer it: whether a horizontal or vertical sweep looks better depends on how
the content moves. It also changes how many regions there are — a 1080p
picture in 64×64 blocks is 17 rows but 30 columns, so `columns` allows a
longer cycle and thus less intra per picture.
### Rate control
`cbr` holds every frame to the same size, which is what a link with a fixed
budget wants. `vbr` holds the same *average* while letting a frame that needs
it spend up to the ceiling — a scene change is coded rather than smeared, at
the cost of a burst the path has to absorb. `cqp` holds quality constant and
lets the bitrate go wherever the content takes it, which is a recording
setting rather than a streaming one.
The command socket carries a target bitrate but has no way to name a mode
beyond CBR and constant QP. A target arriving while the encode is `vbr` is
therefore applied as a target, leaving the mode and the ceiling alone — so the
ceiling asked for at launch survives a session, and a congestion controller
adjusts underneath it. Retargeting costs no rebuild and no key frame.
---
## Per-app shader-hash config
@@ -145,10 +216,10 @@ src/
├── framebuffer.rs image view and framebuffer tracking
├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
├── swapchain.rs vkCreateSwapchainKHR, image enumeration
├── capture.rs GPU blit to the capture image, DMA-BUF export
├── capture.rs GPU blit into the capture ring, CPU readback fallback
├── present.rs vkQueuePresentKHR, encode dispatch
├── encode.rs pixelforge pipeline, codec probing, IPC send
├── dmabuf_import.rs cross-device zero-copy import
├── shared.rs creating the game's device for the encoder to share
├── config.rs per-app TOML shader-hash config
└── discovery.rs draw-call logging for shader discovery
```