mirror of
https://github.com/nestriness/nestri.git
synced 2026-09-19 09:15:19 +03:00
docs: the guest READMEs described code that is gone
All three came across with their standalone repos and none had been touched since the transport changed. nescapture claimed to packetize with Reed-Solomon FEC and stream RTP/UDP to a Moonlight client. It sends encoded frames to neshub over a Unix socket. packetizer.rs, control.rs and shard_batch.rs do not exist, and neither does ARCHITECTURE.md. Nine documented environment variables are not read by anything -- NESCAPTURE_RTP_HOST was listed as *required* -- and five that are read were undocumented, including the one that says where frames go. Someone following that quick start would have set a required variable that does nothing and got no output. Documented the three sockets, since nescapture binds one and connects to two and that was written down nowhere. neswire documented --rtp-addr and --channels against a gstreamer pipeline. It has --ipc-path, --channels, --packet-duration-ms and --bitrate-per-channel, and hub-stub exists precisely so it can be tested without a hub. Kept the reason hub-stub decodes rather than counts bytes: Opus codes silence at ~3 kbps, so a dead sink looks alive on a meter. nescope was mostly right. It called the capture layer "vkcapture", listed seven of fifteen modules and five of nine flags, and its TODO list was three items that neshub and nescapture now do. Added compositor mode, which is the shape a real session uses and was undocumented. All three licence lines were "TBD" or "See project repository".
This commit is contained in:
@@ -1,209 +1,176 @@
|
||||
# nescapture
|
||||
|
||||
A Vulkan implicit layer that captures frames from a running game, encodes them
|
||||
with **Vulkan Video** hardware acceleration (H.264 / H.265 / AV1), packetizes with
|
||||
Reed-Solomon FEC, and streams over RTP/UDP to a Moonlight-compatible client —
|
||||
all with **zero CPU copies** — fully GPU from game rendering to RTP output.
|
||||
A Vulkan implicit layer that captures frames from inside the workload's own
|
||||
process, encodes them with **Vulkan Video** on the GPU that drew them, and
|
||||
sends them to [`neshub`](../neshub) over a Unix socket — no copy out to the
|
||||
CPU and back.
|
||||
|
||||
Being a layer rather than a screen-scraper is the whole point: the frame is
|
||||
already on the GPU when we get it, and it never leaves.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
## Where it sits
|
||||
|
||||
```
|
||||
Game process
|
||||
│ Vulkan calls
|
||||
▼
|
||||
┌───────────────────────────────────────────────┐
|
||||
│ nescapture Vulkan implicit layer │
|
||||
│ │
|
||||
│ vkCreateShaderModule → SHA-256 hash │
|
||||
│ vkCreateGraphicsPipelines → track hashes │
|
||||
│ vkCmdBindPipeline → detect HUD shaders │
|
||||
│ vkCmdEndRenderPass/Rendering → inject copy │
|
||||
│ vkQueuePresentKHR → capture + encode │
|
||||
└───────────────────────────────────────────────┘
|
||||
│ GPU blit (same device)
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ nescapture implicit layer │
|
||||
│ │
|
||||
│ vkCreateShaderModule → SHA-256 hash │
|
||||
│ vkCreateGraphicsPipelines → track hashes │
|
||||
│ vkCmdBindPipeline → detect HUD │
|
||||
│ vkQueuePresentKHR → capture+encode │
|
||||
└──────────────────────────────────────────────┘
|
||||
│ GPU blit, same device
|
||||
▼
|
||||
final_image (DMA-BUF exportable)
|
||||
│ get_dmabuf_fd(final_memory)
|
||||
▼
|
||||
DmaBufImporter (pixelforge VkDevice)
|
||||
DmaBufImporter (pixelforge VkDevice)
|
||||
│ import_or_reuse() → vk::Image
|
||||
▼
|
||||
ColorConverter (GPU compute shader)
|
||||
│ BGRA/RGB10/FP16 → NV12/P010/YUV444
|
||||
▼
|
||||
Encoder (Vulkan Video, hardware H.264/H.265)
|
||||
│ encode()
|
||||
Encoder (Vulkan Video: H.264 / H.265 / AV1)
|
||||
│ Annex-B packets
|
||||
▼
|
||||
EncodedPacket (Annex-B)
|
||||
│
|
||||
▼
|
||||
Packetizer (Moonlight wire format, Reed-Solomon FEC)
|
||||
│ UDP datagrams
|
||||
▼
|
||||
Moonlight client (or any RTP/UDP receiver)
|
||||
|
||||
CPU fallback: only when DMA-BUF export unavailable (rare driver config)
|
||||
Unix datagram → neshub → the client
|
||||
```
|
||||
|
||||
CPU fallback exists only for driver configurations without DMA-BUF external
|
||||
memory export.
|
||||
|
||||
---
|
||||
|
||||
## Sockets
|
||||
|
||||
| path | direction | carries |
|
||||
| --- | --- | --- |
|
||||
| `/tmp/nestri-video.sock` | nescapture → neshub | encoded frames |
|
||||
| `/tmp/nestri-stats.sock` | nescapture → neshub | capture fps, encode ms, drops |
|
||||
| `/tmp/nescapture-cmd.sock` | neshub → nescapture | IDR requests, encode settings |
|
||||
|
||||
nescapture binds the command socket and connects to the other two. The stats
|
||||
path is derived from `NESCAPTURE_IPC_PATH`'s directory, so moving the video
|
||||
socket moves both.
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# 1. Build
|
||||
cargo build --release
|
||||
cargo build --release -p nescapture
|
||||
|
||||
# 2. Install the layer manifest
|
||||
sudo cp manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
|
||||
# Edit the manifest's `library_path` to point at target/release/libnescapture.so
|
||||
sudo cp apps/nescapture/manifest/VK_LAYER_nescapture.json /usr/share/vulkan/implicit_layer.d/
|
||||
# Point the manifest's `library_path` at target/release/libnescapture_layer.so
|
||||
|
||||
# 3. Configure and launch a game
|
||||
# 3. Run something that draws
|
||||
export NESCAPTURE_ENABLE=1
|
||||
export NESCAPTURE_RTP_HOST=192.168.1.50 # Moonlight / receiver IP
|
||||
export NESCAPTURE_RTP_PORT=47998 # default
|
||||
export NESCAPTURE_CODEC=h265 # h264 | h265 | av1 (auto-probes if unset)
|
||||
export NESCAPTURE_BITRATE=10000 # kbps (CBR; ignored if NESCAPTURE_QP is set)
|
||||
export NESCAPTURE_FPS=60
|
||||
export RUST_LOG=info # or NESCAPTURE_LOG=debug
|
||||
export NESCAPTURE_CODEC=h265
|
||||
export NESCAPTURE_BITRATE=10000
|
||||
export RUST_LOG=info
|
||||
|
||||
wine MyGame.exe # or native Vulkan game
|
||||
./my-vulkan-app
|
||||
```
|
||||
|
||||
The layer is inert unless `NESCAPTURE_ENABLE=1`. That is deliberate — an
|
||||
implicit layer is loaded into *every* Vulkan process on the system.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| ------------------------- | -------------- | ------------------------------------------------------ |
|
||||
| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer |
|
||||
| `NESCAPTURE_RTP_HOST` | _(required)_ | Destination IP / hostname for RTP stream |
|
||||
| `NESCAPTURE_RTP_PORT` | `47998` | Destination UDP port |
|
||||
| `NESCAPTURE_CODEC` | auto | `h264`, `h265` or `av1` — falls back to h264 if unsupported |
|
||||
| `NESCAPTURE_BITRATE` | `10000` | CBR target bitrate in kbps |
|
||||
| `NESCAPTURE_QP` | _(unset)_ | If set, use CQP with this quality level instead of CBR |
|
||||
| `NESCAPTURE_FPS` | `60` | Target frame rate |
|
||||
| `NESCAPTURE_IDR_INTERVAL` | `120` | Force an IDR keyframe every N frames |
|
||||
| `NESCAPTURE_FEC_PCT` | `20` | Reed-Solomon FEC percentage |
|
||||
| `NESCAPTURE_MIN_FEC` | `2` | Minimum FEC packets per block |
|
||||
| `NESCAPTURE_PACKET_SIZE` | `1392` | Max UDP payload size (bytes) |
|
||||
| `NESCAPTURE_CTRL_PORT` | `47999` | UDP port for the control stream |
|
||||
| `NESCAPTURE_CAPTURE_HUDLESS` | _(unset)_ | Set to `1` to also capture HUDless frames |
|
||||
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to per-game shader-hash TOML config |
|
||||
| `NESCAPTURE_GAME_NAME` | (exe basename) | Override game identification |
|
||||
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to enable discovery mode (logs all draws) |
|
||||
| `NESCAPTURE_LOG` | `info` | Log level (`error`, `warn`, `info`, `debug`, `trace`) |
|
||||
| `NESCAPTURE_RTP_FORMAT` | `moonlight` | `standard` for RFC 6184/7798 RTP (GStreamer/FFmpeg compatible), `moonlight` for Moonlight wire format with FEC |
|
||||
| Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `NESCAPTURE_ENABLE` | _(unset)_ | Set to `1` to activate the layer. Nothing happens otherwise |
|
||||
| `NESCAPTURE_IPC_PATH` | `/tmp/nestri-video.sock` | Where to send encoded frames |
|
||||
| `NESCAPTURE_CODEC` | best available | `h264`, `h265` or `av1`; probes if unset |
|
||||
| `NESCAPTURE_FORMAT` | `yuv420` | `yuv420` or `yuv444` |
|
||||
| `NESCAPTURE_DEPTH` | auto | `8` or `10`; inferred from the swapchain `VkFormat` if unset |
|
||||
| `NESCAPTURE_BITRATE` | `10000` | CBR target in kbps. Ignored when `NESCAPTURE_QP` is set |
|
||||
| `NESCAPTURE_QP` | _(unset)_ | Constant QP instead of CBR |
|
||||
| `NESCAPTURE_FPS` | `60` | Target frame rate |
|
||||
| `NESCAPTURE_IDR_INTERVAL` | `4` | Force an IDR every N **seconds** |
|
||||
| `NESCAPTURE_TUNE` | _(unset)_ | `highquality`, `lowlatency`, `ultralowlatency`, `lossless` |
|
||||
| `NESCAPTURE_CONFIG` | _(unset)_ | Path to the per-app shader-hash TOML |
|
||||
| `NESCAPTURE_GAME_NAME` | exe basename | Override app identification for that config |
|
||||
| `NESCAPTURE_DISCOVER` | _(unset)_ | Set to `1` to log every draw, for finding HUD shaders |
|
||||
| `RUST_LOG` | `error` | Standard `env_logger` filter, e.g. `nescapture_layer=debug` |
|
||||
|
||||
Everything else is decided at runtime: the client asks `neshub` for a codec or
|
||||
bitrate change and it arrives on the command socket, so the encoder is
|
||||
reconfigured without a restart.
|
||||
|
||||
---
|
||||
|
||||
## Control stream
|
||||
## Per-app shader-hash config
|
||||
|
||||
Send single-byte UDP datagrams to `NESCAPTURE_CTRL_PORT` (default 47999):
|
||||
|
||||
| Byte | Command |
|
||||
| ------ | -------------------- |
|
||||
| `0x01` | Start streaming |
|
||||
| `0x02` | Stop streaming |
|
||||
| `0x03` | Request IDR keyframe |
|
||||
|
||||
Example with netcat:
|
||||
|
||||
```bash
|
||||
# Request IDR
|
||||
printf '\x03' | nc -u -q1 localhost 47999
|
||||
```
|
||||
|
||||
The control handle wiring in `present.rs` is currently a commented-out stub.
|
||||
See `control.rs` for the full wiring instructions.
|
||||
|
||||
---
|
||||
|
||||
## Per-game shader-hash config
|
||||
|
||||
HUD shader detection requires a per-game config. Run the game once with
|
||||
`NESCAPTURE_DISCOVER=1` to log all shaders, then identify HUD pipelines by the
|
||||
`[SUSPECT]` marker (blend=true, depth=false, ≤6 vertices).
|
||||
HUD detection needs to know which pipelines draw the HUD, and that is
|
||||
per-application. Run once with `NESCAPTURE_DISCOVER=1` to log every shader,
|
||||
then pick out the HUD pipelines by the `[SUSPECT]` marker — blend on, depth
|
||||
off, six vertices or fewer.
|
||||
|
||||
```toml
|
||||
# ~/.config/nescapture/games.toml
|
||||
[game."GameName.exe"]
|
||||
# ~/.config/nescapture/apps.toml
|
||||
[game."MyApp.exe"]
|
||||
hud_fragment_shaders = ["0xaabbccddeeff0011"]
|
||||
hud_vertex_shaders = ["0x1a2b3c4d5e6f7890"]
|
||||
skip_fragment_shaders = ["0x1122334455667788"]
|
||||
```
|
||||
|
||||
```bash
|
||||
export NESCAPTURE_CONFIG=~/.config/nescapture/games.toml
|
||||
export NESCAPTURE_GAME_NAME=GameName.exe
|
||||
export NESCAPTURE_CONFIG=~/.config/nescapture/apps.toml
|
||||
export NESCAPTURE_GAME_NAME=MyApp.exe
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Modules
|
||||
|
||||
```
|
||||
src/
|
||||
├── lib.rs entry points, dispatch routing
|
||||
├── dispatch.rs Vulkan function-pointer types and tables
|
||||
├── state.rs global DashMaps, DeviceState, CbState
|
||||
├── instance.rs vkCreateInstance / vkDestroyInstance
|
||||
├── device.rs vkCreateDevice / vkDestroyDevice
|
||||
├── shader.rs SPIR-V hashing
|
||||
├── pipeline.rs graphics pipeline tracking
|
||||
├── framebuffer.rs image view and framebuffer tracking
|
||||
├── commands.rs vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
|
||||
├── swapchain.rs vkCreateSwapchainKHR, image enumeration
|
||||
├── capture.rs GPU blit to the capture image, DMA-BUF export
|
||||
├── present.rs vkQueuePresentKHR, encode dispatch
|
||||
├── encode.rs pixelforge pipeline, codec probing, IPC send
|
||||
├── dmabuf_import.rs cross-device zero-copy import
|
||||
├── config.rs per-app TOML shader-hash config
|
||||
└── discovery.rs draw-call logging for shader discovery
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Crate | Purpose |
|
||||
| ---------------------- | -------------------------------------------- |
|
||||
| `pixelforge` | Vulkan Video hardware encode (H.264 / H.265) |
|
||||
| `ash` | Vulkan bindings |
|
||||
| `reed-solomon-erasure` | FEC for RTP packetizer |
|
||||
| `sha2` + `bytemuck` | SPIR-V shader fingerprinting |
|
||||
| `dashmap` | Lock-free concurrent state maps |
|
||||
| `serde` + `toml` | Per-game shader config |
|
||||
| Crate | Purpose |
|
||||
| --- | --- |
|
||||
| `pixelforge` | Vulkan Video hardware encode |
|
||||
| `ash` | Vulkan bindings |
|
||||
| `nesprotocol` | The IPC frame format `neshub` reads |
|
||||
| `sha2`, `bytemuck` | SPIR-V fingerprinting |
|
||||
| `dashmap`, `once_cell` | Lock-free concurrent state |
|
||||
| `serde`, `toml` | Per-app shader config |
|
||||
|
||||
`ash` and `pixelforge` are pinned to git revisions — both track Vulkan Video
|
||||
support that has not landed in a release.
|
||||
|
||||
---
|
||||
|
||||
## Zero-copy GPU pipeline
|
||||
## Licence
|
||||
|
||||
The full GPU path is implemented — no CPU color conversion:
|
||||
|
||||
1. `final_image` allocated with `DMA_BUF_EXT` external memory (`capture.rs`).
|
||||
2. `get_dmabuf_fd()` exports the DMA-BUF fd (`capture.rs`).
|
||||
3. `DmaBufImporter::import_or_reuse()` imports the fd as a `vk::Image` (`dmabuf_import.rs`).
|
||||
4. `ColorConverter::convert()` runs a GPU compute shader:
|
||||
BGRA/RGBA/RGB10/FP16 → NV12/P010/YUV444 (`encode.rs`).
|
||||
5. `Encoder::encode()` encodes from the converter's output directly.
|
||||
6. RTP packetizer + UDP send.
|
||||
|
||||
CPU fallback exists only for rare driver configurations that don't support
|
||||
DMA-BUF external memory export.
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
nescapture/
|
||||
├── Cargo.toml
|
||||
├── README.md
|
||||
├── ARCHITECTURE.md
|
||||
├── manifest/
|
||||
│ └── VK_LAYER_nescapture.json
|
||||
└── src/
|
||||
├── lib.rs — entry points, dispatch routing
|
||||
├── dispatch.rs — Vulkan function-pointer types + tables
|
||||
├── state.rs — global DashMaps, DeviceState, CbState
|
||||
├── instance.rs — vkCreateInstance / vkDestroyInstance
|
||||
├── device.rs — vkCreateDevice / vkDestroyDevice
|
||||
├── shader.rs — SPIR-V hashing
|
||||
├── pipeline.rs — graphics pipeline tracking
|
||||
├── framebuffer.rs — image view / framebuffer tracking
|
||||
├── commands.rs — vkCmdBind*, vkCmdDraw*, vkCmdBeginRenderPass
|
||||
├── swapchain.rs — vkCreateSwapchainKHR, image enumeration
|
||||
├── capture.rs — GPU blit to capture image, DMA-BUF export
|
||||
├── present.rs — vkQueuePresentKHR, encode dispatch
|
||||
├── encode.rs — pixelforge pipeline, codec probing, RTP send
|
||||
├── dmabuf_import.rs — DmaBufImporter (cross-device zero-copy import)
|
||||
├── packetizer.rs — Moonlight RTP packetizer (Reed-Solomon FEC)
|
||||
├── shard_batch.rs — zero-alloc shard buffer
|
||||
├── control.rs — UDP control stream (IDR / start / stop)
|
||||
├── config.rs — per-game TOML shader-hash config
|
||||
└── discovery.rs — draw-call logging for shader discovery
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
TBD
|
||||
Apache 2.0. See [LICENSE](../../LICENSE).
|
||||
|
||||
@@ -24,7 +24,7 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com
|
||||
│ frame capture
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ vkcapture layer │ ← External Vulkan interception
|
||||
│ nescapture │ ← Vulkan layer, inside the game's own process
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
@@ -32,15 +32,22 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com
|
||||
|
||||
### Core Components
|
||||
|
||||
| Module | Purpose |
|
||||
| ------------------ | -------------------------------------------------------------------------------- |
|
||||
| `main.rs` | CLI entry point, event loop, process management |
|
||||
| `state.rs` | `NescopeState` — central compositor state, Wayland globals |
|
||||
| `handlers.rs` | Smithay protocol handlers (`CompositorHandler`, `XdgShellHandler`, `XwmHandler`) |
|
||||
| `input.rs` | Programmatic input injection via `calloop::channel` |
|
||||
| `hdr.rs` | HDR/color management protocol handlers |
|
||||
| `focus.rs` | Keyboard focus routing for X11 windows |
|
||||
| `protocols/mod.rs` | Generated gamescope swapchain bindings |
|
||||
| Module | Purpose |
|
||||
| -------------------- | -------------------------------------------------------------------------------- |
|
||||
| `main.rs` | CLI entry point, event loop, process management |
|
||||
| `state.rs` | `NescopeState` — central compositor state, Wayland globals |
|
||||
| `handlers.rs` | Smithay protocol handlers (`CompositorHandler`, `XdgShellHandler`) |
|
||||
| `xwm.rs` | X11 window management |
|
||||
| `focus.rs` | Keyboard focus routing for X11 windows |
|
||||
| `input.rs` | Input decoding and injection via `calloop::channel` |
|
||||
| `input_ipc.rs` | The client side of `neshub`'s input socket |
|
||||
| `libinput_backend.rs`| Real input devices, when there are any |
|
||||
| `hdr.rs` | HDR / colour management protocol handlers |
|
||||
| `gpu_readback.rs` | Reading a dmabuf back to the CPU, for screenshots |
|
||||
| `screenshot_ipc.rs` | Serving captures to whoever is listening |
|
||||
| `screenshot_wire.rs` | The screenshot frame format |
|
||||
| `protocols/` | Generated gamescope swapchain bindings |
|
||||
| `bin/nescope-shot.rs`| Ask a running nescope what is on screen |
|
||||
|
||||
### Design Decisions
|
||||
|
||||
@@ -50,7 +57,9 @@ nescope is built on top of [Smithay](https://smithay.org/), the Rust Wayland com
|
||||
|
||||
3. **Input via Channel**: Input events arrive through a `calloop::channel::Sender<InputEvent>` rather than being decoded from a proxy connection. This allows external code (streaming servers, test harnesses) to inject input.
|
||||
|
||||
4. **Process Subreaper**: nescope registers as a subreaper (`PR_SET_CHILD_SUBREAPER`) so orphaned descendants (e.g., Steam launcher → game client) are reparented to it, enabling reliable cleanup.
|
||||
4. **Process Subreaper**: nescope registers as a subreaper (`PR_SET_CHILD_SUBREAPER`) so orphaned descendants — a launcher that execs the real client and exits, say — are reparented to it rather than to PID 1, and can be cleaned up reliably.
|
||||
|
||||
5. **Two shapes, one binary**: given a command after `--`, nescope launches it and exits when it and its windows are gone. Given none, it comes up as a plain compositor and waits for something to connect. The second is what a real session needs, where the processes that draw are started by something else entirely.
|
||||
|
||||
## Building
|
||||
|
||||
@@ -61,16 +70,22 @@ cargo build --release
|
||||
## Usage
|
||||
|
||||
```text
|
||||
nescope [OPTIONS] -- <command> [args...]
|
||||
nescope [OPTIONS] [-- <command> [args...]]
|
||||
|
||||
Options:
|
||||
--width <N> Output width [default: 1920]
|
||||
--height <N> Output height [default: 1080]
|
||||
--fps <N> Virtual refresh rate [default: 60]
|
||||
--hdr Enable HDR protocols
|
||||
--socket <NAME> Wayland socket name [default: nescope-0]
|
||||
--width <N> Output width [default: 1920]
|
||||
--height <N> Output height [default: 1080]
|
||||
--fps <N> Virtual refresh rate [default: 60]
|
||||
--hdr Enable HDR protocols
|
||||
--socket <NAME> Wayland socket name [default: nescope-0]
|
||||
--input-ipc <PATH> neshub's input socket [default: /tmp/nestri-input.sock]
|
||||
--screenshot-ipc <PATH> Serve screenshots here [default: off]
|
||||
--render-device <PATH> GPU to pin the workload to, e.g. /dev/dri/renderD128
|
||||
--x-display <N> XWayland display number [default: 1]
|
||||
```
|
||||
|
||||
The command is optional; see design decision 5 above.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Effect |
|
||||
@@ -81,18 +96,37 @@ Options:
|
||||
| `NESCOPE_FPS` | Override `--fps` |
|
||||
| `NESCOPE_HDR` | Enable HDR |
|
||||
| `NESCOPE_SOCKET` | Override socket name |
|
||||
| `NESCOPE_INPUT_IPC` | Override `--input-ipc` |
|
||||
| `NESCOPE_SCREENSHOT_IPC` | Override `--screenshot-ipc` |
|
||||
| `NESCOPE_RENDER_DEVICE` | Override `--render-device` |
|
||||
| `NESCOPE_X_DISPLAY` | Override `--x-display` |
|
||||
| `RUST_LOG` | Tracing filter (e.g., `nescope=debug`) |
|
||||
|
||||
### Example
|
||||
|
||||
```sh
|
||||
# Launch a game at 1440p with HDR
|
||||
# Wrap one program at 1440p with HDR
|
||||
nescope --width 2560 --height 1440 --hdr -- %command%
|
||||
|
||||
# 1080p with debug logging
|
||||
# Plain compositor mode: come up and wait for clients
|
||||
nescope --x-display 1
|
||||
|
||||
# Debug logging
|
||||
RUST_LOG=nescope=debug nescope -- %command%
|
||||
```
|
||||
|
||||
### Seeing what is on screen
|
||||
|
||||
nescope is headless, so a black stream, a window that never mapped and a client
|
||||
rendering fine all look identical from outside. `nescope-shot` is the listener
|
||||
side of the screenshot socket:
|
||||
|
||||
```sh
|
||||
# start this first — nescope dials out
|
||||
nescope-shot --socket /tmp/nestri-screenshot.sock --watch --out shot.ppm
|
||||
nescope --screenshot-ipc /tmp/nestri-screenshot.sock -- <program>
|
||||
```
|
||||
|
||||
## HDR Support
|
||||
|
||||
nescope supports two HDR signaling paths:
|
||||
@@ -123,12 +157,14 @@ nescope waits 5 seconds after the last mapped window disappears before exiting,
|
||||
- **clap** — CLI parsing
|
||||
- **x11rb** — X11 atom management
|
||||
|
||||
## License
|
||||
## What nescope does not do
|
||||
|
||||
See project repository.
|
||||
Frames reach the client through [`nescapture`](../nescapture), which captures
|
||||
inside the workload's process, and [`neshub`](../neshub), which muxes and
|
||||
sends. Input arrives the same way in reverse. nescope provides the display
|
||||
those components need and injects what they hand it; it never touches the
|
||||
network.
|
||||
|
||||
## TODO
|
||||
## Licence
|
||||
|
||||
- [ ] Handle sending frames thru to the client
|
||||
- [ ] Handle packetizing and sharding
|
||||
- [ ] Handle mouse and kb input from the client
|
||||
Apache 2.0. See [LICENSE](../../LICENSE).
|
||||
|
||||
@@ -1,36 +1,81 @@
|
||||
## neswire
|
||||
# neswire
|
||||
|
||||
A small custom PipeWire sink for cloud gaming audio capture.
|
||||
A PipeWire sink that Opus-encodes guest audio and sends it to
|
||||
[`neshub`](../neshub).
|
||||
|
||||
Currently for debugging uses RTP to send Opus (with FEC enabled by default) over to target address.
|
||||
Audio in the guest has no speakers to reach, so neswire registers itself as an
|
||||
output device and takes what anything plays into it. From the application's
|
||||
side it is an ordinary sink.
|
||||
|
||||
---
|
||||
|
||||
### Testing
|
||||
## Running it
|
||||
|
||||
#### Mono/Stereo
|
||||
|
||||
Launch gstreamer pipeline to receive audio as so (will save incoming RTP audio into test.mkv):
|
||||
```bash
|
||||
gst-launch-1.0 udpsrc port=12345 caps="application/x-rtp,media=audio,encoding-name=OPUS,clock-rate=48000,payload=111" ! rtpopusdepay2 ! opusdec ! matroskamux ! filesink location=test.mkv sync=false
|
||||
cargo run --release --bin neswire
|
||||
```
|
||||
|
||||
Then run neswire like so for example:
|
||||
Then select the **neswire** sink as the system output, or point an application
|
||||
at it.
|
||||
|
||||
| Flag | Env | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `--ipc-path` | `NESWIRE_IPC_PATH` | `/tmp/nestri-audio.sock` | Where to send Opus packets |
|
||||
| `--channels` | `NESWIRE_CHANNELS` | `2` | `2` stereo, `6` for 5.1, `8` for 7.1 |
|
||||
| `--packet-duration-ms` | `NESWIRE_PACKET_DURATION_MS` | `5` | Opus frame size in ms |
|
||||
| `--bitrate-per-channel` | `NESWIRE_BITRATE_PER_CHANNEL` | `64` | kbps per channel |
|
||||
|
||||
`RUST_LOG` takes a standard tracing filter, e.g. `RUST_LOG=neswire=debug`.
|
||||
|
||||
Sample rate is fixed at 48 kHz, which is what Opus wants and what every
|
||||
consumer device runs at anyway.
|
||||
|
||||
---
|
||||
|
||||
## Testing without a hub
|
||||
|
||||
neswire's only output is a Unix datagram socket that `neshub` binds, so running
|
||||
it on a desktop means it has nothing to talk to — it retries `connect` forever
|
||||
and there is no way to see what it would have sent. `hub-stub` binds that
|
||||
socket and reports what arrives:
|
||||
|
||||
```bash
|
||||
cargo run --release --bin neswire -- --rtp-addr 127.0.0.1:12345
|
||||
# terminal 1
|
||||
cargo run --bin hub-stub
|
||||
|
||||
# terminal 2
|
||||
cargo run --bin neswire
|
||||
```
|
||||
|
||||
#### Surround
|
||||
It **decodes** the Opus rather than counting bytes, and that distinction
|
||||
matters more than it looks. Opus codes digital silence in about two bytes a
|
||||
packet, so a sink that is receiving nothing but zeros still produces a steady
|
||||
~3 kbps and looks correct on every meter downstream. Peak amplitude is what
|
||||
tells a working sink from a silent one.
|
||||
|
||||
Launch gstreamer pipeline to receive audio as so (will save incoming RTP audio into test_multi.mkv):
|
||||
```bash
|
||||
gst-launch-1.0 udpsrc port=12345 caps='application/x-rtp,media=audio,encoding-name=MULTIOPUS,clock-rate=48000,payload=111,encoding-params=(string)8,num_streams=(string)5,coupled_streams=(string)3,channel_mapping=(string)"0,6,1,2,3,4,5,7"' ! rtpopusdepay2 ! opusdec ! matroskamux ! filesink location=test.mkv sync=false
|
||||
cargo run --bin hub-stub -- --channels 8 # match neswire's channel count
|
||||
```
|
||||
|
||||
Then run neswire like so for example:
|
||||
```bash
|
||||
cargo run --release --bin neswire -- --rtp-addr 127.0.0.1:12345 --channels 8
|
||||
`hub-stub` mirrors `ipc_listener::run_audio_listener` in `neshub`: same bind,
|
||||
same `0o666`, same stream-type and codec checks. Where the two disagree,
|
||||
`neshub` is right and the stub should be corrected.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/
|
||||
├── main.rs CLI, wiring, the sink→encoder→IPC chain
|
||||
├── sink.rs the PipeWire sink itself
|
||||
├── encoder.rs Opus encode, including the multichannel mappings
|
||||
└── bin/
|
||||
└── hub-stub.rs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
For both afterwards, set the neswire sink as audio output source in your system (or specify it as output in some app/game),
|
||||
then play some audio, stop gst pipeline and sink, listen to results after.
|
||||
## Licence
|
||||
|
||||
Apache 2.0. See [LICENSE](../../LICENSE).
|
||||
|
||||
Reference in New Issue
Block a user