feat: pyrowave support (#354)

At this point, why not..

Calling it `nespyro` since it's slightly modified, uses slang shader
language instead, along other things..

---------

Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kristian Ollikainen
2026-10-01 23:53:19 +03:00
committed by GitHub
co-authored by DatCaptainHorse Claude Opus 5.5
parent ca7115dba8
commit cf46bf4d12
62 changed files with 15649 additions and 589 deletions
@@ -0,0 +1,448 @@
# nespyro: a PyroWave codec crate
Status: approved design, 2026-09-27. Part 1 of 4.
## Why
Nestri streams with hardware H.264, H.265 and AV1 through Vulkan Video. PyroWave
(Hans-Kristian Arntzen, MIT) is an intra-only wavelet codec that runs entirely in
compute shaders. It spends far more bandwidth (about 170 Mbit/s at 1080p60 4:2:0)
and far less time (upstream claims under 0.1 ms encode and decode at 1080p) than
any hardware encoder, which makes it the LAN codec: a direct path, a wired
network, and latency as the only thing that matters.
## The four parts
Each part gets its own spec and is built complete; nothing half-wired lands.
1. **nespyro** (this spec): the codec as a Rust crate, both directions, proven
on one device and against the C++ reference.
2. **Protocol and transport**: a codec id, a self-contained-frame flag so intra
frames do not ride the reliable keyframe path, datagrams cut on PyroWave
packet boundaries, partial frames delivered to the decoder, and a bitrate
ceiling that admits hundreds of Mbit/s.
3. **nescapture**: `PerFrameEncoder` behind an abstraction over pixelforge or
nespyro, and nespyro's device requirements merged into the game's device.
4. **nesrecon / nesvideo**: a nespyro decoder beside the others, a shared device
and compute queue no longer gated on H.264 Vulkan Video decode, a
three-plane convert, and the convert shader's format bug fixed while it is
open (see part 4 notes below).
Parts 3 and 4 depend on 1. Part 2 is independent of both, but nothing crosses
the network without it.
## Decisions
- **Separate crate, not pixelforge.** pixelforge is shaped by Vulkan Video at
every layer: a closed `Codec` enum matched throughout, a `VideoCodec` trait of
sessions and DPBs, and device selection that refuses a GPU without a video
queue. nescore also pins upstream pixelforge rather than the fork, and a
wavelet codec would never go upstream there.
- **Lives at `nestri/crates/nespyro`**, MIT like PyroWave itself. It is open;
there is nothing to gate.
- **Bitstream is exactly upstream's**, frozen at PyroWave `89f7e47`
(2026-09-25). Codec bugs stay reportable upstream, and the C++ reference stays
a usable oracle. nesprotocol (part 2) carries a PyroWave format version, since
the bitstream itself has none and is marked draft.
- **Shaders are Slang**, ported from upstream's GLSL, compiled by `build.rs`
with `slangc` at build time only. Nothing compiles at runtime.
- **Desktop GPUs only** (path A). See "Future work: mobile path".
- **Vulkan 1.3 minimum.** Its core features are used freely.
- **The API mirrors pixelforge** so nescapture plugs it in the same way it
plugs in the pixelforge encoder.
## Scope
- Chroma 4:2:0 and 4:4:4.
- 8-bit (R8) and 16-bit (R16) planes: SDR, and HDR10 PQ.
- Precision mode 1 only, which is upstream's default: FP16 wavelet storage for
levels 0 and 1, FP32 for levels 2 to 4. Modes 0 and 2 are tuning knobs that do
not affect the bitstream.
- The decoder carries the non-FP16 variants of the inverse transform and
chooses at runtime. The encoder requires `shaderFloat16`, because upstream's
quantizer and rate-control shaders use it unconditionally.
- Colour: full range always, BT.709 or BT.2020 PQ. Limited range and BT.601
are not produced.
## Crate layout
```
crates/nespyro/
build.rs slangc over an explicit entry-point list
shaders/ Slang
src/
lib.rs
bitstream/ pure Rust, no Vulkan
device.rs requirements, Context
pipeline.rs layouts, spec constants, subgroup sizes, cache
encoder/ EncodeConfig, Encoder, EncodeFuture, EncodedFrame
decoder/ Decoder, DecodedFrame
sync.rs TimelinePoint, QueueLock
tests/
```
### build.rs and shaders
`build.rs` runs `slangc` (or `$SLANGC`) for each entry point, with
`-target spirv -profile spirv_1_5 -emit-spirv-directly -fvk-use-entrypoint-name`.
It does not pass `-fvk-invert-y`, because everything here is compute. Output
goes to `OUT_DIR`, is embedded with `include_bytes!`, and a missing or failing
`slangc` fails the build with the command it ran.
The guest image builder (`build/Containerfile`, builder stage) gains
`shader-slang` in its `pacman -S` list. The runtime stages do not.
Shaders, ported one to one from upstream:
| Slang | Upstream |
|---|---|
| `common.slang` | `dwt_common.h`, `dwt_swizzle.h`, `dwt_quant_scale.h`, `constants.h` |
| `rgb_to_ycbcr.slang` | none, ours |
| `dwt.slang` | `dwt.comp` |
| `wavelet_quant.slang` | `wavelet_quant.comp` |
| `analyze_rate_control.slang` | `analyze_rate_control.comp`, `analyze_rate_control_finalize.comp` |
| `resolve_rate_control.slang` | `resolve_rate_control.comp` |
| `block_packing.slang` | `block_packing.comp` |
| `wavelet_dequant.slang` | `wavelet_dequant.comp`, SSBO mode only |
| `idwt.slang` | `idwt.comp`, FP16 and FP32 variants |
Buffers are reached through buffer device addresses carried in push constants.
Images and samplers are bound with push descriptors (`VK_KHR_push_descriptor`),
so no descriptor pool exists. Every pipeline declares the subgroup size
upstream requires, and `computeFullSubgroups` where upstream asks for full
groups.
### device
`DeviceRequirements` lists what nespyro needs, in a form a caller merges into a
device it is about to create, as nescapture does for the game's device:
- Vulkan 1.3 core: `synchronization2`, `subgroupSizeControl`,
`computeFullSubgroups`, `maintenance4`.
- Vulkan 1.2 core: `timelineSemaphore`, `bufferDeviceAddress`,
`storageBuffer8BitAccess`, `shaderFloat16` (encoder only).
- Vulkan 1.1 core: `storageBuffer16BitAccess`.
- Vulkan 1.0: `shaderInt16`, `shaderStorageImageWriteWithoutFormat`.
- `VK_KHR_push_descriptor`.
- Subgroup operations checked against the device's properties: the encoder
needs basic, vote, arithmetic, ballot, shuffle, shuffle relative and
clustered. The decoder needs the same without clustered. Each pipeline's
subgroup size range, taken from upstream (for example 16..64 for block
packing, and exactly 64, 16 or 32 for the resolve pass), must intersect the
device's `minSubgroupSize..maxSubgroupSize`, with compute in
`requiredSubgroupSizeStages`.
`Context::from_existing(instance, physical_device, device, queue, family,
QueueLock)` wraps a device the caller owns. `Context::supports_encode()` and
`supports_decode()` answer from the device's real features and properties,
never from its name. There is no context that creates its own device; both
callers already have one.
### sync
`TimelinePoint { semaphore, value }` has the same shape as pixelforge's. Waits
use `ALL_COMMANDS`, for the reason pixelforge gives: nespyro's submissions begin
with transitions whose stages the caller cannot know.
`QueueLock` serialises nespyro's submits against anything else on the same
queue. A caller whose queue is not shared passes an uncontended lock.
## Encode
```rust
let mut enc = Encoder::new(ctx, EncodeConfig::new(w, h)
.with_chroma(Chroma::Yuv444)
.with_depth(Depth::Sixteen)
.with_colour(ColourDescription::bt2020_pq())
.with_frame_rate(120, 1)
.with_target_bitrate(400_000_000)
.with_packet_size(1200))?;
let fut = enc.encode_after(src_image, src_format, src_layout, &[blit_done])?;
let frame: EncodedFrame = fut.await?;
```
**Input.** The caller's RGB image, sampled in place: `B8G8R8A8`, `R8G8B8A8`,
`A2B10G10R10` or `R16G16B16A16_SFLOAT`. No copy is made first.
**Per slot** (two in flight): the Y, Cb and Cr plane images, the block
metadata and bitstream buffers, and a persistently mapped host-cached readback
buffer. Upstream recreates its buffers on every call; nespyro allocates them
once.
**Shared across slots**: the wavelet images, the block statistics, the quant
and bucket buffers, and the pipelines. The GPU runs the encode work in order,
so these need no copies.
**One submission per frame:**
1. Wait on the caller's points.
2. `rgb_to_ycbcr` writes the three planes. The matrix and the full range are
applied here, and so is the 4:2:0 subsampling, with centre siting.
3. `dwt`, 13 dispatches for 4:2:0 and 15 for 4:4:4.
4. `wavelet_quant` then `analyze_rate_control`, per band (42 bands for 4:2:0,
48 for 4:4:4).
5. `analyze_rate_control_finalize`, then `resolve_rate_control`.
6. `block_packing` per band. Finished 32×32 block packets land in GPU order.
7. Copy the metadata table and the first `target_bytes + headroom` bytes of the
bitstream into the readback buffer. Rate control caps the frame at its
target, so the copy is sized to the target, not to the whole buffer.
8. Signal the encoder's timeline.
**Completion thread.** It waits for that value, then packetizes on the CPU:
the 8-byte sequence header first, carrying the configured colour fields and
the frame counter, then every non-empty block in block-index order. A new
packet starts when the next block would pass `packet_size`, and blocks are
never split. It resolves the future with:
```rust
struct EncodedFrame {
data: Vec<u8>,
packets: Vec<Range<usize>>, // each independently parseable
critical_packets: usize, // leading packets covering the lowest bands
pts: u64,
stats: EncodeStats, // GPU timestamps per pass
}
```
**Rate control.** The target bits per frame is bitrate ÷ fps.
`set_target_bitrate` takes effect on the next frame with no rebuild. There is
no IDR and no GOP, because every frame stands alone. `set_colour_description`
changes the matrix and the header bits. Changing the size, chroma or depth
means building a new `Encoder`, as in pixelforge.
**Errors.** A block's header size must equal its size in the metadata table,
or packetizing fails with an error, as upstream checks it. A block cannot
exceed the 12-bit `payload_words` limit: the largest possible 32×32 block is
about 620 words (8-byte header, 16 × 3 bytes of codes, 16 × 8 subblocks × 18
planes, 128 bytes of signs), well under 4095. A test pins that bound. A lost
device or failed submit is an error from the future.
## Decode
```rust
let mut dec = Decoder::new(ctx, DecodeConfig::new(w, h, chroma, depth)
.with_consumer_queue_family(graphics_family))?;
match dec.push_packet(datagram)? { Push::Accepted | Push::Stale => {},
Push::Reconfigure(info) => { /* rebuild the decoder */ } }
if dec.readiness().is_complete_enough() {
let frame = dec.decode_after(&[])?;
}
```
**Packets.** `push_packet(&[u8]) -> Result<Push>` parses one packet on the CPU,
copying only block payloads. It handles the 3-bit frame counter as upstream
does: an older frame's packet is `Stale` and dropped, a newer frame's clears
the state. A header that disagrees with the decoder's size, chroma or depth
returns `Reconfigure`, and nothing is rebuilt behind the caller's back. A
truncated packet, or one whose block sizes run past its end, is `Err(Malformed)`,
never skipped.
**Readiness.** `readiness()` reports blocks received against the header's
total, and whether the two lowest bands are complete. When to decode is the
caller's decision (part 2's policy). `is_complete_enough()` is upstream's rule:
every block, or at least 90% with the two lowest bands complete.
**One submission per frame:**
1. Write block payloads and the offset table into the slot's mapped buffer. On
memory that is both device-local and host-visible (ReBAR, or an integrated
GPU) the shader reads it there. Otherwise a staging copy moves it to
device-local memory first.
2. `wavelet_dequant` per band. A missing block has offset `u32::MAX` and
decodes as zeros.
3. `idwt` per level per component, FP16 or FP32 as chosen at creation. The last
level adds the DC offset back and writes straight into the output planes.
4. Signal the decoder's timeline.
**Output.**
```rust
struct DecodedFrame {
y: PlaneView, cb: PlaneView, cr: PlaneView,
width: u32, height: u32, chroma: Chroma, depth: Depth,
colour: ColourDescription, // from the sequence header
missing_blocks: u32,
ready: TimelinePoint,
}
```
The planes come from a ring of three slots, R8 or R16, in `GENERAL`, with
sampled and storage usage. When the consumer's queue family differs, they are
created `CONCURRENT` across the decode and consumer families, as pixelforge
does for `with_consumer_queue_family`, so no ownership transfer is needed. The
consumer waits on `ready` in its own submit; nothing waits on the CPU.
Dropping the frame returns its slot. If all three slots are held,
`decode_after` returns `Busy` instead of stalling, which makes a consumer
holding frames too long visible.
A frame decoded with blocks missing succeeds and says so in `missing_blocks`,
so the loss is visible to the caller rather than looking like a whole frame.
## Testing
GPU and reference tests carry `#[ignore = "needs …"]` naming exactly what they
need, and they fail, never skip, when that is absent.
**Bitstream (pure Rust, always run):**
- Header pack and unpack against `bitstream.md`'s field layouts.
- Block-index ordering and the per-level/band tables against values taken from
`metal/pyrowave_bitstream.cpp`.
- Packetize then parse reproduces every block exactly. No packet passes
`packet_size` unless one block alone is larger, and no block is split.
- Frame counter wrap: stale packets drop, a new frame clears state.
- A truncated packet or an impossible `payload_words` returns `Malformed`, and
the parser never reads past the end.
- The largest possible block fits in 12 bits of words.
**GPU (one device):**
- Round trip over synthetic sources (gradients, noise, hard text-like edges,
flat colour) at 1920×1080, 1366×768 (not a multiple of 32) and one under the
128 minimum. Each runs in 4:2:0 and 4:4:4, and in R8 and R16. Decoded PSNR
must clear a threshold for each bitrate.
- Rate control really limits: encoded size is at or under the target at every
tested rate, and above a floor, so a nearly empty frame decoding to grey
cannot pass. A low target must give measurably lower PSNR than a high one.
- Determinism: the same input encodes to byte-identical output twice, despite
the GPU's unordered block writes.
- Colour: known RGB values map to the expected full-range YCbCr for BT.709 and
BT.2020 PQ. The header's colour bits match the configuration, and the decoder
reads the same back.
- Loss: dropping non-critical packets still decodes, `missing_blocks` equals
what was dropped, and the lowest bands are intact. Dropping a critical packet
shows in `readiness()`.
**Reference cross-check.** A small C++ harness is built from PyroWave
`89f7e47` and Granite `1b2d1801` (`checkout_granite.sh`), outside cargo. Its
path is given in `NESPYRO_REFERENCE`, and the test fails if that is set but the
binary is missing.
- Our encoder into the reference decoder, and the reference encoder into ours.
Pass is upstream's own tolerance from `pyrowave_device_validation.cpp`: at
most 1 LSB luma error and at most 2 chroma.
- Our encoder beside theirs on identical YCbCr input. Slang and glslang may
round floating point differently, so byte identity is not required. The test
reports the fraction of blocks that match exactly, and the size and PSNR
difference, as a diagnostic.
**Latency.** A benchmark takes GPU timestamps around every pass, ours beside
the reference, at 1080p and 4K. It measures GPU time, not command recording.
## Future work: mobile path (path B)
Left out on purpose; the device check refuses such devices with a clear error
rather than degrading. Adding them means:
- The fragment inverse transform: `idwt.vert` and `idwt.frag` with its three
`CHROMA_CONFIG` variants, separate vertical and horizontal passes into
R16F/R32F (RG for chroma) intermediates, and three scissored draws per pass
with the `EDGE_CONDITION` specialization constant.
- Its output views need `COLOR_ATTACHMENT` usage, and the pass uses dynamic
rendering.
- `wavelet_dequant` storage modes 1 (R8/R16/R32 uniform texel buffers, for
devices without 8-bit storage) and 2 (linear-tiled images aliased on one
buffer).
- Upstream's vendor preference table (`pyrowave_decoder.cpp:913-930`) choosing
fragment over compute for Mali, Qualcomm and PanVK.
- Hardware to test on, which this project does not yet have.
## Part 4 notes, carried forward
Found while reading nesrecon, to fix when part 4 opens the convert shader:
- The Vulkan Video path creates its RGBA target as `R8G8B8A8_UNORM`
(`renderer.rs:616,624`), but `yuv_convert.slang:3` writes it as `rgba16f`,
and `probe_signal` reads it as `PASS_FORMAT` (`renderer.rs:1493-1499`). The
DMA-BUF path correctly uses `PASS_FORMAT`.
- One `vk::Queue` per family is shared between the render thread and
pixelforge's decode thread with no lock (`vulkan/mod.rs:555-578`). nespyro's
`QueueLock` is where that gets fixed.
## Found while building
Recorded after part 1 was built, for parts 2 to 4 to start from.
### Where the crate differs from this design
- `Context::from_existing` takes the `Roles` the device was created for.
Vulkan cannot say which features a device was created with, and the decoder
needs to know whether `shaderFloat16` is on before choosing its FP16 tile.
Decode-only requirements ask for FP16 only where the device has it.
- `Encoder::encode_planes_after` encodes the caller's own YCbCr planes. It
exists so the encoder can be compared with the reference on identical input,
and it lets a caller with its own conversion skip ours.
- Decoder output planes are padded to the codec's alignment. The final inverse
transform writes whole 32×32 tiles; upstream relies on out-of-bounds stores
being discarded, which core Vulkan only promises with `robustImageAccess`.
`PlaneView` gives both the picture size and the image size.
- `Decoder::stats` reports the GPU time of the last finished decode.
- Both halves record their many dispatches once and reuse them. On Mesa's ANV
recording costs 6 to 14 µs per dispatch; per frame it was 416 µs of a
decode. Per-frame values reach the shaders through a small parameters buffer.
- Device requirements also check each plane and wavelet format for sampled
and format-less storage use, which core Vulkan guarantees only for
`R32_SFLOAT`.
### Measured, Arc A310, Mesa ANV
| | nespyro | reference |
|---|---|---|
| encode 1080p 4:2:0, GPU, excluding colour conversion | 0.94 ms | 1.05 ms |
| encode 4K 4:2:0 | 3.07 ms | 3.22 ms |
| encode 1080p 4:4:4 | 1.36 ms | 1.50 ms |
| decode 1080p 4:2:0, GPU | 0.56 ms | |
| `encode_after` / `decode_after` CPU, 1080p 4:2:0 | 140 / 134 µs | |
Both decoders give bit-identical planes on every stream either encoder wrote.
On identical input the encoders tie on smooth content and nespyro is level or
ahead on edges and noise.
Subgroup sizes follow Granite's rule, which upstream's shaders were tuned
under: varying when a pass's range covers the device's, otherwise the smallest
in range. Taking the largest made rate-control analysis 73% slower on Intel.
### Upstream bugs, worth reporting to PyroWave
1. **Scratch payload undersized.** The quantizer's scratch is sized at aligned
width × height × 2 bytes. High-entropy 4:4:4 content needs more (3.4 MB
against 2.1 MB at 1366×768), and the encoder then packs blocks from bytes it
never wrote: a block whose header claims 115 words holds 95. nespyro sizes
it by the real bound, 136 bytes per 8×8 block.
2. **Rate-control prefix sum stops short.** `analyze_rate_control_finalize`
steps while `step < 512 / 2`, so the upper half of its lanes miss part of
the running total. Frames land under target, not over. Ported unchanged.
3. **Out-of-bounds shared read.** `analyze_rate_control` reads
`shared_rate_cost[gl_SubgroupInvocationID]` past its 16 entries on wider
subgroups. The value is unused; nespyro clamps the index.
4. **Stale bytes in the stream.** A block's unused sign bits come from
uninitialised shared memory and its padding from whatever an earlier frame
left. Harmless to decoders, but non-deterministic and leaks old memory;
nespyro writes zeros.
### Toolchain and driver bugs
- **Slang 2026.17** emits `OpGroupNonUniformShuffleXor` without declaring
`GroupNonUniformShuffle` when nothing else in the shader does. `spirv-val`
catches it; `shuffle_xor` in `common.slang` works around it. `build.rs`
requires `spirv-val` for that reason.
- **Mesa ANV** spends 12.6 ms of CPU recording `vkCmdFillBuffer` of an
8,396,864-byte buffer, and 3 µs for a 16,785,472-byte one. nespyro records
its fills once, so it no longer matters here, but it is worth a Mesa report.
### Found by part 3
- **`Readiness::critical_complete` cannot be trusted.** A block the encoder
quantized to nothing is never sent, and its absence is indistinguishable from
a loss. Noise leaves the coarsest high-pass bands empty often enough that a
frame with every packet delivered read as critically incomplete, so no
partial frame would ever have been decoded. Upstream has the same flaw and
answers it with an out-of-band mask of active blocks. Here the transport
knows which datagrams carried the critical packets and says so
(`CollectedFrame::critical_complete`); `Readiness::is_complete_enough_with`
takes that answer. Found by the real-frame transport test.
### For the parts that follow
- Part 2: a frame at 200 Mbit/s and 60 fps is about 416 KB in 1200-byte
packets, a few hundred per frame. `EncodedFrame::critical_packets` is small,
typically one to a few packets.
- Part 4: nesrecon's convert shader takes two planes; nespyro gives three, in
`GENERAL`, padded, with the picture size in `PlaneView`.
@@ -0,0 +1,122 @@
# PyroWave decode in nesrecon / nesvideo
Status: built, 2026-10-01 (nescore `e20a4d2`, `0e729ef`). Part 4 of 4; parts 1-3 are
`2026-09-27-nespyro-codec-crate-design.md`,
`2026-10-01-pyrowave-transport-design.md`,
`2026-10-01-pyrowave-nescapture-design.md`. Code lives in nescore
(`crates/nesrecon`, `apps/nesvideo`); nespyro is a git dep on nestri at the
same rev nesprotocol is pinned to.
## Decisions
- **nespyro decodes on nesrecon's device**, from the decode thread, as one
more backend beside Vulkan Video / VA-API / CPU. Its planes stay on the GPU
and are converted by the render thread like a Vulkan Video picture.
- **The device is no longer gated on H.264 Vulkan Video decode.**
`shared_decode_device` exists when either pixelforge or nespyro fits.
- **Planes are always `Depth::Sixteen`.** PyroWave decodes in float and the
stream carries no depth; R16 costs little and never bands an HDR stream.
- **Converter: a second entry point**, three planes, chroma shift in the
uniform's spare word. The two-plane path is untouched, so its output and
tests do not move.
- **Fix while open:** the Vulkan-path RGBA target is `R8G8B8A8_UNORM` while the
shader writes `rgba16f` and the probe reads `PASS_FORMAT`; it becomes
`PASS_FORMAT`. That changes the Vulkan Video path's output (it was undefined
behaviour), so the PSNR baseline is re-taken for that path.
- **Deblock is off for PyroWave**: it has no block grid. `block_grid_for`
returns none, as for anything without one.
- **Busy is not an error.** nespyro has three output slots and the renderer
can hold all three (hand-off slot, current picture, retire queue); a frame
that finds none free is dropped and counted, and the next decodes.
## Device (`vulkan/mod.rs`)
- `nespyro::DeviceRequirements::query(.., Roles::DECODE)` beside pixelforge's,
independently. Device preference also accepts a nespyro-capable device.
- Features: the 1.0 ones in `PhysicalDeviceFeatures`, 16-bit storage in the
existing `Vulkan11Features`, subgroup size control and full subgroups in the
existing `Vulkan13Features`, and a new `Vulkan12Features` for 8-bit
storage, int8, float16 (when available), buffer device address and timeline
semaphores — replacing the standalone timeline struct, which may not be
chained beside it.
- Queue: nespyro gets the compute-only family's queue 0, which nesrecon
never submits to; pixelforge may use it too, but only from the same decode
thread, so the two are serialised by that thread. Without a compute-only
family, a second graphics queue if there is one. Without either -- every
Intel GPU under ANV, whose one family has one queue -- the render queue
itself, under a `QueueLock` the render thread holds around every submit,
present and `vkDeviceWaitIdle` (`VkCtx::queue_lock`, `VkCtx::wait_idle`).
The first build refused PyroWave in that case, which on this machine meant
the client never advertised it and the hub refused every request.
- `SharedVulkanDevice` carries `pyro: Option<(family, index)>`.
- `nesrecon::decodes(PyroWave, _)` answers from a flag set when the device was
created with nespyro's additions; nesvideo advertises PyroWave only then.
Still open, out of scope: pixelforge's transfer queue can alias the graphics
queue on a device without a transfer or compute family, and pixelforge has no
lock to share. Unchanged by this part.
## Decode thread (`decode/`)
- The channel carries `enum VideoUnit { Coded(Vec<u8>, VideoCodec),
PyroWave(PyroUnit) }`, `PyroUnit { packets: Vec<Bytes>, critical_complete,
ts_ms }`. `VideoCodec` gains `PyroWave`; every closed match says what it
means there.
- `PyroDecoder` (new backend): built lazily for the stream's size and chroma,
rebuilt on `Push::Reconfigure`. Per unit: push every packet; decode when
`readiness().is_complete_enough_with(critical_complete)`; otherwise drop
and count. Output `DecodedFrame::PyroWave(PyroPicture)` into the same sink.
- `decode_ns` is CPU time as for the others; nespyro's GPU time goes to the
log at trace once a second.
## Render (`vulkan/renderer.rs`, `yuv_convert.*`)
- `import_frame` takes `PyroPicture` into the same slot the Vulkan picture
uses; retired the same way, after the frame that sampled it.
- The render submit waits on the picture's `ready` point when it has one.
- `yuv_convert_planar` entry: three `Texture2D` loads, chroma at
`(x >> shift, y >> shift)`, same matrix/range/peak code. R16 full range.
- Matrix and range from the picture's `ColourDescription`, per frame.
## nesvideo
- `PyroCollector` frames go to the decode channel as `VideoUnit::PyroWave`.
A full channel drops and counts; no IDR, nothing to resync.
- The sequence header (first packet) gives size and colour: `stream_dims`, and
`bitstream_colour` as a `StreamColour`, so the existing colour logic picks
the swapchain (HDR10 for PQ) exactly as for the other codecs.
- `client_capabilities` asks `nesrecon::decodes` for PyroWave like the rest.
- The overlay's "not decoded yet" note goes; the PyroWave rate line stays.
## Tests (last step)
- Unit: channel/unit plumbing, colour mapping, `block_grid_for`, the planar
shader's text checks beside the existing ones.
- Existing nesrecon/nesvideo tests stay green.
- Full stack by eye: PyroWave on screen, switching to and from a hardware
codec, 4:2:0 and 4:4:4, SDR and HDR.
## Found while building
- **The channel type stayed.** A PyroWave unit is serialised into the existing
`Vec<u8>` (`[flags][count]` then length-prefixed packets,
`nesrecon::encode_pyro_unit`), so PyroWave is one more `VideoDecoder`
variant and nothing else on the channel changed. One copy a frame, about
37 MB/s at 300 Mbit/s.
- **A picture keeps its decoder alive.** Dropping a `nespyro::Decoder`
destroys its output images even while a `DecodedFrame` of them is held, and
a codec switch or resize drops one while the renderer still shows its last
picture. `PyroPicture` holds an `Arc` of the decoder, released after the
slot. Worth fixing in nespyro itself when it is next opened: the output
images should outlive the decoder or the frame should own them.
- **Capabilities are sent twice.** The first message goes out at connect,
before the renderer has created the device that decides whether PyroWave
decodes. A second follows once it has; the host renegotiates to the codec
it was already on. A `--codec pyrowave` request waits for the second, or the
hub would refuse it.
- **Deblock is suppressed, not switched off.** `PostProcessChain` gained a
per-type suppression, so a person's own pass toggles are untouched and the
pass comes back on a switch to a block codec.
- **A device shared for PyroWave alone** is not offered to the Vulkan Video
decoder (`SharedVulkanDevice::video_decode`), which would otherwise fail and
log an error on every switch to H.264.
@@ -0,0 +1,89 @@
# PyroWave in nescapture
Status: approved design, 2026-10-01. Part 3 of 4. Parts 1 and 2:
`2026-09-27-nespyro-codec-crate-design.md`, `2026-10-01-pyrowave-transport-design.md`.
## Decisions
- **Vulkan 1.3 instance.** `vkCreateInstance` raises anything below 1.3 to 1.3
where the loader has it (today: 1.0 → 1.1). Only the client app has to care
about old Vulkan; nescapture runs on the game's machine, which we control.
- **Shared device only.** nespyro runs on the game's device or not at all. On
the own-device fallback a PyroWave request is refused and logged; a CPU round
trip of a frame defeats the codec's reason to exist.
- **Decoupled from Vulkan Video.** The game's device gets pixelforge's
additions, nespyro's, both, or neither, each checked on its own. A GPU with
no video encode can still stream PyroWave once a client asks; until then
nothing is encoded, and the log says why.
## Device (`instance.rs`, `shared.rs`, `device.rs`)
- `prepare` asks pixelforge `encode_device_requirements` and nespyro
`DeviceRequirements::query(Roles::ENCODE)` independently.
`Additions { video: Option<pixelforge::DeviceRequirements>, pyro:
Option<nespyro::DeviceRequirements>, queues }`. The effective version
(min of instance and physical device) must be 1.3 for `pyro`.
- `Feature` gains nespyro's bits: Storage16Bit, Storage8Bit, ShaderInt8,
ShaderFloat16, BufferDeviceAddress, SubgroupSizeControl,
ComputeFullSubgroups, and the 1.0 core ShaderInt16 and
ShaderStorageImageWriteWithoutFormat. 1.0 bits live in
`VkPhysicalDeviceFeatures`: patched in place inside a chained
`VkPhysicalDeviceFeatures2`, else in a copy of `*pEnabledFeatures` the create
info points at instead, else in a copy of our own. Restore covers all three.
- Extension `VK_KHR_push_descriptor` added with nespyro.
- `plan_queues` plans encode only with pixelforge, compute with either,
transfer with pixelforge. nespyro uses the compute queue, `QueueLock::none()`:
one encoder thread submits for both, at most one queue of ours per family,
internally synchronized where shared with the game.
- `SharedDevice::pyro_context() -> Option<nespyro::Context>`.
`SharedDevice::video_context()` returns an error when pixelforge's additions
were not made.
## Encoder thread (`encode.rs`)
- `enum Codec { Hw(HwCodec), PyroWave }` in `EncoderConfig`,
`EncodeSettingsChange`, `current_codec` (protocol id). `HwCodec` unchanged.
Every match says what PyroWave does; depth clamping, GOP, intra refresh and
IDR apply only to `Hw`. IDR on PyroWave is a no-op.
- `enum Backend { Hw(PerFrameEncoder), Pyro(PyroEncoder) }`. `PyroEncoder`
owns `nespyro::Encoder` and its rebuild key (w, h, depth, chroma, colour,
packet size). Bitrate-only change → `set_target_bitrate`, no rebuild.
- Input: the slot image in `GENERAL`, `encode_after(view, Source, blit)`.
`Source` from the same `source_spec`/`stream_spec`; only BT.709 SDR and
BT.2020 PQ, anything else refused. pixelforge `TimelinePoint` converted at
the boundary.
- Slot lifetime: a PyroWave frame's slot is held until its future resolves; up
to two held (nespyro's in-flight limit). Hardware path unchanged.
- `EncodedFrame.future` becomes an enum of the two futures.
- Codec switches use the existing `drop_encoder` + `needs_reconfig_flag`.
## Commands, IPC, stats
- `MSG_ENCODE_SETTINGS` via `decode_encode_settings_ext`. Codec 4 →
`PyroWave`, with chroma and packet size. Refused (warned) with no nespyro
context, no or too small packet size, or rate 0.
- `host_caps` includes PyroWave bits only with a nespyro context.
- IPC: PyroWave frames via `chunk_frame`, each message under the existing 250
ms timeout. A timed-out message drops the rest of the frame and counts it in
`dropped`; other errors reconnect as today. Width/height from the frame, not
frozen at pipeline creation (fixes a stale-size bug for every codec).
- `enc_ms` from nespyro `gpu_ns`. Scratch overflow warned once per episode.
- Fix while open: the CPU fallback's hardcoded `Codec::H264` in
`InputImage::new`.
## Tests
- Unit: planning with one or both sets; 1.0 feature bits in `pEnabledFeatures`,
in `Features2`, in neither, and restoration; the instance raise decision;
command decode and refusals; the two-slot hold rule.
- GPU `#[ignore]`, in `crates/nespyro/tests/transport.rs` beside nespyro's GPU
harness: real frames → `chunk_frame` → `FrameAssembler` → `pack_datagrams` →
`PyroCollector` → `Depacketizer`; readiness equals the encoder's own packets,
clean and with loss. (Moved from part 2.) nescapture's own mapping of an
`EncodedFrame` onto `chunk_frame` (`pyro_messages`) is unit-tested on the
CPU, since `EncodedFrame` can be built by hand.
- Found by it: `Readiness::critical_complete` false-rejects frames with empty
coarse blocks; the datagram header gained a critical count so the collector
can answer exactly (see parts 1 and 2).
- Guest image: rebuild, run a title, send a PyroWave `MSG_ENCODE_SETTINGS` to
`/tmp/nescapture-cmd.sock`, and see PyroWave frames leave neshub.
@@ -0,0 +1,176 @@
# PyroWave transport
Status: approved design, 2026-10-01. Part 2 of 4; part 1 is
`2026-09-27-nespyro-codec-crate-design.md`.
## Why
The media path was built for hardware codecs over the internet: a frame is one
IPC message, the hub cuts it into byte slices, the client needs every slice,
keyframes ride reliable streams, and a controller ratchets the bitrate. None of
that fits PyroWave:
- A unix datagram tops out near 416 KB under the default `net.core.wmem_max`,
and the hub reads into 2 MiB. A PyroWave frame is 400 KB at 200 Mbit/s and
2 MB at 1 Gbit/s.
- Byte slicing throws away what PyroWave is built around: packets that each
decode on their own, and a decoder that shows a frame missing some of them.
- Every PyroWave frame is intra. Flagged as a keyframe it would take the
two-slot reliable path every frame; unflagged, `ResyncGate` withholds it.
## Decisions
- **Opt-in only.** `ClientCaps` advertises PyroWave, but `best()` and
`CODEC_PREFERENCE` never return it. A session reaches it only by the client
sending `MSG_ENCODE_SETTINGS` naming it.
- **No controller.** PyroWave runs at a static rate: what the request names, or
the hub's `--pyrowave-kbps` default (300 000) when it names 0. The request is
CBR, which already sets the controller to Manual (`note_manual_target`), so
it stands down with no change to `control.rs`. The existing controller is
slated for replacement and gets nothing new.
- **Critical packets sent twice**, first and again last, as datagrams. No
reliable stream anywhere on the PyroWave path.
- **Shared logic in nesprotocol, pure.** Chunking, reassembly of IPC messages,
datagram packing and the client-side collector are functions and state
machines with an explicit clock. neshub wires them to sockets. nesprotocol
keeps zero dependencies and does not depend on nespyro: packets are byte
ranges to it.
- **The 4 MiB datagram buffer stays** for every codec. A hub-side skip guard
keeps PyroWave's queue to about two frames instead.
## Scope
In part 2: nesprotocol, neshub, tests, a throughput bench. Not in part 2:
- nescapture (part 3) calls `chunk_frame`, reads the packet size from encode
settings, and checks the format version it was built with.
- nesvideo (part 4) dispatches datagram kind 2 to `PyroCollector` and feeds a
decoder. Moved from part 2: without a decoder the client never advertises
PyroWave, so the glue would be unreachable. The collector itself is built and
tested here, end to end through a real iroh connection.
## Wire
### Negotiation
- `CODEC_PYROWAVE = 4`. `ClientCaps` bit `4*2 + DEPTH_8` = decodes PyroWave SDR,
`+ DEPTH_10` = decodes PyroWave HDR. Chroma is not a capability: anything
running nespyro decodes 4:2:0 and 4:4:4.
- `PYROWAVE_FORMAT: u8 = 1`, meaning upstream `89f7e47`'s bitstream. The client
appends it to `MSG_CLIENT_CAPS` as byte 3; `decode_client_caps` keeps reading
the first two. `decode_pyrowave_format(payload) -> Option<u8>` reads it.
- Encode settings grow two appended fields, read only when present:
`[codec][rc][value u32][depth][chroma][packet_size u16]`. `CHROMA_420 = 0`,
`CHROMA_444 = 1`. The client sends packet size 0; the hub fills it in.
`decode_encode_settings` keeps its signature; a new
`decode_encode_settings_ext` returns a struct with the two options.
The hub refuses (logs, does not forward) a PyroWave request from a client
whose caps lacked PyroWave or carried a different `PYROWAVE_FORMAT`.
### IPC (nescapture → neshub), stream type `STREAM_PYROWAVE = 7`
The same 20-byte IPC header (codec `CODEC_PYROWAVE`, ts, width, height), and
its data is:
```text
[frame u32][msg index u16][msg count u16][critical u16][packet count u16]
then per packet: [len u16][bytes]
```
`frame` is a counter, `critical` the frame's total of leading critical packets
(the same in every message). Each message holds at most `IPC_PYROWAVE_MAX`
(64 KiB) of data and whole packets only, in frame order. A packet is at most
`max(packet_size, 2488)` bytes, the largest block, so `u16` lengths suffice.
### Datagram kind `DGRAM_PYROWAVE = 2`
```text
[1B kind][2B seq][2B index][2B total][2B critical][4B ts_ms][1B flags][payload]
```
14 bytes. `seq` is the frame, `index`/`total` count datagrams in it.
`critical` is how many leading indexes carry the critical packets: a receiver
cannot work that out from the blocks it got, because an empty block is never
sent and looks like a lost one (found in part 3; the header was 12 bytes
before). The collector reports `critical_complete` from it. A payload
is one whole packet, or one piece of a packet too large for a datagram:
- `FLAG_DUPLICATE` (bit 0): a second copy of a critical datagram. Same index.
- `FLAG_PART` (bit 1): a piece of one packet. Pieces occupy consecutive
indexes from one marked `FLAG_FIRST_PART` (bit 2) to one marked
`FLAG_LAST_PART` (bit 3). Both ends are marked so a receiver missing a piece
can tell an orphaned middle from the start of the next run. Only a block
larger than the payload limit does this, at most two or three pieces.
`total` is capped at `PYRO_MAX_DATAGRAMS = 16384`, about 19 MB a frame.
## neshub
- **Packet size.** On forwarding a PyroWave request, the hub fills
`packet_size` with the smallest `max_datagram_size() - 12` across streaming
clients (1100 with none). The command forwarder in `main.rs` does it, since
it already serialises every command to nescapture. If the path later shrinks
below that, `pack_datagrams` sends the now-oversized packets in pieces like
any oversized block; nothing is dropped.
- **Listener.** The video socket accepts `STREAM_PYROWAVE` beside
`STREAM_VIDEO`. `FrameAssembler` collects a frame's messages; an incomplete
frame is abandoned when the next frame starts or after 100 ms, and counted.
The receive buffer stays 2 MiB, which is 32 messages' worth.
- **Broadcast.** The per-session video channel carries an enum: a hardware
frame as `Bytes`, or a PyroWave frame as `Arc<PyroFrame>`. Both are
refcounted, which also removes today's per-client copy of every frame.
- **Writer.** PyroWave frames skip `ResyncGate` and `KeyframeSender` and keep a
`seq` of their own. `pack_datagrams` lays them out (critical, rest, critical
again) into one `BytesMut`, split without copying.
- **Skip guard** (`pyrowave::should_skip`, shared with the bench). Before sending, with `backlog = DGRAM_BUFFER_BYTES -
datagram_send_buffer_space()`: skip the whole frame if the backlog exceeds
this frame's size, or if backlog plus frame exceeds the buffer. The skipped
frame's `seq` is reused (the transport doc's rule). A frame larger than the
buffer itself can never go out; that is warned once. Skips are counted and
logged at debug.
## The collector (client side, in nesprotocol)
`PyroCollector::push(datagram, now) -> Vec<PyroFrame>`, plus `poll(now)`.
Frames in flight by `seq`, each a slot per index (duplicates dropped by index).
A frame is released when:
1. every index has arrived, or
2. a newer frame has started and `GRACE` (2 ms) has passed since, or
3. `IDLE` (20 ms) has passed since its *last* datagram.
Not a deadline from the first datagram: on a saturated path a frame takes a
frame interval to arrive, so such a deadline tears every frame once the path is
slower than it assumes. The end-to-end burst test found this.
Released frames carry `seq`, `ts_ms`, packets in index order (piece runs joined,
broken runs dropped), received/total, and `critical_complete`. A datagram for a `seq` at or behind
the newest released one is stale and dropped. No reorder window, no IDR: every
frame stands alone. Whether a frame is decodable is the decoder's call.
Counts per frame: whole, partial, and datagrams lost, duplicated, stale.
## Testing
- **nesprotocol units:** chunk/assemble round trips including over 4 MB;
packing never splits a packet that fits; oversized packets come back whole
from pieces and a broken run is dropped; critical datagrams repeat with the
same index; every release trigger; stale and duplicate datagrams; the
negatives (`best()` never picks PyroWave, a wrong format is refused).
- **neshub end to end (CPU):** synthetic packets in as IPC messages through the
real listener and writer, over an iroh loopback connection, into a collector.
Byte-exact, clean and with dropped datagrams; the skip guard engages under a
blocked receiver.
- **End to end with real nespyro frames:** moved to part 3, where nescapture
produces them. Every released frame must give a `Depacketizer` the same
readiness as the encoder's own packets.
- **Throughput:** `examples/pyro_bench.rs` sends PyroWave-shaped frames at a
given rate over iroh. Loopback measures CPU cost; serve/connect across the
LAN measures the real thing. If iroh cannot carry the rates, that comes back
as a design question before part 3.
## Order
nestri `feat/pyrowave`: nesprotocol, then neshub, then the bench. nescore is
untouched until part 4 pins the rev.