mirror of
https://github.com/nestriness/nestri.git
synced 2026-11-19 05:29:49 +02:00
feat: pyrowave support (#354)
At this point, why not.. Calling it `nespyro` since it's slightly modified, uses slang shader language instead, along other things.. --------- Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
DatCaptainHorse
Claude Opus 5.5
parent
ca7115dba8
commit
cf46bf4d12
@@ -0,0 +1,448 @@
|
||||
# nespyro: a PyroWave codec crate
|
||||
|
||||
Status: approved design, 2026-09-27. Part 1 of 4.
|
||||
|
||||
## Why
|
||||
|
||||
Nestri streams with hardware H.264, H.265 and AV1 through Vulkan Video. PyroWave
|
||||
(Hans-Kristian Arntzen, MIT) is an intra-only wavelet codec that runs entirely in
|
||||
compute shaders. It spends far more bandwidth (about 170 Mbit/s at 1080p60 4:2:0)
|
||||
and far less time (upstream claims under 0.1 ms encode and decode at 1080p) than
|
||||
any hardware encoder, which makes it the LAN codec: a direct path, a wired
|
||||
network, and latency as the only thing that matters.
|
||||
|
||||
## The four parts
|
||||
|
||||
Each part gets its own spec and is built complete; nothing half-wired lands.
|
||||
|
||||
1. **nespyro** (this spec): the codec as a Rust crate, both directions, proven
|
||||
on one device and against the C++ reference.
|
||||
2. **Protocol and transport**: a codec id, a self-contained-frame flag so intra
|
||||
frames do not ride the reliable keyframe path, datagrams cut on PyroWave
|
||||
packet boundaries, partial frames delivered to the decoder, and a bitrate
|
||||
ceiling that admits hundreds of Mbit/s.
|
||||
3. **nescapture**: `PerFrameEncoder` behind an abstraction over pixelforge or
|
||||
nespyro, and nespyro's device requirements merged into the game's device.
|
||||
4. **nesrecon / nesvideo**: a nespyro decoder beside the others, a shared device
|
||||
and compute queue no longer gated on H.264 Vulkan Video decode, a
|
||||
three-plane convert, and the convert shader's format bug fixed while it is
|
||||
open (see part 4 notes below).
|
||||
|
||||
Parts 3 and 4 depend on 1. Part 2 is independent of both, but nothing crosses
|
||||
the network without it.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Separate crate, not pixelforge.** pixelforge is shaped by Vulkan Video at
|
||||
every layer: a closed `Codec` enum matched throughout, a `VideoCodec` trait of
|
||||
sessions and DPBs, and device selection that refuses a GPU without a video
|
||||
queue. nescore also pins upstream pixelforge rather than the fork, and a
|
||||
wavelet codec would never go upstream there.
|
||||
- **Lives at `nestri/crates/nespyro`**, MIT like PyroWave itself. It is open;
|
||||
there is nothing to gate.
|
||||
- **Bitstream is exactly upstream's**, frozen at PyroWave `89f7e47`
|
||||
(2026-09-25). Codec bugs stay reportable upstream, and the C++ reference stays
|
||||
a usable oracle. nesprotocol (part 2) carries a PyroWave format version, since
|
||||
the bitstream itself has none and is marked draft.
|
||||
- **Shaders are Slang**, ported from upstream's GLSL, compiled by `build.rs`
|
||||
with `slangc` at build time only. Nothing compiles at runtime.
|
||||
- **Desktop GPUs only** (path A). See "Future work: mobile path".
|
||||
- **Vulkan 1.3 minimum.** Its core features are used freely.
|
||||
- **The API mirrors pixelforge** so nescapture plugs it in the same way it
|
||||
plugs in the pixelforge encoder.
|
||||
|
||||
## Scope
|
||||
|
||||
- Chroma 4:2:0 and 4:4:4.
|
||||
- 8-bit (R8) and 16-bit (R16) planes: SDR, and HDR10 PQ.
|
||||
- Precision mode 1 only, which is upstream's default: FP16 wavelet storage for
|
||||
levels 0 and 1, FP32 for levels 2 to 4. Modes 0 and 2 are tuning knobs that do
|
||||
not affect the bitstream.
|
||||
- The decoder carries the non-FP16 variants of the inverse transform and
|
||||
chooses at runtime. The encoder requires `shaderFloat16`, because upstream's
|
||||
quantizer and rate-control shaders use it unconditionally.
|
||||
- Colour: full range always, BT.709 or BT.2020 PQ. Limited range and BT.601
|
||||
are not produced.
|
||||
|
||||
## Crate layout
|
||||
|
||||
```
|
||||
crates/nespyro/
|
||||
build.rs slangc over an explicit entry-point list
|
||||
shaders/ Slang
|
||||
src/
|
||||
lib.rs
|
||||
bitstream/ pure Rust, no Vulkan
|
||||
device.rs requirements, Context
|
||||
pipeline.rs layouts, spec constants, subgroup sizes, cache
|
||||
encoder/ EncodeConfig, Encoder, EncodeFuture, EncodedFrame
|
||||
decoder/ Decoder, DecodedFrame
|
||||
sync.rs TimelinePoint, QueueLock
|
||||
tests/
|
||||
```
|
||||
|
||||
### build.rs and shaders
|
||||
|
||||
`build.rs` runs `slangc` (or `$SLANGC`) for each entry point, with
|
||||
`-target spirv -profile spirv_1_5 -emit-spirv-directly -fvk-use-entrypoint-name`.
|
||||
It does not pass `-fvk-invert-y`, because everything here is compute. Output
|
||||
goes to `OUT_DIR`, is embedded with `include_bytes!`, and a missing or failing
|
||||
`slangc` fails the build with the command it ran.
|
||||
|
||||
The guest image builder (`build/Containerfile`, builder stage) gains
|
||||
`shader-slang` in its `pacman -S` list. The runtime stages do not.
|
||||
|
||||
Shaders, ported one to one from upstream:
|
||||
|
||||
| Slang | Upstream |
|
||||
|---|---|
|
||||
| `common.slang` | `dwt_common.h`, `dwt_swizzle.h`, `dwt_quant_scale.h`, `constants.h` |
|
||||
| `rgb_to_ycbcr.slang` | none, ours |
|
||||
| `dwt.slang` | `dwt.comp` |
|
||||
| `wavelet_quant.slang` | `wavelet_quant.comp` |
|
||||
| `analyze_rate_control.slang` | `analyze_rate_control.comp`, `analyze_rate_control_finalize.comp` |
|
||||
| `resolve_rate_control.slang` | `resolve_rate_control.comp` |
|
||||
| `block_packing.slang` | `block_packing.comp` |
|
||||
| `wavelet_dequant.slang` | `wavelet_dequant.comp`, SSBO mode only |
|
||||
| `idwt.slang` | `idwt.comp`, FP16 and FP32 variants |
|
||||
|
||||
Buffers are reached through buffer device addresses carried in push constants.
|
||||
Images and samplers are bound with push descriptors (`VK_KHR_push_descriptor`),
|
||||
so no descriptor pool exists. Every pipeline declares the subgroup size
|
||||
upstream requires, and `computeFullSubgroups` where upstream asks for full
|
||||
groups.
|
||||
|
||||
### device
|
||||
|
||||
`DeviceRequirements` lists what nespyro needs, in a form a caller merges into a
|
||||
device it is about to create, as nescapture does for the game's device:
|
||||
|
||||
- Vulkan 1.3 core: `synchronization2`, `subgroupSizeControl`,
|
||||
`computeFullSubgroups`, `maintenance4`.
|
||||
- Vulkan 1.2 core: `timelineSemaphore`, `bufferDeviceAddress`,
|
||||
`storageBuffer8BitAccess`, `shaderFloat16` (encoder only).
|
||||
- Vulkan 1.1 core: `storageBuffer16BitAccess`.
|
||||
- Vulkan 1.0: `shaderInt16`, `shaderStorageImageWriteWithoutFormat`.
|
||||
- `VK_KHR_push_descriptor`.
|
||||
- Subgroup operations checked against the device's properties: the encoder
|
||||
needs basic, vote, arithmetic, ballot, shuffle, shuffle relative and
|
||||
clustered. The decoder needs the same without clustered. Each pipeline's
|
||||
subgroup size range, taken from upstream (for example 16..64 for block
|
||||
packing, and exactly 64, 16 or 32 for the resolve pass), must intersect the
|
||||
device's `minSubgroupSize..maxSubgroupSize`, with compute in
|
||||
`requiredSubgroupSizeStages`.
|
||||
|
||||
`Context::from_existing(instance, physical_device, device, queue, family,
|
||||
QueueLock)` wraps a device the caller owns. `Context::supports_encode()` and
|
||||
`supports_decode()` answer from the device's real features and properties,
|
||||
never from its name. There is no context that creates its own device; both
|
||||
callers already have one.
|
||||
|
||||
### sync
|
||||
|
||||
`TimelinePoint { semaphore, value }` has the same shape as pixelforge's. Waits
|
||||
use `ALL_COMMANDS`, for the reason pixelforge gives: nespyro's submissions begin
|
||||
with transitions whose stages the caller cannot know.
|
||||
|
||||
`QueueLock` serialises nespyro's submits against anything else on the same
|
||||
queue. A caller whose queue is not shared passes an uncontended lock.
|
||||
|
||||
## Encode
|
||||
|
||||
```rust
|
||||
let mut enc = Encoder::new(ctx, EncodeConfig::new(w, h)
|
||||
.with_chroma(Chroma::Yuv444)
|
||||
.with_depth(Depth::Sixteen)
|
||||
.with_colour(ColourDescription::bt2020_pq())
|
||||
.with_frame_rate(120, 1)
|
||||
.with_target_bitrate(400_000_000)
|
||||
.with_packet_size(1200))?;
|
||||
let fut = enc.encode_after(src_image, src_format, src_layout, &[blit_done])?;
|
||||
let frame: EncodedFrame = fut.await?;
|
||||
```
|
||||
|
||||
**Input.** The caller's RGB image, sampled in place: `B8G8R8A8`, `R8G8B8A8`,
|
||||
`A2B10G10R10` or `R16G16B16A16_SFLOAT`. No copy is made first.
|
||||
|
||||
**Per slot** (two in flight): the Y, Cb and Cr plane images, the block
|
||||
metadata and bitstream buffers, and a persistently mapped host-cached readback
|
||||
buffer. Upstream recreates its buffers on every call; nespyro allocates them
|
||||
once.
|
||||
|
||||
**Shared across slots**: the wavelet images, the block statistics, the quant
|
||||
and bucket buffers, and the pipelines. The GPU runs the encode work in order,
|
||||
so these need no copies.
|
||||
|
||||
**One submission per frame:**
|
||||
|
||||
1. Wait on the caller's points.
|
||||
2. `rgb_to_ycbcr` writes the three planes. The matrix and the full range are
|
||||
applied here, and so is the 4:2:0 subsampling, with centre siting.
|
||||
3. `dwt`, 13 dispatches for 4:2:0 and 15 for 4:4:4.
|
||||
4. `wavelet_quant` then `analyze_rate_control`, per band (42 bands for 4:2:0,
|
||||
48 for 4:4:4).
|
||||
5. `analyze_rate_control_finalize`, then `resolve_rate_control`.
|
||||
6. `block_packing` per band. Finished 32×32 block packets land in GPU order.
|
||||
7. Copy the metadata table and the first `target_bytes + headroom` bytes of the
|
||||
bitstream into the readback buffer. Rate control caps the frame at its
|
||||
target, so the copy is sized to the target, not to the whole buffer.
|
||||
8. Signal the encoder's timeline.
|
||||
|
||||
**Completion thread.** It waits for that value, then packetizes on the CPU:
|
||||
the 8-byte sequence header first, carrying the configured colour fields and
|
||||
the frame counter, then every non-empty block in block-index order. A new
|
||||
packet starts when the next block would pass `packet_size`, and blocks are
|
||||
never split. It resolves the future with:
|
||||
|
||||
```rust
|
||||
struct EncodedFrame {
|
||||
data: Vec<u8>,
|
||||
packets: Vec<Range<usize>>, // each independently parseable
|
||||
critical_packets: usize, // leading packets covering the lowest bands
|
||||
pts: u64,
|
||||
stats: EncodeStats, // GPU timestamps per pass
|
||||
}
|
||||
```
|
||||
|
||||
**Rate control.** The target bits per frame is bitrate ÷ fps.
|
||||
`set_target_bitrate` takes effect on the next frame with no rebuild. There is
|
||||
no IDR and no GOP, because every frame stands alone. `set_colour_description`
|
||||
changes the matrix and the header bits. Changing the size, chroma or depth
|
||||
means building a new `Encoder`, as in pixelforge.
|
||||
|
||||
**Errors.** A block's header size must equal its size in the metadata table,
|
||||
or packetizing fails with an error, as upstream checks it. A block cannot
|
||||
exceed the 12-bit `payload_words` limit: the largest possible 32×32 block is
|
||||
about 620 words (8-byte header, 16 × 3 bytes of codes, 16 × 8 subblocks × 18
|
||||
planes, 128 bytes of signs), well under 4095. A test pins that bound. A lost
|
||||
device or failed submit is an error from the future.
|
||||
|
||||
## Decode
|
||||
|
||||
```rust
|
||||
let mut dec = Decoder::new(ctx, DecodeConfig::new(w, h, chroma, depth)
|
||||
.with_consumer_queue_family(graphics_family))?;
|
||||
match dec.push_packet(datagram)? { Push::Accepted | Push::Stale => {},
|
||||
Push::Reconfigure(info) => { /* rebuild the decoder */ } }
|
||||
if dec.readiness().is_complete_enough() {
|
||||
let frame = dec.decode_after(&[])?;
|
||||
}
|
||||
```
|
||||
|
||||
**Packets.** `push_packet(&[u8]) -> Result<Push>` parses one packet on the CPU,
|
||||
copying only block payloads. It handles the 3-bit frame counter as upstream
|
||||
does: an older frame's packet is `Stale` and dropped, a newer frame's clears
|
||||
the state. A header that disagrees with the decoder's size, chroma or depth
|
||||
returns `Reconfigure`, and nothing is rebuilt behind the caller's back. A
|
||||
truncated packet, or one whose block sizes run past its end, is `Err(Malformed)`,
|
||||
never skipped.
|
||||
|
||||
**Readiness.** `readiness()` reports blocks received against the header's
|
||||
total, and whether the two lowest bands are complete. When to decode is the
|
||||
caller's decision (part 2's policy). `is_complete_enough()` is upstream's rule:
|
||||
every block, or at least 90% with the two lowest bands complete.
|
||||
|
||||
**One submission per frame:**
|
||||
|
||||
1. Write block payloads and the offset table into the slot's mapped buffer. On
|
||||
memory that is both device-local and host-visible (ReBAR, or an integrated
|
||||
GPU) the shader reads it there. Otherwise a staging copy moves it to
|
||||
device-local memory first.
|
||||
2. `wavelet_dequant` per band. A missing block has offset `u32::MAX` and
|
||||
decodes as zeros.
|
||||
3. `idwt` per level per component, FP16 or FP32 as chosen at creation. The last
|
||||
level adds the DC offset back and writes straight into the output planes.
|
||||
4. Signal the decoder's timeline.
|
||||
|
||||
**Output.**
|
||||
|
||||
```rust
|
||||
struct DecodedFrame {
|
||||
y: PlaneView, cb: PlaneView, cr: PlaneView,
|
||||
width: u32, height: u32, chroma: Chroma, depth: Depth,
|
||||
colour: ColourDescription, // from the sequence header
|
||||
missing_blocks: u32,
|
||||
ready: TimelinePoint,
|
||||
}
|
||||
```
|
||||
|
||||
The planes come from a ring of three slots, R8 or R16, in `GENERAL`, with
|
||||
sampled and storage usage. When the consumer's queue family differs, they are
|
||||
created `CONCURRENT` across the decode and consumer families, as pixelforge
|
||||
does for `with_consumer_queue_family`, so no ownership transfer is needed. The
|
||||
consumer waits on `ready` in its own submit; nothing waits on the CPU.
|
||||
Dropping the frame returns its slot. If all three slots are held,
|
||||
`decode_after` returns `Busy` instead of stalling, which makes a consumer
|
||||
holding frames too long visible.
|
||||
|
||||
A frame decoded with blocks missing succeeds and says so in `missing_blocks`,
|
||||
so the loss is visible to the caller rather than looking like a whole frame.
|
||||
|
||||
## Testing
|
||||
|
||||
GPU and reference tests carry `#[ignore = "needs …"]` naming exactly what they
|
||||
need, and they fail, never skip, when that is absent.
|
||||
|
||||
**Bitstream (pure Rust, always run):**
|
||||
- Header pack and unpack against `bitstream.md`'s field layouts.
|
||||
- Block-index ordering and the per-level/band tables against values taken from
|
||||
`metal/pyrowave_bitstream.cpp`.
|
||||
- Packetize then parse reproduces every block exactly. No packet passes
|
||||
`packet_size` unless one block alone is larger, and no block is split.
|
||||
- Frame counter wrap: stale packets drop, a new frame clears state.
|
||||
- A truncated packet or an impossible `payload_words` returns `Malformed`, and
|
||||
the parser never reads past the end.
|
||||
- The largest possible block fits in 12 bits of words.
|
||||
|
||||
**GPU (one device):**
|
||||
- Round trip over synthetic sources (gradients, noise, hard text-like edges,
|
||||
flat colour) at 1920×1080, 1366×768 (not a multiple of 32) and one under the
|
||||
128 minimum. Each runs in 4:2:0 and 4:4:4, and in R8 and R16. Decoded PSNR
|
||||
must clear a threshold for each bitrate.
|
||||
- Rate control really limits: encoded size is at or under the target at every
|
||||
tested rate, and above a floor, so a nearly empty frame decoding to grey
|
||||
cannot pass. A low target must give measurably lower PSNR than a high one.
|
||||
- Determinism: the same input encodes to byte-identical output twice, despite
|
||||
the GPU's unordered block writes.
|
||||
- Colour: known RGB values map to the expected full-range YCbCr for BT.709 and
|
||||
BT.2020 PQ. The header's colour bits match the configuration, and the decoder
|
||||
reads the same back.
|
||||
- Loss: dropping non-critical packets still decodes, `missing_blocks` equals
|
||||
what was dropped, and the lowest bands are intact. Dropping a critical packet
|
||||
shows in `readiness()`.
|
||||
|
||||
**Reference cross-check.** A small C++ harness is built from PyroWave
|
||||
`89f7e47` and Granite `1b2d1801` (`checkout_granite.sh`), outside cargo. Its
|
||||
path is given in `NESPYRO_REFERENCE`, and the test fails if that is set but the
|
||||
binary is missing.
|
||||
- Our encoder into the reference decoder, and the reference encoder into ours.
|
||||
Pass is upstream's own tolerance from `pyrowave_device_validation.cpp`: at
|
||||
most 1 LSB luma error and at most 2 chroma.
|
||||
- Our encoder beside theirs on identical YCbCr input. Slang and glslang may
|
||||
round floating point differently, so byte identity is not required. The test
|
||||
reports the fraction of blocks that match exactly, and the size and PSNR
|
||||
difference, as a diagnostic.
|
||||
|
||||
**Latency.** A benchmark takes GPU timestamps around every pass, ours beside
|
||||
the reference, at 1080p and 4K. It measures GPU time, not command recording.
|
||||
|
||||
## Future work: mobile path (path B)
|
||||
|
||||
Left out on purpose; the device check refuses such devices with a clear error
|
||||
rather than degrading. Adding them means:
|
||||
|
||||
- The fragment inverse transform: `idwt.vert` and `idwt.frag` with its three
|
||||
`CHROMA_CONFIG` variants, separate vertical and horizontal passes into
|
||||
R16F/R32F (RG for chroma) intermediates, and three scissored draws per pass
|
||||
with the `EDGE_CONDITION` specialization constant.
|
||||
- Its output views need `COLOR_ATTACHMENT` usage, and the pass uses dynamic
|
||||
rendering.
|
||||
- `wavelet_dequant` storage modes 1 (R8/R16/R32 uniform texel buffers, for
|
||||
devices without 8-bit storage) and 2 (linear-tiled images aliased on one
|
||||
buffer).
|
||||
- Upstream's vendor preference table (`pyrowave_decoder.cpp:913-930`) choosing
|
||||
fragment over compute for Mali, Qualcomm and PanVK.
|
||||
- Hardware to test on, which this project does not yet have.
|
||||
|
||||
## Part 4 notes, carried forward
|
||||
|
||||
Found while reading nesrecon, to fix when part 4 opens the convert shader:
|
||||
|
||||
- The Vulkan Video path creates its RGBA target as `R8G8B8A8_UNORM`
|
||||
(`renderer.rs:616,624`), but `yuv_convert.slang:3` writes it as `rgba16f`,
|
||||
and `probe_signal` reads it as `PASS_FORMAT` (`renderer.rs:1493-1499`). The
|
||||
DMA-BUF path correctly uses `PASS_FORMAT`.
|
||||
- One `vk::Queue` per family is shared between the render thread and
|
||||
pixelforge's decode thread with no lock (`vulkan/mod.rs:555-578`). nespyro's
|
||||
`QueueLock` is where that gets fixed.
|
||||
|
||||
## Found while building
|
||||
|
||||
Recorded after part 1 was built, for parts 2 to 4 to start from.
|
||||
|
||||
### Where the crate differs from this design
|
||||
|
||||
- `Context::from_existing` takes the `Roles` the device was created for.
|
||||
Vulkan cannot say which features a device was created with, and the decoder
|
||||
needs to know whether `shaderFloat16` is on before choosing its FP16 tile.
|
||||
Decode-only requirements ask for FP16 only where the device has it.
|
||||
- `Encoder::encode_planes_after` encodes the caller's own YCbCr planes. It
|
||||
exists so the encoder can be compared with the reference on identical input,
|
||||
and it lets a caller with its own conversion skip ours.
|
||||
- Decoder output planes are padded to the codec's alignment. The final inverse
|
||||
transform writes whole 32×32 tiles; upstream relies on out-of-bounds stores
|
||||
being discarded, which core Vulkan only promises with `robustImageAccess`.
|
||||
`PlaneView` gives both the picture size and the image size.
|
||||
- `Decoder::stats` reports the GPU time of the last finished decode.
|
||||
- Both halves record their many dispatches once and reuse them. On Mesa's ANV
|
||||
recording costs 6 to 14 µs per dispatch; per frame it was 416 µs of a
|
||||
decode. Per-frame values reach the shaders through a small parameters buffer.
|
||||
- Device requirements also check each plane and wavelet format for sampled
|
||||
and format-less storage use, which core Vulkan guarantees only for
|
||||
`R32_SFLOAT`.
|
||||
|
||||
### Measured, Arc A310, Mesa ANV
|
||||
|
||||
| | nespyro | reference |
|
||||
|---|---|---|
|
||||
| encode 1080p 4:2:0, GPU, excluding colour conversion | 0.94 ms | 1.05 ms |
|
||||
| encode 4K 4:2:0 | 3.07 ms | 3.22 ms |
|
||||
| encode 1080p 4:4:4 | 1.36 ms | 1.50 ms |
|
||||
| decode 1080p 4:2:0, GPU | 0.56 ms | |
|
||||
| `encode_after` / `decode_after` CPU, 1080p 4:2:0 | 140 / 134 µs | |
|
||||
|
||||
Both decoders give bit-identical planes on every stream either encoder wrote.
|
||||
On identical input the encoders tie on smooth content and nespyro is level or
|
||||
ahead on edges and noise.
|
||||
|
||||
Subgroup sizes follow Granite's rule, which upstream's shaders were tuned
|
||||
under: varying when a pass's range covers the device's, otherwise the smallest
|
||||
in range. Taking the largest made rate-control analysis 73% slower on Intel.
|
||||
|
||||
### Upstream bugs, worth reporting to PyroWave
|
||||
|
||||
1. **Scratch payload undersized.** The quantizer's scratch is sized at aligned
|
||||
width × height × 2 bytes. High-entropy 4:4:4 content needs more (3.4 MB
|
||||
against 2.1 MB at 1366×768), and the encoder then packs blocks from bytes it
|
||||
never wrote: a block whose header claims 115 words holds 95. nespyro sizes
|
||||
it by the real bound, 136 bytes per 8×8 block.
|
||||
2. **Rate-control prefix sum stops short.** `analyze_rate_control_finalize`
|
||||
steps while `step < 512 / 2`, so the upper half of its lanes miss part of
|
||||
the running total. Frames land under target, not over. Ported unchanged.
|
||||
3. **Out-of-bounds shared read.** `analyze_rate_control` reads
|
||||
`shared_rate_cost[gl_SubgroupInvocationID]` past its 16 entries on wider
|
||||
subgroups. The value is unused; nespyro clamps the index.
|
||||
4. **Stale bytes in the stream.** A block's unused sign bits come from
|
||||
uninitialised shared memory and its padding from whatever an earlier frame
|
||||
left. Harmless to decoders, but non-deterministic and leaks old memory;
|
||||
nespyro writes zeros.
|
||||
|
||||
### Toolchain and driver bugs
|
||||
|
||||
- **Slang 2026.17** emits `OpGroupNonUniformShuffleXor` without declaring
|
||||
`GroupNonUniformShuffle` when nothing else in the shader does. `spirv-val`
|
||||
catches it; `shuffle_xor` in `common.slang` works around it. `build.rs`
|
||||
requires `spirv-val` for that reason.
|
||||
- **Mesa ANV** spends 12.6 ms of CPU recording `vkCmdFillBuffer` of an
|
||||
8,396,864-byte buffer, and 3 µs for a 16,785,472-byte one. nespyro records
|
||||
its fills once, so it no longer matters here, but it is worth a Mesa report.
|
||||
|
||||
### Found by part 3
|
||||
|
||||
- **`Readiness::critical_complete` cannot be trusted.** A block the encoder
|
||||
quantized to nothing is never sent, and its absence is indistinguishable from
|
||||
a loss. Noise leaves the coarsest high-pass bands empty often enough that a
|
||||
frame with every packet delivered read as critically incomplete, so no
|
||||
partial frame would ever have been decoded. Upstream has the same flaw and
|
||||
answers it with an out-of-band mask of active blocks. Here the transport
|
||||
knows which datagrams carried the critical packets and says so
|
||||
(`CollectedFrame::critical_complete`); `Readiness::is_complete_enough_with`
|
||||
takes that answer. Found by the real-frame transport test.
|
||||
|
||||
### For the parts that follow
|
||||
|
||||
- Part 2: a frame at 200 Mbit/s and 60 fps is about 416 KB in 1200-byte
|
||||
packets, a few hundred per frame. `EncodedFrame::critical_packets` is small,
|
||||
typically one to a few packets.
|
||||
- Part 4: nesrecon's convert shader takes two planes; nespyro gives three, in
|
||||
`GENERAL`, padded, with the picture size in `PlaneView`.
|
||||
@@ -0,0 +1,122 @@
|
||||
# PyroWave decode in nesrecon / nesvideo
|
||||
|
||||
Status: built, 2026-10-01 (nescore `e20a4d2`, `0e729ef`). Part 4 of 4; parts 1-3 are
|
||||
`2026-09-27-nespyro-codec-crate-design.md`,
|
||||
`2026-10-01-pyrowave-transport-design.md`,
|
||||
`2026-10-01-pyrowave-nescapture-design.md`. Code lives in nescore
|
||||
(`crates/nesrecon`, `apps/nesvideo`); nespyro is a git dep on nestri at the
|
||||
same rev nesprotocol is pinned to.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **nespyro decodes on nesrecon's device**, from the decode thread, as one
|
||||
more backend beside Vulkan Video / VA-API / CPU. Its planes stay on the GPU
|
||||
and are converted by the render thread like a Vulkan Video picture.
|
||||
- **The device is no longer gated on H.264 Vulkan Video decode.**
|
||||
`shared_decode_device` exists when either pixelforge or nespyro fits.
|
||||
- **Planes are always `Depth::Sixteen`.** PyroWave decodes in float and the
|
||||
stream carries no depth; R16 costs little and never bands an HDR stream.
|
||||
- **Converter: a second entry point**, three planes, chroma shift in the
|
||||
uniform's spare word. The two-plane path is untouched, so its output and
|
||||
tests do not move.
|
||||
- **Fix while open:** the Vulkan-path RGBA target is `R8G8B8A8_UNORM` while the
|
||||
shader writes `rgba16f` and the probe reads `PASS_FORMAT`; it becomes
|
||||
`PASS_FORMAT`. That changes the Vulkan Video path's output (it was undefined
|
||||
behaviour), so the PSNR baseline is re-taken for that path.
|
||||
- **Deblock is off for PyroWave**: it has no block grid. `block_grid_for`
|
||||
returns none, as for anything without one.
|
||||
- **Busy is not an error.** nespyro has three output slots and the renderer
|
||||
can hold all three (hand-off slot, current picture, retire queue); a frame
|
||||
that finds none free is dropped and counted, and the next decodes.
|
||||
|
||||
## Device (`vulkan/mod.rs`)
|
||||
|
||||
- `nespyro::DeviceRequirements::query(.., Roles::DECODE)` beside pixelforge's,
|
||||
independently. Device preference also accepts a nespyro-capable device.
|
||||
- Features: the 1.0 ones in `PhysicalDeviceFeatures`, 16-bit storage in the
|
||||
existing `Vulkan11Features`, subgroup size control and full subgroups in the
|
||||
existing `Vulkan13Features`, and a new `Vulkan12Features` for 8-bit
|
||||
storage, int8, float16 (when available), buffer device address and timeline
|
||||
semaphores — replacing the standalone timeline struct, which may not be
|
||||
chained beside it.
|
||||
- Queue: nespyro gets the compute-only family's queue 0, which nesrecon
|
||||
never submits to; pixelforge may use it too, but only from the same decode
|
||||
thread, so the two are serialised by that thread. Without a compute-only
|
||||
family, a second graphics queue if there is one. Without either -- every
|
||||
Intel GPU under ANV, whose one family has one queue -- the render queue
|
||||
itself, under a `QueueLock` the render thread holds around every submit,
|
||||
present and `vkDeviceWaitIdle` (`VkCtx::queue_lock`, `VkCtx::wait_idle`).
|
||||
The first build refused PyroWave in that case, which on this machine meant
|
||||
the client never advertised it and the hub refused every request.
|
||||
- `SharedVulkanDevice` carries `pyro: Option<(family, index)>`.
|
||||
- `nesrecon::decodes(PyroWave, _)` answers from a flag set when the device was
|
||||
created with nespyro's additions; nesvideo advertises PyroWave only then.
|
||||
|
||||
Still open, out of scope: pixelforge's transfer queue can alias the graphics
|
||||
queue on a device without a transfer or compute family, and pixelforge has no
|
||||
lock to share. Unchanged by this part.
|
||||
|
||||
## Decode thread (`decode/`)
|
||||
|
||||
- The channel carries `enum VideoUnit { Coded(Vec<u8>, VideoCodec),
|
||||
PyroWave(PyroUnit) }`, `PyroUnit { packets: Vec<Bytes>, critical_complete,
|
||||
ts_ms }`. `VideoCodec` gains `PyroWave`; every closed match says what it
|
||||
means there.
|
||||
- `PyroDecoder` (new backend): built lazily for the stream's size and chroma,
|
||||
rebuilt on `Push::Reconfigure`. Per unit: push every packet; decode when
|
||||
`readiness().is_complete_enough_with(critical_complete)`; otherwise drop
|
||||
and count. Output `DecodedFrame::PyroWave(PyroPicture)` into the same sink.
|
||||
- `decode_ns` is CPU time as for the others; nespyro's GPU time goes to the
|
||||
log at trace once a second.
|
||||
|
||||
## Render (`vulkan/renderer.rs`, `yuv_convert.*`)
|
||||
|
||||
- `import_frame` takes `PyroPicture` into the same slot the Vulkan picture
|
||||
uses; retired the same way, after the frame that sampled it.
|
||||
- The render submit waits on the picture's `ready` point when it has one.
|
||||
- `yuv_convert_planar` entry: three `Texture2D` loads, chroma at
|
||||
`(x >> shift, y >> shift)`, same matrix/range/peak code. R16 full range.
|
||||
- Matrix and range from the picture's `ColourDescription`, per frame.
|
||||
|
||||
## nesvideo
|
||||
|
||||
- `PyroCollector` frames go to the decode channel as `VideoUnit::PyroWave`.
|
||||
A full channel drops and counts; no IDR, nothing to resync.
|
||||
- The sequence header (first packet) gives size and colour: `stream_dims`, and
|
||||
`bitstream_colour` as a `StreamColour`, so the existing colour logic picks
|
||||
the swapchain (HDR10 for PQ) exactly as for the other codecs.
|
||||
- `client_capabilities` asks `nesrecon::decodes` for PyroWave like the rest.
|
||||
- The overlay's "not decoded yet" note goes; the PyroWave rate line stays.
|
||||
|
||||
## Tests (last step)
|
||||
|
||||
- Unit: channel/unit plumbing, colour mapping, `block_grid_for`, the planar
|
||||
shader's text checks beside the existing ones.
|
||||
- Existing nesrecon/nesvideo tests stay green.
|
||||
- Full stack by eye: PyroWave on screen, switching to and from a hardware
|
||||
codec, 4:2:0 and 4:4:4, SDR and HDR.
|
||||
|
||||
## Found while building
|
||||
|
||||
- **The channel type stayed.** A PyroWave unit is serialised into the existing
|
||||
`Vec<u8>` (`[flags][count]` then length-prefixed packets,
|
||||
`nesrecon::encode_pyro_unit`), so PyroWave is one more `VideoDecoder`
|
||||
variant and nothing else on the channel changed. One copy a frame, about
|
||||
37 MB/s at 300 Mbit/s.
|
||||
- **A picture keeps its decoder alive.** Dropping a `nespyro::Decoder`
|
||||
destroys its output images even while a `DecodedFrame` of them is held, and
|
||||
a codec switch or resize drops one while the renderer still shows its last
|
||||
picture. `PyroPicture` holds an `Arc` of the decoder, released after the
|
||||
slot. Worth fixing in nespyro itself when it is next opened: the output
|
||||
images should outlive the decoder or the frame should own them.
|
||||
- **Capabilities are sent twice.** The first message goes out at connect,
|
||||
before the renderer has created the device that decides whether PyroWave
|
||||
decodes. A second follows once it has; the host renegotiates to the codec
|
||||
it was already on. A `--codec pyrowave` request waits for the second, or the
|
||||
hub would refuse it.
|
||||
- **Deblock is suppressed, not switched off.** `PostProcessChain` gained a
|
||||
per-type suppression, so a person's own pass toggles are untouched and the
|
||||
pass comes back on a switch to a block codec.
|
||||
- **A device shared for PyroWave alone** is not offered to the Vulkan Video
|
||||
decoder (`SharedVulkanDevice::video_decode`), which would otherwise fail and
|
||||
log an error on every switch to H.264.
|
||||
@@ -0,0 +1,89 @@
|
||||
# PyroWave in nescapture
|
||||
|
||||
Status: approved design, 2026-10-01. Part 3 of 4. Parts 1 and 2:
|
||||
`2026-09-27-nespyro-codec-crate-design.md`, `2026-10-01-pyrowave-transport-design.md`.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Vulkan 1.3 instance.** `vkCreateInstance` raises anything below 1.3 to 1.3
|
||||
where the loader has it (today: 1.0 → 1.1). Only the client app has to care
|
||||
about old Vulkan; nescapture runs on the game's machine, which we control.
|
||||
- **Shared device only.** nespyro runs on the game's device or not at all. On
|
||||
the own-device fallback a PyroWave request is refused and logged; a CPU round
|
||||
trip of a frame defeats the codec's reason to exist.
|
||||
- **Decoupled from Vulkan Video.** The game's device gets pixelforge's
|
||||
additions, nespyro's, both, or neither, each checked on its own. A GPU with
|
||||
no video encode can still stream PyroWave once a client asks; until then
|
||||
nothing is encoded, and the log says why.
|
||||
|
||||
## Device (`instance.rs`, `shared.rs`, `device.rs`)
|
||||
|
||||
- `prepare` asks pixelforge `encode_device_requirements` and nespyro
|
||||
`DeviceRequirements::query(Roles::ENCODE)` independently.
|
||||
`Additions { video: Option<pixelforge::DeviceRequirements>, pyro:
|
||||
Option<nespyro::DeviceRequirements>, queues }`. The effective version
|
||||
(min of instance and physical device) must be 1.3 for `pyro`.
|
||||
- `Feature` gains nespyro's bits: Storage16Bit, Storage8Bit, ShaderInt8,
|
||||
ShaderFloat16, BufferDeviceAddress, SubgroupSizeControl,
|
||||
ComputeFullSubgroups, and the 1.0 core ShaderInt16 and
|
||||
ShaderStorageImageWriteWithoutFormat. 1.0 bits live in
|
||||
`VkPhysicalDeviceFeatures`: patched in place inside a chained
|
||||
`VkPhysicalDeviceFeatures2`, else in a copy of `*pEnabledFeatures` the create
|
||||
info points at instead, else in a copy of our own. Restore covers all three.
|
||||
- Extension `VK_KHR_push_descriptor` added with nespyro.
|
||||
- `plan_queues` plans encode only with pixelforge, compute with either,
|
||||
transfer with pixelforge. nespyro uses the compute queue, `QueueLock::none()`:
|
||||
one encoder thread submits for both, at most one queue of ours per family,
|
||||
internally synchronized where shared with the game.
|
||||
- `SharedDevice::pyro_context() -> Option<nespyro::Context>`.
|
||||
`SharedDevice::video_context()` returns an error when pixelforge's additions
|
||||
were not made.
|
||||
|
||||
## Encoder thread (`encode.rs`)
|
||||
|
||||
- `enum Codec { Hw(HwCodec), PyroWave }` in `EncoderConfig`,
|
||||
`EncodeSettingsChange`, `current_codec` (protocol id). `HwCodec` unchanged.
|
||||
Every match says what PyroWave does; depth clamping, GOP, intra refresh and
|
||||
IDR apply only to `Hw`. IDR on PyroWave is a no-op.
|
||||
- `enum Backend { Hw(PerFrameEncoder), Pyro(PyroEncoder) }`. `PyroEncoder`
|
||||
owns `nespyro::Encoder` and its rebuild key (w, h, depth, chroma, colour,
|
||||
packet size). Bitrate-only change → `set_target_bitrate`, no rebuild.
|
||||
- Input: the slot image in `GENERAL`, `encode_after(view, Source, blit)`.
|
||||
`Source` from the same `source_spec`/`stream_spec`; only BT.709 SDR and
|
||||
BT.2020 PQ, anything else refused. pixelforge `TimelinePoint` converted at
|
||||
the boundary.
|
||||
- Slot lifetime: a PyroWave frame's slot is held until its future resolves; up
|
||||
to two held (nespyro's in-flight limit). Hardware path unchanged.
|
||||
- `EncodedFrame.future` becomes an enum of the two futures.
|
||||
- Codec switches use the existing `drop_encoder` + `needs_reconfig_flag`.
|
||||
|
||||
## Commands, IPC, stats
|
||||
|
||||
- `MSG_ENCODE_SETTINGS` via `decode_encode_settings_ext`. Codec 4 →
|
||||
`PyroWave`, with chroma and packet size. Refused (warned) with no nespyro
|
||||
context, no or too small packet size, or rate 0.
|
||||
- `host_caps` includes PyroWave bits only with a nespyro context.
|
||||
- IPC: PyroWave frames via `chunk_frame`, each message under the existing 250
|
||||
ms timeout. A timed-out message drops the rest of the frame and counts it in
|
||||
`dropped`; other errors reconnect as today. Width/height from the frame, not
|
||||
frozen at pipeline creation (fixes a stale-size bug for every codec).
|
||||
- `enc_ms` from nespyro `gpu_ns`. Scratch overflow warned once per episode.
|
||||
- Fix while open: the CPU fallback's hardcoded `Codec::H264` in
|
||||
`InputImage::new`.
|
||||
|
||||
## Tests
|
||||
|
||||
- Unit: planning with one or both sets; 1.0 feature bits in `pEnabledFeatures`,
|
||||
in `Features2`, in neither, and restoration; the instance raise decision;
|
||||
command decode and refusals; the two-slot hold rule.
|
||||
- GPU `#[ignore]`, in `crates/nespyro/tests/transport.rs` beside nespyro's GPU
|
||||
harness: real frames → `chunk_frame` → `FrameAssembler` → `pack_datagrams` →
|
||||
`PyroCollector` → `Depacketizer`; readiness equals the encoder's own packets,
|
||||
clean and with loss. (Moved from part 2.) nescapture's own mapping of an
|
||||
`EncodedFrame` onto `chunk_frame` (`pyro_messages`) is unit-tested on the
|
||||
CPU, since `EncodedFrame` can be built by hand.
|
||||
- Found by it: `Readiness::critical_complete` false-rejects frames with empty
|
||||
coarse blocks; the datagram header gained a critical count so the collector
|
||||
can answer exactly (see parts 1 and 2).
|
||||
- Guest image: rebuild, run a title, send a PyroWave `MSG_ENCODE_SETTINGS` to
|
||||
`/tmp/nescapture-cmd.sock`, and see PyroWave frames leave neshub.
|
||||
@@ -0,0 +1,176 @@
|
||||
# PyroWave transport
|
||||
|
||||
Status: approved design, 2026-10-01. Part 2 of 4; part 1 is
|
||||
`2026-09-27-nespyro-codec-crate-design.md`.
|
||||
|
||||
## Why
|
||||
|
||||
The media path was built for hardware codecs over the internet: a frame is one
|
||||
IPC message, the hub cuts it into byte slices, the client needs every slice,
|
||||
keyframes ride reliable streams, and a controller ratchets the bitrate. None of
|
||||
that fits PyroWave:
|
||||
|
||||
- A unix datagram tops out near 416 KB under the default `net.core.wmem_max`,
|
||||
and the hub reads into 2 MiB. A PyroWave frame is 400 KB at 200 Mbit/s and
|
||||
2 MB at 1 Gbit/s.
|
||||
- Byte slicing throws away what PyroWave is built around: packets that each
|
||||
decode on their own, and a decoder that shows a frame missing some of them.
|
||||
- Every PyroWave frame is intra. Flagged as a keyframe it would take the
|
||||
two-slot reliable path every frame; unflagged, `ResyncGate` withholds it.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Opt-in only.** `ClientCaps` advertises PyroWave, but `best()` and
|
||||
`CODEC_PREFERENCE` never return it. A session reaches it only by the client
|
||||
sending `MSG_ENCODE_SETTINGS` naming it.
|
||||
- **No controller.** PyroWave runs at a static rate: what the request names, or
|
||||
the hub's `--pyrowave-kbps` default (300 000) when it names 0. The request is
|
||||
CBR, which already sets the controller to Manual (`note_manual_target`), so
|
||||
it stands down with no change to `control.rs`. The existing controller is
|
||||
slated for replacement and gets nothing new.
|
||||
- **Critical packets sent twice**, first and again last, as datagrams. No
|
||||
reliable stream anywhere on the PyroWave path.
|
||||
- **Shared logic in nesprotocol, pure.** Chunking, reassembly of IPC messages,
|
||||
datagram packing and the client-side collector are functions and state
|
||||
machines with an explicit clock. neshub wires them to sockets. nesprotocol
|
||||
keeps zero dependencies and does not depend on nespyro: packets are byte
|
||||
ranges to it.
|
||||
- **The 4 MiB datagram buffer stays** for every codec. A hub-side skip guard
|
||||
keeps PyroWave's queue to about two frames instead.
|
||||
|
||||
## Scope
|
||||
|
||||
In part 2: nesprotocol, neshub, tests, a throughput bench. Not in part 2:
|
||||
|
||||
- nescapture (part 3) calls `chunk_frame`, reads the packet size from encode
|
||||
settings, and checks the format version it was built with.
|
||||
- nesvideo (part 4) dispatches datagram kind 2 to `PyroCollector` and feeds a
|
||||
decoder. Moved from part 2: without a decoder the client never advertises
|
||||
PyroWave, so the glue would be unreachable. The collector itself is built and
|
||||
tested here, end to end through a real iroh connection.
|
||||
|
||||
## Wire
|
||||
|
||||
### Negotiation
|
||||
|
||||
- `CODEC_PYROWAVE = 4`. `ClientCaps` bit `4*2 + DEPTH_8` = decodes PyroWave SDR,
|
||||
`+ DEPTH_10` = decodes PyroWave HDR. Chroma is not a capability: anything
|
||||
running nespyro decodes 4:2:0 and 4:4:4.
|
||||
- `PYROWAVE_FORMAT: u8 = 1`, meaning upstream `89f7e47`'s bitstream. The client
|
||||
appends it to `MSG_CLIENT_CAPS` as byte 3; `decode_client_caps` keeps reading
|
||||
the first two. `decode_pyrowave_format(payload) -> Option<u8>` reads it.
|
||||
- Encode settings grow two appended fields, read only when present:
|
||||
`[codec][rc][value u32][depth][chroma][packet_size u16]`. `CHROMA_420 = 0`,
|
||||
`CHROMA_444 = 1`. The client sends packet size 0; the hub fills it in.
|
||||
`decode_encode_settings` keeps its signature; a new
|
||||
`decode_encode_settings_ext` returns a struct with the two options.
|
||||
|
||||
The hub refuses (logs, does not forward) a PyroWave request from a client
|
||||
whose caps lacked PyroWave or carried a different `PYROWAVE_FORMAT`.
|
||||
|
||||
### IPC (nescapture → neshub), stream type `STREAM_PYROWAVE = 7`
|
||||
|
||||
The same 20-byte IPC header (codec `CODEC_PYROWAVE`, ts, width, height), and
|
||||
its data is:
|
||||
|
||||
```text
|
||||
[frame u32][msg index u16][msg count u16][critical u16][packet count u16]
|
||||
then per packet: [len u16][bytes]
|
||||
```
|
||||
|
||||
`frame` is a counter, `critical` the frame's total of leading critical packets
|
||||
(the same in every message). Each message holds at most `IPC_PYROWAVE_MAX`
|
||||
(64 KiB) of data and whole packets only, in frame order. A packet is at most
|
||||
`max(packet_size, 2488)` bytes, the largest block, so `u16` lengths suffice.
|
||||
|
||||
### Datagram kind `DGRAM_PYROWAVE = 2`
|
||||
|
||||
```text
|
||||
[1B kind][2B seq][2B index][2B total][2B critical][4B ts_ms][1B flags][payload]
|
||||
```
|
||||
|
||||
14 bytes. `seq` is the frame, `index`/`total` count datagrams in it.
|
||||
`critical` is how many leading indexes carry the critical packets: a receiver
|
||||
cannot work that out from the blocks it got, because an empty block is never
|
||||
sent and looks like a lost one (found in part 3; the header was 12 bytes
|
||||
before). The collector reports `critical_complete` from it. A payload
|
||||
is one whole packet, or one piece of a packet too large for a datagram:
|
||||
|
||||
- `FLAG_DUPLICATE` (bit 0): a second copy of a critical datagram. Same index.
|
||||
- `FLAG_PART` (bit 1): a piece of one packet. Pieces occupy consecutive
|
||||
indexes from one marked `FLAG_FIRST_PART` (bit 2) to one marked
|
||||
`FLAG_LAST_PART` (bit 3). Both ends are marked so a receiver missing a piece
|
||||
can tell an orphaned middle from the start of the next run. Only a block
|
||||
larger than the payload limit does this, at most two or three pieces.
|
||||
|
||||
`total` is capped at `PYRO_MAX_DATAGRAMS = 16384`, about 19 MB a frame.
|
||||
|
||||
## neshub
|
||||
|
||||
- **Packet size.** On forwarding a PyroWave request, the hub fills
|
||||
`packet_size` with the smallest `max_datagram_size() - 12` across streaming
|
||||
clients (1100 with none). The command forwarder in `main.rs` does it, since
|
||||
it already serialises every command to nescapture. If the path later shrinks
|
||||
below that, `pack_datagrams` sends the now-oversized packets in pieces like
|
||||
any oversized block; nothing is dropped.
|
||||
- **Listener.** The video socket accepts `STREAM_PYROWAVE` beside
|
||||
`STREAM_VIDEO`. `FrameAssembler` collects a frame's messages; an incomplete
|
||||
frame is abandoned when the next frame starts or after 100 ms, and counted.
|
||||
The receive buffer stays 2 MiB, which is 32 messages' worth.
|
||||
- **Broadcast.** The per-session video channel carries an enum: a hardware
|
||||
frame as `Bytes`, or a PyroWave frame as `Arc<PyroFrame>`. Both are
|
||||
refcounted, which also removes today's per-client copy of every frame.
|
||||
- **Writer.** PyroWave frames skip `ResyncGate` and `KeyframeSender` and keep a
|
||||
`seq` of their own. `pack_datagrams` lays them out (critical, rest, critical
|
||||
again) into one `BytesMut`, split without copying.
|
||||
- **Skip guard** (`pyrowave::should_skip`, shared with the bench). Before sending, with `backlog = DGRAM_BUFFER_BYTES -
|
||||
datagram_send_buffer_space()`: skip the whole frame if the backlog exceeds
|
||||
this frame's size, or if backlog plus frame exceeds the buffer. The skipped
|
||||
frame's `seq` is reused (the transport doc's rule). A frame larger than the
|
||||
buffer itself can never go out; that is warned once. Skips are counted and
|
||||
logged at debug.
|
||||
|
||||
## The collector (client side, in nesprotocol)
|
||||
|
||||
`PyroCollector::push(datagram, now) -> Vec<PyroFrame>`, plus `poll(now)`.
|
||||
Frames in flight by `seq`, each a slot per index (duplicates dropped by index).
|
||||
A frame is released when:
|
||||
|
||||
1. every index has arrived, or
|
||||
2. a newer frame has started and `GRACE` (2 ms) has passed since, or
|
||||
3. `IDLE` (20 ms) has passed since its *last* datagram.
|
||||
|
||||
Not a deadline from the first datagram: on a saturated path a frame takes a
|
||||
frame interval to arrive, so such a deadline tears every frame once the path is
|
||||
slower than it assumes. The end-to-end burst test found this.
|
||||
|
||||
Released frames carry `seq`, `ts_ms`, packets in index order (piece runs joined,
|
||||
broken runs dropped), received/total, and `critical_complete`. A datagram for a `seq` at or behind
|
||||
the newest released one is stale and dropped. No reorder window, no IDR: every
|
||||
frame stands alone. Whether a frame is decodable is the decoder's call.
|
||||
|
||||
Counts per frame: whole, partial, and datagrams lost, duplicated, stale.
|
||||
|
||||
## Testing
|
||||
|
||||
- **nesprotocol units:** chunk/assemble round trips including over 4 MB;
|
||||
packing never splits a packet that fits; oversized packets come back whole
|
||||
from pieces and a broken run is dropped; critical datagrams repeat with the
|
||||
same index; every release trigger; stale and duplicate datagrams; the
|
||||
negatives (`best()` never picks PyroWave, a wrong format is refused).
|
||||
- **neshub end to end (CPU):** synthetic packets in as IPC messages through the
|
||||
real listener and writer, over an iroh loopback connection, into a collector.
|
||||
Byte-exact, clean and with dropped datagrams; the skip guard engages under a
|
||||
blocked receiver.
|
||||
- **End to end with real nespyro frames:** moved to part 3, where nescapture
|
||||
produces them. Every released frame must give a `Depacketizer` the same
|
||||
readiness as the encoder's own packets.
|
||||
- **Throughput:** `examples/pyro_bench.rs` sends PyroWave-shaped frames at a
|
||||
given rate over iroh. Loopback measures CPU cost; serve/connect across the
|
||||
LAN measures the real thing. If iroh cannot carry the rates, that comes back
|
||||
as a design question before part 3.
|
||||
|
||||
## Order
|
||||
|
||||
nestri `feat/pyrowave`: nesprotocol, then neshub, then the bench. nescore is
|
||||
untouched until part 4 pins the rev.
|
||||
Reference in New Issue
Block a user