fix(nescapture): stop defaulting unreadable formats to BGRA (#315)

Stacked on #313, which this depends on.

Found while checking whether the 10-bit HDR path actually works now that
a
client can obtain an HDR swapchain (see #314). **It does** — verified
end to
end rather than from the format list: a client requesting `A2B10G10R10`
+
`HDR10_ST2084` produces

```
pix_fmt=yuv420p10le   color_range=pc
color_space=bt2020nc  color_transfer=smpte2084  color_primaries=bt2020
```

which is a correctly tagged HDR10 stream, and the first pixel-level HDR
check
here with 10-bit rather than 8-bit input. Three ways it could have gone
wrong
instead.

## Unrecognised formats defaulted to BGRA

`vk_format_to_input_format` returned `BGRA` for anything it did not
know, which
reads a packed 10-bit or FP16 buffer as eight-bit channels. It now
returns
`None`, and the encode loop drops those frames with one log line per
format.

A stalled stream is a complaint. A stream at full frame rate carrying
nonsense
is not, and that is the failure this area keeps producing.

## Bit depth and input format had drifted apart

They were two separate matches on the same `VkFormat`. `A2R10G10B10`
counted as
ten-bit in one and had no entry in the other, so it fell back to
eight-bit
BGRA — the encoder configured for ten bits while the converter read
eight.

Depth now derives from the input format, so that disagreement is
unrepresentable. `A2R10G10B10` stays unmapped deliberately: a WSI layer
offers
it as one of its HDR pairs and the compositor dmabuf list advertises it,
but
the converter has no red-first 10-bit input, so there is nothing correct
to map
it to.

## The CPU fallback could not read either HDR format

It read four bytes per pixel for every format and encoded eight-bit
regardless, so a packed 10-bit buffer became garbage and an FP16 one was
half
an image of misread floats. It now refuses what it cannot read.

## Recorded, not fixed: the colour space we see is not always the one
requested

A FIXME at the point the value is read. A WSI layer rewrites
`imageColorSpace`
to `SRGB_NONLINEAR` before calling down — deliberately, since it carries
the
real colour space to the compositor out of band. We sit below it, so we
read
the rewrite. Measured, all three lines from one run:

```
[Gamescope WSI] ... colorspace: VK_COLOR_SPACE_HDR10_ST2084_EXT
swapchain created — format=A2B10G10R10 colorspace=SRGB_NONLINEAR
(re)init encoder: H265 Yuv420 Ten Bt709 → P010
```

Ten-bit right, BT.709 wrong: PQ samples encoded and tagged as SDR. The
same
client *without* the layer gives `Ten Bt2020` and an smpte2084 stream,
so this
is specific to the layer path — which is the path Proton titles take.

The fix cannot be local; the true colour space only exists in the
compositor,
which does receive it, so it needs a channel from there. Layer ordering
is not
a fix — we do not control it, and the non-layer path still needs the
Vulkan
value. Left out of this PR as a design change rather than a bug fix.

## Verification

- 4 new tests, 12 total, all passing.
- No new clippy warnings (diffed against the base branch).
- 10-bit HDR path: unchanged, still `Ten Bt2020` / smpte2084.
- SDR path: `verify-chain.sh` passes, 835 frames, 8-bit BGRA, brightness
  agreement 0.57.





<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR makes Vulkan format handling fail safely instead of interpreting
unsupported swapchain buffers as BGRA.
- Maps supported Vulkan formats to explicit converter inputs and derives
bit depth from that mapping.
- Drops unsupported GPU frames with rate-limited logging.
- Rejects unsupported HDR formats in the eight-bit CPU fallback.
- Documents the color-space limitation caused by rewritten WSI metadata.
- Adds tests covering unsupported, eight-bit, 10-bit, and FP16 formats.

<h3>Confidence Score: 5/5</h3>

The PR appears safe to merge, with no new actionable issues introduced
since the previous review.

The changes since the previous review are empty, the sole previous
finding was manually resolved after Greptile conceded it based on the
stacked PR dependency, and the full PR introduces no confirmed rule
violations or remaining correctness failures.

<h3>Important Files Changed</h3>




| Filename | Overview |
|----------|----------|
| apps/nescapture/src/encode.rs | Replaces unsafe BGRA fallback behavior
with explicit format validation, consistent bit-depth derivation,
guarded CPU fallback, and focused tests. |
| apps/nescapture/src/swapchain.rs | Documents the known WSI color-space
rewrite limitation at the point where swapchain metadata is recorded. |


<h3>Flowchart</h3>

```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Captured Vulkan frame] --> B{Known converter input?}
  B -->|No| C[Log format change and drop frame]
  B -->|Yes| D[Derive input format and bit depth]
  D --> E{DMA-BUF path available?}
  E -->|Yes| F[GPU color conversion and encoding]
  E -->|No| G{Eight-bit RGBA or BGRA?}
  G -->|Yes| H[CPU conversion and encoding]
  G -->|No| I[Return recoverable error]
```

<sub>Reviews (3): Last reviewed commit: ["docs(nescapture): the
colour-space note
..."](https://github.com/nestrilabs/nestri/commit/7b05908e0a94739aa7ee5bd7fe5764a0a2b95071)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=60377562)</sub>

**Context used:**

- Knowledge Base — [Vulkan capture
layer](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/capture-layer.md)

<!-- /greptile_comment -->

apps/nescapture/src/swapchain.rs

@@ -50,6 +50,31 @@ pub unsafe extern "system" fn vkCreateSwapchainKHR(
     *ds.swapchain.lock().unwrap() = Some(unsafe { *p_swapchain });
     *ds.swapchain_format.lock().unwrap() = ci.image_format;
     *ds.swapchain_extent.lock().unwrap() = ci.image_extent;
+    // The colour space the layer below us was asked for, which is the one the
+    // game asked for in every configuration we ship.
+    //
+    // The exception is worth knowing about, because it is silent. A WSI layer
+    // of the gamescope kind rewrites `imageColorSpace` to SRGB_NONLINEAR before
+    // calling down -- deliberately, since it carries the real colour space to
+    // the compositor out of band instead. We sit below such a layer, so we
+    // would read the rewrite. Measured, all three lines from one run of a
+    // client requesting HDR10 PQ:
+    //
+    //     [Gamescope WSI] ... colorspace: VK_COLOR_SPACE_HDR10_ST2084_EXT
+    //     swapchain created — format=A2B10G10R10 colorspace=SRGB_NONLINEAR
+    //     (re)init encoder: H265 Yuv420 Ten Bt709 → P010
+    //
+    // Ten-bit right, BT.709 wrong: PQ samples encoded and tagged as SDR, at
+    // full frame rate, decoding cleanly.
+    //
+    // This is not a bug to fix here. That route predates Wayland colour
+    // management and the compositor no longer enables it -- HDR comes from
+    // `wp_color_manager_v1` on a Wayland surface, where this value is correct
+    // and the same client yields Bt2020 and an smpte2084 stream. It is recorded
+    // because it is the reason the route stays off: enabling it would trade no
+    // HDR for wrong HDR. Anyone re-enabling it has to give this process a
+    // channel to the compositor first, since the true colour space exists only
+    // there.
     ds.swapchain_colorspace
         .store(ci.image_color_space.as_raw() as u32, Ordering::Relaxed);