fix(nescapture): tag encoded streams full-range to match the samples written (#313)

## The bug `nescapture` sets the colour converter full-range unconditionally, but the video usability information carried pixelforge's **default limited-range flag**. A compliant decoder then expanded 16–235 out of samples that already covered 0–255 — darkening midtones and clipping both ends. pixelforge keeps two separate flags for this, one on the converter and one on the colour description, and its own documentation says they must agree. Only the first was being set. The two lines are about forty apart, each is correct on its own, and the comment above the second states the right intent while the call below it does the opposite: ```rust // GPU framebuffer captures are always full-range — use BT.709 full-range // so the decoder doesn't apply limited-range expansion. enc_cfg = enc_cfg.with_color_description(ColorDescription::bt709()); // ^ this constructor is limited-range ``` ## Evidence Measured on a Radeon RX 9060 XT, comparing the encoded result against the compositor's own readback of the same frames: | ground truth = `51` | before | after | |---|---|---| | flat background, decoded | **`38`** | `49–51` | | mean luma, capture path vs readback | **10.41 apart** | **0.55 apart** | | luma histogram intersection | **0.090** | **0.913** | | declared `color_range` | `tv` | `pc` | **The encoded luma is byte-identical before and after** — `Y = 51.00`, standard deviation `0.00` on both runs. Only the tag changed, which is what identifies this as a signalling bug rather than a conversion one, and why nothing short of a comparison against ground truth could see it: the stream was valid, the frame rate was right, the picture was recognisable, and every liveness check passed. The HDR arm (`bt2020_pq`) carried the same defect and is fixed the same way, but **has not been run** — no 10-bit verification here. ## `scripts/verify-chain.sh` Runs a Vulkan workload under `nescope` with the layer active and compares the encoded output against `nescope-shot`'s readback of the same frames. Two paths that share almost no code see the same content, so disagreement localises the fault; a single path cannot tell a correct frame from a plausible-looking wrong one. **Confirmed it fails when this change is reverted** — both the tag check and the brightness-agreement check fire. One note on its thresholds, since it is easy to get backwards: the not-blank check is a low absolute floor plus a comparison against the readback's own structure, rather than a fixed number. A fixed number was tried first and was wrong in the worst way — the **broken** build scored 20.49 on it and the **fixed** build 17.74, because the range defect stretched contrast and that reads as more detail. How much structure a correct frame carries depends on what the workload drew, so the only stable reference is ground truth measured in the same run. ## Not covered `vkcube` rather than a real workload; 720p, H.264, 8-bit; one card, one driver. XWayland, HUD detection and real swapchain formats are untouched.  <h3>Greptile Summary</h3> The PR aligns encoded-stream color metadata with the full-range samples produced by nescapture and updates the CPU fallback to BT.709 full-range conversion. - Updates pixelforge and configures matching converter color space, range, and SDR reference white. - Corrects Vulkan color-space mapping and adds regression tests for SDR, HDR, and CPU fallback behavior. - Adds SDR capture-chain and HDR comparison verification scripts. <h3>Confidence Score: 5/5</h3> The PR appears safe to merge. No blocking failure remains. <h3>Important Files Changed</h3> | Filename | Overview | |----------|----------| | apps/nescapture/src/encode.rs | Aligns GPU and CPU conversion output with encoded color metadata and adds focused regression coverage. | | apps/nescapture/scripts/verify-chain.sh | Adds an end-to-end SDR verifier using a static corner patch to avoid the previously reported temporal mismatch. | | apps/nescapture/scripts/verify-hdr.sh | Adds an HDR comparison harness for inspecting conversion behavior across builds. | | apps/nescapture/Cargo.toml | Advances pixelforge to the revision providing the required color-conversion configuration. | | Cargo.lock | Records the pixelforge update and resulting transitive dependency refresh. | <sub>Reviews (5): Last reviewed commit: ["test(nescapture): check the HDR conversi..."](2f9773c4b7) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=60147076)</sub> **Context used:** - Knowledge Base — [Vulkan capture layer](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/capture-layer.md)  --------- Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 17:25:19 +03:00 · 2026-09-04 19:03:13 +03:00
parent aaa1bbd0f4
commit 3389e6065f
5 changed files with 682 additions and 84 deletions
--- a/apps/nescapture/scripts/verify-chain.sh
+++ b/apps/nescapture/scripts/verify-chain.sh
@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# Verify the capture chain end to end on this machine's GPU.
+#
+# Runs a Vulkan workload under the compositor with the layer active, then checks
+# the encoded result against the compositor's own readback of the same frames.
+# Two independent paths see the same content: the compositor reads the surface
+# back to the CPU, the layer exports it as a DMA-BUF and encodes it on the GPU.
+# Agreement between them is the evidence; a single path cannot tell a correct
+# frame from a plausible-looking wrong one.
+#
+# The failure this is really aimed at is silent: a black or mis-levelled frame
+# arrives as a valid stream at the right frame rate, and every liveness check
+# passes. So the checks below are about pixel values, not about whether bytes
+# moved.
+#
+# This covers the SDR path only. The HDR arms need a swapchain this workload
+# cannot ask for, and the check that matters there is a different one — absolute
+# sample values against the standard, rather than two instruments against each
+# other. See verify-hdr.sh.
+#
+# Usage: apps/nescapture/scripts/verify-chain.sh [seconds]
+set -euo pipefail
+
+SECS="${1:-16}"
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"; kill $(jobs -p) 2>/dev/null || true' EXIT
+
+: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
+export XDG_RUNTIME_DIR
+
+for tool in ffmpeg ffprobe vkcube python3; do
+  command -v "$tool" >/dev/null || { echo "missing required tool: $tool" >&2; exit 1; }
+done
+
+echo "building…"
+cargo build --release -p nescope -p nescapture --manifest-path "$ROOT/Cargo.toml" >/dev/null
+
+LAYER="$ROOT/target/release/libnescapture_layer.so"
+MANIFEST_DIR="$WORK/implicit_layer.d"
+mkdir -p "$MANIFEST_DIR"
+sed "s#\"library_path\": \".*\"#\"library_path\": \"$LAYER\"#" \
+  "$ROOT/apps/nescapture/manifest/VK_LAYER_nescapture.json" > "$MANIFEST_DIR/VK_LAYER_nescapture.json"
+export VK_ADD_IMPLICIT_LAYER_PATH="$MANIFEST_DIR"
+
+VIDEO_SOCK="$WORK/video.sock"
+SHOT_SOCK="$WORK/shot.sock"
+STREAM="$WORK/capture.h264"
+
+cat > "$WORK/recv.py" <<'PY'
+import os, socket, struct, sys, time
+sock, out, secs = sys.argv[1], sys.argv[2], float(sys.argv[3])
+s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 8 << 20)
+s.bind(sock); os.chmod(sock, 0o777); s.settimeout(1.0)
+n = 0; end = time.time() + secs
+with open(out, "wb") as f:
+    while time.time() < end:
+        try: buf = s.recv(8 << 20)
+        except socket.timeout: continue
+        if len(buf) < 20 or buf[:4] != b"NSTR" or buf[4] != 0: continue
+        (_, _, _, dl) = struct.unpack("<IHHI", buf[8:20])
+        f.write(buf[20:20 + dl]); n += 1
+print(n)
+PY
+
+echo "capturing for ${SECS}s…"
+"$ROOT/target/release/nescope-shot" --socket "$SHOT_SOCK" --watch --interval 1000 \
+  --keep 3 --out "$WORK/shot.ppm" >/dev/null 2>&1 &
+python3 "$WORK/recv.py" "$VIDEO_SOCK" "$STREAM" "$SECS" > "$WORK/frames.txt" &
+RECV=$!
+sleep 1
+
+NESCAPTURE_ENABLE=1 NESCAPTURE_CODEC=h264 NESCAPTURE_BITRATE=20000 NESCAPTURE_FPS=60 \
+NESCAPTURE_IPC_PATH="$VIDEO_SOCK" RUST_LOG=nescapture_layer=debug \
+timeout "$((SECS - 2))" "$ROOT/target/release/nescope" \
+  --width 1280 --height 720 --fps 60 --screenshot-ipc "$SHOT_SOCK" \
+  -- vkcube --c 100000 > "$WORK/run.log" 2>&1 || true
+wait $RECV || true
+
+FRAMES="$(cat "$WORK/frames.txt")"
+echo
+echo "frames encoded:   $FRAMES"
+grep -m1 "First import" "$WORK/run.log" || echo "  (no DMA-BUF import logged)"
+
+python3 - "$WORK" "$STREAM" "$FRAMES" <<'PY'
+import glob, subprocess, sys
+import numpy as np
+from PIL import Image
+
+work, stream, frames = sys.argv[1], sys.argv[2], int(sys.argv[3])
+fails = []
+
+if frames < 30:
+    fails.append(f"only {frames} frames encoded (want >= 30)")
+
+probe = subprocess.run(
+    ["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
+     "stream=color_range", "-of", "default=noprint_wrappers=1:nokey=1", stream],
+    capture_output=True, text=True).stdout.strip()
+print(f"declared range:   {probe or '(none)'}")
+if probe != "pc":
+    fails.append(f"stream declares color_range={probe or 'unset'}; the converter "
+                 "writes full-range samples, so the tag must be 'pc'")
+
+subprocess.run(["ffmpeg", "-v", "error", "-y", "-i", stream, "-vf",
+                r"select='eq(n\,120)+eq(n\,240)'", "-fps_mode", "passthrough",
+                f"{work}/dec_%02d.png"], check=True)
+
+def luma(a):
+    return 0.2126 * a[..., 0] + 0.7152 * a[..., 1] + 0.0722 * a[..., 2]
+
+dec = [np.asarray(Image.open(p).convert("RGB")).astype(float)
+       for p in sorted(glob.glob(f"{work}/dec_*.png"))]
+shots = [np.asarray(Image.open(p).convert("RGB")).astype(float)
+         for p in sorted(glob.glob(f"{work}/shot-*.ppm"))]
+
+if not dec:
+    fails.append("nothing decoded from the stream")
+if not shots:
+    fails.append("compositor readback produced no frames")
+
+if dec:
+    worst = min(luma(d).std() for d in dec)
+    print(f"decoded luma std: {worst:.2f}")
+    # An absolute floor only has to catch a blank frame, which sits near zero.
+    # How much structure a *correct* frame carries depends entirely on what the
+    # workload drew, so the real check is the relative one below, against the
+    # readback of the same content.
+    if worst < 5.0:
+        fails.append(f"decoded frames are near-uniform (luma std {worst:.2f}) — "
+                     "the classic silent failure is a blank frame at full frame rate")
+
+if dec and shots:
+    ds, ss = min(luma(d).std() for d in dec), min(luma(s).std() for s in shots)
+    print(f"readback luma std:{ss:.2f}")
+    if ss > 1.0 and abs(ds - ss) / ss > 0.25:
+        fails.append(f"decoded structure {ds:.2f} vs readback {ss:.2f} — the two "
+                     "paths saw the same frames, so they should carry the same detail")
+
+if dec and shots:
+    # Compare a corner, not the whole frame. The two instruments sample at
+    # different moments -- the readback is on a 1 s timer, the decoded frames are
+    # picked by index -- so any whole-frame statistic also carries whatever the
+    # workload was doing at each instant. The workload draws a centred object on
+    # a flat background, so a corner patch is the same colour in every frame and
+    # the comparison stops depending on lining them up.
+    #
+    # This is the measurement that catches a range or matrix error: a flat patch
+    # of known colour, decoded, against the same patch read back from the
+    # compositor. It is where a full-range/limited-range mismatch shows up as a
+    # constant offset.
+    def corner(a):
+        return a[8:72, 8:72]
+
+    def spread(patches):
+        m = [luma(p).mean() for p in patches]
+        return max(m) - min(m)
+
+    dc, sc = [corner(d) for d in dec], [corner(s) for s in shots]
+    a = float(np.mean([luma(p).mean() for p in sc]))
+    b = float(np.mean([luma(p).mean() for p in dc]))
+    print(f"readback corner:  {a:.2f}")
+    print(f"decoded corner:   {b:.2f}")
+    print(f"difference:       {abs(a - b):.2f}")
+
+    # If the corner is not actually flat across frames, the assumption above does
+    # not hold for this workload and the comparison would be measuring animation.
+    # Say so rather than reporting a number that means nothing.
+    drift = max(spread(dc), spread(sc))
+    if drift > 3.0:
+        fails.append(f"the corner patch varies by {drift:.2f} between frames, so it "
+                     "is not background here; the brightness check assumes a "
+                     "workload that leaves its corners alone")
+    elif abs(a - b) > 4.0:
+        fails.append(f"the two paths disagree on brightness by {abs(a-b):.2f}; "
+                     "they are looking at the same content, so one of them is wrong")
+
+print()
+if fails:
+    print("FAIL")
+    for f in fails:
+        print(f"  - {f}")
+    sys.exit(1)
+print("PASS")
+PY
--- a/apps/nescapture/scripts/verify-hdr.sh
+++ b/apps/nescapture/scripts/verify-hdr.sh
@@ -0,0 +1,173 @@
+#!/usr/bin/env bash
+# Verify that the HDR path *converts*, not merely that it says it did.
+#
+# The defect this exists to catch: the colour space reached the encoder's
+# configuration and its VUI, but not the conversion shader. The stream then
+# declares BT.2020 NCL while the samples under it were written with the BT.709
+# matrix, and nothing downstream can tell.
+#
+# Two rules follow from that, and they are the whole design:
+#
+#   1. Compare raw Y/U/V samples, never a decode to RGB. If the shader and the
+#      VUI agree on the *wrong* matrix, an RGB round trip inverts exactly what
+#      it applied and returns the original colour. It scores the broken build
+#      perfect.
+#   2. Use a saturated colour, never grey. Achromatic input gives identical
+#      results under every matrix here, so a grey patch cannot see this defect
+#      at any tolerance.
+#
+# `ffprobe` output is byte-identical between a correct and a broken build --
+# it reads the declaration, which is the half that was already right.
+#
+# Needs a probe that can drive an HDR swapchain on purpose. It is not vendored:
+# it pulls winit and ash, which is a lot of build for a test fixture. Point
+# HDRPROBE at one that accepts `--color R,G,B`, `--width`, `--height`, `--sdr`.
+#
+# Usage: HDRPROBE=/path/to/hdrprobe apps/nescapture/scripts/verify-hdr.sh
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+WORK="$(mktemp -d /tmp/nshdr.XXXXXX)"   # short path: an AF_UNIX socket has ~108 bytes
+trap 'rm -rf "$WORK"; kill $(jobs -p) 2>/dev/null || true' EXIT
+
+: "${XDG_RUNTIME_DIR:=/run/user/$(id -u)}"
+export XDG_RUNTIME_DIR
+: "${SECS:=12}"
+W=1920; H=1080
+
+if [ -z "${HDRPROBE:-}" ] || [ ! -x "${HDRPROBE:-}" ]; then
+  echo "set HDRPROBE to a probe that can present a known colour in a chosen colour space" >&2
+  exit 2
+fi
+for t in ffmpeg python3; do
+  command -v "$t" >/dev/null || { echo "missing required tool: $t" >&2; exit 1; }
+done
+python3 -c "import numpy" 2>/dev/null || { echo "missing python numpy" >&2; exit 1; }
+
+echo "building…"
+cargo build --release -p nescope -p nescapture --manifest-path "$ROOT/Cargo.toml" >/dev/null
+
+# Point the loader at this build. A stale layer installed system-wide otherwise
+# wins the lookup and the run silently measures whatever is in /usr/lib.
+mkdir -p "$WORK/lay"
+sed "s#\"library_path\": \".*\"#\"library_path\": \"$ROOT/target/release/libnescapture_layer.so\"#" \
+  "$ROOT/apps/nescapture/manifest/VK_LAYER_nescapture.json" > "$WORK/lay/VK_LAYER_nescapture.json"
+export VK_ADD_IMPLICIT_LAYER_PATH="$WORK/lay"
+
+run_case() {
+  local tag=$1 color=$2 mode=$3
+  local sock="$WORK/$tag.sock" out="$WORK/$tag.h264"
+  local hdrflag="" probeflag=""
+  [ "$mode" = "hdr" ] && hdrflag="--hdr" || probeflag="--sdr"
+
+  python3 - "$sock" "$out" "$SECS" > "$WORK/$tag.frames" <<'PY' &
+import os, socket, struct, sys, time
+sock, out, secs = sys.argv[1], sys.argv[2], float(sys.argv[3])
+s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 8 << 20)
+s.bind(sock); os.chmod(sock, 0o777); s.settimeout(1.0)
+n = 0; end = time.time() + secs
+with open(out, "wb") as f:
+    while time.time() < end:
+        try: b = s.recv(8 << 20)
+        except socket.timeout: continue
+        if len(b) < 20 or b[:4] != b"NSTR" or b[4] != 0: continue
+        f.write(b[20:20 + struct.unpack("<I", b[16:20])[0]]); n += 1
+print(n)
+PY
+  local recv=$!
+  sleep 1
+  NESCAPTURE_ENABLE=1 NESCAPTURE_CODEC=h264 NESCAPTURE_BITRATE=20000 NESCAPTURE_FPS=60 \
+  NESCAPTURE_IPC_PATH="$sock" RUST_LOG=nescapture_layer=info \
+  timeout "$((SECS - 2))" "$ROOT/target/release/nescope" \
+    --width $W --height $H --fps 60 $hdrflag \
+    -- "$HDRPROBE" --width $W --height $H --color "$color" $probeflag --frames 100000 \
+    > "$WORK/$tag.log" 2>&1 || true
+  wait $recv || true
+
+  # Decode to raw planes in the format the stream already is, so ffmpeg inserts
+  # no scaler and applies no matrix. What is compared is what the shader wrote.
+  ffmpeg -v error -y -i "$out" -pix_fmt yuv420p -f rawvideo "$WORK/$tag.yuv" 2>/dev/null || true
+  printf "  %-10s %s frames, %s\n" "$tag" "$(cat "$WORK/$tag.frames")" \
+    "$(grep -m1 -o 'CHOSEN: .*' "$WORK/$tag.log" || echo 'no format line')"
+}
+
+echo "running ${SECS}s per case…"
+run_case red_hdr   255,0,0 hdr
+run_case green_hdr 0,255,0 hdr
+run_case red_sdr   255,0,0 sdr
+
+python3 - "$WORK" "$W" "$H" <<'PY'
+import os, sys
+import numpy as np
+
+work, W, H = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
+FRAME = W * H * 3 // 2
+
+# Luma coefficients are the published ones (ITU-R BT.709-6, BT.2020-2 Table 4).
+# Cb/Cr come from the standard relations rather than from the shader's own
+# constants, so agreement is a cross-check and not a restatement.
+KR_KB = {"bt709": (0.2126, 0.0722), "bt2020": (0.2627, 0.0593)}
+
+def expect(rgb, matrix):
+    kr, kb = KR_KB[matrix]
+    r, g, b = (c / 255.0 for c in rgb)
+    y = kr * r + (1.0 - kr - kb) * g + kb * b
+    q = lambda x: min(255.0, max(0.0, x * 255.0))
+    return q(y), q((b - y) / (2 * (1 - kb)) + 0.5), q((r - y) / (2 * (1 - kr)) + 0.5)
+
+def centre(a):
+    h, w = a.shape
+    return a[h // 2 - 100:h // 2 + 100, w // 2 - 100:w // 2 + 100]
+
+def decide(tag, rgb, want):
+    path = f"{work}/{tag}.yuv"
+    n = os.path.getsize(path) // FRAME if os.path.exists(path) else 0
+    if n < 3:
+        return tag, "INCONCLUSIVE", f"only {n} decoded frames"
+    ys, us, vs = [], [], []
+    for i in (n - 3, n - 2, n - 1):
+        buf = np.fromfile(path, dtype=np.uint8, count=FRAME, offset=i * FRAME)
+        ys.append(centre(buf[:W * H].reshape(H, W).astype(float)))
+        us.append(centre(buf[W * H:W * H + W * H // 4].reshape(H // 2, W // 2).astype(float)))
+        vs.append(centre(buf[W * H + W * H // 4:].reshape(H // 2, W // 2).astype(float)))
+    y = float(np.mean([p.mean() for p in ys]))
+    std = max(p.std() for p in ys)
+    u = float(np.mean([p.mean() for p in us]))
+    v = float(np.mean([p.mean() for p in vs]))
+
+    e709, e2020 = expect(rgb, "bt709"), expect(rgb, "bt2020")
+    d709, d2020 = abs(y - e709[0]), abs(y - e2020[0])
+    print(f"  {tag:10} Y={y:7.2f} U={u:6.2f} V={v:6.2f}  (std {std:.2f})")
+    print(f"  {'':10} BT.709 Y={e709[0]:6.2f} off {d709:5.2f} | "
+          f"BT.2020 Y={e2020[0]:6.2f} off {d2020:5.2f}")
+
+    # The shader quantises with uint(), which truncates rather than rounds, so a
+    # correct sample sits up to one code low. The two hypotheses are ~13 codes
+    # apart, so a 2-code window cannot admit both.
+    if std >= 2.0:
+        return tag, "INCONCLUSIVE", f"centre patch not flat (std {std:.2f})"
+    hit, miss = (d2020, d709) if want == "bt2020" else (d709, d2020)
+    other = "bt709" if want == "bt2020" else "bt2020"
+    if hit < 2.0 and miss > 6.0:
+        return tag, "PASS", f"converted on the {want} matrix (off {hit:.2f})"
+    if miss < 2.0:
+        return tag, "FAIL", f"converted with the {other} matrix (off {miss:.2f}), not {want}"
+    return tag, "INCONCLUSIVE", f"near neither (off {hit:.2f} / {miss:.2f})"
+
+print()
+results = [
+    decide("red_hdr",   (255, 0, 0), "bt2020"),
+    decide("green_hdr", (0, 255, 0), "bt2020"),
+    decide("red_sdr",   (255, 0, 0), "bt709"),
+]
+print("-" * 60)
+for tag, verdict, why in results:
+    print(f"  {verdict:13} {tag:10} {why}")
+print()
+if all(v == "PASS" for _, v, _ in results):
+    print("PASS")
+    sys.exit(0)
+print("FAIL")
+sys.exit(1)
+PY