1 Commits

Author	SHA1	Message	Date
KAAL1 (Bingus)	e70f05e245	fix(nescope): make HDR reachable — start XWayland, advertise the opaque FourCCs (#314) ... Three related fixes. Together they take HDR from unreachable to working end to end on the XWayland path. ## XWayland was never started Three lines had been commented out since the initial import: the call that spawns XWayland, the guard that waits for it, and the `DISPLAY` handed to the child. Every game therefore launched as a native Wayland client. Nothing reported it -- the compositor still logged the X display it was telling clients to point at, which is why it read as working. That is also why HDR never fired. The colour space is signalled over a protocol whose Vulkan layer lives in the game process and finds the compositor through the X11 root window, so the one path able to carry it was the one path no game was on. `ENABLE_GAMESCOPE_WSI` and `DXVK_HDR` were already being set, which switched that layer on and then handed it a display it could not use. Restoring the guard also fixes the ordering it was written for: the launch now happens after XWayland reports ready rather than ~40ms before it. ## Mesa was dropping every format we advertised alpha-only Mesa tracks two flags per VkFormat -- one contributed by a format alpha FourCC, one by its opaque FourCC -- and skips any format carrying only one: ```c if (!(disp_fmt->flags & WSI_WL_FMT_ALPHA) || !(disp_fmt->flags & WSI_WL_FMT_OPAQUE)) continue; ``` We advertised `ARGB8888` and `XRGB8888`, so `B8G8R8A8` survived and made the list look like it was working. Everything else was alpha-only and was dropped in silence -- `ABGR8888` had been advertised all along while `R8G8B8A8` never once appeared on a surface. Adding the opaque spellings takes the surface from 6 formats to 21 and restores the three that carry HDR. The comment above that list claimed it was for XWayland DRI3 and that a game swapchain format was independent of it. It was the opposite: the list decides what a game can select, and deleting an entry removes that format from every client. ## Verified against swapchains, not format lists A client asking for `A2B10G10R10` + `HDR10_ST2084` now gets a swapchain and the compositor is told colorspace `1000104008`; one asking for `R16G16B16A16_SFLOAT` + scRGB linear gets `1000104002`. Previously both were refused at creation -- the WSI layer re-checks the requested format against the driver own surface list, so the colour space and the pixel format arrive from two different places and only one was being supplied. `apps/nescope/scripts/verify-hdr-formats.sh` asks what a client is offered from inside a child process, keeping the XCB and Wayland surfaces apart since a game presents through the XCB one. The default mode guards both halves of what the compositor controls; `--expect-layer` states the full target and passes once a WSI layer is present. No new dependencies (`vulkaninfo` + `python3`). ## Still open HDR is XWayland-only, documented as a FIXME in `hdr.rs`. A WSI layer binds the swapchain factory on its own Wayland connection while a native client surface lives on the client one, and object IDs do not cross connections. The FIXME records the fix both reference implementations point at, and the trap to avoid when we take it: gating format injection on "the compositor supports HDR" rather than on being able to signal the surface hands a client PQ pixels that arrive tagged as SDR, with nothing reporting an error. nescope ships no WSI layer of its own; the above was verified with the stock gamescope one, which drives our protocol unmodified. <!-- greptile_comment --> <h3>Greptile Summary</h3> This PR makes HDR-capable native Wayland presentation reachable, adds the alpha/opaque DMA-BUF FourCC pairs Mesa requires, makes XWayland an explicit compatibility mode, and adds an HDR surface-format diagnostic. - Starts XWayland only with `--xwayland`, waits for readiness before launching the child, and stops cleanly if startup fails or times out. - Routes Proton through Wayland unconditionally while retaining `DXVK_HDR` as an HDR-specific setting. - Advertises paired alpha and opaque FourCC variants with portable modifiers. - Separates XCB and Wayland probe results, selects one hardware adapter, and distinguishes probe failures from format regressions. - Documents the limitations of the legacy gamescope WSI path and the external-layer dependency. <h3>Confidence Score: 5/5</h3> The PR appears safe to merge; no outstanding correctness, security, or repository-rule issue remains. All previous findings are resolved in the current code, including the XWayland failure lifecycle, removal of unsupported vendor modifiers, corrected HDR documentation, reliable diagnostic failure handling, per-GPU format selection, and unconditional Proton Wayland routing. The changes since the previous review preserve diagnostic output handling without introducing a new failure. <h3>Important Files Changed</h3> | Filename | Overview | |----------|----------| | apps/nescope/src/main.rs | Adds opt-in XWayland lifecycle handling, readiness timeout, conditional DISPLAY propagation, and unconditional Proton Wayland routing; the previous launch-environment finding is fixed. | | apps/nescope/src/state.rs | Stops the event loop on reported XWayland startup failure and advertises portable paired alpha/opaque DMA-BUF formats without vendor-specific modifiers. | | apps/nescope/src/hdr.rs | Documents the working native Wayland HDR path and accurately distinguishes it from the external, deliberately disabled gamescope WSI route. | | apps/nescope/scripts/verify-hdr-formats.sh | Adds a diagnostic that keeps GPU and surface paths separate and now preserves the intended exit behavior when filtered Vulkan diagnostics contain no matching lines. | <h3>Flowchart</h3> ```mermaid %%{init: {'theme': 'neutral'}}%% flowchart LR Launch[nescope child launch] --> Mode{--xwayland?} Mode -->|No| Wayland[Native Wayland surface] Mode -->|Yes| Wait[Start and await XWayland] Wait -->|Ready| XCB[XCB / XWayland surface] Wait -->|Error or 10s timeout| Stop[Log failure and stop] Wayland --> Formats[Paired alpha and opaque FourCCs] Formats --> HDR[HDR10 and scRGB formats available] XCB --> SDR[X11 compatibility path without native HDR] Proton[Proton child] -->|PROTON_ENABLE_WAYLAND=1| Wayland ``` <sub>Reviews (9): Last reviewed commit: ["nescope/scripts: guard the diagnostic pi..."](f8bdd68f87) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=60368324)</sub> **Context used:** - Knowledge Base — [Compositor, display, and input control](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/compositor-input.md) - Knowledge Base — [Streaming host runtime](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/streaming-runtime.md) <!-- /greptile_comment -->	2026-09-04 19:18:46 +03:00