github-mirrors/netris-nestri
1
0
Fork 0
mirror of https://github.com/nestriness/nestri.git synced 2026-09-24 19:42:24 +03:00
Code Issues Packages Projects Releases Wiki Activity

fix(nescope): make HDR reachable — start XWayland, advertise the opaque FourCCs (#314)

Browse Source 
Three related fixes. Together they take HDR from unreachable to working
end to end on the XWayland path.

## XWayland was never started

Three lines had been commented out since the initial import: the call
that spawns XWayland, the guard that waits for it, and the `DISPLAY`
handed to the child. Every game therefore launched as a native Wayland
client. Nothing reported it -- the compositor still logged the X display
it was telling clients to point at, which is why it read as working.

That is also why HDR never fired. The colour space is signalled over a
protocol whose Vulkan layer lives in the game process and finds the
compositor through the X11 root window, so the one path able to carry it
was the one path no game was on. `ENABLE_GAMESCOPE_WSI` and `DXVK_HDR`
were already being set, which switched that layer on and then handed it
a
display it could not use.

Restoring the guard also fixes the ordering it was written for: the
launch
now happens after XWayland reports ready rather than ~40ms before it.

## Mesa was dropping every format we advertised alpha-only

Mesa tracks two flags per VkFormat -- one contributed by a format alpha
FourCC, one by its opaque FourCC -- and skips any format carrying only
one:

```c
if (!(disp_fmt->flags & WSI_WL_FMT_ALPHA) ||
   !(disp_fmt->flags & WSI_WL_FMT_OPAQUE))
   continue;
```

We advertised `ARGB8888` and `XRGB8888`, so `B8G8R8A8` survived and made
the list look like it was working. Everything else was alpha-only and
was
dropped in silence -- `ABGR8888` had been advertised all along while
`R8G8B8A8` never once appeared on a surface. Adding the opaque spellings
takes the surface from 6 formats to 21 and restores the three that carry
HDR.

The comment above that list claimed it was for XWayland DRI3 and that a
game swapchain format was independent of it. It was the opposite: the
list decides what a game can select, and deleting an entry removes that
format from every client.

## Verified against swapchains, not format lists

A client asking for `A2B10G10R10` + `HDR10_ST2084` now gets a swapchain
and the compositor is told colorspace `1000104008`; one asking for
`R16G16B16A16_SFLOAT` + scRGB linear gets `1000104002`. Previously both
were refused at creation -- the WSI layer re-checks the requested format
against the driver own surface list, so the colour space and the pixel
format arrive from two different places and only one was being supplied.

`apps/nescope/scripts/verify-hdr-formats.sh` asks what a client is
offered
from inside a child process, keeping the XCB and Wayland surfaces apart
since a game presents through the XCB one. The default mode guards both
halves of what the compositor controls; `--expect-layer` states the full
target and passes once a WSI layer is present. No new dependencies
(`vulkaninfo` + `python3`).

## Still open

HDR is XWayland-only, documented as a FIXME in `hdr.rs`. A WSI layer
binds
the swapchain factory on its own Wayland connection while a native
client
surface lives on the client one, and object IDs do not cross
connections.
The FIXME records the fix both reference implementations point at, and
the
trap to avoid when we take it: gating format injection on "the
compositor
supports HDR" rather than on being able to signal the surface hands a
client PQ pixels that arrive tagged as SDR, with nothing reporting an
error.

nescope ships no WSI layer of its own; the above was verified with the
stock gamescope one, which drives our protocol unmodified.















<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR makes HDR-capable native Wayland presentation reachable, adds
the alpha/opaque DMA-BUF FourCC pairs Mesa requires, makes XWayland an
explicit compatibility mode, and adds an HDR surface-format diagnostic.
- Starts XWayland only with `--xwayland`, waits for readiness before
launching the child, and stops cleanly if startup fails or times out.
- Routes Proton through Wayland unconditionally while retaining
`DXVK_HDR` as an HDR-specific setting.
- Advertises paired alpha and opaque FourCC variants with portable
modifiers.
- Separates XCB and Wayland probe results, selects one hardware adapter,
and distinguishes probe failures from format regressions.
- Documents the limitations of the legacy gamescope WSI path and the
external-layer dependency.

<h3>Confidence Score: 5/5</h3>

The PR appears safe to merge; no outstanding correctness, security, or
repository-rule issue remains.

All previous findings are resolved in the current code, including the
XWayland failure lifecycle, removal of unsupported vendor modifiers,
corrected HDR documentation, reliable diagnostic failure handling,
per-GPU format selection, and unconditional Proton Wayland routing. The
changes since the previous review preserve diagnostic output handling
without introducing a new failure.

<h3>Important Files Changed</h3>




| Filename | Overview |
|----------|----------|
| apps/nescope/src/main.rs | Adds opt-in XWayland lifecycle handling,
readiness timeout, conditional DISPLAY propagation, and unconditional
Proton Wayland routing; the previous launch-environment finding is
fixed. |
| apps/nescope/src/state.rs | Stops the event loop on reported XWayland
startup failure and advertises portable paired alpha/opaque DMA-BUF
formats without vendor-specific modifiers. |
| apps/nescope/src/hdr.rs | Documents the working native Wayland HDR
path and accurately distinguishes it from the external, deliberately
disabled gamescope WSI route. |
| apps/nescope/scripts/verify-hdr-formats.sh | Adds a diagnostic that
keeps GPU and surface paths separate and now preserves the intended exit
behavior when filtered Vulkan diagnostics contain no matching lines. |


<h3>Flowchart</h3>

```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Launch[nescope child launch] --> Mode{--xwayland?}
  Mode -->|No| Wayland[Native Wayland surface]
  Mode -->|Yes| Wait[Start and await XWayland]
  Wait -->|Ready| XCB[XCB / XWayland surface]
  Wait -->|Error or 10s timeout| Stop[Log failure and stop]
  Wayland --> Formats[Paired alpha and opaque FourCCs]
  Formats --> HDR[HDR10 and scRGB formats available]
  XCB --> SDR[X11 compatibility path without native HDR]
  Proton[Proton child] -->|PROTON_ENABLE_WAYLAND=1| Wayland
```

<sub>Reviews (9): Last reviewed commit: ["nescope/scripts: guard the
diagnostic
pi..."](https://github.com/nestrilabs/nestri/commit/f8bdd68f87b3e29e4195f325b3c98800076a55d5)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=60368324)</sub>

**Context used:**

- Knowledge Base — [Compositor, display, and input
control](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/compositor-input.md)
- Knowledge Base — [Streaming host
runtime](https://app.greptile.com/nestri/-/custom-context/knowledge-base/nestrilabs/nestri/-/docs/streaming-runtime.md)

<!-- /greptile_comment -->
This commit is contained in:
KAAL1 (Bingus)
2026-09-04 19:18:46 +03:00
committed by GitHub
parent 200bc9c75f
commit e70f05e245
4 changed files with 537 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/nescope/src/main.rs
+130 -54
View File
@@ -25,7 +25,8 @@
//! | Variable | Effect |
//! |----------------|-----------------------------------------------|
//! | `WAYLAND_DISPLAY` | Set by nescope before spawning the game |
//! | `DISPLAY` | Set to the XWayland display (`:N`) |
//! | `DISPLAY` | XWayland display (`:N`), only with `--xwayland` |
//! | `PROTON_ENABLE_WAYLAND` | Set to `1` always, so Proton renders through Wayland |
//! | `XCURSOR_THEME` | XCursor theme name for the software cursor |
//! | `XCURSOR_SIZE` | XCursor size in pixels |
//! | `RUST_LOG` | Tracing filter (e.g. `nescope=debug`) |
@@ -45,6 +46,11 @@ use std::os::unix::process::CommandExt;
use std::sync::Arc;
use std::time::Duration;
/// How long to wait for XWayland to report a display before giving up. Startup
/// is normally tens of milliseconds; this only has to be longer than a slow
/// machine's worst case, not tuned.
const XWAYLAND_TIMEOUT_SECS: u64 = 10;
use calloop::generic::Generic;
use calloop::signals::{Signal, Signals};
use calloop::timer::Timer;
@@ -96,6 +102,17 @@ struct Args {
#[arg(long, env = "NESCOPE_HDR")]
hdr: bool,
/// Run XWayland, for Linux-native software with no Wayland support.
///
/// Off by default, and that is the point. XWayland costs input latency and
/// a compositing hop, which is the wrong trade for a streaming box. Windows
/// titles do not need it -- Proton renders through Wayland when told to,
/// which is what the launch environment does -- and HDR is only offered on
/// the Wayland surface, so a game routed through XWayland loses it too.
/// Turn this on for the shrinking set of X11-only native software.
#[arg(long, env = "NESCOPE_XWAYLAND")]
xwayland: bool,
/// Wayland socket name (created in $XDG_RUNTIME_DIR).
#[arg(long, default_value = "nescope-0", env = "NESCOPE_SOCKET")]
socket: String,
@@ -278,7 +295,9 @@ fn main() {
args.hdr,
args.render_device.clone(),
);
//state.init_xwayland(&loop_handle, Some(args.x_display));
if args.xwayland {
state.init_xwayland(&loop_handle, Some(args.x_display));
}
// Said out loud because in compositor mode nothing else can work them out.
// A process started by the hub rather than by nescope has no inherited
@@ -286,7 +305,11 @@ fn main() {
if args.command.is_empty() {
tracing::info!(
wayland_display = %socket_name.to_string_lossy(),
display = format!(":{}", args.x_display),
display = if args.xwayland {
format!(":{}", args.x_display)
} else {
"(none — XWayland off; pass --xwayland if you need it)".to_string()
},
"compositor mode — point clients at these and they will connect"
);
}
@@ -402,6 +425,11 @@ fn main() {
// Run with a 1-second timeout so the idle closure fires even when no
// Wayland events arrive (needed for zombie reaping and auto-exit checks).
// Deadline for XWayland to come up. The launch below waits on it, so if it
// never arrives there is nothing to wait for and no game to run.
let startup = std::time::Instant::now();
let mut xwayland_timed_out = false;
event_loop
.run(Some(Duration::from_secs(1)), &mut data, move |data| {
// ── Reap zombie children ──────────────────────────────────
@@ -415,62 +443,110 @@ fn main() {
&& data.primary_pid.is_none()
&& !data.state.game_launched
{
//if let Some(xdisplay) = data.state.xdisplay {
data.state.game_launched = true;
tracing::info!("Launching {:?}", command[0]);
// Only wait on XWayland when we are the ones providing it.
// Without --xwayland there is no display coming, so waiting
// would mean never launching.
if !args.xwayland || data.state.xdisplay.is_some() {
data.state.game_launched = true;
tracing::info!("Launching {:?}", command[0]);
let mut cmd = std::process::Command::new(&command[0]);
let mut cmd = std::process::Command::new(&command[0]);
cmd.args(&command[1..])
//.env("DISPLAY", format!(":{xdisplay}"))
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::inherit())
.stderr(std::process::Stdio::inherit())
// Put the game in its own process group so we can
// kill the whole tree at once with kill(-pgid, …).
.process_group(0)
// Provide also WAYLAND_DISPLAY, so if the game or application
// is Wayland-native and doesn't support older X11 it'll still run.
.env("WAYLAND_DISPLAY", &gamescope_wayland_socket);
cmd.args(&command[1..])
.stdin(std::process::Stdio::null())
.stdout(std::process::Stdio::inherit())
.stderr(std::process::Stdio::inherit())
// Put the game in its own process group so we can
// kill the whole tree at once with kill(-pgid, …).
.process_group(0)
.env("WAYLAND_DISPLAY", &gamescope_wayland_socket);
if args.hdr {
tracing::debug!(
gamescope_wayland_socket,
"Setting GAMESCOPE_WAYLAND_DISPLAY for application"
// DISPLAY only if XWayland is actually running. Setting it
// otherwise points clients at a server that is not there,
// which is what the compositor used to do.
if let Some(xdisplay) = data.state.xdisplay {
cmd.env("DISPLAY", format!(":{xdisplay}"));
}
// Proton renders through XWayland unless this is set, and
// XWayland is off by default -- so without this a Windows
// title has no display at all. Unconditional for that
// reason: it is how the game reaches the compositor, not
// an HDR switch. It is also what makes HDR reachable, since
// colour management only exists on the Wayland surface --
// measured here, that surface offers 21 formats including
// HDR10 over A2B10G10R10 while the XWayland one offers two,
// both 8-bit sRGB.
cmd.env("PROTON_ENABLE_WAYLAND", "1");
if args.hdr {
// DXVK's dxgi.dll gates HDR colour space exposure on
// this. Without it neither DX11 nor DX12 (vkd3d-proton
// through DXVK's dxgi) sees HDR as available.
cmd.env("DXVK_HDR", "1");
// Left set, but deliberately without ENABLE_GAMESCOPE_WSI
// alongside it, so it is inert unless somebody opts in.
//
// That pair activates gamescope's WSI layer, which
// predates Wayland colour management and works by
// hiding HDR from the driver and reporting it to the
// compositor out of band. We do not want it: it needs a
// layer this image does not ship, it only helps the
// XWayland path, and capture reads the colour space it
// hides -- measured, a game asking for HDR10 through it
// has its PQ samples encoded and tagged BT.709 SDR.
// Enabling it would trade no HDR for wrong HDR.
tracing::debug!(
gamescope_wayland_socket,
"HDR: Wayland colour management; gamescope WSI not enabled"
);
cmd.env("GAMESCOPE_WAYLAND_DISPLAY", &gamescope_wayland_socket);
}
// Detect GPU vendor from render device and set VK_DRIVER_FILES
// so the game uses the same GPU as nescope.
if let Some(ref rd) = args.render_device {
if let Some(icd_path) = detect_gpu_icd(rd) {
cmd.env("VK_ICD_FILENAMES", &icd_path);
cmd.env("VK_DRIVER_FILES", &icd_path); // Mesa fallback
tracing::info!("GPU ICD → {icd_path}");
}
}
match cmd.spawn() {
Ok(child) => {
let pid = child.id();
tracing::info!("Game process spawned (pid {pid})");
data.primary_pid = Some(pid as i32);
data.game_pgid = Some(pid as i32); // PGID == PID due to .process_group(0)
data.game_process = Some(child);
}
Err(e) => {
tracing::error!("Failed to launch {:?}: {e}", command[0]);
data.loop_signal.stop();
return;
}
}
} else if args.xwayland
&& !xwayland_timed_out
&& startup.elapsed() > Duration::from_secs(XWAYLAND_TIMEOUT_SECS)
{
// Waiting forever is the outcome to avoid: the auto-exit
// below only runs once a game has been launched, so a
// display that never arrives leaves nescope polling with no
// game and nothing logged. Smithay does not always report a
// failed XWayland as an error -- an Xwayland that exits
// immediately simply never becomes ready -- so this is a
// deadline, not an error handler.
xwayland_timed_out = true;
tracing::error!(
"XWayland did not become ready within {XWAYLAND_TIMEOUT_SECS}s — \
cannot launch a game without a display"
);
cmd.env("GAMESCOPE_WAYLAND_DISPLAY", &gamescope_wayland_socket);
cmd.env("ENABLE_GAMESCOPE_WSI", "1");
// DXVK's dxgi.dll gates HDR color space exposure on this env var.
// Without it, both DX11 (DXVK) and DX12 (vkd3d-proton via DXVK dxgi)
// games will not see HDR as available.
cmd.env("DXVK_HDR", "1");
kill_all_children();
data.loop_signal.stop();
}
// Detect GPU vendor from render device and set VK_DRIVER_FILES
// so the game uses the same GPU as nescope.
if let Some(ref rd) = args.render_device {
if let Some(icd_path) = detect_gpu_icd(rd) {
cmd.env("VK_ICD_FILENAMES", &icd_path);
cmd.env("VK_DRIVER_FILES", &icd_path); // Mesa fallback
tracing::info!("GPU ICD → {icd_path}");
}
}
match cmd.spawn() {
Ok(child) => {
let pid = child.id();
tracing::info!("Game process spawned (pid {pid})");
data.primary_pid = Some(pid as i32);
data.game_pgid = Some(pid as i32); // PGID == PID due to .process_group(0)
data.game_process = Some(child);
}
Err(e) => {
tracing::error!("Failed to launch {:?}: {e}", command[0]);
data.loop_signal.stop();
return;
}
}
//}
}
// ── Poll primary process ──────────────────────────────────