The host can now ask a box for a shell on a pseudo-terminal, the way a container
runtime attaches to a container. `HostToGuest::Attach` starts the process, and
the guest then dials the host on a vsock port of its own (7001) and carries the
terminal's bytes over a small framed stream: data both ways, a resize, a hangup,
and the exit status last. A screenful of output on the JSON-lines control channel
would hold every other message in the box behind it, which is why it is not
there. A failure to start is the exception, and is said on the control channel
because nothing has connected to the data port yet.
A shell is not a launch: it never ends the box, and the box ending ends it. It
is started through the same reaper as everything else, so it cannot become the
zombie that exists to be prevented. With a terminal on the other end the PTY is
ordinary; without one it is put in raw mode so a command run for its output gets
its bytes back unrewritten. Closing the stream hangs the shell up, process group
and all.
The frame codec is a pure bytes-in, frames-out module in nesprotocol with a
payload cap and no skipping of unknown tags, since a length read from a guest is
read across a trust boundary. No version bump: adding a message does not need
one, and a guest that predates it simply never dials.
The runtime stage gains strace, lsof and less, so a root shell in a box is not
one that cannot trace a process.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There is no service manager in a box, so nothing applies the distribution's
sysctl configuration, and the guest ran on the kernel's own defaults. Two of
them are too low for a modern engine under Proton:
- vm.max_map_count defaults to 65530. A large Unreal Engine 5 title maps far
more regions than that, and the failure is a null allocation deep inside the
engine or Wine rather than a message about mappings. It is set to 2147483642,
the value SteamOS and the distributions that package Proton use. It is read
back after the write, because an accepted write that did not take reports
success while the box is still unconfigured.
- RLIMIT_NOFILE is raised on init, which every process in the box inherits. A
child can lower a limit but cannot raise its hard one, so the unprivileged
workload could never get above what it started with. 524288 is the hard limit
Proton's documentation asks for. It is only ever raised.
Both are best effort and each failure says what it costs.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The confirm screen named the client asking to sign in. That name is
chosen by whoever started the request, so it let a stranger's request
introduce itself as anything at all; and to the person reading, it is
just a device. It now says "A device is asking to sign in", and a test
checks that a client's self-chosen name never appears.
The warning about choosing Deny moves under the two buttons, as small
print in the same style as the terms line on the email screen, so the
question and its two answers read first.
bun --watch watches each imported file, and many editors save by
writing a new file and renaming it over the old one. The watch does not
follow the rename, so the first such save reloaded the preview and every
later one was silently ignored.
The preview now runs under a small supervisor that watches the source
directories and restarts the server, which sees a save however it was
made. It also stops the server when it is stopped itself, so the port
is not left held.
bun run preview (in apps/auth) serves a gallery of all 18 states the
sign-in flow can show, at desktop and phone sizes, each one also
viewable full size. It needs no database or mail: each state is the
real screen value, drawn by the real renderer with the real theme.
bun --watch restarts the server on any change to the renderer, the
stylesheet or the copy, and every page polls for the restart and
reloads itself, so styling is edit, save, look.
The issuer's screens were written inline in its route handlers, so the
only way to look at one was to reach the condition that produces it: an
expired cookie, a refused grant, ten wrong guesses. They now live in
ui/issuer.ts and the handlers call them, with no change to copy, status
or markup.
The app's theme moves to its own module for the same reason: it can be
used without importing the issuer and its database stores.
The boot descriptor gains `images`: read-only EROFS images mounted on their
own, with no writable layer over them. nesinit mounts them first, before
overlays and shares, with the same nosuid/nodev guards every layer has and
without noexec because what is in one is run. CONTROL_VERSION moves to 5: the
descriptor refuses fields it does not know, and a guest image without Proton is
useless to a host that does not send one, so the mismatch is named at the
handshake.
The guest rootfs no longer contains Proton. The checks that used to guard the
in-image copy (every library its programs and Wine's core link, an interpreter
behind the entry point) move to a compat_check stage that verifies the Proton
named by PROTON_IMAGE and leaves a marker in the image saying which one. The
leaf stages copy that marker out of it so the stage cannot be skipped as
unused. With nothing at a Proton path, a box started without one fails saying
where it looked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
At this point, why not..
Calling it `nespyro` since it's slightly modified, uses slang shader
language instead, along other things..
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /install.sh serves a POSIX script, embedded in the API at build time
so the script and the routes that redeem its token ship together. It
checks the platform, asks where box images should live, downloads the
host agent at a pinned version, verifies it against SHA256SUMS, installs
it for the calling user and hands over with the token in the environment
rather than argv.
GET /install/:component/:version/:asset redirects to a one-minute signed
URL on a private S3-compatible bucket, so every download passes through a
route that can be logged or switched off. The SigV4 signer is written
against Web Crypto and checked against AWS's published example.
POST /machine/install-token mints a token for a team the caller belongs
to, with the same team resolution and membership rule as /register.
POST /machine/install needs no session: it spends the token and returns
the same id, slug and one-time secret as registering directly, refusing
unknown, expired and used tokens identically.
Adds `nesgamepad`, basically "diet-coke vimputti" for Nestri's microVM
needs here.
Comes with direct mapping for:
- Sony
- DualShock 4 (v2), DualSense
- Microsoft
- Xbox 360 pad, Xbox One S pad
- Nintendo
- Pro Controller
- Generic pad for unknown controllers
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
"No suitable memory type for DMA-BUF import" names neither mask, so it cannot
tell a driver that refused the descriptor outright (fd_type_bits=0) from one
whose memory types simply do not overlap the image's — which are different
bugs in different components. The three numbers were already computed for a
debug! two lines above, and a debug line only fires at a level nobody runs in
the field; this one did not print at all while a neighbouring crate's did.
Now the failure carries image_type_bits, fd_type_bits, their intersection, and
the size, format, modifier and stride it was importing. On a virtualised NVIDIA
node this turned three unknowns into one measurement and pointed straight at
the kernel side.
The comment explaining the `cmd /C start` bug spelled out the cut submit
link in full. This repository is public and is pull-mirrored onto forges
that crawlers index far more eagerly than GitHub does, so GPTBot found it
there and followed it -- twice. The worker classed both as truncated
submissions, correctly, and raised an alert apiece for a bug fixed in
0.2.1.
Describe the cut URL instead of writing it, and say why, so the next
person documenting a bad link does not paste a working one.
Three comments named a file in the internal repo: a contract, a design
guide, and a user-research document with section numbers. A filename plus
a section tells a reader what to ask for, which is the escape the marker
convention exists to prevent.
Each comment now states the requirement on its own terms. The render-node
requirement carries a decision marker; the other two have no decision
behind them, so they cite nothing.
Nine commits, in the order they are easiest to read.
## The shared operator secret is gone
`x-nestri-admin-token` had no caller left — the device pairing it
existed for is
on hold, and nothing in any tree sent it. What remained was a key that
bypassed
authentication entirely and was required to boot.
Every route behind it had a better answer. The two Steam sync routes
took a
`userId` **in the body**, so one secret could write into anybody's
library; they
now authenticate as the host holding that person's Steam sign-in, and
the claim
is checked against the enrolment record. Download-state narrows to hosts
alone.
Creating a game by hand is deleted (syncing already upserts the
catalogue), as is
reading the waitlist — every address on it belongs to someone who has
not agreed
to anything, and answering it over HTTP made that list something a
leaked key
could drain.
Nothing in the API now accepts a credential standing for more than one
caller.
## An organisation owns hardware
Two kinds of machine were modelled as one. A host somebody brings is
theirs,
reached through a team, and should die with their account. A host bought
to serve
other people's workloads is none of those things, and had to be
registered under
an employee's personal team — where their account going away took it
with them.
Ownership is now an either/or, enforced by a check constraint rather
than a
convention: both null is a host nothing can bill, and both set is two
answers to
"whose is this?". Membership of an organisation is derived from a
verified email
domain rather than stored, so signing in with a personal address still
gets an
ordinary personal account.
Not a billing subject. A team pays for what it uses either way.
## Burn, and the three windows
The unit is one second of a reference session, so an allowance is
measured in
time and a bar prints the stored number rather than converting into it.
Counters are stored beside the time they began, and a total whose stamp
has
rolled outside its window reads as zero — so a window clears without
anything
running. No schedule to misfire, no race between a reset and a write.
Two rules on the allowances are enforced rather than remembered: an
allowance
must exceed its own window, or one uninterrupted session hits a wall;
and each
longer one must sit under what the shorter already permits, or it never
binds and
is decoration.
Burn is recorded as segments at one rate, because a run's rate does not
survive
its own lifetime. On our own hardware a bigger tier costs more; on a
caller's own
card it does not, since there is no share of a card of ours being spent.
The gate is at the one moment it may speak — before a run starts, never
again. A
limit refuses the next run and never interrupts one going.
## Payment
Checkout, portal, and a webhook. No price, no currency and no card
detail is
stored: a subscription's existence and its state are the whole of what
crosses
back. Free is a real subscription too, created outright since nothing a
month
needs no payment, so an upgrade changes a subscription rather than
inventing a
customer.
The webhook is the only route no session protects. A signature over the
raw body
stands in for one, checked before the body is parsed, and with no secret
configured it refuses everything.
## Breaking
- `x-nestri-admin-token` is no longer accepted; `ADMIN_SHARED_SECRET` is
no longer read
- `POST /games`, `GET /waitlist` and the `/pairing-code` routes are gone
- `POST /games/sync` and `POST /library/sync` now need host credentials
and take `userId` in the body
- `POST /steam/link` no longer accepts `userId`; `POST
/games/download-state` no longer accepts `hostId`
- `POST /session` responds `{ data, billing }` rather than `{ data }`
## Checks
342 tests pass. Typecheck unchanged from before the branch — the two
pre-existing
errors in `utils/hook.ts` and `utils/validator.ts` are untouched.
Billing is inert until configured: unset `POLAR_*` means every team is
free, no
checkout starts, and the webhook refuses every delivery.
Checkout, the customer portal, and the webhook that moves a team's plan.
Nothing money-shaped is stored. No price, no currency, no card detail — a
subscription's existence and its state are the whole of what crosses back,
because they are the only two facts the product needs and anything more would
be a second copy of a record somebody else is authoritative for.
Currency is deliberately not ours to hold. A product carries a price per
currency on their side and the customer's location picks one at checkout, so
there is no figure in this API that could drift from the one somebody is
charged. The team id travels as the customer's external id, which keeps the
mapping on their side rather than putting a foreign primary key in our schema.
Access follows their state, and the interesting cases are where that is not the
same as "paying right now". Cancelling keeps the plan: they paid to the end of
the period and turning them off when they click it takes something they bought.
A failed card keeps it too, because a retry that ends in payment should not
have cost them access in the middle. Only a revoked subscription takes it away,
which is the one moment nothing is left that was paid for. An event we do not
recognise changes nothing at all — new types are added by people who do not
know what we do with them, and a default that moved a plan would eventually
cancel an account nobody cancelled.
The webhook is the only route here no session protects, because its caller has
no account and never will. A signature over the raw body stands in for one, and
it is checked before the body is looked at — a body that has been parsed and
re-serialized is not the body that was signed. With no secret configured it
refuses everything rather than accepting anything, since otherwise knowing the
URL would be enough to set somebody's plan. Note also what is absent: no route
sets a plan, so there is no endpoint for granting yourself a subscription.
The product is written down as a definition with a script rather than clicked
into a dashboard, because the two environments are separate servers and nothing
made in one can be moved to the other. Promoting it is running the same script
with the other token, which is the only version of that which cannot drift. It
writes nothing without --apply and refuses to add a second product with a name
already taken.
The rate was a constant. That was correct for everything the system can
currently run and wrong the moment it can run anything else, because a tier
buys a share of a card — so a bigger one on our own hardware is more of
something we bought being spent, and a flat rate there sells a whole card for
the price of a quarter of one.
So a run's rate now comes from what the run is: its size tier, and whose
hardware it sits on.
On the caller's own hardware the tier changes nothing. There is no share of a
card of ours in play, so a run costs one unit a second whatever size it asked
for. Charging somebody more for taking more of a GPU they bought is a tax on
their own hardware, and not doing that is most of what this model is for.
This exposed a bug in what went before. Resegmenting recomputed one shared rate
and wrote it to every open stretch, which was harmless while all runs cost the
same and would have quietly repriced an expensive run as whatever the last one
to start was. Each stretch now keeps its own rate, which is also the more
honest shape: a run's rate is a property of that run, and nothing about it
changed because a sibling appeared or the clock ticked.
The account's total is now the sum of what its runs cost rather than a count
times one rate — an expensive run and a cheap one alongside it are not two of
anything. Concurrency still lands exactly where it did, as there being more to
add, and no run gets dearer because another started.
There is no hardware factor yet and its absence is deliberate: nothing records
which card a host has, so a table keyed on a model would be keyed on nothing.
A faster card should cost more, and that starts with a column.
The reference tier is pinned at exactly one unit a second, checked rather than
assumed. The unit is a second of a reference session, so moving it would
silently redefine every allowance — the same stored number would mean a
different number of hours.
The counters this fills are the ones the windows already knew how to read.
What was missing was anything that put a number in them.
Burn is recorded as segments: one stretch of one run at one unchanging rate,
opened when the rate becomes true and closed when it stops being. Not a row per
session, because a session's rate does not survive its own lifetime — a second
run changes what the account spends per second while the first is still going,
and a rate that applied from that moment must not be backdated over the time
before it. Not a row per event either, because burn accrues against an envelope
that is held rather than per thing consumed.
Closing a segment is what moves burn into the counters, so a long run lands
incrementally instead of all at the end. Burn that only arrives when a session
stops is burn that cannot refuse the next one, and a bar that does not move
while something is running is a bar nobody believes.
The counters are written with the staleness rule as a single statement: add to
the total if its stamp is still inside the window, otherwise start again from
this amount. Reading and then deciding would be two statements with a gap, and
the gap is where a concurrent tick doubles or vanishes. The first tick for a
team and the thousandth are the same call, for the same reason.
The gate sits at the one moment it is allowed to speak — before a run starts,
never again. A limit refuses the next run and never interrupts one already
going; someone losing a session mid-game to a meter does not come back. Every
window is checked rather than the shortest, because they protect different
things over different spans.
The answer comes back with the created run rather than being thrown away: the
response carries where each window stands, what the account spends per second
now, and what one more run would cost. Every surface that can start a run has
to show that before the click, and a second call for it is a call nobody makes.
Asking twice would also let the number shown and the number billed disagree.
Accrual is wired to the run's own state transition, in the same transaction
that moves it. A session that went live without its meter starting is free
hardware; one that ended without its meter stopping bills forever. Both are
silent, so neither may be a second write that might not happen.
Two kinds of machine were modelled as one. A host somebody brings is theirs,
reached through a team, and should die with their account. A host bought to
serve other people's workloads is none of those things — and there was nowhere
to put it, so it had to be registered under an employee's personal team, where
it was that person's property and their account going away took it with them.
Ownership becomes an either/or. A machine names a team or an organisation,
exactly one, enforced by a check constraint rather than by convention: both
null is a host nothing can bill, and both set is two answers to "whose is
this?" where whichever join a query happens to take decides who pays. Hardware
an organisation owns has no team and no person at all, which is the point.
The organisation is deliberately not a billing subject and has no plan columns.
It says who owns the metal; a team pays for what it uses either way.
Membership is derived from a verified email domain rather than stored. An
address is already the root identity, so a second record of who belongs where
is a second answer that can disagree with the first — and deriving it means
signing in with a personal address still gets an ordinary personal account,
which is what lets one person hold a company account and use the consumer
product. Nothing is granted on an unverified domain or an unverified address:
either one is a string somebody typed.
Entitlement on fleet hardware refuses everyone for now, with a reason that says
so. What grants a run on metered hardware is a plan, and there is nothing to
ask yet, so it fails closed rather than giving the expensive case away. The
branch is written out so the plan check has one obvious place to land.
Routes are read-only, and nothing seeds an organisation. Creating one grants
membership to everyone who can receive mail at a domain, so it is an operator
action against the database — a migration that inserted one would insert it
into every deployment, including ones we have nothing to do with. See
docs/deploy.md.
The formatter had never been run over these files, so `oxfmt` on a couple of
touched files rewrote two dozen others and buried the real change. Running it
everywhere once makes the next diff mean something.
No behaviour changes: import ordering, line joining, and reindented JSON in
the generated migration snapshots. Both test suites and both typechecks give
the same answers as before, including the two type errors this does not fix.
A single secret that turned any request into an operator was the only
credential several routes accepted, and it had no caller left: the device
pairing it existed for is on hold, and nothing in this tree or any client
sent it. What remained was a key that bypassed authentication entirely,
required to boot, and checked by nobody.
Every route behind it had a better answer available:
- Library and game sync move to host credentials. Both took a `userId` in
the body, which meant one secret could write into anybody's library. A
host now says which of its enrolled users a batch is for, and that claim
is checked against the Steam sign-ins it actually holds — one box carries
several people's accounts, so the pair is the unit.
- Download-state reporting narrows to hosts alone, and the body that could
name a different host is gone. Which host is reporting comes from its own
credentials, and a body that still names one is refused rather than
ignored.
- Linking a Steam account is always for the caller.
- Creating a game by hand is deleted; syncing already upserts the catalogue.
- Reading the waitlist is deleted. Every address on it belongs to someone
who has not agreed to anything, and answering it over HTTP made that list
something a leaked key could drain.
- The pairing-code routes are deleted with the flow they served. The domain
module and its table stay, so returning to it is a route file rather than
a migration.
Nothing in the API now accepts a credential that stands for more than one
caller: every request resolves to a specific user or a specific host, which
is what lets a route say "the caller's own library" and mean it.
BREAKING CHANGE: the `x-nestri-admin-token` header is no longer accepted and
`ADMIN_SHARED_SECRET` is no longer read. `POST /games`, `GET /waitlist` and
the `/pairing-code` routes are gone; `POST /games/sync` and `POST /library/sync`
now require host credentials and take `userId` in the body; `POST /steam/link`
no longer accepts `userId`; `POST /games/download-state` no longer accepts
`hostId`.
A host's hostname was its primary key. That worked and disclosed three
things it should not have: ids here are monotonic, so an id in a hostname
tells anyone who reads a URL roughly when that machine was registered and
where it falls among its owner's others; an id is the primary key, so a
name that had to change could only change by re-registering the machine,
which is changing its identity to fix its name; and the hostname is also
the OAuth audience and the cookie scope, so the id travelled into redirect
URLs and browser history.
Machines now carry a minted name -- two words and four digits, unique
across the fleet, DNS-safe by construction, which an id was not. The words
are a curated list rather than a dictionary, because every pair is shown to
strangers. Names the fleet's own infrastructure answers on are refused at
mint time: one minted onto the edge's own label would take the published
key path away from every host at once.
Minting retries on the unique index rather than checking first, because two
registrations in the same instant both read "free" and both write. Only a
name collision retries; a duplicate id or secret means something a new name
cannot fix.
The migration adds the column in three steps. Generated as a NOT NULL
column it fails outright against a populated table, and a default would be
worse: every row would share one value on a routing key.
The email field sat a little taller than the action below it. Two comments
also described this page as rendering inside a worker, which is not how the
control plane runs; what they were reaching for is that nothing preprocesses
this file, so the design tokens are written out longhand.
The screen was still the upstream template's: its font, its accent, its
logo, and a theme that tried to serve a light and a dark scheme from one
set of colours by deriving each one from the background's lightness.
That derivation is replaced with stated values, and the page is dark only.
Black, a neutral grey ramp, one brand accent, Mona Sans for the display
line and Geist for anything read or typed; two dashed bands closing into a
box on a wide screen, a dashed vertical either side of the column, the
wordmark, and the line of copy the rest of the product opens with.
The email field keeps the fix from the previous commit and takes the brand
colour on focus, which is the only place it appears besides the wordmark.
Measured against a render of the same design built from its own source:
the band rules, the column rules, the wordmark box and the button height
land on identical pixels.
The email field computed its own background one step lighter than the page
and never set `color`. Form controls do not inherit it, so the text someone
typed was the UA default — black, over a near-black field. There was no
`color-scheme` either, so the browser rendered the control in light
appearance to begin with.
The theme was a second, separate loss: `issuer()` still takes one and calls
`setTheme`, but nothing had passed one since `packages/auth` became a
vendored fork, so every sign-in rendered as OpenAuth — its font, its
periwinkle, its logo. Restored from the pre-fork config, with the logo and
favicon repointed because both URLs it carried now 404 and a broken `logo`
renders a broken image rather than falling back.
Both servers bound `0.0.0.0`. That was harmless while the only deployment was
docker-compose.yml, which publishes these ports on 127.0.0.1 and makes the
container's own bind irrelevant. As ordinary processes on a rented machine
there is no such wrapper, and `0.0.0.0` is a listener on the internet — in
front of an issuer that sets cookies without `Secure` and mints sign-in codes,
because it expects something else to be terminating TLS.
So the default is `127.0.0.1` and `HOST` is there for the deployment that
genuinely needs every interface. compose now sets `HOST: 0.0.0.0` explicitly,
which is not a workaround: inside a container, binding loopback is what would
make the published port unreachable. The right answer differs by deployment,
which is why it is a variable rather than a constant.
Both now log the address they bound, not the one they hoped for.
BREAKING CHANGE: api and auth no longer listen on every interface by default.
A deployment that relied on that must set HOST=0.0.0.0.
Get this thing going..
<!-- greptile_comment -->
<!-- greptile_summary -->
<h2><a
href="https://app.greptile.com/api/retrigger?id=63134761"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/RetriggerDark.svg?v=1"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"><img
alt="Retrigger"
src="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=1"
align="right"></picture></a>Confidence Score: 5/5</h2>
The PR appears safe to merge; all previous findings are resolved and the
latest readiness change introduces no established actionable regression.
<h3>Summary</h3>
- Establishes required guest filesystems, runtime directories, device
permissions, and service processes.
- Reports initialization and service deaths over the lifecycle channel.
- Supports launch, restart, and shutdown commands for a resident guest.
- Separates service and workload identities and configures per-launch
runtime environments.
- Removes the currently inactive nescope screenshot option and makes
capture-chain verification fail explicitly when compositor readback is
unavailable.
- Reworks the guest image around `nesinit` as PID 1 without a
distribution service manager.
<h3>Diagram</h3>
```mermaid
sequenceDiagram
participant Host
participant Init as nesinit
participant FS as Guest filesystems
participant Services as Service stack
participant Workload
Init->>Host: Ready(protocol version)
Host->>Init: Boot(mount descriptors)
Init->>FS: Establish and mount shares
Init->>Services: Spawn services in order
Services-->>Init: Required sockets ready
Init->>Host: Initialized(service names)
Host->>Init: Launch(id, exec, on_exit)
Init->>Workload: Spawn with isolated UID/runtime
Init->>Host: Started(id)
Workload-->>Init: Exit status
Init->>Host: WorkloadExited(id, status)
Host->>Init: Launch / Restart / Shutdown
```
<sub>Reviews (4) · Last reviewed commit: ["fix(nesinit): readiness is a
socket
that..."](https://github.com/nestrilabs/nestri/commit/731d34df9df30463f67963363424cb9487e89196)</sub>
<!-- /greptile_comment -->
---------
Co-authored-by: DatCaptainHorse <DatCaptainHorse@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
A host reporting an endpoint id another machine already holds hit the unique
index, and the raw refusal reached the global handler as a 500 -- telling a
host its beat broke the server rather than that the id is taken. It is now the
409 every other conflict here gives, and the route documents it.
Checked-then-written would be worse rather than better: two hosts reporting the
same id in the same instant both read "nobody holds it" and both write, which
is precisely what the index is for. The read would add a query and remove
nothing.
Before:
expect(res.status).toBe(409)
Received: 500
Two changes to who may start a flow here, and where a refusal is delivered.
A host reached at its own hostname sits on a different registrable domain from
this issuer, deliberately: that is what stops a cookie set there from ever
reaching this one. The default rule allows a redirect back to whatever hostname
the request arrived on, so it refused exactly the case the separation created.
Which is a real problem rather than a theoretical one, because a session cookie
without a Domain attribute is host-only, so a browser arriving at one of those
hostnames for the first time carries no cookie whether or not it is signed in,
and sending it here to sign in again changes nothing.
So a client id that is a single hostname under that zone, whose redirect_uri is
https and that same hostname at one reserved path, is allowed. Making the
client id the hostname is the load-bearing part: a token's audience is its
client id, so the session that comes back is bound to the host it will live on
and is not a credential anywhere else.
Separately, and worth its own paragraph: a refused client's redirect_uri was
still used to report the refusal. The check that approves that URI is the one
that just failed, so /authorize was an open redirector to anywhere at all --
no sign-in required, on the hostname people are asked to type a password into.
It is now a page here. Before:
GET /authorize?client_id=web&redirect_uri=https://somewhere.example/callback
-> 302 https://somewhere.example/callback?error=unauthorized_client
The machine table said who owns a host, which team it belongs to and when it
was last seen, and nothing about how to reach it. Anything standing in front of
a host and authenticating browsers on its behalf could therefore authorise a
request perfectly and then have nowhere to send it.
Reported, never assigned. A host holds the secret half of this identity and is
the only thing that can know the public half first, so it rides on the beat it
already sends as itself. Omitting the field leaves the stored value alone --
an agent that does not mention where it is has not moved, and an absent field
must never read as "nowhere", which would take every host shipped before this
field off the map on its next beat.
Nullable, because "has never reported one" is a real state that every host
registered before today is in. Unique, because an endpoint id belongs to one
host: two rows claiming the same one would send a request addressed to one
machine to another machine's agent, which is the one mistake here that the
authorisation in front of it cannot catch.
## What was missing
`neshub` serves the session's address on a socket, and its own flag has
always
said how that address gets out:
> *"neshub listens; nesinit dials and carries the ticket to the host,
because
> the person who needs it is outside this VM and stdout here is a log
file
> inside one."*
Nothing dialled it. `grep -rn ticket apps/nesinit/src` returned nothing
at all,
so the address never left the guest — and the one lifecycle message for
it,
`Ticket`, had no sender.
This adds the carrier.
## Three decisions worth reading
**Polled, not read once.** An address is not a value, it is the best
answer so
far: an endpoint discovers more ways to reach it after it binds — a
local one
immediately, a relayed one seconds later. Reading once means whoever
asked first
decides, and the first answer is the one that works on a local network
and fails
from anywhere else. Only a *changed* answer is forwarded, so an
unchanged one
costs nothing.
**Dials, does not listen.** The opposite of the payload relay next door,
and
deliberately so. There the guest listens because the workload starts
later; here
the server is the long-lived one. Dialling also makes "not bound yet" an
error to
retry rather than a connection to wait for without knowing whether it is
coming
— which is the ordinary case at boot, since this starts before the
server does.
**The address is never logged.** It is a capability to reach the
session, and a
log inside the guest is the one place it has no reason to be. The log
line says
whether it is the first one and nothing else.
## Failing first
The carrier's tests fail against unmodified code by not compiling:
`ticket.rs`
does not exist and `session::run` takes three arguments. Said plainly
rather
than manufactured. The behavioural gap is better shown as the `grep`
above —
nothing in the guest ever sent a `Ticket`, so the message had one end.
`nesinit`: **35 tests passing**, up from 27. Four on the carrier itself
(an address arrives; a better one replaces it; a socket that is not
there yet is
waited out rather than failed; an empty answer is not an address) and
two on the
session (an address reaches the caller as `Ticket`; a carrier that stops
does
not end the session).
## Verified in a real guest
Built static for musl, run as PID 1 in a real microVM under a real VMM
with a
real vsock. It dialled out, completed the handshake at version 2, took a
boot
descriptor, started its workload, read the address that workload
published and
sent it up the channel — twice, the second time because a better one
appeared.
The caller saw `nestri:local-only` and then `nestri:with-relays`.
Guest boot to init was **310 ms**.
## Review round (f74de9b, on top of two fixes from a hardware run)
`18864b9` and `00a2bda` came from running this on real hardware and are
the
reason it works at all: nothing was mounting `/proc` or anywhere
writable, so
the relay and the address socket both failed with `EROFS` and the
session was
reported as a workload that ran and published nothing. And one look at
the
socket had a size cap but no time cap, so a peer that accepted and then
said
nothing stopped the search for a better address permanently.
Four review findings on top, all real:
| finding | outcome |
|---|---|
| a tmpfs over the share tree hid the install, user, work and log-share
directories the image prepares | moved to `/run`, where a runtime socket
belongs; a test asserts the share tree is never mounted over, and
another that nothing is mounted before the mount containing it |
| the relay directory's mode stopped the workload reaching the socket |
the directory is searchable and writable only by init — which is what
makes the socket unreplaceable — and the socket itself is what the
workload may connect to |
| the address was built once and served forever, so polling could only
return the first one | rebuilt per read from the endpoint, which is what
makes polling worth doing |
| the workload could replace the socket and publish an address of its
choosing | `SO_PEERCRED` before a byte is read; an address from the
workload's user is refused and logged loudly. **Fails closed**, so the
host's default workload uid moved off the services' uid and the image
grew a second user — see the thread |
**These two PRs are now order-dependent.** This one merged alone,
against a host
still defaulting the workload to the services' uid, refuses every
legitimate
address.
Also bumps `tinyvec` by a patch release. It does not build on this
toolchain —
`vec` resolves to the module, not the macro — which made every crate
depending
on an endpoint unbuildable, `neshub` included. Pre-existing: the
lockfile named
the same version before this branch. It is why `neshub` could be built
and
tested here at all.
**40 tests** in `nesinit` (was 27), 3 in `neshub`, fmt clean, and no
clippy
warning in any file this branch touches.
## What this does not verify
- **The separated-uid pairing has not been run on hardware.** The rig
has no
init system, so the only process it can start is the workload — which
means
the address producer *is* the workload and the two cannot be given
different
uids in it. The refusal and the root stand-down are both verified on a
real
VM; the legitimate combination is covered by a unit test over a real
socket
and real kernel credentials, and by nothing else.
- **The image was not built.** The second user is one `useradd` in a
Dockerfile
that needs a base image this machine does not have.
- **The real address server was not in the loop.** The socket was served
by a
stand-in written for the test, which speaks the same one-line protocol.
The
real one has never had anything read from it, and its behaviour when it
learns
a new address — whether it re-serves the newer one on the next dial,
which is
the entire premise of polling — is **asserted, not measured**.
- **No media.** Nothing rendered or encoded; the address pointed at
nothing.
- **Nobody connected to it.** The address left the guest. It was never
dialled.
- **Not the real guest image.** A 3 MB rootfs built for this, with no
init
system, no compositor and no Mesa. The image build does not ship this
binary
at all — its `default` runlevel still registers services on the
assumption
that something else starts the payload.
- **The relay was down for the whole run**, because the test root was
read-only
and `/nestri` could not be created. That is non-fatal by design and the
session
ran anyway, but it means the payload layer and this carrier have not
been
exercised in the same boot.
- **One address, one guest, one session.** Nothing concurrent.
<!-- greptile_comment -->
<h3>Greptile Summary</h3>
This PR carries changing session tickets from `neshub` through `nesinit`
to the host and establishes writable runtime filesystems for guest
sockets.
- Rebuilds tickets from the endpoint on every IPC connection so newly
learned addresses can propagate.
- Polls the ticket socket and forwards changed tickets over the
lifecycle channel.
- Moves the payload socket into a dedicated `/run/nestri` runtime
directory.
- Adds workload/service UID separation infrastructure and
peer-credential checks.
- Fixes the previously reported stale-ticket and filesystem-mount
failures.
<h3>Confidence Score: 3/5</h3>
The PR is not yet safe to merge because the unresolved ticket-source
impersonation issue remains for root workloads and for deployments that
do not select the new workload UID.
The stale-ticket issue and both filesystem findings are fixed. However,
the unresolved ticket-source finding is only partially addressed:
`Untrusted::refuse` deliberately disables peer rejection when the
descriptor selects UID 0, allowing a root workload to replace the `/tmp`
socket and supply an attacker-chosen ticket. The new `nesplay` account
also does not enforce separation because the workload UID still comes
from an external descriptor producer; if that producer continues using
UID 1000, the carrier rejects the legitimate UID-1000 `neshub` peer as
well.
**Files Needing Attention:** apps/nesinit/src/ticket.rs,
build/Dockerfile
<h3>Important Files Changed</h3>
| Filename | Overview |
|----------|----------|
| apps/nesinit/src/ticket.rs | Adds ticket polling and peer-credential
rejection, but the protection is disabled for root workloads and depends
on external UID separation. |
| apps/nesinit/src/session.rs | Integrates ticket forwarding into the
lifecycle loop and records the descriptor-provided workload UID before
startup. |
| apps/nesinit/src/filesystems.rs | Establishes early mounts without
hiding the session share tree and makes the payload socket directory
traversable. |
| apps/nesinit/src/payload.rs | Relocates the payload socket under
`/run/nestri` and grants workloads permission to connect. |
| apps/neshub/src/ipc_listener.rs | Rebuilds the served ticket per
connection so later endpoint addresses are included. |
| build/Dockerfile | Adds a distinct `nesplay` account, although the
external boot-descriptor producer must select that UID for separation to
hold. |
<h3>Sequence Diagram</h3>
```mermaid
sequenceDiagram
participant Hub as neshub
participant Socket as Ticket Unix socket
participant Init as nesinit carrier
participant Session as nesinit session
participant Host as Host controller
loop Every polling interval
Init->>Socket: Connect
Socket->>Hub: Accept connection
Hub->>Hub: Rebuild ticket from endpoint.addr()
Hub-->>Init: Current ticket
Init->>Init: Check peer UID and compare with prior ticket
alt Trusted and changed
Init->>Session: Queue ticket
Session->>Host: GuestToHost::Ticket
end
end
```
<sub>Reviews (3): Last reviewed commit: ["fix(nesinit): do not mount
over the
shar..."](https://github.com/nestrilabs/nestri/commit/f74de9beb8f72402c74ad42fb792864b6d0fe009)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=60934382)</sub>
<!-- /greptile_comment -->
Four findings from review, all of them real.
The relay's directory was mounted on the tree a session's shares live in. A
fresh tmpfs there hides every directory the image prepared underneath it: the
install, the user state, the work directory, and the mount point the log share
is attached to from fstab. A box would have come up with a socket and without
any of the places its workload looks for its files, and the exact-path check
could not notice, because what fstab mounts is a directory inside that tree
rather than the tree itself. It moves to /run, which is where a runtime socket
belongs, is a tmpfs already, and has nothing else mounted inside it.
It was also owned by this process and closed to everyone else, which stopped
the workload traversing it to reach the relay at all. The directory is now
readable and searchable, and still writable by nothing but this process, which
is what makes the socket in it unreplaceable; the socket itself is what the
workload is allowed to connect to. The permission belongs on the socket rather
than on the path.
The address served to a reader was built once at startup and served forever, so
a reader that polls for a better one could only ever get the first. An endpoint
does not know all of its own addresses when it binds: the first is the one that
works on the same network and fails from anywhere else. It is now rebuilt per
read, which is what makes polling for it worth doing.
And the address was taken from whoever held a path in a directory the workload
can write. Workload code could unlink the socket a service was listening on,
bind its own, and every read afterwards would hand the client an address of its
choosing -- a session given to somebody else rather than a session that fails.
The peer's credentials are now checked before a byte is read, from the kernel
rather than from anything the peer says about itself, and an address served by
the workload's own user is refused and said loudly.
That check is only worth something while the workload has a user of its own, so
the image grows one. Two users, and they must stay two: one runs the services
that ship in the image, the other is who a workload runs as. Sharing one does
not weaken the check, it makes every session fail it.
A workload running as root is every user at once and cannot be told apart from
anything; the check stands down there and says so at boot instead, because
refusing root would refuse whatever legitimately serves the address as well.
Also bumps tinyvec by a patch release. It does not build on this toolchain --
`vec` resolves to the module and not the macro -- which made every crate that
depends on an endpoint, including this one, unbuildable. Pre-existing and
nothing to do with this change; the lockfile said the same version before it.
There was a cap on how much this would read and none on how long it would
wait. The far end can accept a connection and then write nothing, and a
read with no deadline turns that into a poll loop that never runs again:
the address already forwarded stays correct, and the better one that
arrives afterwards is never seen. That is the failure this polls to avoid,
reached by a different route.
Five seconds, against a two second interval, so an answer that is merely
slow still lands and one that is never coming is abandoned.
The test hangs against the code as it was, which is the whole point of it.
The root arrives read-only and this process is PID 1, so until it mounts
them there is no /proc and nowhere in the filesystem to put a socket.
Nothing else in the guest is an init system, so nothing else was going to.
The symptom was three failures that look unrelated and share one cause. On
a real box the payload relay could not bind, with EROFS; whatever serves
the session's address could not bind either, the same way; and this process
could not make itself ineligible for the OOM killer, because /proc was not
there to write to. What the caller saw was a workload that ran and
published nothing, which is true and says nothing about why.
/proc is mounted first and unconditionally: finding out what an image
already mounted requires it, and it is therefore the one entry that cannot
be checked that way itself. Everything after it is skipped when it is
already present, so an image that does this properly is not mounted over.
Failures warn rather than abort. Refusing to boot would replace a session
that fails with a reason by a guest that never dialled out at all, and the
second is harder to diagnose from the outside.
The relay's directory is named by the module that owns the socket rather
than spelled again here, with a test tying the two together: a rename that
reached one and not the other would put the relay back exactly as it was.
The example generator emitted twenty-five payload characters where an id has
twenty-six, so every documented id was twenty-nine characters — one short of
the width the column holds and, since last commit, one short of what the schema
publishing it will accept. Nothing caught it because an example is never
parsed: it is copied into documentation and read by people.
The width now comes from the generator's own constant instead of being typed
out, in the two places that had counted it by hand. Counting twenty-six of
anything by eye is a thing people get right once and never re-check.
A test pins the three together — a generated id, the schema for one, and the
documented example must all agree, for every prefix. It fails on the
off-by-one that prompted this, and on a prefix without its separator, which
would otherwise read as an id of that type because it starts with the same
three letters.
Whatever serves media in a box knows how it can be reached, and the person who
needs to know is not in the box. Standard output here is a log file inside a
VM, so the control channel is the delivery path rather than a convenience --
which makes this init's job and not a detail of whichever component happens to
bind the port. The socket it reads was already documented as being read this
way; nothing read it.
Polled rather than read once, because an address is not a value but the best
answer so far. An endpoint discovers more ways to reach it after it binds, so
the first answer is the one that works on a local network and fails from
anywhere else. Only a changed answer is forwarded.
It dials rather than listens, which is the opposite of the relay next door and
deliberate: there the guest listens because the workload starts later, and here
the server is the long-lived one. Dialling also makes a server that has not
bound yet something to retry rather than something to wait for without knowing
whether it is coming.
The address itself is never logged. It is a capability to reach the session,
and a log inside the guest is the one place it has no reason to be.
A carrier that stops does not end a session: whatever was already reported is
still correct, and the workload's exit still has to be.
Ids are stored in a fixed-width column, so an overlong one is refused by
Postgres rather than simply matching nothing. That refusal is not a foreign-key
violation, so it fell through to the global error boundary and reached the
caller as a 500 — telling a host to retry something that can never succeed.
Measured: a 44-character user id returned 500, where an absent but well-formed
one correctly returned 404.
`Identifier.schema` is the natural place for the check and had no callers yet,
so it now asserts the exact width an id has as well as its prefix — including
the separator, without which `usrsomething` reads as a user id. The enrolment
schema uses it for both foreign keys, so the refusal happens where the input
arrives and names the field.
Also index `steam_enrolment.user_id`. The primary key begins with the machine,
which answers what one host holds and nothing else, so neither of the two
things that read by user alone can use it: the cascade behind deleting a user,
and asking which hosts hold a token for one person. The table's migration has
not been released, so this is folded into it rather than following it with a
correction.
A host that signs a person into Steam ends up holding a refresh token. The
control plane needs to know that happened — to show it, and so a host that
lost its disk can find out what it is expected to hold — but it must not know
the credential, because the token is bound to the address that obtained it and
a copy anywhere else is the account-theft signal Steam watches for.
So `steam_enrolment` stores the outcome and has no token column, no encrypted
token column, and no column that could hold one later. The safeguard is that
the credential is never sent here at all; a nullable column would be the first
step in undoing it, so a test asserts the column list exactly and fails if one
appears. Three machine-authenticated routes go with it: report a completed
sign-in, report that Steam refused the token, and list what this host should
have. All three take the host from its own credentials, so a box can neither
report onto nor read another box's hardware. Their bodies are strict, so a
host that sends a token is told it is wrong rather than quietly believed —
which also keeps the value out of the request log.
The Steam id is deliberately not unique. One account signed in on two hosts is
two rows and two tokens, and a unique index there would look like hygiene while
refusing somebody their second box.
There is no `pending` state: a sign-in challenge lives about two minutes inside
one process, and nothing outside it needs to know it exists. Nothing revokes
yet, and `last_ok_at` has no writer — a successful logon happens where there is
no credential to report it with — so the column exists with the shape it will
need and stays null rather than being filled with the nearest event that was
easy to observe.
A host agent already sends a full inventory snapshot on a cadence, and
nothing served the endpoint it sends it to — so every one of those calls
answered 404. It fails quietly by design, because a dropped snapshot is
meant to be corrected by the next one, which is exactly why nobody
noticed: the only symptom is a line in the agent's own log.
Kept separate from the heartbeat because the two have different loss
tolerance. A dropped beat moves a host towards offline and unplaces it;
a dropped snapshot costs nothing until the next one arrives. Folding them
together would let a malformed inventory field make a healthy host look
dead.
Three rules decide what a snapshot may do, and the last two are why this
is one core function rather than a loop in the route:
- a box we know, that the snapshot names, takes the reported state
- a box we know that was running, and that the snapshot omits, is
stopped and says so — absence inside a snapshot is information
- a box the snapshot names that is not placed on the calling host is
never created, only reported back as a divergence
The scope is in the `where` clause and not in the agent asking politely
about its own boxes: a machine credential is a long-lived secret sitting
on hardware in somebody's living room.
`pid` and `uptimeS` are accepted and deliberately dropped. A pid is a
number in another machine's namespace, and uptime is derivable from a
run's start time, which is already stored and already trustworthy.
Two defects, both found by running it. Neither affects the container path,
which is why the images passed: they run the same entrypoints from the
repository root, and that turns out to be the load-bearing detail.
**The issuer would not start at all.** `apps/auth/tsconfig.json` named
React as the JSX runtime — left over from an earlier scaffold; there is no
React anywhere in this repository. It only bit when something transpiled
from that directory, and then the process died resolving
`react/jsx-dev-runtime` from a sign-in screen in `packages/auth` before it
bound a port. The package holding those components already said `hono/jsx`,
and so did the root; this is the third place agreeing with them.
**Neither process could see `.env`.** They ran with the working directory
set to their own app, so the environment file the repository documents —
the one at the root — was not the one they were offered. The issuer then
correctly refused to send a sign-in code rather than logging one, which is
the right behaviour and an opaque way to discover a path problem. Both now
run from the root, which is also exactly what the images run.
Six defects found by running the thing rather than reading it. The
previous change was verified by bundling, by tests, and by the container
images — none of which start a Worker, so every one of these was invisible.
**`bun dev` did not start.** It ran one multi-config process, which does
not connect a service binding between the workers it loads; the API
reported `AUTH [not connected]` and could not verify a token. It is two
processes now, which is what the dev registry connects, and the second is
backgrounded with the first killed on exit so stopping the pair stops both.
**Neither server could bind.** Wrangler resolves `localhost` and takes
`::1` first; a host with no IPv6 address on its loopback dies with a bind
error from inside the runtime that names neither the app nor the port.
`dev.ip` is pinned to `127.0.0.1`, and `inspector_port` is now distinct per
app — it is not derived from the port above, so the second server to start
died on an address already in use.
**The API worker failed to evaluate.** A specifier ending in `.sql` is
claimed by the bundler as a module of its own, so the schema file was
emitted verbatim beside the bundle and the runtime threw on an export it
could not find. The route was reaching past the domain module into the
schema to spell a status; it now asks the domain module, which is the rule
everywhere else here and happens to also avoid the hazard.
**Signing in failed on the second request that touched the database.** A
pool is cached per connection string, and on a Worker an I/O object created
while handling one request may not be touched while handling another. The
first request always succeeded, which is why it went unnoticed — a sign-in
is several. The cache is now kept only where a process outlives its
requests, which is the case it was added for.
**The images named a base that podman will not resolve.** A short name
needs a registry; the database service alongside them already spelled one.
**Compose pinned container names.** The name is not scoped to the project,
so a second checkout got the same three, and `down` in one stopped the
other's containers. This is not hypothetical — it stopped a running
development database while this was being tested.
Verified by signing in end to end against both dev servers: a code
requested over HTTP, read from the issuer's log, redeemed, exchanged for
tokens, and presented to the API, which resolved it to the account the
sign-in had just created.
Three things review caught, and one shape correction.
**No credential has a default any more.** The compose file shipped
`ADMIN_SHARED_SECRET` falling back to a value written in this repository —
and that header bypasses token verification entirely, so anyone reading
the file could act as an operator against any deployment that had not
overridden it. A default is worth less than it looks here: the deployment
that never set the variable is exactly the one where the default is public.
Every credential now comes from `.env`, and compose refuses to start naming
the variable it wanted. That also takes the last literal password out of a
tracked file.
**The origin ports are on loopback.** Both services speak plain HTTP and
mark no cookie `Secure`, because both expect to sit behind something that
terminates TLS. Published on every interface they were a way to reach the
issuer around that proxy, with sign-in codes and tokens in clear text.
**Mail settings are passed through rather than fixed.** The issuer was
pinned to printing sign-in codes to its log, and the three delivery
settings never reached it — so the documented way to configure mail could
not work, and every code and recipient went to the container log instead.
Printing codes is now asked for in `.env` like everything else, and with
nothing configured the issuer refuses to send rather than logging.
**Sandbox becomes a domain rather than a prefix.** `api.sandbox.nestri.io`
and `auth.sandbox.nestri.io`, because sandbox holds whatever is not
production and that set grows. One certificate for `*.sandbox.nestri.io`
then covers all of it, including unpredictable per-pull-request names,
and cannot be presented for production's own domain — which the zone-wide
wildcard the previous shape leaned on could.
Also drops `STEAM_API_KEY`. It was declared in two type definitions and
read by nothing: linking an account makes no outbound call that needs it.
Moving the issuer's state into Postgres removed the last thing that tied
either app to one hosting provider. What was left was a deployment tool
describing resources that no longer existed — so this replaces it with
`wrangler`, which is what actually deploys a Worker, and adds a second way
to run each app that involves no provider at all.
Each app now has a `wrangler.jsonc` with an environment per stage, and a
`Dockerfile` beside it. The handler is the same one in both cases; what
differs is only where its settings come from. Two of them gained a second
spelling so that nothing has to branch on the runtime: Postgres arrives as
a pooled binding or as `DATABASE_URL`, and the route to the issuer is a
service binding or `AUTH_INTERNAL_URL`.
That last one is new, and it is a split the binding was already making
without saying so. `AUTH_ISSUER_URL` has to be the issuer's public name,
because it is compared literally against every token's `iss` claim — but
the public name is often not routable from inside a deployment. So the
name and the route are two settings now rather than one that cannot be
both.
DNS moves out of code and into `docs/dns.md`, which lists every hostname
and what it is for. Six records that change roughly never did not need a
tool, and the table outlives whatever is answering the names — which is
the point, since some of them will stop being Workers. The sandbox
hostnames are hyphenated rather than nested for the same reason: a
certificate covering `*.nestri.io` covers one label and not two, so
`api-sandbox.nestri.io` can become an ordinary origin later without a
certificate having to be ordered for it first.
Also drops `EMAIL_DEV_LOG` from committed configuration into `.dev.vars`,
which `wrangler deploy` cannot upload. Printing a live sign-in code to a
log should not be one forgotten override away from production.
The issuer kept everything behind one get/set/remove/scan interface, which
is what a library that must run on any provider's cache can offer. Three of
the things kept there could not actually be served by it.
An authorization code must be redeemable once and a refresh token spendable
once, and through get and set the check and the write are separate steps —
so two requests arriving together both read an unspent record, and both mint
a session. In the refresh case that also means the reuse which reveals a
stolen token is never recorded, because recording it is the write that the
second caller overwrites. Each now has a table and an interface of its own:
redeeming is one `delete ... returning`, spending is one
`update ... where time_used is null returning *`, so exactly one caller is
ever told it went first. This is the same argument the device grant already
made, applied to the two records that had it too.
Signing keys move for a different reason. Nothing races for them; they are
the one record whose loss ends every session at once, and a cache is a place
things may be evicted from. They are retired by setting a column rather than
deleted, so the tokens they signed stay verifiable until they expire.
Both credential tables store a hash and never the credential, as the device
grant does. An authorization code travels in a query string and so passes
through history, referrer headers and any log along the redirect; a refresh
token resumes a session outright.
What is left in the generic store is the rate-limit counters — written far
more often than read, meaningless within the hour, and allowed to be
approximate, since a lost increment costs one guess out of ten. Those move
to Postgres too, so the only key-value binding this deploys with is gone and
the control plane's state is one database. That was the point: nothing here
now depends on a primitive a self-hoster cannot run.
The generic scan also gained the separator on its prefix, so scanning `a`
cannot return what is under `ab` — subjects and email addresses are both
prefixes of longer subjects and email addresses.
Deploying this signs everyone out. The signing keys and refresh tokens are
in a store that is being left behind, so the issuer starts with a fresh key
set and every existing token stops verifying.
DatCaptainHorse
Wanjohi
KAAL1
