A single secret that turned any request into an operator was the only
credential several routes accepted, and it had no caller left: the device
pairing it existed for is on hold, and nothing in this tree or any client
sent it. What remained was a key that bypassed authentication entirely,
required to boot, and checked by nobody.
Every route behind it had a better answer available:
- Library and game sync move to host credentials. Both took a `userId` in
the body, which meant one secret could write into anybody's library. A
host now says which of its enrolled users a batch is for, and that claim
is checked against the Steam sign-ins it actually holds — one box carries
several people's accounts, so the pair is the unit.
- Download-state reporting narrows to hosts alone, and the body that could
name a different host is gone. Which host is reporting comes from its own
credentials, and a body that still names one is refused rather than
ignored.
- Linking a Steam account is always for the caller.
- Creating a game by hand is deleted; syncing already upserts the catalogue.
- Reading the waitlist is deleted. Every address on it belongs to someone
who has not agreed to anything, and answering it over HTTP made that list
something a leaked key could drain.
- The pairing-code routes are deleted with the flow they served. The domain
module and its table stay, so returning to it is a route file rather than
a migration.
Nothing in the API now accepts a credential that stands for more than one
caller: every request resolves to a specific user or a specific host, which
is what lets a route say "the caller's own library" and mean it.
BREAKING CHANGE: the `x-nestri-admin-token` header is no longer accepted and
`ADMIN_SHARED_SECRET` is no longer read. `POST /games`, `GET /waitlist` and
the `/pairing-code` routes are gone; `POST /games/sync` and `POST /library/sync`
now require host credentials and take `userId` in the body; `POST /steam/link`
no longer accepts `userId`; `POST /games/download-state` no longer accepts
`hostId`.
A host's hostname was its primary key. That worked and disclosed three
things it should not have: ids here are monotonic, so an id in a hostname
tells anyone who reads a URL roughly when that machine was registered and
where it falls among its owner's others; an id is the primary key, so a
name that had to change could only change by re-registering the machine,
which is changing its identity to fix its name; and the hostname is also
the OAuth audience and the cookie scope, so the id travelled into redirect
URLs and browser history.
Machines now carry a minted name -- two words and four digits, unique
across the fleet, DNS-safe by construction, which an id was not. The words
are a curated list rather than a dictionary, because every pair is shown to
strangers. Names the fleet's own infrastructure answers on are refused at
mint time: one minted onto the edge's own label would take the published
key path away from every host at once.
Minting retries on the unique index rather than checking first, because two
registrations in the same instant both read "free" and both write. Only a
name collision retries; a duplicate id or secret means something a new name
cannot fix.
The migration adds the column in three steps. Generated as a NOT NULL
column it fails outright against a populated table, and a default would be
worse: every row would share one value on a routing key.
The email field sat a little taller than the action below it. Two comments
also described this page as rendering inside a worker, which is not how the
control plane runs; what they were reaching for is that nothing preprocesses
this file, so the design tokens are written out longhand.
The screen was still the upstream template's: its font, its accent, its
logo, and a theme that tried to serve a light and a dark scheme from one
set of colours by deriving each one from the background's lightness.
That derivation is replaced with stated values, and the page is dark only.
Black, a neutral grey ramp, one brand accent, Mona Sans for the display
line and Geist for anything read or typed; two dashed bands closing into a
box on a wide screen, a dashed vertical either side of the column, the
wordmark, and the line of copy the rest of the product opens with.
The email field keeps the fix from the previous commit and takes the brand
colour on focus, which is the only place it appears besides the wordmark.
Measured against a render of the same design built from its own source:
the band rules, the column rules, the wordmark box and the button height
land on identical pixels.
The email field computed its own background one step lighter than the page
and never set `color`. Form controls do not inherit it, so the text someone
typed was the UA default — black, over a near-black field. There was no
`color-scheme` either, so the browser rendered the control in light
appearance to begin with.
The theme was a second, separate loss: `issuer()` still takes one and calls
`setTheme`, but nothing had passed one since `packages/auth` became a
vendored fork, so every sign-in rendered as OpenAuth — its font, its
periwinkle, its logo. Restored from the pre-fork config, with the logo and
favicon repointed because both URLs it carried now 404 and a broken `logo`
renders a broken image rather than falling back.
`release-prod.yml` for api and auth, mirroring the edge's. The gate is not
"you may not merge" — it is that merging does not deploy: tests run against a
real Postgres, both binaries are executed, and a failure anywhere means no
release exists, so the machine keeps serving what it has.
The new piece is `nestri-migrate`, because the deploy runs migrations *before*
it swaps a release into place and had nothing to run.
`drizzle-kit migrate` reads the migrations folder at runtime, which is right on
a laptop and wrong on a server: the deploy ships flat, checksummed files into
bin/, and a migrator that needs a directory beside it can be pointed at the
wrong directory. So the folder is baked into the binary — generated on every
build rather than committed, so it cannot drift — and the artefact's checksum
then covers every statement it will run.
It reimplements drizzle's bookkeeping in thirty lines of SQL rather than
calling into `db.dialect.migrate`: same table, same schema, same sha256 over
the whole file, same high-water-mark comparison. That equivalence is the one
thing that must not rot, because the production database was first migrated by
drizzle-kit and a disagreement means a migration applied twice. Two checks
hold it: CI applies the migrations with drizzle-kit and then asserts the
embedded set reports nothing pending, and the hashes were verified by hand
against the live database — all fourteen match to the byte.
Proven before shipping, against the real database: nothing pending on the
deployed schema, 14 unchanged rows, and a scratch database migrated from empty
to the same 22 tables and the same high-water mark, idempotent on a second run.
Refuses with exit 2 when DATABASE_URL is absent rather than defaulting to
localhost, which would be a migrator reporting success having migrated nothing.
setup-bun is pinned to a commit and not to `v2`. A moving major tag is fine
everywhere else in this repository; this workflow is the only thing between a
merge and a process serving users, and there is no first-party bun action to
prefer instead.
A host reporting an endpoint id another machine already holds hit the unique
index, and the raw refusal reached the global handler as a 500 -- telling a
host its beat broke the server rather than that the id is taken. It is now the
409 every other conflict here gives, and the route documents it.
Checked-then-written would be worse rather than better: two hosts reporting the
same id in the same instant both read "nobody holds it" and both write, which
is precisely what the index is for. The read would add a query and remove
nothing.
Before:
expect(res.status).toBe(409)
Received: 500
Two changes to who may start a flow here, and where a refusal is delivered.
A host reached at its own hostname sits on a different registrable domain from
this issuer, deliberately: that is what stops a cookie set there from ever
reaching this one. The default rule allows a redirect back to whatever hostname
the request arrived on, so it refused exactly the case the separation created.
Which is a real problem rather than a theoretical one, because a session cookie
without a Domain attribute is host-only, so a browser arriving at one of those
hostnames for the first time carries no cookie whether or not it is signed in,
and sending it here to sign in again changes nothing.
So a client id that is a single hostname under that zone, whose redirect_uri is
https and that same hostname at one reserved path, is allowed. Making the
client id the hostname is the load-bearing part: a token's audience is its
client id, so the session that comes back is bound to the host it will live on
and is not a credential anywhere else.
Separately, and worth its own paragraph: a refused client's redirect_uri was
still used to report the refusal. The check that approves that URI is the one
that just failed, so /authorize was an open redirector to anywhere at all --
no sign-in required, on the hostname people are asked to type a password into.
It is now a page here. Before:
GET /authorize?client_id=web&redirect_uri=https://somewhere.example/callback
-> 302 https://somewhere.example/callback?error=unauthorized_client
The machine table said who owns a host, which team it belongs to and when it
was last seen, and nothing about how to reach it. Anything standing in front of
a host and authenticating browsers on its behalf could therefore authorise a
request perfectly and then have nowhere to send it.
Reported, never assigned. A host holds the secret half of this identity and is
the only thing that can know the public half first, so it rides on the beat it
already sends as itself. Omitting the field leaves the stored value alone --
an agent that does not mention where it is has not moved, and an absent field
must never read as "nowhere", which would take every host shipped before this
field off the map on its next beat.
Nullable, because "has never reported one" is a real state that every host
registered before today is in. Unique, because an endpoint id belongs to one
host: two rows claiming the same one would send a request addressed to one
machine to another machine's agent, which is the one mistake here that the
authorisation in front of it cannot catch.
The example generator emitted twenty-five payload characters where an id has
twenty-six, so every documented id was twenty-nine characters — one short of
the width the column holds and, since last commit, one short of what the schema
publishing it will accept. Nothing caught it because an example is never
parsed: it is copied into documentation and read by people.
The width now comes from the generator's own constant instead of being typed
out, in the two places that had counted it by hand. Counting twenty-six of
anything by eye is a thing people get right once and never re-check.
A test pins the three together — a generated id, the schema for one, and the
documented example must all agree, for every prefix. It fails on the
off-by-one that prompted this, and on a prefix without its separator, which
would otherwise read as an id of that type because it starts with the same
three letters.
Ids are stored in a fixed-width column, so an overlong one is refused by
Postgres rather than simply matching nothing. That refusal is not a foreign-key
violation, so it fell through to the global error boundary and reached the
caller as a 500 — telling a host to retry something that can never succeed.
Measured: a 44-character user id returned 500, where an absent but well-formed
one correctly returned 404.
`Identifier.schema` is the natural place for the check and had no callers yet,
so it now asserts the exact width an id has as well as its prefix — including
the separator, without which `usrsomething` reads as a user id. The enrolment
schema uses it for both foreign keys, so the refusal happens where the input
arrives and names the field.
Also index `steam_enrolment.user_id`. The primary key begins with the machine,
which answers what one host holds and nothing else, so neither of the two
things that read by user alone can use it: the cascade behind deleting a user,
and asking which hosts hold a token for one person. The table's migration has
not been released, so this is folded into it rather than following it with a
correction.
A host that signs a person into Steam ends up holding a refresh token. The
control plane needs to know that happened — to show it, and so a host that
lost its disk can find out what it is expected to hold — but it must not know
the credential, because the token is bound to the address that obtained it and
a copy anywhere else is the account-theft signal Steam watches for.
So `steam_enrolment` stores the outcome and has no token column, no encrypted
token column, and no column that could hold one later. The safeguard is that
the credential is never sent here at all; a nullable column would be the first
step in undoing it, so a test asserts the column list exactly and fails if one
appears. Three machine-authenticated routes go with it: report a completed
sign-in, report that Steam refused the token, and list what this host should
have. All three take the host from its own credentials, so a box can neither
report onto nor read another box's hardware. Their bodies are strict, so a
host that sends a token is told it is wrong rather than quietly believed —
which also keeps the value out of the request log.
The Steam id is deliberately not unique. One account signed in on two hosts is
two rows and two tokens, and a unique index there would look like hygiene while
refusing somebody their second box.
There is no `pending` state: a sign-in challenge lives about two minutes inside
one process, and nothing outside it needs to know it exists. Nothing revokes
yet, and `last_ok_at` has no writer — a successful logon happens where there is
no credential to report it with — so the column exists with the shape it will
need and stays null rather than being filled with the nearest event that was
easy to observe.
The helper told you to use "an isolated database for tests", which reads
as a database of its own and is not what the suite wants. Route tests
reach the database through the app and core tests reach it directly, so
two different values put the fixtures in one database and the assertions
in the other — around forty failures, none of them in the code that
caused it, and nothing in the output naming the setting.
Also drops a type import nothing uses.
A host agent already sends a full inventory snapshot on a cadence, and
nothing served the endpoint it sends it to — so every one of those calls
answered 404. It fails quietly by design, because a dropped snapshot is
meant to be corrected by the next one, which is exactly why nobody
noticed: the only symptom is a line in the agent's own log.
Kept separate from the heartbeat because the two have different loss
tolerance. A dropped beat moves a host towards offline and unplaces it;
a dropped snapshot costs nothing until the next one arrives. Folding them
together would let a malformed inventory field make a healthy host look
dead.
Three rules decide what a snapshot may do, and the last two are why this
is one core function rather than a loop in the route:
- a box we know, that the snapshot names, takes the reported state
- a box we know that was running, and that the snapshot omits, is
stopped and says so — absence inside a snapshot is information
- a box the snapshot names that is not placed on the calling host is
never created, only reported back as a divergence
The scope is in the `where` clause and not in the agent asking politely
about its own boxes: a machine credential is a long-lived secret sitting
on hardware in somebody's living room.
`pid` and `uptimeS` are accepted and deliberately dropped. A pid is a
number in another machine's namespace, and uptime is derivable from a
run's start time, which is already stored and already trustworthy.
Six defects found by running the thing rather than reading it. The
previous change was verified by bundling, by tests, and by the container
images — none of which start a Worker, so every one of these was invisible.
**`bun dev` did not start.** It ran one multi-config process, which does
not connect a service binding between the workers it loads; the API
reported `AUTH [not connected]` and could not verify a token. It is two
processes now, which is what the dev registry connects, and the second is
backgrounded with the first killed on exit so stopping the pair stops both.
**Neither server could bind.** Wrangler resolves `localhost` and takes
`::1` first; a host with no IPv6 address on its loopback dies with a bind
error from inside the runtime that names neither the app nor the port.
`dev.ip` is pinned to `127.0.0.1`, and `inspector_port` is now distinct per
app — it is not derived from the port above, so the second server to start
died on an address already in use.
**The API worker failed to evaluate.** A specifier ending in `.sql` is
claimed by the bundler as a module of its own, so the schema file was
emitted verbatim beside the bundle and the runtime threw on an export it
could not find. The route was reaching past the domain module into the
schema to spell a status; it now asks the domain module, which is the rule
everywhere else here and happens to also avoid the hazard.
**Signing in failed on the second request that touched the database.** A
pool is cached per connection string, and on a Worker an I/O object created
while handling one request may not be touched while handling another. The
first request always succeeded, which is why it went unnoticed — a sign-in
is several. The cache is now kept only where a process outlives its
requests, which is the case it was added for.
**The images named a base that podman will not resolve.** A short name
needs a registry; the database service alongside them already spelled one.
**Compose pinned container names.** The name is not scoped to the project,
so a second checkout got the same three, and `down` in one stopped the
other's containers. This is not hypothetical — it stopped a running
development database while this was being tested.
Verified by signing in end to end against both dev servers: a code
requested over HTTP, read from the issuer's log, redeemed, exchanged for
tokens, and presented to the API, which resolved it to the account the
sign-in had just created.
Three things review caught, and one shape correction.
**No credential has a default any more.** The compose file shipped
`ADMIN_SHARED_SECRET` falling back to a value written in this repository —
and that header bypasses token verification entirely, so anyone reading
the file could act as an operator against any deployment that had not
overridden it. A default is worth less than it looks here: the deployment
that never set the variable is exactly the one where the default is public.
Every credential now comes from `.env`, and compose refuses to start naming
the variable it wanted. That also takes the last literal password out of a
tracked file.
**The origin ports are on loopback.** Both services speak plain HTTP and
mark no cookie `Secure`, because both expect to sit behind something that
terminates TLS. Published on every interface they were a way to reach the
issuer around that proxy, with sign-in codes and tokens in clear text.
**Mail settings are passed through rather than fixed.** The issuer was
pinned to printing sign-in codes to its log, and the three delivery
settings never reached it — so the documented way to configure mail could
not work, and every code and recipient went to the container log instead.
Printing codes is now asked for in `.env` like everything else, and with
nothing configured the issuer refuses to send rather than logging.
**Sandbox becomes a domain rather than a prefix.** `api.sandbox.nestri.io`
and `auth.sandbox.nestri.io`, because sandbox holds whatever is not
production and that set grows. One certificate for `*.sandbox.nestri.io`
then covers all of it, including unpredictable per-pull-request names,
and cannot be presented for production's own domain — which the zone-wide
wildcard the previous shape leaned on could.
Also drops `STEAM_API_KEY`. It was declared in two type definitions and
read by nothing: linking an account makes no outbound call that needs it.
Moving the issuer's state into Postgres removed the last thing that tied
either app to one hosting provider. What was left was a deployment tool
describing resources that no longer existed — so this replaces it with
`wrangler`, which is what actually deploys a Worker, and adds a second way
to run each app that involves no provider at all.
Each app now has a `wrangler.jsonc` with an environment per stage, and a
`Dockerfile` beside it. The handler is the same one in both cases; what
differs is only where its settings come from. Two of them gained a second
spelling so that nothing has to branch on the runtime: Postgres arrives as
a pooled binding or as `DATABASE_URL`, and the route to the issuer is a
service binding or `AUTH_INTERNAL_URL`.
That last one is new, and it is a split the binding was already making
without saying so. `AUTH_ISSUER_URL` has to be the issuer's public name,
because it is compared literally against every token's `iss` claim — but
the public name is often not routable from inside a deployment. So the
name and the route are two settings now rather than one that cannot be
both.
DNS moves out of code and into `docs/dns.md`, which lists every hostname
and what it is for. Six records that change roughly never did not need a
tool, and the table outlives whatever is answering the names — which is
the point, since some of them will stop being Workers. The sandbox
hostnames are hyphenated rather than nested for the same reason: a
certificate covering `*.nestri.io` covers one label and not two, so
`api-sandbox.nestri.io` can become an ordinary origin later without a
certificate having to be ordered for it first.
Also drops `EMAIL_DEV_LOG` from committed configuration into `.dev.vars`,
which `wrangler deploy` cannot upload. Printing a live sign-in code to a
log should not be one forgotten override away from production.
Two problems found in review, both in the key store.
Nothing stopped a kind from having two live keys, and the bootstrap path
walks straight into it: two workers starting against an empty table both
find no key and both insert one. From then on each signs and encrypts with
its own. That is not the harmless split the comment here claimed — the
issuer reaches for a single key rather than the published set when it
decrypts a session cookie and when it verifies an access token, so a cookie
written by one worker is unreadable to the other and a token minted by one
is rejected by the other. It stays silent until someone cannot sign in.
A partial unique index over the kind, where the key has not been retired,
makes the second insert a dropped write instead. Both workers then read the
table again and use the key that won, which is all that matters. The
conflict clause stops naming a target: both indexes on the table mean the
same thing at this call site, that the row already exists in some form.
Creating a key is now attempted once rather than retried, because a store
declining the write is an expected answer and spinning on it would hang the
request instead of failing it.
Separately, a key pair reported the algorithm the issuer currently uses
rather than the one stored on the key it was built from, so a retained key
would advertise the wrong algorithm in a token header and in the JWKS after
a rotation — which defeats keeping it. The material was already being
imported with the stored value; only what was handed back disagreed.
Retiring a key and creating its replacement now have to happen together, so
that a kind never has two live keys and never has none.
The issuer kept everything behind one get/set/remove/scan interface, which
is what a library that must run on any provider's cache can offer. Three of
the things kept there could not actually be served by it.
An authorization code must be redeemable once and a refresh token spendable
once, and through get and set the check and the write are separate steps —
so two requests arriving together both read an unspent record, and both mint
a session. In the refresh case that also means the reuse which reveals a
stolen token is never recorded, because recording it is the write that the
second caller overwrites. Each now has a table and an interface of its own:
redeeming is one `delete ... returning`, spending is one
`update ... where time_used is null returning *`, so exactly one caller is
ever told it went first. This is the same argument the device grant already
made, applied to the two records that had it too.
Signing keys move for a different reason. Nothing races for them; they are
the one record whose loss ends every session at once, and a cache is a place
things may be evicted from. They are retired by setting a column rather than
deleted, so the tokens they signed stay verifiable until they expire.
Both credential tables store a hash and never the credential, as the device
grant does. An authorization code travels in a query string and so passes
through history, referrer headers and any log along the redirect; a refresh
token resumes a session outright.
What is left in the generic store is the rate-limit counters — written far
more often than read, meaningless within the hour, and allowed to be
approximate, since a lost increment costs one guess out of ten. Those move
to Postgres too, so the only key-value binding this deploys with is gone and
the control plane's state is one database. That was the point: nothing here
now depends on a primitive a self-hoster cannot run.
The generic scan also gained the separator on its prefix, so scanning `a`
cannot return what is under `ab` — subjects and email addresses are both
prefixes of longer subjects and email addresses.
Deploying this signs everyone out. The signing keys and refresh tokens are
in a store that is being left behind, so the issuer starts with a fresh key
set and every existing token stops verifying.
The mutual exclusion was asserted only through sequential calls, where
the winner had already committed before the rival began. That never
reaches the case the design is for: both attempts reading the run as
unclaimed before either writes.
Two tests, because the first can pass for the wrong reason. The
concurrent transitions depend on how the transactions interleave; the
paired updates skip the read entirely, so nothing but the predicate in
the where clause can refuse the second.
Both fail with two winners if the check is moved out of the write and
left in the read above it.
The agent side sends a claim token on every write; this side rejected the
field outright, so every state report and every ticket publish answered
400. Both bodies now take it.
Underneath that, nothing compared a holder. A run was reachable by any
caller on the right machine, and a box names exactly one machine — so two
attempts polling the same job presented identical credentials and were
told apart only by which one's select landed first. That is timing, not a
rule, and no caller could be told which case it was in.
The row now remembers which attempt holds it. Taking a claim requires
there to be no holder; every write after it requires the caller to be the
holder. The same state reported by a different attempt is a lost race and
not a retry, and is refused whatever the state is - which is the only
thing that separates the two 200s from the 409s.
The ticket is held to the claim too, for a worse reason than a double
start: the client re-reads the address rather than keeping the first, so
a ticket written by a losing attempt produces a client that connects,
successfully, to a machine running nothing.
The holder is never cleared, including on a terminal state, so a settled
claim cannot be replayed and a finished run still records which attempt
ran it. It is not in what goes out - holding one permits writing to a
run, and the owner reading their own session is not the holder.
The cap on how many codes a sign-in could ask for was held per attempt,
keyed by a value in the caller's own cookie. That bounds nothing. The
caller decides how many attempts to start, and starting a fresh one costs
them a discarded cookie — so either replaying an older cookie or simply
beginning again walked straight around it, and the only thing left
spacing the mail out was the interval between sends.
The count now sits against the claim, over a window. That is the thing
being protected: the mailbox belongs to somebody who did not ask to hear
from us, and whoever is pointing at it is not the party to trust with the
tally.
A resend also left the previous code live, with a budget of guesses of
its own. Several resends therefore meant several working codes and
several times the chances at them, which made asking for a new code the
cheapest way to buy more tries at the old one. A new code now retires the
one before it.
Reported against the replay path. The replay was real and the same hole
was wider than that: starting a new attempt needed no replay at all.
A user code is eight characters from a twenty-five character alphabet,
which is a large space but a fixed one, and the endpoint that checked
them had no opinion about how often you asked. That is the guessing
attack RFC 8628 section 5.2 asks implementations to limit, and nothing
here did.
Wrong codes are now counted per caller address over a rolling window,
and the endpoint stops answering once the budget is gone. Getting a code
right is not charged for, so somebody who mistypes once and then succeeds
is not walking towards a lockout. A caller whose address cannot be
established shares one bucket with every other such caller, which makes
stripping the headers that say where you are buy a smaller budget rather
than an unlimited one.
The counter lives in the general-purpose store and is approximate. The
number that decides this is whether somebody is working through the code
space, and a handful either way does not change that answer.
A six-digit code has a million values, and nothing was counting how many
of them a caller tried. The code travelled in an encrypted cookie the
caller held, verification compared against that cookie, and a wrong
answer simply re-rendered the form. Nobody has to be the person the code
was mailed to: type somebody else's address into the first screen and the
code goes to their mailbox while the cookie stays with you. At that point
the only thing between a stranger and an account is a million requests,
and the constant-time comparison protecting the code was guarding a door
you could just keep knocking on.
Guesses are now counted on the server, under a name that changes with
every code. That placement is the point: a counter kept beside the code,
in the cookie, is a counter the guesser can wind back by replaying an
older copy. Starting over is still allowed and still costs a fresh code
sent to the mailbox being aimed at, which is where somebody notices. A
correct code spends its record too, so its remaining guesses do not carry
into the next one.
The cookie also lived for twenty-four hours, which made the pin a
password with a million possible values and a day to try them. Ten
minutes now, and the code stops being accepted when the clock says so
rather than when the cookie happens to go away.
Resend had no limit either, so the button was a way to mail a stranger as
fast as requests go out. Codes to one address are spaced, and one attempt
at signing in can only ask for so many.
Both refusals say the same thing on purpose. Which of the two it was is a
fact about somebody else's mailbox.
Anybody could ask for a device code and be handed a link with the user
code already in it. Following that link started a sign-in, and finishing
the sign-in approved the grant. So sending somebody the link was enough:
they saw an ordinary sign-in prompt, completed it, and whoever kept the
device code polled and collected their access and refresh tokens. The
victim never saw a question, because there was not one.
There is now. Signing in says who the browser belongs to; it does not say
the person meant to hand an account to a program somewhere else. Those
are two questions and only the second authorizes anything, so the flow
ends at a page that names the program, shows the code back so it can be
compared with what the device is displaying, and offers Approve and Deny.
Approving is a POST carrying a value from the cookie, so another site
cannot submit it on somebody's behalf. Denial moved onto the same page:
it used to be a GET anyone could fire, which meant a link scanner could
cancel a real sign-in and a stranger with a user code could grief one.
Three more things that were wrong underneath.
The grant was read, modified and written back as a whole record. A poll
that read a pending grant and then wrote its bookkeeping erased an
approval that landed in between, and the client polled a dead grant until
it expired. Grants moved to a table, where approving is one conditional
update and redeeming is one delete that returns what it deleted, so
neither party can undo the other and two polls cannot both be served.
Tokens were minted when the person clicked and left sitting in storage
until collected. They are minted at redemption now, so the lifetime the
client is told about starts when it receives them, and a grant nobody
collects leaves no usable refresh token behind.
The client identifier was never checked, at either end. It is validated
when the grant is created and has to match when the code is redeemed —
without that, a leaked code is redeemable by anyone, and the identifier
the token carries is whatever the last caller claimed. The device code
is also stored as a hash now, since it is the credential the tokens are
handed to.
The store is an interface because the issuer cannot reach the database,
and because the guarantees are the point: every method is one operation,
and no caller reads a grant, decides, and writes it back.
Three rules here are enforced across a lookup and then a write, and each
was only as good as whatever stopped the two from interleaving. Nothing
did.
The connection cap counted with `select ... for update` over the
connections a user already had. That locks the rows it finds, and when
it finds none it locks nothing — there are no gap locks under read
committed — so several first-time links all counted zero and all
inserted. Six concurrent links against a cap of four produced six. The
count now happens under a lock on the account's own row, which is the
one thing every caller for that account is guaranteed to contend on.
Creating an account from a verified address looked the address up and
then inserted. Two tabs finishing the same sign-in both found nothing,
and the loser got the driver's constraint violation instead of the
account the winner had just made. The unique index is the thing that
actually arbitrates, so the loser now reads back what the winner wrote.
Claiming an address on an older account had the same shape and now gives
the same sentence a screen would have shown a moment earlier.
The tests run each call several times at once against a real database,
because run one at a time all three pass whether or not any of this
exists.
The migration adds session.claim_token, but neither the schema nor the
snapshot knew about it. Nothing breaks today because the two agree with
each other; it breaks the moment someone declares the field, because
generate then diffs against a snapshot without it and emits
ALTER TABLE "session" ADD COLUMN "claim_token" text;
which fails on every database the migration has already run against.
Declared with no writer yet, so the schema, the snapshot and the
database say the same thing.
Connecting a Steam account wrote the row itself, so the limit on how many one
person may connect was enforced on the sign-in path and nowhere else — and
this is the path the settings screen calls, which makes it the one that would
have gone over. It now resolves who is asking and hands over to the single
place the rule lives.
Two things fall out of that. A Steam account already connected to somebody
else is a conflict rather than a silent success returning the other person's
row id, and a Steam id of the wrong shape is refused before a lookup.
A program with no browser — the desktop app — had a client for RFC 8628 and
nothing to point it at. This serves the other half: a device authorization
request that hands back a code, a page a person enters that code on, and a
token endpoint that answers the poll.
Both of the paths the client already implements are now reachable. Polling
faster than the advertised interval gets slow_down, and each warning widens
the interval so ignoring one costs more than the last; refusing gets
access_denied, so a request nobody started stops instead of being polled until
it ages out. The interval is capped, because it only ever grows and a code has
to stay pollable for the whole of its life.
The codes live in the same storage as the other short-lived grants rather than
in a table, since that is what they are. User codes are drawn from an alphabet
with no vowels and no look-alike pairs, and are accepted back in whatever case
and spacing a person retyped them in.
Runs against a database where every user was created by a gaming sign-in, so
most rows have no email at all and nothing has ever stopped two rows from
sharing one. The address is normalized first, duplicates are separated before
the unique index exists — the older row keeps the address, the newer one is
asked for a new one and loses nothing else — and the index is partial so that
accounts with no address do not collide with each other.
Verified against a database built to contain the awkward rows rather than
against an empty schema, by the script alongside it: an account with no
address, one with both, one with two connections, a duplicated address in two
different cases, an account already over the connection cap, and a deleted row
holding an address a live row also holds. Removing the de-duplication makes
the index creation fail, which is how we know the fixtures are load-bearing.
Also adds a nullable column recording which attempt holds a session run. It is
not part of the change above and carries no reason of its own; the endpoint
that reads and writes it arrives separately, and it is here because a schema
change has one owner at a time.
Signing in with Steam used to create the account. That made a second Steam
account a second person, and it made losing a Steam account lose everything
attached to it — the boxes, the team, the billing history.
Invert it. A user comes into existence by verifying an email address and
nothing else; a Steam account hangs off a user that already exists, capped at
four. Signing in with Steam resolves an account and refuses when there is
none, so the accounts made before this keep working — they already have the
connection this looks for — while nothing new is created behind a persona.
The cap lives here rather than in the schema because a unique index cannot
count the rows sharing a foreign key. The email column gains a partial unique
index instead, which is the constraint that can be expressed, and the address
is trimmed and lower-cased at the edge so two spellings are not two accounts.
Four things the session endpoints did not do, or did wrongly.
The box had three states and nothing wrote them. A box read `created`
while a run on it was `live`, so every screen showing a person what their
hardware is doing was reading a column no code had ever moved. A run
reaching `live` now makes its box `running`, and a terminal run stops it:
`ended` cleanly, `failed` not, carrying the reason the agent gave. Not
every run state maps — a box has no `starting` on purpose, because that
transition is synchronous from the agent's side and a state nobody sets
is a state that lies. Both writes are one transaction, since "this run is
live" and "the box under it is running" are one fact in two tables, and a
box stuck `running` with nothing on it has nothing to correct it.
`POST /session` accepted any game in the catalog. A run launches as a
Steam account that has to own the game, so one outside the caller's
library is a box that starts, tries to launch and fails minutes later
with nothing to point at; it is now refused up front. Told apart from a
game that does not exist rather than hidden, because the catalog is
public and "you do not own this" is a sentence a person can act on. The
library is a synced copy, so this refuses a game bought since the last
sync — that is a staleness bug in the sync, not a reason to start runs
that cannot work.
Publishing a ticket only refused terminal runs, so a host could publish
an address for a run it had never claimed. A ticket is the address of
something being brought up, so only `starting` and `live` accept one, and
the state is in the write rather than only in the check above it. The two
refusals stay separate answers because they are different mistakes: one
agent skipped a step, the other has nothing left to reach.
The migration that adds the one-active-run index stopped older duplicate
runs without clearing the ticket they had published, which is the
invariant that same migration exists to establish. It clears it now,
verified against a box carrying two unstopped runs.
Nine tests, each checked against the unfixed code first.
Two invariants the session endpoint stated but did not hold.
A box runs one thing at a time. `POST /session` read `activeForBox` and
refused when something was already running, but the read and the insert
are two statements with nothing between them: two requests that both saw
"nothing is running" each got a row, and the job poll then handed the
host the same box to start twice. Demonstrated at 2 rows and 2 jobs from
one box. That is the failure the state claim exists to prevent, one step
earlier, and it takes the same answer — a partial unique index on the
predicate the read asks about, so the database refuses the second insert.
`Session.request` turns that refusal into the same 409 in the same words,
so a caller cannot tell which of the two caught it.
The migration resolves any existing duplicates before creating the index,
keeping each box's newest unstopped run because that is the one a person
is waiting on, and stopping the rest rather than deleting them.
Separately, a run that reached `ended` or `failed` kept the last ticket
it published. Publishing a new one is already refused, so the stale
address was both the only ticket a client could read for a dead run and
the one nothing was allowed to replace — and a client that polls would
dial it. Terminal transitions now clear it, in `setState` as well as in
the compare-and-set, so the invariant does not depend on which writer
stopped the run.
Seven tests, each checked against the unfixed code first. The published
descriptions for the ticket field and the read endpoint now say that a
stopped run has no address.
A run of a box had core support and no HTTP surface. This adds both halves
of it: a person asks for a run and reads it back, and the host agent the box
is placed on is handed the work and reports what happened.
The access rule is the point. An agent may only see or touch a run whose box
is placed on its own hardware, and that is a `where` clause on every one of
the three agent endpoints rather than a check next to them — host credentials
are long-lived secrets sitting on hardware in somebody's home, so what one
leaking can reach has to be decided by the query. "No such run" and "not your
run" are the same refusal, so ids cannot be discovered by reporting states
at them.
`Session.setState` updated on the id alone, which means two agents polling
the same work both succeed and both start the same box. There is one host
today, which is exactly why that would have been built wrong and stayed
wrong. The state a run is moving out of is now part of the `where` clause,
so the database picks the winner; the loser gets a conflict rather than a
silent no-op. Three cases that look alike are kept apart: re-reporting a
state you already reported changes nothing and is not an error, a transition
that does not exist is refused with the run left where it was, and another
host reporting anything is forbidden.
Asking for a run makes no decision about where it happens — a box already
names its hardware, so the run inherits it by join. Placement therefore
gets an interface at box creation, where the decision actually is, with the
single-host case as its implementation and a deliberate refusal when there
is more than one candidate and no policy to choose with.
Tests cover the wire shape from both sides, the query scoping, the claim,
and the timestamp idempotence a run's billing rests on.
Comments and served API descriptions here had grown references that only make
sense to someone with our internal notes: relative paths that escape this
tree, filenames and titles of documents nobody outside can open, quoted prose
from them, and the name of a component that has no public surface — once in an
OpenAPI description, which is published output rather than source.
None of it was load-bearing. Every case restates as what the code actually
requires, and every rewrite came out shorter: "in the words the host agent
reports" for a component name, "republished as addresses are discovered" for a
quoted phrase, "a size tier sets vCPU, RAM and the output geometry" for a
sentence that had been carrying a path.
Internal reasoning is now cited exactly one way, ref(d-NNNN) in a source
comment, with the rule that the sentence must still stand if the marker is
deleted. CLAUDE.md leads with it, because the previous version of this mistake
was made by people who knew the repo was public and it still took ten
occurrences to notice, so "be careful" is not a mechanism.
Commit messages get the stricter rule and carry no references at all: a
comment can be fixed by the next commit and a published message cannot be
fixed at all. Git hooks now enforce both halves.
The check caught a real one while being written: the CLAUDE.md table spelled
out the paths it was prohibiting, which discloses them to exactly the reader
it protects against.
138 tests, 0 fail.
Second half of G1's "neslet registers against api.nestri.io and heartbeats".
Registration already worked; there was no heartbeat endpoint at all — grep for
it across apps/api and packages/core returned nothing, and neslet's own
main.rs says the same from its side.
POST /machine/heartbeat, machine credentials only. Two decisions worth stating
because neither is obvious from the diff:
**It returns the interval.** The auth middleware already touches lastSeen on
every authenticated machine request, so an endpoint that only did that would
add an endpoint and no capability. What a host cannot know on its own is how
often the control plane wants to hear from it, so the response carries the
cadence. A fleet whose interval can only change by shipping a new agent is a
fleet whose interval never changes.
**It takes no body.** neslet has a HostSummary ready to send, and week 2 owns
box state reporting. Accepting fields nothing acts on yet would mean a wire
shape we would have to keep, chosen before the thing that consumes it exists.
Online-ness is derived from lastSeen rather than stored: a host that stops
beating goes offline through the passage of time, which is the one mechanism
that cannot itself fail. Three missed beats, not one — a single missed beat is
a lost packet, and treating that as offline would make placement flap.
Also: the machine actor's teamID stops being optional. It was `...(teamId ? {}
: {})` in the middleware, a branch for a state that cannot exist now that
machine.team_id is notNull.
134 tests, 0 fail.
Migration 1 of 0048, and the first of the seven weeks — nothing about a live
feed works without these two tables, so it is not a cleanup during them.
box a VM someone owns: an id that is also its DNS label, an editable
label, an owning user, the machine it sits on, a tier and a state.
Owned by a person and placed on a team's hardware, which are two
different relationships, hence both userId and machineId.
session one run of one box by one linked Steam account, and what costs
money. Separate from box because the ticket changes after bind as
addresses are discovered — the vsock contract calls it "a stream,
not one value" — so it is a column a client polls, not a value it
is handed once.
Box states are neslet's own three and no more. `starting` and `stopping` are
the obvious additions and both are omitted because nothing would ever write
them; a failed box is `stopped` with stopClean false, which is how neslet
models it too.
The generated migration would have failed on live rows in three ways, so it
is hand-written and tested against a database seeded at the old schema:
- machine.team_id becomes notNull, and *every existing row is null* because
the old registration path passed null. Personal teams are backfilled for
machine owners first, reusing a team they already own rather than minting
a second, with the owner membership row repaired where missing.
- game_download.host_id becomes a foreign key. It held free-form strings,
so unattributable rows are deleted before the cast — the only destructive
statement here, and a considered loss: it is a progress report neslet
re-derives from disk.
- Team.createPersonal was written and documented in packages/core/CLAUDE.md
as part of the login flow and never actually called, so no user has a
team. ensurePersonal is idempotent and now runs on every login, which is
what backfills accounts the migration does not reach.
Verified on a seeded legacy database: three null-team machines backfilled, an
existing team reused rather than duplicated, a blank display name handled, and
both unattributable download rows dropped while the attributable one survived.
Also fixes two things this work ran into rather than caused:
- Database.client() built a new postgres pool on every call, and use()
called it twice per invocation — pools of ten connections held for a 30s
idle timeout. Invisible in a Worker where requests are short; the suite
crossed 100 connections and Postgres said "sorry, too many clients
already" in whichever file ran last, which reads as a flaky test rather
than a leak. Now one pool per connection string.
- download.test.ts asserted against `hst_…` host ids, which is exactly the
unattributable row the new foreign key exists to refuse.
There is no "no team" any more: PATCH /machine/:id took teamId null to mean
"mine alone" and now requires a team, because the personal team is the one to
name. Its test is updated to the new contract rather than deleted.
113 → 128 tests, 0 fail.
`bun run db:push` has been failing on **every pull request** with
`error: Script not found "db:push"`. A red check has meant nothing for
as long as that's been true.
There are **two independent causes**, and fixing only the reported one
leaves the job red.
## 1. The script isn't at the root
`db:push` lives in `packages/core/package.json`; CI runs from the root.
Added root passthroughs for `db:migrate` and `db:push`, so the command
CI runs is also the one a human can run.
## 2. `drizzle.config.ts` enabled TLS for any `DATABASE_URL`
```ts
ssl: !!process.env.DATABASE_URL ? { rejectUnauthorized: false } : false
```
That's true for *any* URL — so it failed against every plain Postgres,
**including CI's own `postgres:18-alpine` service container**. And
`drizzle-kit` reports that failure as a spinner and a non-zero exit with
no message attached, which is why it would have been maddening to find
from a log.
Measured against a local container:
| | result |
|---|---|
| `DATABASE_URL` set (TLS on) | migrations fail, no error text |
| `DATABASE_URL` unset, same database | all seven apply |
TLS is now decided by the connection string: an explicit `sslmode` wins,
otherwise a local host gets none (it doesn't speak TLS at all) and any
other host gets TLS without chain verification, which is what a hosted
Postgres usually needs. The URL is parsed once rather than eight times.
## CI now applies migrations instead of `push`
`drizzle-kit push` diffs the schema against whatever is already in the
database and is a development tool — CI wants exactly what's committed
in `packages/core/migrations`. And `push` under `strict: true` asks for
confirmation, which on a runner is a **hang**, not a failure.
## Verified
Locally against `postgres:18-alpine` from an empty database, running
exactly what the workflow runs:
```
bun install --frozen-lockfile ✓
bun run db:migrate ✓ 7 migrations applied
bun test ✓ 113 pass, 0 fail, 297 expect() calls
```
Worth landing ahead of #310 so that a red check starts meaning something
again.
<!-- greptile_comment -->
<h3>Greptile Summary</h3>
This PR repairs the database-backed web CI job by exposing core database
commands at the workspace root, applying committed migrations instead of
schema push, and selecting PostgreSQL TLS behavior from the connection
URL.
- Adds root passthrough scripts for database migration and schema push
commands.
- Adds the core `drizzle-kit migrate` command and runs it in CI.
- Disables TLS for local PostgreSQL while honoring explicit `sslmode`
settings.
- Keeps migration and test steps pointed at the same temporary CI
database.
<h3>Confidence Score: 5/5</h3>
The PR appears safe to merge, with the migration command, working
directory, connection settings, and test database remaining aligned.
The changed CI path reaches the committed migration history through the
intended core package configuration, uses plaintext for the local
PostgreSQL service, and then tests against the same migrated database;
no changed-code defect remains.
<h3>Important Files Changed</h3>
| Filename | Overview |
|----------|----------|
| .github/workflows/ci.yml | Replaces schema push with committed
migration execution while preserving the shared CI database URL. |
| package.json | Adds root-level passthroughs to the core package's
database commands. |
| packages/core/drizzle.config.ts | Parses the database URL once and
selects TLS based on explicit mode or local-versus-remote host
inference. |
| packages/core/package.json | Adds the `drizzle-kit migrate` script
consumed by the root command and CI workflow. |
<h3>Flowchart</h3>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart LR
PR[Pull request or main push] --> CI[Web CI job]
CI --> PG[PostgreSQL 18 service]
CI --> Install[Bun frozen install]
Install --> Root[Root db:migrate script]
Root --> Core[packages/core db:migrate]
Core --> Config[drizzle.config.ts]
Config --> Migrations[Committed migrations]
Migrations --> PG
PG --> Tests[Bun tests]
```
<sub>Reviews (1): Last reviewed commit: ["fix(ci): the web job has been
failing
on..."](203e882fbd)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=59430932)</sub>
<!-- /greptile_comment -->
CLAUDE.md was 1,324 lines and all of it was about the TypeScript half, written
before there was another half. Every line of it loaded on every turn regardless
of what was being worked on, which is a real cost paid constantly for context
that is usually irrelevant.
Split by where it applies, so each guide loads when you are in the directory it
describes:
packages/core/CLAUDE.md 694 domain modules, fn(), actor, errors, auth
apps/api/CLAUDE.md 284 routes, registration, error flow
docs/alchemy.md 345 stages, bindings, secrets, the CLI
CLAUDE.md 72 the repo, both toolchains, two hard rules
Nothing was rewritten or dropped — the three files are the original text,
verified identical after the split. What the root file now carries is only what
is true repo-wide: the layout, the commands, where the detail lives, and the two
rules that are not style preferences. One of those is that nothing closed may
enter this repo, which is here because it has already been caught once.
The README described a streaming platform in four bullets and did not mention
that half the repository is Rust that runs inside a virtual machine. It now says
what each component does, why a micro-VM rather than a container, what is
deliberately absent, and what decides whether a thing is open — data is, capacity
is not.
It also says plainly that this is mid-rewrite and the docs are behind. Someone
arriving at a repo whose documentation does not match its tree should be told
that by the README rather than discover it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Squashes the current state of the internal working tree onto this history.
The two trees had grown apart with no common ancestor, so this is a content
sync rather than a merge, and the published history is preserved rather than
rewritten — a force-push here would break every existing fork and clone to no
benefit.
What lands:
- Waitlist: API route, core module, and migration 0006 alongside game aliases.
- User verification.
- CI, oxfmt config, editor settings.
- Assorted fixes across the API routes and core modules.
The repository's own README, the wordmark and the per-package READMEs are kept
from this side; the internal tree had dropped them and they are what a stranger
arriving here reads first.
The marketing site in the internal tree is deliberately not here. It is a
separate product with its own repo and its own licence, and this repo is the
open one — a closed component does not belong in it regardless of how convenient
the directory looked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
