From 4668c98785735ce05b1eac197baa26ce488ea904 Mon Sep 17 00:00:00 2001 From: Wanjohi Date: Thu, 17 Sep 2026 01:44:16 +0300 Subject: [PATCH 1/2] ci: a merge to prod releases the control plane, migrator included MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `release-prod.yml` for api and auth, mirroring the edge's. The gate is not "you may not merge" — it is that merging does not deploy: tests run against a real Postgres, both binaries are executed, and a failure anywhere means no release exists, so the machine keeps serving what it has. The new piece is `nestri-migrate`, because the deploy runs migrations *before* it swaps a release into place and had nothing to run. `drizzle-kit migrate` reads the migrations folder at runtime, which is right on a laptop and wrong on a server: the deploy ships flat, checksummed files into bin/, and a migrator that needs a directory beside it can be pointed at the wrong directory. So the folder is baked into the binary — generated on every build rather than committed, so it cannot drift — and the artefact's checksum then covers every statement it will run. It reimplements drizzle's bookkeeping in thirty lines of SQL rather than calling into `db.dialect.migrate`: same table, same schema, same sha256 over the whole file, same high-water-mark comparison. That equivalence is the one thing that must not rot, because the production database was first migrated by drizzle-kit and a disagreement means a migration applied twice. Two checks hold it: CI applies the migrations with drizzle-kit and then asserts the embedded set reports nothing pending, and the hashes were verified by hand against the live database — all fourteen match to the byte. Proven before shipping, against the real database: nothing pending on the deployed schema, 14 unchanged rows, and a scratch database migrated from empty to the same 22 tables and the same high-water mark, idempotent on a second run. Refuses with exit 2 when DATABASE_URL is absent rather than defaulting to localhost, which would be a migrator reporting success having migrated nothing. setup-bun is pinned to a commit and not to `v2`. A moving major tag is fine everywhere else in this repository; this workflow is the only thing between a merge and a process serving users, and there is no first-party bun action to prefer instead. --- .github/workflows/release-prod.yml | 274 ++++++++++++++++++++++ .gitignore | 5 + packages/core/scripts/embed-migrations.ts | 56 +++++ packages/core/src/migrate.ts | 113 +++++++++ 4 files changed, 448 insertions(+) create mode 100644 .github/workflows/release-prod.yml create mode 100644 packages/core/scripts/embed-migrations.ts create mode 100644 packages/core/src/migrate.ts diff --git a/.github/workflows/release-prod.yml b/.github/workflows/release-prod.yml new file mode 100644 index 00000000..20ffdf49 --- /dev/null +++ b/.github/workflows/release-prod.yml @@ -0,0 +1,274 @@ +# What a merge to `prod` produces: the control plane, as three binaries. +# +# The gate is not "you may not merge" — it is **"merging does not deploy"**. +# Every step below runs before anything is published, and a failure anywhere +# means no release exists, so the machine has nothing new to pull and keeps +# serving what it already has. There is no staging environment; this file is +# the substitute, and it is only a real gate if it is willing to refuse. +# +# Unlike the edge, this half has tests and a database, so the gate is a real +# test run against a real Postgres and not only a smoke test. +# +# These are deployed components, not released ones: they are identified by a +# git sha and carry no version number. The release tag is `prod-` and +# there is deliberately no semver anywhere in here. +name: release prod + +on: + push: + branches: [prod] + # For iterating on this file without merging. Builds, tests, and publishes + # nothing — which is what you want while it is being written. + workflow_dispatch: + +permissions: + contents: write + +concurrency: + # Two merges in quick succession: the newer wins, the older is cancelled. + # Deploying the older sha after the newer one is a silent rollback nobody + # asked for. + group: release-prod + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + services: + postgres: + image: postgres:18-alpine + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: nestri + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + # Pinned to a commit rather than to `v2`. Everywhere else in this + # repository a moving major tag is fine; this workflow is the only thing + # between a merge and a process serving real users, and a tag is a + # mutable pointer someone else controls. There is no first-party bun + # action to prefer instead, so the mitigation is the pin. + # oven-sh/setup-bun v2.2.0 + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 + with: + bun-version: 1.3.11 + + - name: Install + run: bun install --frozen-lockfile + + # The committed migrations, applied by drizzle-kit — the same way + # `web.yml` does it, and the same way a developer does it locally. This + # is the reference against which the embedded set is checked below. + - name: Apply migrations + run: bun run db:migrate + env: + DATABASE_URL: postgres://postgres:postgres@localhost:5432/nestri + + - name: Test + run: bun test + env: + TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/nestri + + # Generated on every build rather than committed, so the binary's + # migration set cannot drift from `packages/core/migrations`. + - name: Embed migrations + run: bun run packages/core/scripts/embed-migrations.ts + + - name: Build + run: | + set -euo pipefail + mkdir -p dist + bun build --compile --minify --target=bun-linux-x64 \ + apps/api/app/server.ts --outfile dist/nestri-api + bun build --compile --minify --target=bun-linux-x64 \ + apps/auth/src/server.ts --outfile dist/nestri-auth + bun build --compile --minify --target=bun-linux-x64 \ + packages/core/src/migrate.ts --outfile dist/nestri-migrate + ls -lh dist + + # The two things worth asserting about the migrator are that it agrees + # with drizzle-kit and that it will not run without being told where. + # + # The agreement check is the important one. `nestri-migrate` reimplements + # drizzle's bookkeeping — same table, same sha256, same high-water mark — + # because the production database was first migrated by drizzle-kit and + # the two have to agree about what has already run. If they ever diverge, + # the failure is a migration applied twice against production. Here, the + # database has just been migrated by drizzle-kit, so the embedded set + # must report nothing pending. It is the cheapest possible test for the + # most expensive possible bug. + - name: Smoke test — the migrator agrees with drizzle-kit + env: + DATABASE_URL: postgres://postgres:postgres@localhost:5432/nestri + run: | + set -euo pipefail + out="$(./dist/nestri-migrate --check)" + echo "$out" + echo "$out" | grep -q "nothing to do" || { + echo "::error::the embedded migrations disagree with drizzle-kit — a migration would be applied twice" + exit 1 + } + + # No DATABASE_URL must be a refusal, not a default: configuration + # that is absent means refuse, never assume. A migrator that falls + # back to localhost is one that reports success having migrated + # nothing that matters. + if env -u DATABASE_URL ./dist/nestri-migrate; then + echo "::error::ran with no DATABASE_URL" + exit 1 + fi + + # Both servers are executed here, because `bun build --compile` embeds a + # runtime and a binary that builds can still fail the moment it runs. + # Neither check touches the database: `/` and the metadata document are + # answered without it, which is what makes this a test of the binary + # rather than of the runner's Postgres. + - name: Smoke test — both servers answer + run: | + set -euo pipefail + + PORT=13999 HOST=127.0.0.1 \ + DATABASE_URL="postgres://nobody@127.0.0.1:1/none" \ + AUTH_ISSUER_URL=https://auth.nestri.io \ + ADMIN_SHARED_SECRET=smoke \ + ./dist/nestri-api & api=$! + # NOT `/`, which the issuer answers 404 — it has no root route. NOT + # `openid-configuration` either, also a 404: this is an OAuth 2.0 + # authorization server, not an OIDC provider. A health check written + # from habit fails every deploy forever. + PORT=13998 HOST=127.0.0.1 \ + DATABASE_URL="postgres://nobody@127.0.0.1:1/none" \ + ./dist/nestri-auth & auth=$! + trap 'kill $api $auth 2>/dev/null || true' EXIT + sleep 3 + + code=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:13999/ || echo 000) + [ "$code" = "200" ] || { echo "::error::api / returned $code"; exit 1; } + echo "api / -> 200" + + code=$(curl -s -o /dev/null -w '%{http_code}' \ + http://127.0.0.1:13998/.well-known/oauth-authorization-server || echo 000) + [ "$code" = "200" ] || { echo "::error::auth metadata returned $code"; exit 1; } + echo "auth /.well-known/oauth-authorization-server -> 200" + + # These sit behind a tunnel and authenticate nothing at the network + # layer, so the bind default is a security control. Both defaulted to + # 0.0.0.0 until 2026-09-16, which was harmless inside a container and + # a public listener as a bare process. + # The unreachable database is named `nowhere.invalid` and not a + # loopback address on purpose: the assertion below greps for + # 127.0.0.1, and a connection string containing it would satisfy the + # grep from the wrong line. + env -u HOST PORT=13997 DATABASE_URL="postgres://nobody@nowhere.invalid:1/none" \ + ADMIN_SHARED_SECRET=smoke ./dist/nestri-api > /tmp/bind.log 2>&1 & probe=$! + sleep 3 + kill $probe 2>/dev/null || true + grep -q "listening on http://127.0.0.1:" /tmp/bind.log || { + echo "::error::the default bind is no longer loopback" + cat /tmp/bind.log + exit 1 + } + echo "default bind -> loopback" + + - name: Package + id: package + run: | + set -euo pipefail + cd dist + sha256sum nestri-api nestri-auth nestri-migrate > SHA256SUMS + cat SHA256SUMS + + # The manifest is what the machine reads: the sha it came from, and + # the checksum of every artefact, so the agent can verify what it + # downloaded before it swaps anything into place. + # + # `unit` is what gets restarted. `nestri-migrate` has none on purpose + # — it is run, not served — and `migrate: true` is what makes the + # deploy run it *before* the swap. That ordering is why migrations + # must be additive: a rollback moves code and never schema, so every + # migration has to be readable by the release it replaces. + cat > manifest.json <> "$GITHUB_OUTPUT" + + - uses: actions/upload-artifact@v4 + with: + name: control-plane-prod + path: dist/* + if-no-files-found: error + + outputs: + short: ${{ steps.package.outputs.short }} + + publish: + # Only a real merge publishes. A manual run builds, tests and stops. + if: github.event_name == 'push' && github.ref == 'refs/heads/prod' + needs: build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/download-artifact@v4 + with: + name: control-plane-prod + path: dist + + # One immutable release per sha, and the tag is the sha. Rollback is + # therefore "point the machine at the previous release", not "rebuild an + # older commit and hope it produces the same bytes". + - name: Release + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + TAG="prod-${{ needs.build.outputs.short }}" + if gh release view "$TAG" >/dev/null 2>&1; then + echo "release $TAG already exists — replacing its assets" + gh release upload "$TAG" dist/* --clobber + else + gh release create "$TAG" dist/* \ + --target "${{ github.sha }}" \ + --title "$TAG" \ + --notes "Deployed build of the control plane from ${{ github.sha }}. + + No version number: these are deployed components, identified by a git + sha. The machine reads \`manifest.json\` from this release, + verifies each artefact against its \`sha256\`, runs \`nestri-migrate\` + before the swap, and restarts only the units whose binary changed. + + Built by ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" + fi + gh release view "$TAG" diff --git a/.gitignore b/.gitignore index 7134a86a..2e39519f 100644 --- a/.gitignore +++ b/.gitignore @@ -47,3 +47,8 @@ lunora/_generated # accident; never again. nesdoctor.json *.nesdoctor.json + +# Baked from packages/core/migrations by scripts/embed-migrations.ts on every +# build. Committing it would let the binary's migration set drift from the +# folder, which is the one thing embedding exists to prevent. +packages/core/src/migrations.generated.ts diff --git a/packages/core/scripts/embed-migrations.ts b/packages/core/scripts/embed-migrations.ts new file mode 100644 index 00000000..7189263d --- /dev/null +++ b/packages/core/scripts/embed-migrations.ts @@ -0,0 +1,56 @@ +/** + * Bake `migrations/` into a TypeScript module so the migrator can be a single + * self-contained binary. + * + * `drizzle-kit migrate` reads the folder at runtime, which is the right shape + * on a laptop and the wrong one on a server: the deploy agent ships flat, + * checksummed files into `bin/`, and a migrator that needs a directory beside + * it is a migrator that can be run against the wrong directory. Embedding + * removes the question — the binary *is* the migration set, and its checksum + * covers every statement it will run. + * + * Run before `bun build --compile`. It runs on every build rather than being + * committed, so the embedded set cannot drift from the folder. + * + * bun run packages/core/scripts/embed-migrations.ts + */ +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import path from 'node:path'; + +const here = path.dirname(new URL(import.meta.url).pathname); +const folder = path.join(here, '..', 'migrations'); +const out = path.join(here, '..', 'src', 'migrations.generated.ts'); + +type JournalEntry = { idx: number; when: number; tag: string; breakpoints: boolean }; + +const journal = JSON.parse( + fs.readFileSync(path.join(folder, 'meta', '_journal.json'), 'utf8') +) as { entries: JournalEntry[] }; + +// Deliberately identical to drizzle-orm's own `readMigrationFiles`: the same +// `--> statement-breakpoint` split, the same sha256 over the whole file, the +// same `when` as the ordering key. The live database was migrated by +// drizzle-kit, so anything else here would re-apply migrations that have +// already run. +const migrations = journal.entries.map((e) => { + const sql = fs.readFileSync(path.join(folder, `${e.tag}.sql`), 'utf8'); + return { + tag: e.tag, + sql: sql.split('--> statement-breakpoint'), + bps: e.breakpoints, + folderMillis: e.when, + hash: crypto.createHash('sha256').update(sql).digest('hex') + }; +}); + +fs.writeFileSync( + out, + `// Generated by scripts/embed-migrations.ts. Do not edit.\n` + + `// ${migrations.length} migrations, ${migrations.at(-1)?.tag ?? 'none'} last.\n` + + `export type Embedded = {\n` + + `\ttag: string;\n\tsql: string[];\n\tbps: boolean;\n\tfolderMillis: number;\n\thash: string;\n};\n\n` + + `export const MIGRATIONS: Embedded[] = ${JSON.stringify(migrations, null, '\t')};\n` +); + +console.log(`embedded ${migrations.length} migrations -> ${path.relative(process.cwd(), out)}`); diff --git a/packages/core/src/migrate.ts b/packages/core/src/migrate.ts new file mode 100644 index 00000000..b5b18860 --- /dev/null +++ b/packages/core/src/migrate.ts @@ -0,0 +1,113 @@ +/** + * The migrator, as a binary. + * + * nestri-migrate apply everything pending + * nestri-migrate --check say what is pending, change nothing + * + * The deploy agent runs this **before** it swaps the new release into place, + * which is what makes the additive-only rule load-bearing: a rollback moves + * code and never schema, so every migration has to be readable by the release + * it is replacing. Add a column, deploy, stop writing the old one, drop it in + * a later release — never in the same one. + * + * The bookkeeping is deliberately identical to `drizzle-orm`'s own + * `PgDialect.migrate`, down to the table, the schema and the high-water-mark + * comparison, because this database was first migrated by `drizzle-kit` and + * both must agree about what has already run. Verified 2026-09-17 against the + * live database: all fourteen hashes and timestamps match to the byte. + * + * It talks to Postgres directly rather than through `drizzle-orm` — this is + * thirty lines of SQL, and the version that goes near production schema should + * be the one you can read in full. + */ +import postgres from 'postgres'; + +import { MIGRATIONS } from './migrations.generated.js'; + +const SCHEMA = 'drizzle'; +const TABLE = '__drizzle_migrations'; + +async function main() { + const check = process.argv.includes('--check'); + + const url = process.env.DATABASE_URL; + if (!url) { + console.error('nestri-migrate: DATABASE_URL is not set'); + process.exit(2); + } + + // One connection, and no idle timeout worth the name: this process exists + // for a few seconds and then stops. + const sql = postgres(url, { + max: 1, + idle_timeout: 5, + connect_timeout: 30, + // `CREATE ... IF NOT EXISTS` raises a NOTICE every single run, and + // postgres-js prints the whole struct by default -- so the steady state + // of this binary was eighteen lines of noise around one line of fact. + // Errors are unaffected; they are thrown, not noticed. + onnotice: () => {} + }); + + try { + await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS "${SCHEMA}"`); + await sql.unsafe(` + CREATE TABLE IF NOT EXISTS "${SCHEMA}"."${TABLE}" ( + id SERIAL PRIMARY KEY, + hash text NOT NULL, + created_at bigint + ) + `); + + const rows = await sql.unsafe( + `select id, hash, created_at from "${SCHEMA}"."${TABLE}" order by created_at desc limit 1` + ); + const last = rows[0] ? Number(rows[0].created_at) : null; + + // A high-water mark, not a set of hashes. That is drizzle's rule and + // changing it here would make the two disagree about a migration that + // was applied out of order. + const pending = MIGRATIONS.filter((m) => last === null || last < m.folderMillis); + + if (pending.length === 0) { + console.log(`nestri-migrate: nothing to do (${MIGRATIONS.length} applied)`); + return; + } + + if (check) { + console.log(`nestri-migrate: ${pending.length} pending`); + for (const m of pending) console.log(` ${m.tag}`); + return; + } + + // All of them in one transaction, as drizzle does. Postgres has + // transactional DDL, so a failure half way through leaves the schema + // exactly as it was rather than half-migrated with a release about to + // be swapped in on top of it. + await sql.begin(async (tx) => { + for (const m of pending) { + console.log(`nestri-migrate: applying ${m.tag}`); + for (const stmt of m.sql) { + if (stmt.trim() === '') continue; + await tx.unsafe(stmt); + } + await tx.unsafe( + `insert into "${SCHEMA}"."${TABLE}" ("hash", "created_at") values($1, $2)`, + [m.hash, m.folderMillis] + ); + } + }); + + console.log(`nestri-migrate: applied ${pending.length}`); + } finally { + await sql.end(); + } +} + +main().catch((err) => { + // Non-zero and loud. The agent refuses to swap the release in when this + // fails, so the thing that matters most is that it cannot fail quietly. + console.error('nestri-migrate: failed'); + console.error(err); + process.exit(1); +}); From 7859c8e13aa3f55677fa2de794f4f92a41422372 Mon Sep 17 00:00:00 2001 From: Wanjohi Date: Thu, 17 Sep 2026 01:44:50 +0300 Subject: [PATCH 2/2] ci: restart the issuer before the API Units come back in the order their artefacts appear in the manifest, and the API reaches the issuer over AUTH_INTERNAL_URL -- so api-then-auth means the API spends a moment talking to a service that is restarting. --- .github/workflows/release-prod.yml | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release-prod.yml b/.github/workflows/release-prod.yml index 20ffdf49..2be50e64 100644 --- a/.github/workflows/release-prod.yml +++ b/.github/workflows/release-prod.yml @@ -190,6 +190,10 @@ jobs: # the checksum of every artefact, so the agent can verify what it # downloaded before it swaps anything into place. # + # The order matters: units are restarted in the order they appear + # here, and the API's AUTH_INTERNAL_URL names the issuer, so the + # issuer comes back first. + # # `unit` is what gets restarted. `nestri-migrate` has none on purpose # — it is run, not served — and `migrate: true` is what makes the # deploy run it *before* the swap. That ordering is why migrations @@ -205,16 +209,16 @@ jobs: "run": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}", "migrate": true, "artefacts": [ - { - "name": "nestri-api", - "unit": "nestri-api", - "sha256": "$(sha256sum nestri-api | cut -d' ' -f1)" - }, { "name": "nestri-auth", "unit": "nestri-auth", "sha256": "$(sha256sum nestri-auth | cut -d' ' -f1)" }, + { + "name": "nestri-api", + "unit": "nestri-api", + "sha256": "$(sha256sum nestri-api | cut -d' ' -f1)" + }, { "name": "nestri-migrate", "sha256": "$(sha256sum nestri-migrate | cut -d' ' -f1)"