clone

one developer profile for links and private tcp services, reached peer to peer over iroh with no open ports

Run the control plane on Fly

Dami committed d74882f
One machine, one volume, SQLite on it. A Fly volume attaches to a single machine
in a single region and nothing here wants a second writer; since the heartbeat
went away the database is read-mostly, a few hundred bytes per account and a
handful of writes per user per lifetime, so this shape has room in it.

The signing key becomes a secret rather than a file. Issuer::from_hex takes it
directly, Config prefers it over the state directory, and an unusable value is
fatal rather than a reason to generate a fresh one — silently minting a new key
would break every daemon in the field while looking healthy from this end. Losing
that key is the only unrecoverable failure in the system, and tying its survival
to one volume on one host was the wrong place to put it. Local development is
unchanged: no secret set, same file as before.

Two SQLite settings that a host which SIGKILLs you deserves:

  - WAL and synchronous=NORMAL. A deploy replaces the machine rather than asking
    the process to stop, so an abrupt exit is the normal ending. WAL replays; the
    trade is that a crash can lose the last transactions, which here means a
    profile edit someone can redo, not a corrupt database.

  - foreign_keys=ON, on every open. It was set inside SCHEMA, which migrations
    skip once user_version has moved, so every process after the one that created
    the database ran with ON DELETE CASCADE quietly doing nothing. A share
    outliving its resource was possible the whole time. There is a test.

The image builds both halves from source. web/pkg/ is gitignored, so a deploy
that copied it in would ship whatever happened to be on the machine running
`fly deploy`; building the bundle and the binary in the same image means they
always match. That needed one portability fix — build-wasm.sh reached for
shasum, which is a perl thing, and Debian has sha256sum.

.dockerignore excludes data/ and *.key. The build context is a place secrets go
to become image layers.

Not included, deliberately: IROH_SERVICES_API_SECRET. iroh_services::Client
needs an iroh Endpoint and devsite-server has no iroh in its dependency tree at
all — the control plane signs capabilities and serves HTML, and never joins the
network. The endpoints worth observing are the daemon, which runs on other
people's machines, and the browser, where a secret would be public. Parked.

fly.toml validates. The image is unbuilt locally — no Docker daemon on this
machine — so the first real build of it will be Fly's remote builder.
commit
d74882ffe18d519d7547d281732f41cda86f2acc
parent
dffe6c6b379889be26aed1ed735804d514222af1
author
Dami <dami.etoile@gmail.com>
9 files changed, +369 -42
A.dockerignore22+++++
ADockerfile47+++++++++++
MREADME.md17+++-
Mcrates/devsite-server/src/db.rs47+++++++++++
Mcrates/devsite-server/src/issuer.rs59+++++++++++++-
Mcrates/devsite-server/src/main.rs20++++-
Mdeploy/README.md132+++++++++++++++++++++++--------
Afly.toml60++++++++++++++
Mscripts/build-wasm.sh7+-
A .dockerignore +22 -0
A Dockerfile +47 -0
@@ -0,0 +1,47 @@
1+# The control plane, built from source in one go.2+#3+# Both halves are built here rather than uploaded from a laptop: web/pkg/ is4+# gitignored, so a deploy that copied it in would ship whatever happened to be on5+# the machine that ran `fly deploy`. The image is the build.6+7+FROM rust:1.96-bookworm AS build8+9+# `ring` (via iroh's tls-ring feature) compiles C for wasm32, so the wasm build10+# needs a clang with the WebAssembly backend. Debian's has one; Apple's does not,11+# which is what scripts/build-wasm.sh goes looking for on a laptop.12+RUN apt-get update \13+ && apt-get install -y --no-install-recommends clang llvm \14+ && rm -rf /var/lib/apt/lists/*15+16+RUN rustup target add wasm32-unknown-unknown17+# The installer script, rather than `cargo install wasm-pack`, which builds it18+# from source and roughly doubles the length of a cold image build.19+RUN curl -sSfL https://rustwasm.github.io/wasm-pack/installer/init.sh | sh20+21+WORKDIR /src22+COPY . .23+24+# --release: the bundle is ~4x smaller than the dev build and this is the copy25+# every visitor downloads.26+RUN ./scripts/build-wasm.sh --release27+RUN cargo build --release -p devsite-server28+29+FROM debian:bookworm-slim30+31+# TLS roots, for verifying Shoo's JWKS endpoint.32+RUN apt-get update \33+ && apt-get install -y --no-install-recommends ca-certificates \34+ && rm -rf /var/lib/apt/lists/*35+36+COPY --from=build /src/target/release/devsite-server /usr/local/bin/devsite-server37+COPY --from=build /src/web /app/web38+39+# Defaults for running in a container. Everything that differs per deployment —40+# the public origin above all — is set in fly.toml or as a secret.41+ENV DEVSITE_BIND=0.0.0.0:8080 \42+ DEVSITE_WEB_ROOT=/app/web \43+ DEVSITE_DB=/data/devsite.db \44+ DEVSITE_STATE_DIR=/data/state45+46+EXPOSE 808047+CMD ["devsite-server"]
M README.md +11 -6
@@ -96,20 +96,25 @@ applies to, the whitelist, and the grammars are in
9696 | `DEVSITE_PUBLIC_ORIGIN` | *(required)* | The exact origin browsers load. **Changing it orphans every account** — Shoo derives both `client_id` and the pairwise subject from it. |9797 | `DEVSITE_BIND` | `127.0.0.1:4000` | |9898 | `DEVSITE_DB` | `devsite.db` | |99-| `DEVSITE_STATE_DIR` | `.devsite-state` | Holds the capability signing key. |99+| `DEVSITE_STATE_DIR` | `.devsite-state` | Holds the capability signing key, when it is not supplied directly. |100+| `DEVSITE_SIGNING_KEY` | *(unset)* | The capability signing key as 64 hex characters. Takes precedence over the state directory; a bad value is fatal rather than a reason to generate a new key. For hosts where a secret store beats a file. |100101 | `DEVSITE_HOME` | platform config dir | Daemon identity and config. Set it to run several daemons on one machine. |101102 102103 ## Deploying103104 104-`deploy/` publishes the control plane at `https://dev.site` through Cloudflare Tunnel — so105-it has no inbound port either. See `deploy/README.md`.105+The control plane runs on Fly: one machine, one volume, SQLite on it. `fly.toml` and106+`Dockerfile` are the deployment; `fly deploy` builds both the wasm bundle and the binary107+inside the image, so nothing is uploaded from a laptop.106108 107109 ```bash108-cloudflared tunnel login # once, opens a browser109-./deploy/setup-tunnel.sh # creates the tunnel + DNS110-./deploy/run.sh # control plane + tunnel110+fly deploy111111 ```112112 113+Two settings cannot be undone — the public origin, and the capability signing key, which is114+a Fly secret rather than a file so that losing a volume does not lose every daemon's trust.115+First-time setup, DNS, certificates and backups are in116+[`deploy/README.md`](deploy/README.md).117+113118 ## Layout114119 115120 | Path | What it is |
M crates/devsite-server/src/db.rs +47 -0
@@ -138,9 +138,34 @@ impl ResourceKind {
138138 }139139 }140140 141+/// Connection settings, applied on every open.142+///143+/// `foreign_keys` is per-connection and defaults to off, so setting it inside144+/// `SCHEMA` only ever covered the connection that created the database — every145+/// later process ran with `ON DELETE CASCADE` silently doing nothing. It belongs146+/// here, where it runs each time.147+///148+/// The rest is about being killed rather than asked to stop, which is the normal149+/// way a process ends on a host that redeploys by replacing the machine. WAL150+/// keeps readers off the writer's back and survives an abrupt exit by replaying151+/// the log; `synchronous = NORMAL` is the documented companion to it, trading a152+/// fsync per commit for one per checkpoint. A crash can then lose the last153+/// transactions, which for this data means a profile edit someone can redo — not154+/// a corrupt database.155+fn configure(conn: &Connection) -> Result<()> {156+ conn.execute_batch(157+ "PRAGMA foreign_keys = ON;158+ PRAGMA journal_mode = WAL;159+ PRAGMA synchronous = NORMAL;160+ PRAGMA busy_timeout = 5000;",161+ )162+ .context("configuring the database connection")163+}164+141165 impl Db {142166 pub fn open(path: &str) -> Result<Self> {143167 let conn = Connection::open(path).with_context(|| format!("opening database {path}"))?;168+ configure(&conn)?;144169 migrate(&conn)?;145170 Ok(Self { conn })146171 }
@@ -148,6 +173,7 @@ impl Db {
148173 #[cfg(test)]149174 pub fn open_in_memory() -> Result<Self> {150175 let conn = Connection::open_in_memory()?;176+ configure(&conn)?;151177 migrate(&conn)?;152178 Ok(Self { conn })153179 }
@@ -608,6 +634,27 @@ mod tests {
608634 assert_ne!(a, b, "names are scoped to their owner");609635 }610636 637+ #[test]638+ fn foreign_keys_are_enforced_on_every_connection() {639+ // Not a hypothetical: the pragma lived in SCHEMA, which migrations skip640+ // for an existing database, so every process after the first ran without641+ // it. A share pointing at a deleted resource would have been left behind.642+ let (db, dami, frank) = seeded();643+ let agent = db644+ .create_resource(dami.id, "Agent", ResourceKind::Service, Visibility::Shared, None, 0)645+ .unwrap();646+ db.share_with(agent, frank.id).unwrap();647+648+ db.conn649+ .execute("DELETE FROM resources WHERE id = ?1", params![agent.to_string()])650+ .unwrap();651+652+ assert!(653+ db.resources_shared_with(frank.id).unwrap().is_empty(),654+ "the share should have been cascaded away with its resource"655+ );656+ }657+611658 #[test]612659 fn a_theme_round_trips_and_can_be_cleared() {613660 let (db, dami, _) = seeded();
M crates/devsite-server/src/issuer.rs +58 -1
@@ -18,7 +18,33 @@ pub struct Issuer {
1818 }1919 2020 impl Issuer {21- /// Load the signing key, generating it on first run.21+ /// Load a signing key supplied directly, as 64 hex characters.22+ ///23+ /// For hosts where the filesystem is the wrong place to keep it. On Fly the24+ /// key is a secret: encrypted at rest, injected into the machine's25+ /// environment, and — the part that matters — not tied to the survival of one26+ /// volume on one host. Losing this key is the single unrecoverable failure in27+ /// the system, because every daemon pinned its public half at `devsite login`28+ /// and refuses capabilities signed by anything else.29+ ///30+ /// An unusable value is an error, never a fallback to generating a fresh key:31+ /// that would break every daemon in the field while looking perfectly healthy32+ /// from here.33+ pub fn from_hex(hex: &str, issuer_name: &str) -> Result<Self> {34+ let raw = data_encoding::HEXLOWER_PERMISSIVE35+ .decode(hex.trim().as_bytes())36+ .context("the signing key is not hex")?;37+ let bytes: [u8; 32] = raw38+ .as_slice()39+ .try_into()40+ .map_err(|_| anyhow::anyhow!("the signing key is not 32 bytes"))?;41+ Ok(Self {42+ issuer_name: issuer_name.to_string(),43+ key: SigningKey::from_bytes(&bytes),44+ })45+ }46+47+ /// Load the signing key from the state directory, generating it on first run.2248 pub fn load_or_create(state_dir: &Path, issuer_name: &str) -> Result<Self> {2349 let path = state_dir.join("capability_signing.key");2450 let key = if path.exists() {
@@ -131,6 +157,37 @@ mod tests {
131157 std::fs::remove_dir_all(&dir).ok();132158 }133159 160+ #[test]161+ fn a_supplied_key_is_used_verbatim() {162+ // The same 32 bytes must always yield the same public half, whether they163+ // arrived from a file or from a secret: a deployment that moves hosts164+ // keeps its identity only if this holds.165+ let hex = data_encoding::HEXLOWER.encode(&[9u8; 32]);166+ let from_secret = Issuer::from_hex(&hex, "https://dev.site").unwrap();167+ let expected = SigningKey::from_bytes(&[9; 32]).verifying_key();168+ assert_eq!(169+ from_secret.public_key_hex(),170+ data_encoding::HEXLOWER.encode(expected.as_bytes())171+ );172+ // Uppercase and stray whitespace are what a shell pipeline produces.173+ assert_eq!(174+ Issuer::from_hex(&format!(" {}\n", hex.to_uppercase()), "https://dev.site")175+ .unwrap()176+ .public_key_hex(),177+ from_secret.public_key_hex()178+ );179+ }180+181+ #[test]182+ fn an_unusable_supplied_key_is_refused_rather_than_replaced() {183+ for bad in ["", "not-hex", &"ab".repeat(31), &"ab".repeat(33)] {184+ assert!(185+ Issuer::from_hex(bad, "https://dev.site").is_err(),186+ "{bad:?} should be refused"187+ );188+ }189+ }190+134191 #[test]135192 fn the_key_survives_a_restart() {136193 let dir = std::env::temp_dir().join(format!("devsite-issuer-p-{}", std::process::id()));
M crates/devsite-server/src/main.rs +19 -1
M deploy/README.md +99 -33
@@ -1,57 +1,123 @@
11 # Deploying dev.site22 3-The control plane runs as an ordinary process; Cloudflare Tunnel publishes it at4-`https://dev.site` with a real certificate and **no inbound port** — the same property the5-product itself is built on.3+The control plane runs on [Fly](https://fly.io): one machine, one volume, SQLite on it.4+`fly.toml` and `Dockerfile` in the repository root are the whole deployment.5+6+One machine is deliberate. A Fly volume attaches to a single machine in a single region,7+and nothing here wants a second writer. Since the heartbeat went away the database is8+read-mostly — a few hundred bytes per account, and a handful of writes per user per9+lifetime — so this shape has a lot of room in it.10+11+## The two things that cannot be undone12+13+**`DEVSITE_PUBLIC_ORIGIN`.** Shoo derives `client_id` and every user's pairwise subject14+from it. Changing it after anyone has signed in does not migrate accounts, it orphans them.15+It lives in `fly.toml` and must match the hostname browsers actually load.16+17+**The capability signing key.** Every daemon pinned its public half at `devsite login` and18+refuses capabilities signed by anything else. Lose it and every daemon in the world stops19+working, with no way to recover but re-running `devsite login` everywhere.20+21+So it is a Fly secret rather than a file on the volume: encrypted at rest, injected into22+the machine's environment, and not tied to the survival of one volume on one host. The23+server prefers `DEVSITE_SIGNING_KEY` when it is set and falls back to `DEVSITE_STATE_DIR`24+when it is not, which is what keeps local development unchanged.625 726 ## One-time setup827 9-Both of these open a browser and need you at the keyboard.28+```bash29+fly apps create devsite30+fly volumes create devsite_data --region sjc --size 131+32+# 32 random bytes as 64 hex characters. Generate it here, keep a copy somewhere33+# you would still have after losing this laptop, and never commit it.34+fly secrets set DEVSITE_SIGNING_KEY="$(openssl rand -hex 32)"35+```36+37+To carry an existing key over instead of generating one:1038 1139 ```bash12-# 1. Authorize cloudflared against the Cloudflare account holding the dev.site zone.13-cloudflared tunnel login40+fly secrets set DEVSITE_SIGNING_KEY="$(xxd -p -c 64 data/state/capability_signing.key)"41+```1442 15-# 2. Create the tunnel and point the hostname at it.16-./deploy/setup-tunnel.sh43+Check it against what daemons pinned before trusting it — `fly logs` prints the public half44+at boot, and `devsite status` prints what the daemon expects.45+46+### The hostname47+48+```bash49+fly ips allocate-v4 --shared50+fly ips allocate-v651+fly ips list # the addresses to point DNS at1752 ```1853 19-`setup-tunnel.sh` creates a tunnel named `devsite`, writes `deploy/cloudflared.yml`, and20-adds the DNS records for `dev.site` and `www.dev.site`.54+At Cloudflare, on the `dev.site` zone, add `A` and `AAAA` records for the apex pointing at55+those addresses, **DNS only — grey cloud, not proxied**. Fly needs to answer the ACME56+challenge on the hostname itself; proxying breaks issuance and puts a second TLS terminator57+in front of a site whose whole argument is about who can see what.2158 22-## Running59+```bash60+fly certs add dev.site61+fly certs show dev.site # until it reports Ready62+```63+64+## Deploying2365 2466 ```bash25-./deploy/run.sh67+fly deploy2668 ```2769 28-That starts the control plane on `127.0.0.1:4000` and the tunnel in front of it.70+Fly builds the image remotely — there is no need for a local Docker daemon. The build runs71+`scripts/build-wasm.sh --release` and `cargo build --release -p devsite-server` inside the72+image, so `web/pkg/` is never uploaded from a laptop and the bundle always matches the73+source that produced the binary.2974 30-## The origin is the one irreversible choice75+A deploy replaces the machine, which means a few seconds where the site is down. That is76+the cost of one volume, and it is the right trade at this size.3177 32-`DEVSITE_PUBLIC_ORIGIN=https://dev.site` is baked into account identity: Shoo derives both33-`client_id` and each user's pairwise subject from it. Changing it later does not migrate34-accounts — it orphans them. It is set in `deploy/env` and should not change once anyone has35-signed in.78+## Backups3679 37-## What Cloudflare does and does not see80+Fly snapshots volumes daily, five days' retention. That covers the machine dying; it does81+not cover "I want the database on my laptop", which is worth having:3882 39-Cloudflare terminates TLS for the control plane, so it sees profile metadata and API calls.40-It never sees private service traffic: the browser's Iroh connection goes to an n0 relay41-directly, and that traffic is end-to-end encrypted between the browser and the daemon.83+```bash84+fly ssh console -C "sqlite3 /data/devsite.db '.backup /data/backup.db'"85+fly sftp get /data/backup.db ./devsite-backup.db86+```4287 43-## Caching88+The signing key is not on the volume and is not in these backups. Keep it separately — it89+is the part that cannot be regenerated.4490 45-`scripts/build-wasm.sh` publishes the wasm bundle under a content hash46-(`/pkg/<hash>/…`), so it is served `immutable` and cached at the edge indefinitely. The47-manifest naming the current hash, the HTML, and every `/api/` response are `no-cache` or48-`no-store`. Redeploying therefore invalidates nothing and never serves a stale bundle.91+## Configuration92+93+Set in `fly.toml`, except the secret.94+95+| Variable | Where | Notes |96+| --- | --- | --- |97+| `DEVSITE_PUBLIC_ORIGIN` | `fly.toml` | `https://dev.site`. See above. |98+| `DEVSITE_SIGNING_KEY` | `fly secrets` | Hex. Never in the repository. |99+| `DEVSITE_BIND` | `Dockerfile` | `0.0.0.0:8080`, matching `internal_port`. |100+| `DEVSITE_DB` | `Dockerfile` | `/data/devsite.db`, on the volume. |101+| `DEVSITE_STATE_DIR` | `Dockerfile` | Unused while the secret is set. |102+| `DEVSITE_WEB_ROOT` | `Dockerfile` | `/app/web`, baked into the image. |103+104+## What Fly sees105+106+Fly terminates TLS for the control plane, so it sees profile metadata and API calls — the107+same position Cloudflare was in before. It never sees private service traffic: that goes108+browser-to-daemon over an iroh relay, end-to-end encrypted, and the control plane is not109+even told where the daemon is.110+111+## Caching49112 50-## Moving off the tunnel later113+`scripts/build-wasm.sh` publishes the wasm bundle under a content hash (`/pkg/<hash>/…`),114+served `immutable`. The manifest naming the current hash, the HTML, and every `/api/`115+response are `no-cache` or `no-store`. Redeploying therefore invalidates nothing and never116+serves a stale bundle.51117 52-Nothing above is load-bearing except the hostname. To move to fly.io or a VPS, run the same53-binary there with the same `DEVSITE_PUBLIC_ORIGIN`, copy `devsite.db` and54-`.devsite-state/capability_signing.key`, and repoint DNS.118+## The previous setup55119 56-**Copy the signing key, do not regenerate it.** Every daemon pinned its public half at57-`devsite login`; a new key makes every existing daemon reject every capability.120+`setup-tunnel.sh`, `run.sh` and `env` publish the control plane from a laptop through121+Cloudflare Tunnel. They still work, and are what `https://dev.site` pointed at before Fly.122+Delete them once the cutover has settled — running both against the same hostname is a way123+to be confused about which one is live.
A fly.toml +60 -0
@@ -0,0 +1,60 @@
1+# The dev.site control plane on Fly.2+#3+# One machine, one volume, SQLite on it. That is a deliberate ceiling, not an4+# oversight: a Fly volume is attached to a single machine in a single region, and5+# nothing here wants more than one writer. The data is tiny — a few hundred bytes6+# per account — and since the heartbeat went away the write path is a handful of7+# rows per user per lifetime, so the constraint that matters is reads, which a8+# shared CPU serves comfortably.9+#10+# What is *not* here, and must not be: DEVSITE_SIGNING_KEY. It is a secret.11+12+app = "devsite"13+primary_region = "sjc"14+15+[build]16+ dockerfile = "Dockerfile"17+18+[env]19+ # The one irreversible setting. Shoo derives client_id and every user's20+ # pairwise subject from this origin, so changing it after anyone has signed in21+ # orphans their account rather than moving it. It must match the hostname22+ # browsers actually load, certificate and all.23+ DEVSITE_PUBLIC_ORIGIN = "https://dev.site"24+ RUST_LOG = "devsite_server=info,tower_http=warn,warn"25+26+[[mounts]]27+ source = "devsite_data"28+ destination = "/data"29+ # Only the database lives here. The signing key comes from a secret, so losing30+ # this volume costs history, not identity — every daemon in the field keeps31+ # working against a restored or even an empty one.32+ initial_size = "1gb"33+34+[http_service]35+ internal_port = 808036+ force_https = true37+ # Never stopped. A suspended machine would serve the first visitor after an38+ # idle period a cold start, and the control plane is on the critical path of39+ # opening any service.40+ auto_stop_machines = "off"41+ auto_start_machines = true42+ min_machines_running = 143+44+ [http_service.concurrency]45+ type = "requests"46+ hard_limit = 25047+ soft_limit = 20048+49+# Cheap, unauthenticated, and touches no state: it reports the origin the process50+# actually booted with, which is the setting most worth noticing is wrong.51+[[http_service.checks]]52+ interval = "30s"53+ timeout = "5s"54+ grace_period = "10s"55+ method = "GET"56+ path = "/api/config"57+58+[[vm]]59+ size = "shared-cpu-1x"60+ memory = "512mb"
M scripts/build-wasm.sh +6 -1