- Separate blob, Git relay, records, room UI and route responsibilities. - Share event projections, privacy policy, signing and typed room reads. - Move schema access behind domain services and compose workers at startup. - Recover optional callback and view planning with bounded, idempotent jobs. - Recheck source and registration identity without resetting completed work. - Verify browser bundles against locked sources and document architecture.
- commit
- 681b50f87af0a59de9583676ed7bedece43b07f0
- parent
- 7d1f192de8284d61782dff10b3bfb5600227faff
- author
- Dami <dami.etoile@gmail.com>
@@ -8,17 +8,17 @@ COPY . .
88 99 ARG TEST_PACKAGES=./...1010 FROM build AS test11-RUN go test -mod=mod -race ${TEST_PACKAGES}11+RUN go test -mod=readonly -race ${TEST_PACKAGES}1212 1313 ARG BENCH_PACKAGES=./internal/storage ./internal/relay1414 ARG BENCHTIME=5s1515 FROM build AS bench1616 ARG BENCH_PACKAGES1717 ARG BENCHTIME18-RUN go test -mod=mod -run '^$' -bench . -benchmem -benchtime=${BENCHTIME} ${BENCH_PACKAGES}18+RUN go test -mod=readonly -run '^$' -bench . -benchmem -benchtime=${BENCHTIME} ${BENCH_PACKAGES}1919 2020 FROM build AS binary21-RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/tiny ./cmd/tiny21+RUN CGO_ENABLED=0 go build -mod=readonly -trimpath -ldflags="-s -w" -o /out/tiny ./cmd/tiny2222 2323 FROM debian:bookworm-slim AS runtime2424 RUN apt-get update \@@ -28,7 +28,7 @@ RUN apt-get update \
2828 && mkdir -p /data \2929 && chown relay:relay /data3030 COPY --from=binary /out/tiny /usr/local/bin/tiny31-COPY LICENSE THIRD_PARTY_NOTICES internal/webui/signer.js.license /usr/share/doc/tiny/31+COPY LICENSE THIRD_PARTY_NOTICES internal/webui/signer.js.license internal/webui/nostr-name.js.license /usr/share/doc/tiny/3232 USER relay3333 WORKDIR /home/relay3434 VOLUME ["/data"]@@ -1,18 +1,28 @@
1-1+22 33 TEST_PACKAGES ?= ./...44 55 test:6- go test -mod=mod ./...6+ go test -mod=readonly ./...77 88 test-race:9- go test -mod=mod -race ./...9+ go test -mod=readonly -race ./...1010 1111 test-internal-race:12- go test -mod=mod -race $(TEST_PACKAGES)12+ go test -mod=readonly -race $(TEST_PACKAGES)1313 1414 benchmark:15- go test -mod=mod ./internal/storage ./internal/relay -run '^$$' -bench . -benchmem -benchtime=$${BENCHTIME:-1s}15+ go test -mod=readonly ./internal/storage ./internal/relay -run '^$$' -bench . -benchmem -benchtime=$${BENCHTIME:-1s}16+17+verify:18+ go test -mod=readonly ./...19+ go vet -mod=readonly ./...20+ npm run verify:bundles21+ npm run test:js22+23+web-test:24+ npm run verify:bundles25+ npm run test:js1626 1727 docker-test:1828 docker build --target test --build-arg TEST_PACKAGES="$(TEST_PACKAGES)" -t tinyrelay-test:local .@@ -79,6 +79,8 @@ See [Wiki](docs/wiki.md) for pages, versions and forks, merge requests and redir
7979 8080 See [Custom views](docs/views.md) for rendering fenced code blocks through a transform you run and keeping the result as signed artifacts.8181 82+See [Architecture](docs/architecture.md) for service responsibilities, event acceptance and dependency checks.83+8284 **Manage > Health** shows the relay and script versions and whether the browser's WebMCP tools are registered. See [WebMCP tools](docs/webmcp.md) for browser-agent integration and authorization behavior.8385 8486 Agents outside the browser connect to `/mcp`, a stateless Model Context Protocol endpoint authenticated with NIP-98, and read `/llms.txt` for a summary of the relay's machine surface. See [MCP](docs/mcp.md). An agent acts under a grant the owner signs, and the owner reviews what it proposes:@@ -38,6 +38,17 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
3838 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN3939 THE SOFTWARE.4040 41+nostr-name browser bundle42+-------------------------43+The generated internal/webui/nostr-name.js bundle is built from44+nostr-web-components 0.2.2, @jsr/nostr__gadgets 0.1.9 and45+@jsr/nostr__tools 2.23.0, as resolved by package-lock.json. The installed46+nostr-web-components package identifies its license as the Unlicense. The47+bundle also retains esbuild license comments for noble-curves and scure-base;48+their installed package metadata identifies them as MIT licensed. The49+corresponding installed license text is recorded in50+internal/webui/nostr-name.js.license.51+4152 Fixi BSD-0 notice4253 -----------------4354 BSD Zero Clause License@@ -323,6 +323,8 @@ Right before each POST the relay checks that the callback still exists and is no
323323 324324 ### Retries and pauses325325 326+A valid event is accepted even when its optional callback work cannot be planned immediately. The relay makes up to three background planning attempts within 15 minutes, rechecks the source and callback, and skips work that is no longer relevant. Recovery fills missing work without restarting completed deliveries or sending earlier events to newly registered callbacks.327+326328 A response outside 2xx, a timeout or a connection failure counts as a failure. The relay tries the event again after one minute and once more after five, three attempts in all, then drops it. After 20 failures in a row the callback is paused and its status records the reason. A callback is also paused when its key is no longer a member. A successful delivery resets the failure count. Resume a paused callback with `resumecallback`.327329 328330 ### Manage callbacks@@ -0,0 +1,31 @@
1+# Architecture2+3+Each tenant has a durable store and a set of services. The daemon constructs those services, admits operations and manages their lifetimes.4+5+## Responsibilities6+7+| Area | Responsibility |8+| --- | --- |9+| `event` and `policy` | Event formats and shared policy decisions. |10+| `storage` | Event persistence, queries and transaction boundaries. |11+| `community` | Membership, moderation, rooms and their projections. |12+| `records` | Relay identity, signed records and notification records. Community data arrives through a required read interface. |13+| `blob` | File storage, upload rules and file HTTP endpoints. |14+| `gitrelay` | Repository admission, storage, synchronization, repair and Git HTTP endpoints. |15+| `replication` | Replication policy, jobs and delivery handlers. |16+| `work` | Durable queue claims, retries and worker lifetimes. The daemon combines handlers from the services. |17+| `webui` | Pages, typed room reads and browser interactions. |18+19+Keep schema changes and data access with the service that owns the data. Share a policy function or a small read contract when several services need the same rule or projection. Avoid introducing a general utility package for unrelated behavior.20+21+## Event acceptance22+23+Client publication and imported events share the persistence and projection path. Each entry point retains its own admission rules and protocol actions. Event projections and durable work records commit together.24+25+Callbacks and custom views are optional. Local planning can recover after acceptance, with bounded retries that recheck the source and captured registrations. Completed work keeps its identity and is not reset by planning recovery. Remote delivery and transforms retain their own retry limits. See [callbacks](agents.md#retries-and-pauses) and [custom views](views.md#failures).26+27+## Browser boundaries28+29+Pages render without JavaScript. The page template owns the shell; page templates own their content sections. Generic components live in `components.js`, room interactions live in `rooms.js`, and signed publication uses the shared `tiny.signing` contract. The verifier checks signed events before publication.30+31+Go module versions are declared in `go.mod`, with dependency checksums in `go.sum`. Browser dependencies are locked in `package-lock.json`. `make verify` checks the Go and JavaScript code and verifies generated browser bundles against their sources. See [Testing](testing.md) for local and Linux checks.@@ -3,13 +3,12 @@
33 The normal development loop is local:44 55 ```sh6-make test-internal-race7-make test8-npm run test:files9-npm run test:collaboration10-npm run test:approvals6+make verify7+make test-race118 ```129 10+`make verify` runs the Go tests, Go vet, JavaScript tests and a check that the bundled browser dependencies match their locked sources. Install JavaScript dependencies with `npm ci` first. `make web-test` runs the browser checks alone.11+1312 The repository also contains a multistage Docker build. `make docker-test` runs `TEST_PACKAGES` (default `./...`) under Go 1.27.1 on Debian Bookworm with the race detector. `make docker-build` builds the self-hosted runtime image.1413 1514 ## Remote Linux host@@ -67,6 +67,8 @@ Each artifact is served at `/views/<name>/<hash>` with its stored media type, un
6767 6868 ## Failures6969 70+A valid event is accepted even when its optional transform work cannot be planned immediately. The relay makes up to three background planning attempts within 15 minutes, rechecks the source and view, and skips work that is no longer relevant. Recovery fills missing work without restarting completed transforms or backfilling views registered after the event.71+7072 A response outside 2xx, a timeout or a connection failure counts as a failure. The relay tries the event again after one minute and once more after five, three attempts in all, then drops it. After 20 failures in a row the view is paused and its status records the reason. A successful run resets the failure count.7173 7274 ## Manage views@@ -1,1615 +0,0 @@
1-// Package blob implements the Blossom and NIP-96 file doors for a local relay.2-package blob3-4-import (5- "context"6- "crypto/sha256"7- "database/sql"8- "encoding/hex"9- "encoding/json"10- "errors"11- "fmt"12- "io"13- "net"14- "net/http"15- "net/url"16- "os"17- "path/filepath"18- "regexp"19- "sort"20- "strconv"21- "strings"22- "sync"23- "time"24-25- "github.com/FelineStateMachine/tinyrelay/internal/event"26- "github.com/FelineStateMachine/tinyrelay/internal/storage"27-)28-29-type Action string30-31-const (32- ActionUpload Action = "upload"33- ActionGet Action = "get"34- ActionDelete Action = "delete"35- ActionList Action = "list"36- ActionMirror Action = "mirror"37- ActionReport Action = "report"38-)39-40-type Authorize func(*http.Request, Action) (string, error)41-type CanRead func(context.Context, string, []string) bool42-type ResolveIP func(context.Context, string) ([]net.IP, error)43-type ReportRecorder func(context.Context, *sql.Tx, string, string, string, string, string) error44-45-type Config struct {46- Root string47- PublicURL string48- Store *storage.Store49- Authorize Authorize50- CanRead CanRead51- HTTPClient *http.Client52- ResolveIP ResolveIP53- IsOwner func(string) bool54- ValidateUpload func(*http.Request, string) error55- // ResolveServers returns the ordered BUD-03 servers for an authorized56- // caller. It is used only for Blossom URI mirror fallback.57- ResolveServers func(context.Context, string) ([]string, error)58- // RecordReport joins native blob reporting to the host moderation ledger.59- // It runs inside the blob report transaction.60- RecordReport ReportRecorder61- // Limits is read for each upload so policy changes apply without reopening62- // the blob service. Zero values mean unlimited.63- Limits func() Limits64- // UploadTerms returns what an uploader's standing imposes on its uploads:65- // an agent grant with a sites ttl gives its claims an expiry, and one with66- // the encrypted flag refuses plain uploads. Zero values mean no terms.67- UploadTerms func(context.Context, string) (UploadTerms, error)68-}69-70-type Limits struct {71- MaxFileBytes int6472- UserStorageBytes int6473-}74-75-// UploadTerms is what an uploader's grant imposes on the blobs it stores.76-// ExpiresAt is the unix time its claims lapse, 0 for never; Encrypted77-// requires the stored bytes to be an encrypted blob or manifest.78-type UploadTerms struct {79- ExpiresAt int6480- Encrypted bool81-}82-83-// encryptedTypes are the content types the browser uploader gives encrypted84-// objects: AES-GCM ciphertext for files and chunks, and the BUD-16 directory85-// type for encrypted manifests. Anything the relay recognizes as a page,86-// image, video, audio or document is plain.87-var encryptedTypes = map[string]bool{"application/octet-stream": true, "application/vnd.blossom.directory+msgpack": true}88-89-type Blob struct {90- SHA256 string `json:"sha256"`91- Size int64 `json:"size"`92- Type string `json:"type"`93- Uploader string `json:"uploader"`94- Uploaded int64 `json:"uploaded"`95-}96-97-type PutOptions struct {98- Reader io.Reader99- Type string100- Uploader string101- Hash string102- // Commit records ownership metadata in the blob transaction. The final103- // argument reports whether the blob is new. An error rolls back all claims.104- // New bytes remain unavailable until their ownership metadata is durable.105- Commit func(context.Context, *sql.Tx, Blob, bool) error106-}107-108-type Service struct {109- config Config110- root string111- store *storage.Store112- resolve ResolveIP113- quotaMu sync.Mutex114- // multipartMu protects partial files and their range metadata. Lock it115- // before quotaMu; network bodies are never read while either is held.116- multipartMu sync.Mutex117- activeParts map[string]int118- // multipartFinalizing prevents another chunk from changing a complete119- // session while its assembled file is being hashed. Hashing is deliberately120- // outside multipartMu because the upload may be as large as 1 GiB.121- multipartFinalizing map[string]bool122- // verifyMultipartHook is used by package tests to pause finalization123- // without allocating a GiB fixture. It is nil in production.124- verifyMultipartHook func(context.Context, string, string, int64) error125-}126-127-var ErrBlocked = errors.New("blocked: this blob was removed by a moderator")128-var ErrHashMismatch = errors.New("conflict: mirrored blob hash does not match the requested hash")129-var ErrFileTooLarge = errors.New("invalid: blob exceeds the per-file size limit")130-var ErrQuotaExceeded = errors.New("restricted: uploader storage quota exceeded")131-var ErrMultipartConflict = errors.New("conflict: upload metadata differs from existing session")132-var ErrNotEncrypted = errors.New("restricted: agent grant requires encrypted uploads")133-134-var shaPattern = regexp.MustCompile(`^[0-9a-f]{64}$`)135-var blobPathPattern = regexp.MustCompile(`^/([0-9a-f]{64})(?:\.[a-z0-9]{1,8})?$`)136-137-// HandlesPath keeps the daemon mount in sync with every supported file door.138-func HandlesPath(path string) bool {139- return path == "/upload" || path == "/mirror" || path == "/report" ||140- path == "/nip96" || path == "/.well-known/nostr/nip96.json" ||141- strings.HasPrefix(path, "/nip96/") || strings.HasPrefix(path, "/list/") ||142- blobPathPattern.MatchString(path)143-}144-145-var extensions = map[string]string{146- "image/png": "png", "image/jpeg": "jpg", "image/gif": "gif", "image/webp": "webp", "image/avif": "avif", "image/svg+xml": "svg",147- "video/mp4": "mp4", "video/webm": "webm", "video/quicktime": "mov", "audio/mpeg": "mp3", "audio/ogg": "ogg", "audio/wav": "wav", "audio/mp4": "m4a",148- "application/pdf": "pdf", "application/json": "json", "text/plain": "txt", "text/markdown": "md",149- "application/vnd.blossom.directory+msgpack": "bdir",150-}151-152-func New(ctx context.Context, config Config) (*Service, error) {153- if config.Root == "" || config.Store == nil {154- return nil, errors.New("blob root and store are required")155- }156- if err := os.MkdirAll(filepath.Join(config.Root, "blobs"), 0o700); err != nil {157- return nil, fmt.Errorf("create blob root: %w", err)158- }159- if _, err := config.Store.DB().ExecContext(ctx, `CREATE TABLE IF NOT EXISTS blobs (160- sha256 TEXT PRIMARY KEY, size INTEGER NOT NULL, type TEXT NOT NULL,161- uploader TEXT NOT NULL, uploaded INTEGER NOT NULL162- ); CREATE TABLE IF NOT EXISTS blob_claims (163- sha256 TEXT NOT NULL, uploader TEXT NOT NULL, claimed_at INTEGER NOT NULL,164- PRIMARY KEY (sha256, uploader), FOREIGN KEY (sha256) REFERENCES blobs(sha256) ON DELETE CASCADE165- ); CREATE INDEX IF NOT EXISTS blobs_uploader ON blobs(uploader, uploaded DESC);166- CREATE INDEX IF NOT EXISTS blob_claims_uploader ON blob_claims(uploader, claimed_at DESC, sha256 DESC);167- CREATE TABLE IF NOT EXISTS blob_reports (168- id TEXT PRIMARY KEY, reporter TEXT NOT NULL, target_pubkey TEXT NOT NULL DEFAULT '',169- target_blob TEXT NOT NULL, type TEXT NOT NULL, content TEXT NOT NULL, at INTEGER NOT NULL,170- status TEXT NOT NULL DEFAULT 'open', resolved_by TEXT NOT NULL DEFAULT '',171- resolved_at INTEGER NOT NULL DEFAULT 0, action TEXT NOT NULL DEFAULT '',172- UNIQUE(reporter,target_blob));173- CREATE TRIGGER IF NOT EXISTS blob_claim_on_insert AFTER INSERT ON blobs174- WHEN NEW.uploader != '' BEGIN INSERT OR IGNORE INTO blob_claims(sha256,uploader,claimed_at) VALUES(NEW.sha256,NEW.uploader,NEW.uploaded); END;175- CREATE TABLE IF NOT EXISTS blob_blocks (sha256 TEXT PRIMARY KEY, reason TEXT NOT NULL DEFAULT '', blocked_at INTEGER NOT NULL DEFAULT 0);176- CREATE TABLE IF NOT EXISTS blob_tombstones (sha256 TEXT PRIMARY KEY, deleted_at INTEGER NOT NULL);177- CREATE TABLE IF NOT EXISTS multipart_uploads (id TEXT PRIMARY KEY, sha256 TEXT NOT NULL, uploader TEXT NOT NULL, length INTEGER NOT NULL, type TEXT NOT NULL, created INTEGER NOT NULL, last_seen INTEGER NOT NULL);178- CREATE TABLE IF NOT EXISTS multipart_parts (upload_id TEXT NOT NULL, offset INTEGER NOT NULL, length INTEGER NOT NULL, PRIMARY KEY(upload_id, offset), FOREIGN KEY(upload_id) REFERENCES multipart_uploads(id) ON DELETE CASCADE);179- INSERT OR IGNORE INTO blob_claims(sha256,uploader,claimed_at)180- SELECT sha256,uploader,uploaded FROM blobs WHERE uploader != ''`); err != nil {181- return nil, fmt.Errorf("initialize blob metadata: %w", err)182- }183- // Claims made under an agent grant with a ttl carry an expiry; every184- // earlier claim keeps 0, which never lapses.185- if _, err := config.Store.DB().ExecContext(ctx, "ALTER TABLE blob_claims ADD COLUMN expires INTEGER NOT NULL DEFAULT 0"); err != nil && !strings.Contains(err.Error(), "duplicate column") {186- return nil, fmt.Errorf("add blob claim expiry: %w", err)187- }188- if _, err := config.Store.DB().ExecContext(ctx, "CREATE INDEX IF NOT EXISTS blob_claims_expires ON blob_claims(expires) WHERE expires>0"); err != nil {189- return nil, fmt.Errorf("index blob claim expiry: %w", err)190- }191- if err := reconcile(ctx, config.Store, filepath.Join(config.Root, "blobs")); err != nil {192- return nil, err193- }194- resolve := config.ResolveIP195- if resolve == nil {196- resolve = func(ctx context.Context, host string) ([]net.IP, error) {197- return net.DefaultResolver.LookupIP(ctx, "ip", host)198- }199- }200- s := &Service{config: config, root: filepath.Join(config.Root, "blobs"), store: config.Store, resolve: resolve, activeParts: make(map[string]int), multipartFinalizing: make(map[string]bool)}201- if err := s.reconcileMultipart(ctx); err != nil {202- return nil, err203- }204- return s, nil205-}206-207-func (s *Service) Handler() http.HandlerFunc { return s.serveHTTP }208-209-// Put stores bytes using their SHA-256 digest as the durable filename.210-func (s *Service) Put(ctx context.Context, options PutOptions) (Blob, error) {211- if options.Reader == nil {212- return Blob{}, errors.New("blob reader is required")213- }214- entry, _, err := s.putOptions(ctx, options)215- return entry, err216-}217-218-func (s *Service) putOptions(ctx context.Context, options PutOptions) (Blob, bool, error) {219- return s.putValidatedWithCommit(ctx, options.Reader, options.Type, options.Uploader, options.Hash, nil, options.Commit)220-}221-222-// Get opens a verified metadata-backed blob for streaming to another service.223-func (s *Service) Get(ctx context.Context, sha string) (Blob, io.ReadCloser, error) {224- if !shaPattern.MatchString(sha) {225- return Blob{}, nil, errors.New("invalid: bad blob sha256")226- }227- entry, err := s.lookup(ctx, sha)228- if err != nil {229- return Blob{}, nil, err230- }231- if s.blocked(ctx, sha) {232- return Blob{}, nil, ErrBlocked233- }234- file, err := os.Open(filepath.Join(s.root, sha))235- if err != nil {236- return Blob{}, nil, err237- }238- return entry, file, nil239-}240-241-// ListMetadata returns all blobs uploaded by the supplied pubkey.242-func (s *Service) ListMetadata(ctx context.Context, uploader string) ([]Blob, error) {243- rows, err := s.store.DB().QueryContext(ctx, `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded244- FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?245- ORDER BY b.uploaded DESC,b.sha256 DESC`, uploader)246- if err != nil {247- return nil, fmt.Errorf("list blob metadata: %w", err)248- }249- defer rows.Close()250- entries := make([]Blob, 0)251- for rows.Next() {252- var entry Blob253- if err := rows.Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded); err != nil {254- return nil, fmt.Errorf("scan blob metadata: %w", err)255- }256- entries = append(entries, entry)257- }258- return entries, rows.Err()259-}260-261-// ListPage returns a BUD-12 cursor page. The cursor is the last blob hash262-// returned by the previous page and is intentionally opaque to callers.263-func (s *Service) ListPage(ctx context.Context, uploader string, limit int, cursor string) ([]Blob, string, error) {264- if limit < 1 {265- limit = 20266- }267- if limit > 100 {268- limit = 100269- }270- query := `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded FROM blobs b271- JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?`272- args := []any{uploader}273- if cursor != "" {274- var uploaded int64275- if err := s.store.DB().QueryRowContext(ctx, `SELECT b.uploaded FROM blobs b276- JOIN blob_claims c ON c.sha256=b.sha256 WHERE b.sha256=? AND c.uploader=?`, cursor, uploader).Scan(&uploaded); err != nil {277- return nil, "", fmt.Errorf("resolve blob cursor: %w", err)278- }279- query += " AND (b.uploaded < ? OR (b.uploaded = ? AND b.sha256 < ?))"280- args = append(args, uploaded, uploaded, cursor)281- }282- query += " ORDER BY b.uploaded DESC,b.sha256 DESC LIMIT ?"283- args = append(args, limit+1)284- rows, err := s.store.DB().QueryContext(ctx, query, args...)285- if err != nil {286- return nil, "", fmt.Errorf("list blob page: %w", err)287- }288- defer rows.Close()289- entries := make([]Blob, 0, limit)290- for rows.Next() {291- var entry Blob292- if err := rows.Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded); err != nil {293- return nil, "", fmt.Errorf("scan blob page: %w", err)294- }295- entries = append(entries, entry)296- }297- if err := rows.Err(); err != nil {298- return nil, "", fmt.Errorf("read blob page: %w", err)299- }300- next := ""301- if len(entries) > limit {302- next = entries[limit-1].SHA256303- entries = entries[:limit]304- }305- return entries, next, nil306-}307-308-// Delete removes a blob's metadata and content. Authorization is the caller's responsibility.309-func (s *Service) Delete(ctx context.Context, sha string) error {310- if !shaPattern.MatchString(sha) {311- return errors.New("invalid: bad blob sha256")312- }313- if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {314- if _, err := tx.ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {315- return fmt.Errorf("record blob deletion: %w", err)316- }317- if _, err := tx.ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {318- return fmt.Errorf("delete blob metadata: %w", err)319- }320- return nil321- }); err != nil {322- return err323- }324- if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {325- return fmt.Errorf("delete blob content: %w", err)326- }327- return nil328-}329-330-// DeleteForUploader removes one owner's claim. The physical object remains331-// while another owner still references it, which makes deduplicated uploads332-// safe to manage independently.333-func (s *Service) DeleteForUploader(ctx context.Context, sha, uploader string) error {334- if !shaPattern.MatchString(sha) {335- return errors.New("invalid: bad blob sha256")336- }337- removed := false338- if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {339- result, err := tx.ExecContext(ctx, "DELETE FROM blob_claims WHERE sha256=? AND uploader=?", sha, uploader)340- if err != nil {341- return fmt.Errorf("delete blob claim: %w", err)342- }343- changed, err := result.RowsAffected()344- if err != nil {345- return fmt.Errorf("check blob claim deletion: %w", err)346- }347- if changed == 0 {348- return sql.ErrNoRows349- }350- var remaining int351- if err := tx.QueryRowContext(ctx, "SELECT count(*) FROM blob_claims WHERE sha256=?", sha).Scan(&remaining); err != nil {352- return fmt.Errorf("count blob claims: %w", err)353- }354- if remaining == 0 {355- removed = true356- if _, err := tx.ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {357- return fmt.Errorf("record blob deletion: %w", err)358- }359- if _, err := tx.ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {360- return fmt.Errorf("delete blob metadata: %w", err)361- }362- }363- return nil364- }); err != nil {365- return err366- }367- if removed {368- if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {369- return fmt.Errorf("delete blob content: %w", err)370- }371- }372- return nil373-}374-375-// Descriptor returns the Blossom descriptor and NIP-94 tags for a blob.376-func (s *Service) Descriptor(baseURL string, entry Blob) map[string]any {377- return map[string]any{378- "url": baseURL + "/" + entry.SHA256 + extensionSuffix(entry.Type),379- "sha256": entry.SHA256,380- "size": entry.Size,381- "type": entry.Type,382- "uploaded": entry.Uploaded,383- "nip94": [][]string{{"url", baseURL + "/" + entry.SHA256 + extensionSuffix(entry.Type)}, {"m", entry.Type}, {"x", entry.SHA256}, {"ox", entry.SHA256}, {"size", strconv.FormatInt(entry.Size, 10)}},384- }385-}386-387-func (s *Service) serveHTTP(w http.ResponseWriter, r *http.Request) {388- w.Header().Set("access-control-allow-origin", "*")389- w.Header().Set("access-control-allow-headers", "authorization, content-type, x-sha-256, x-content-length, x-content-type, upload-type, upload-length, upload-offset")390- w.Header().Set("access-control-allow-methods", "GET, HEAD, PUT, PATCH, POST, DELETE, OPTIONS")391- w.Header().Set("Access-Control-Expose-Headers", "Allow, X-Reason")392- if r.Method == http.MethodOptions {393- if blobPathPattern.MatchString(r.URL.Path) {394- w.Header().Set("Allow", "GET, HEAD, PUT, PATCH, DELETE, OPTIONS")395- }396- w.WriteHeader(http.StatusNoContent)397- return398- }399- if r.URL.Path == "/.well-known/nostr/nip96.json" {400- s.writeJSON(w, http.StatusOK, s.nip96Info(r))401- return402- }403- if r.URL.Path == "/upload" && r.Method == http.MethodHead {404- s.uploadHead(w, r)405- return406- }407- if r.URL.Path == "/upload" && r.Method == http.MethodPut {408- s.upload(w, r)409- return410- }411- if r.URL.Path == "/mirror" && r.Method == http.MethodPut {412- s.mirror(w, r)413- return414- }415- if match := blobPathPattern.FindStringSubmatch(r.URL.Path); match != nil && (r.Method == http.MethodPut || r.Method == http.MethodPatch || r.Method == http.MethodOptions) {416- if r.Method == http.MethodPatch || r.Method == http.MethodOptions {417- s.multipart(w, r, match[1])418- return419- }420- s.pathUpload(w, r, match[1])421- return422- }423- if r.URL.Path == "/report" && r.Method == http.MethodPut {424- s.report(w, r)425- return426- }427- if strings.HasPrefix(r.URL.Path, "/list/") && r.Method == http.MethodGet {428- s.list(w, r, strings.TrimPrefix(r.URL.Path, "/list/"))429- return430- }431- if r.URL.Path == "/nip96" {432- s.nip96(w, r)433- return434- }435- s.blob(w, r)436-}437-438-func (s *Service) upload(w http.ResponseWriter, r *http.Request) {439- pubkey, err := s.authorize(r, ActionUpload)440- if err != nil {441- s.fail(w, http.StatusUnauthorized, err)442- return443- }444- claimed := strings.ToLower(strings.TrimSpace(r.Header.Get("x-sha-256")))445- if claimed != "" && !shaPattern.MatchString(claimed) {446- s.fail(w, http.StatusBadRequest, errors.New("invalid: X-SHA-256 must be lowercase hex"))447- return448- }449- blob, created, err := s.putValidated(r.Context(), r.Body, contentType(r.Header.Get("content-type")), pubkey, claimed, func(actual string) error {450- if s.config.ValidateUpload != nil {451- return s.config.ValidateUpload(r, actual)452- }453- return nil454- })455- if err != nil {456- s.fail(w, statusFor(err), err)457- return458- }459- _ = created460- s.writeJSON(w, http.StatusOK, s.descriptor(r, blob))461-}462-463-func (s *Service) uploadHead(w http.ResponseWriter, r *http.Request) {464- sha := strings.ToLower(strings.TrimSpace(r.Header.Get("x-sha-256")))465- if !shaPattern.MatchString(sha) {466- s.fail(w, http.StatusBadRequest, errors.New("invalid: X-SHA-256 must be lowercase hex"))467- return468- }469- rawLength := strings.TrimSpace(r.Header.Get("x-content-length"))470- if rawLength == "" {471- s.fail(w, http.StatusLengthRequired, errors.New("invalid: X-Content-Length is required"))472- return473- }474- length, err := strconv.ParseInt(rawLength, 10, 64)475- if err != nil || length < 0 {476- s.fail(w, http.StatusBadRequest, errors.New("invalid: X-Content-Length must be a nonnegative integer"))477- return478- }479- pubkey, err := s.authorize(r, ActionUpload)480- if err != nil {481- s.fail(w, http.StatusUnauthorized, err)482- return483- }484- typ := r.Header.Get("x-content-type")485- if typ == "" {486- typ = r.Header.Get("content-type")487- }488- if strings.TrimSpace(typ) == "" {489- s.fail(w, http.StatusBadRequest, errors.New("invalid: X-Content-Type is required"))490- return491- }492- if existing, lookupErr := s.lookup(r.Context(), sha); lookupErr == nil && existing.SHA256 != "" && s.hasClaim(r.Context(), sha, pubkey) {493- w.WriteHeader(http.StatusOK)494- return495- }496- limits := s.currentLimits()497- if limits.MaxFileBytes > 0 && length > limits.MaxFileBytes {498- s.fail(w, http.StatusRequestEntityTooLarge, ErrFileTooLarge)499- return500- }501- // HEAD is advisory for deduplicated uploads, whose final physical size is502- // already present, but it should still reflect the current quota budget.503- if limits.UserStorageBytes > 0 {504- if used, quotaErr := s.quotaUsage(r.Context(), pubkey); quotaErr != nil {505- s.fail(w, http.StatusInternalServerError, quotaErr)506- return507- } else if length > limits.UserStorageBytes-used {508- s.fail(w, http.StatusForbidden, ErrQuotaExceeded)509- return510- }511- }512- if existing, err := s.lookup(r.Context(), sha); err == nil && existing.SHA256 != "" {513- w.WriteHeader(http.StatusOK)514- return515- }516- w.WriteHeader(http.StatusOK)517-}518-519-func (s *Service) blob(w http.ResponseWriter, r *http.Request) {520- path := r.URL.Path521- if strings.HasPrefix(path, "/nip96/") {522- path = strings.TrimPrefix(path, "/nip96")523- }524- match := blobPathPattern.FindStringSubmatch(path)525- if match == nil {526- s.fail(w, http.StatusNotFound, errors.New("not found"))527- return528- }529- sha := match[1]530- if r.Method == http.MethodDelete {531- s.delete(w, r, sha)532- return533- }534- if r.Method != http.MethodGet && r.Method != http.MethodHead {535- s.fail(w, http.StatusMethodNotAllowed, errors.New("method not allowed"))536- return537- }538- if s.config.CanRead != nil {539- pubkey, err := s.authorize(r, ActionGet)540- if err != nil {541- s.fail(w, http.StatusUnauthorized, err)542- return543- }544- if !s.config.CanRead(r.Context(), sha, []string{pubkey}) {545- s.fail(w, http.StatusForbidden, errors.New("restricted: read access denied"))546- return547- }548- }549- entry, err := s.lookup(r.Context(), sha)550- if errors.Is(err, sql.ErrNoRows) {551- s.fail(w, http.StatusNotFound, errors.New("not found"))552- return553- }554- if err != nil {555- s.fail(w, http.StatusInternalServerError, err)556- return557- }558- if s.blocked(r.Context(), sha) {559- s.fail(w, http.StatusForbidden, ErrBlocked)560- return561- }562- file, err := os.Open(filepath.Join(s.root, sha))563- if err != nil {564- s.fail(w, http.StatusNotFound, errors.New("not found"))565- return566- }567- defer file.Close()568- w.Header().Set("etag", `"`+sha+`"`)569- w.Header().Set("Content-Type", entry.Type)570- w.Header().Set("x-content-type-options", "nosniff")571- w.Header().Set("accept-ranges", "bytes")572- w.Header().Set("content-security-policy", "default-src 'none'")573- if s.config.CanRead == nil {574- w.Header().Set("cache-control", "public, max-age=31536000, immutable")575- } else {576- w.Header().Set("cache-control", "private, no-store")577- }578- http.ServeContent(w, r, sha, time.Unix(entry.Uploaded, 0), file)579-}580-581-func (s *Service) delete(w http.ResponseWriter, r *http.Request, sha string) {582- pubkey, err := s.authorize(r, ActionDelete)583- if err != nil {584- s.fail(w, http.StatusUnauthorized, err)585- return586- }587- _, err = s.lookup(r.Context(), sha)588- if errors.Is(err, sql.ErrNoRows) {589- s.fail(w, http.StatusNotFound, errors.New("not found"))590- return591- }592- if err != nil {593- s.fail(w, http.StatusInternalServerError, err)594- return595- }596- if !s.hasClaim(r.Context(), sha, pubkey) && !s.isOwner(pubkey) {597- s.fail(w, http.StatusForbidden, errors.New("restricted: not the uploader"))598- return599- }600- if s.isOwner(pubkey) {601- if err := s.Delete(r.Context(), sha); err != nil {602- s.fail(w, http.StatusInternalServerError, err)603- return604- }605- } else if err := s.DeleteForUploader(r.Context(), sha, pubkey); err != nil {606- if errors.Is(err, sql.ErrNoRows) {607- s.fail(w, http.StatusNotFound, errors.New("not found"))608- return609- }610- s.fail(w, http.StatusInternalServerError, err)611- return612- }613- // Owners and uploaders may remove their claim; a shared physical object is614- // retained until its final claim disappears.615- w.WriteHeader(http.StatusNoContent)616-}617-618-func (s *Service) put(ctx context.Context, source io.Reader, typ, uploader, claimed string) (Blob, bool, error) {619- return s.putValidatedWithCommit(ctx, source, typ, uploader, claimed, nil, nil)620-}621-622-func (s *Service) putValidated(ctx context.Context, source io.Reader, typ, uploader, claimed string, validate func(string) error) (Blob, bool, error) {623- return s.putValidatedWithCommit(ctx, source, typ, uploader, claimed, validate, nil)624-}625-626-func (s *Service) putValidatedWithCommit(ctx context.Context, source io.Reader, typ, uploader, claimed string, validate func(string) error, commit func(context.Context, *sql.Tx, Blob, bool) error) (Blob, bool, error) {627- limits := s.currentLimits()628- streamLimit := limits.MaxFileBytes629- quotaStream := false630- if streamLimit == 0 && limits.UserStorageBytes > 0 && uploader != "" {631- streamLimit = limits.UserStorageBytes632- quotaStream = true633- }634- if streamLimit > 0 && streamLimit < int64(^uint64(0)>>1) {635- source = io.LimitReader(source, streamLimit+1)636- }637- tmp, err := os.CreateTemp(s.root, ".upload-")638- if err != nil {639- return Blob{}, false, fmt.Errorf("create upload temporary file: %w", err)640- }641- tmpName := tmp.Name()642- defer os.Remove(tmpName)643- hasher := sha256.New()644- count, err := io.Copy(io.MultiWriter(tmp, hasher), source)645- if err != nil {646- _ = tmp.Close()647- return Blob{}, false, fmt.Errorf("read upload: %w", err)648- }649- if limits.MaxFileBytes > 0 && count > limits.MaxFileBytes {650- _ = tmp.Close()651- return Blob{}, false, ErrFileTooLarge652- }653- if quotaStream && count > streamLimit {654- _ = tmp.Close()655- return Blob{}, false, ErrQuotaExceeded656- }657- if err := tmp.Sync(); err != nil {658- _ = tmp.Close()659- return Blob{}, false, fmt.Errorf("sync upload: %w", err)660- }661- if err := tmp.Close(); err != nil {662- return Blob{}, false, fmt.Errorf("close upload: %w", err)663- }664- sha := hex.EncodeToString(hasher.Sum(nil))665- if claimed != "" && claimed != sha {666- return Blob{}, false, ErrHashMismatch667- }668- if validate != nil {669- if err := validate(sha); err != nil {670- return Blob{}, false, err671- }672- }673- return s.installUpload(ctx, uploadCandidate{path: tmpName, hash: sha, size: count, typ: typ, uploader: uploader, commit: commit})674-}675-676-type uploadCandidate struct {677- path, hash, typ, uploader, reservation string678- size int64679- commit func(context.Context, *sql.Tx, Blob, bool) error680-}681-682-func (s *Service) installUpload(ctx context.Context, candidate uploadCandidate) (Blob, bool, error) {683- sha, count, typ, uploader := candidate.hash, candidate.size, candidate.typ, candidate.uploader684- // Serialize the lookup, quota check, installation and metadata transaction.685- // This closes the race where concurrent uploads could each observe the same686- // remaining quota before one of them installs a new physical object.687- s.quotaMu.Lock()688- defer s.quotaMu.Unlock()689- if s.blocked(ctx, sha) {690- return Blob{}, false, ErrBlocked691- }692- terms, err := s.uploadTerms(ctx, uploader)693- if err != nil {694- return Blob{}, false, err695- }696- limits := s.currentLimits()697- if limits.MaxFileBytes > 0 && count > limits.MaxFileBytes {698- return Blob{}, false, ErrFileTooLarge699- }700- if limits.UserStorageBytes > 0 && uploader != "" && count > limits.UserStorageBytes {701- return Blob{}, false, ErrQuotaExceeded702- }703- if existing, err := s.lookup(ctx, sha); err == nil {704- if terms.Encrypted && !encryptedTypes[existing.Type] {705- return Blob{}, false, ErrNotEncrypted706- }707- if uploader != "" {708- if limits.UserStorageBytes > 0 && !s.hasClaim(ctx, sha, uploader) {709- used, usageErr := s.quotaUsageExcept(ctx, uploader, candidate.reservation)710- if usageErr != nil {711- return Blob{}, false, fmt.Errorf("check uploader storage quota: %w", usageErr)712- }713- if existing.Size > limits.UserStorageBytes-used {714- return Blob{}, false, ErrQuotaExceeded715- }716- }717- }718- if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {719- if candidate.commit != nil {720- if err := candidate.commit(ctx, tx, existing, false); err != nil {721- return err722- }723- }724- if uploader != "" {725- if _, err := tx.ExecContext(ctx, claimSQL, sha, uploader, time.Now().Unix(), terms.ExpiresAt); err != nil {726- return err727- }728- }729- if candidate.reservation != "" {730- if _, err := tx.ExecContext(ctx, "DELETE FROM multipart_uploads WHERE id=?", candidate.reservation); err != nil {731- return err732- }733- }734- return nil735- }); err != nil {736- return Blob{}, false, err737- }738- // A crash after a scoped metadata commit may leave the bytes missing.739- // A retry supplies those same verified bytes without losing the scope.740- if _, err := os.Stat(filepath.Join(s.root, sha)); errors.Is(err, os.ErrNotExist) {741- if err := s.installBlobFile(candidate.path, sha); err != nil {742- return Blob{}, false, err743- }744- } else if err != nil {745- return Blob{}, false, err746- }747- return existing, false, nil748- } else if !errors.Is(err, sql.ErrNoRows) {749- return Blob{}, false, err750- }751- if limits.UserStorageBytes > 0 && uploader != "" {752- used, err := s.quotaUsageExcept(ctx, uploader, candidate.reservation)753- if err != nil {754- return Blob{}, false, fmt.Errorf("check uploader storage quota: %w", err)755- }756- if count > limits.UserStorageBytes-used {757- return Blob{}, false, ErrQuotaExceeded758- }759- }760- // The type is settled before the bytes are installed so a plain upload761- // under an encrypted-only grant leaves nothing behind. A declared octet762- // stream is sniffed: encrypted bytes stay one, a page or an image does not.763- typ = contentType(typ)764- if typ == "application/octet-stream" {765- typ = detectFileType(candidate.path)766- }767- if terms.Encrypted && !encryptedTypes[typ] {768- return Blob{}, false, ErrNotEncrypted769- }770- stage := ""771- if candidate.commit != nil {772- stage = filepath.Join(s.root, ".scoped-"+sha)773- if err := os.Rename(candidate.path, stage); err != nil {774- return Blob{}, false, fmt.Errorf("stage scoped blob: %w", err)775- }776- if err := syncDirectory(s.root); err != nil {777- return Blob{}, false, fmt.Errorf("sync staged blob directory: %w", err)778- }779- }780- if candidate.commit == nil {781- if err := s.installBlobFile(candidate.path, sha); err != nil {782- return Blob{}, false, err783- }784- }785- now := time.Now().UTC().Unix()786- entry := Blob{SHA256: sha, Size: count, Type: typ, Uploader: uploader, Uploaded: now}787- err = s.store.WithTx(ctx, func(tx *sql.Tx) error {788- if _, err := tx.ExecContext(ctx, "DELETE FROM blob_tombstones WHERE sha256=?", entry.SHA256); err != nil {789- return fmt.Errorf("clear blob deletion tombstone: %w", err)790- }791- if _, err := tx.ExecContext(ctx, "INSERT OR IGNORE INTO blobs(sha256,size,type,uploader,uploaded) VALUES(?,?,?,?,?)", entry.SHA256, entry.Size, entry.Type, entry.Uploader, entry.Uploaded); err != nil {792- return fmt.Errorf("record blob: %w", err)793- }794- if entry.Uploader != "" {795- if _, err := tx.ExecContext(ctx, claimSQL, entry.SHA256, entry.Uploader, entry.Uploaded, terms.ExpiresAt); err != nil {796- return fmt.Errorf("record blob claim: %w", err)797- }798- }799- if candidate.reservation != "" {800- if _, err := tx.ExecContext(ctx, "DELETE FROM multipart_uploads WHERE id=?", candidate.reservation); err != nil {801- return fmt.Errorf("release upload reservation: %w", err)802- }803- }804- if candidate.commit != nil {805- if err := candidate.commit(ctx, tx, entry, true); err != nil {806- return err807- }808- }809- return nil810- })811- if err != nil {812- if stage != "" {813- _ = os.Remove(stage)814- }815- return Blob{}, false, err816- }817- if candidate.commit != nil {818- if err := s.installBlobFile(stage, sha); err != nil {819- return Blob{}, false, err820- }821- }822- if saved, err := s.lookup(ctx, sha); err == nil {823- return saved, saved.Uploader == uploader && saved.Uploaded == now, nil824- }825- return entry, true, nil826-}827-828-func (s *Service) installBlobFile(source, hash string) error {829- if err := os.Rename(source, filepath.Join(s.root, hash)); err != nil {830- return fmt.Errorf("install blob: %w", err)831- }832- if err := syncDirectory(s.root); err != nil {833- return fmt.Errorf("sync blob directory: %w", err)834- }835- return nil836-}837-838-func (s *Service) quotaUsage(ctx context.Context, uploader string) (int64, error) {839- return s.quotaUsageExcept(ctx, uploader, "")840-}841-842-func (s *Service) quotaUsageExcept(ctx context.Context, uploader, reservation string) (int64, error) {843- var used sql.NullInt64844- err := s.store.DB().QueryRowContext(ctx, `SELECT COALESCE((SELECT SUM(b.size)845- FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?),0) +846- COALESCE((SELECT SUM(length) FROM multipart_uploads WHERE uploader=? AND id != ?),0)`, uploader, uploader, reservation).Scan(&used)847- if err != nil {848- return 0, err849- }850- return used.Int64, nil851-}852-853-// claimSQL records a claim. The insert trigger on blobs has usually made the854-// row already, without an expiry, so the newest upload's terms replace it.855-const claimSQL = "INSERT INTO blob_claims(sha256,uploader,claimed_at,expires) VALUES(?,?,?,?) ON CONFLICT(sha256,uploader) DO UPDATE SET expires=excluded.expires"856-857-func (s *Service) claim(ctx context.Context, sha, uploader string, expires int64) error {858- _, err := s.store.DB().ExecContext(ctx, claimSQL, sha, uploader, time.Now().UTC().Unix(), expires)859- if err != nil {860- return fmt.Errorf("record blob claim: %w", err)861- }862- return nil863-}864-865-func (s *Service) uploadTerms(ctx context.Context, uploader string) (UploadTerms, error) {866- if s.config.UploadTerms == nil || uploader == "" {867- return UploadTerms{}, nil868- }869- // A refusal keeps its reason so the door answers with the right status.870- return s.config.UploadTerms(ctx, uploader)871-}872-873-// SweepExpired removes the blobs whose every claim has lapsed and releases874-// lapsed claims on blobs someone else still holds, so an agent's files leave875-// with its site while a person's claim keeps them. It returns how many blobs876-// were removed.877-func (s *Service) SweepExpired(ctx context.Context, now int64) (int, error) {878- s.quotaMu.Lock()879- defer s.quotaMu.Unlock()880- rows, err := s.store.DB().QueryContext(ctx, `SELECT DISTINCT sha256 FROM blob_claims c WHERE c.expires>0 AND c.expires<=?881- AND NOT EXISTS(SELECT 1 FROM blob_claims live WHERE live.sha256=c.sha256 AND (live.expires=0 OR live.expires>?))`, now, now)882- if err != nil {883- return 0, fmt.Errorf("list expired blobs: %w", err)884- }885- var expired []string886- for rows.Next() {887- var sha string888- if err := rows.Scan(&sha); err != nil {889- _ = rows.Close()890- return 0, err891- }892- expired = append(expired, sha)893- }894- if err := errors.Join(rows.Err(), rows.Close()); err != nil {895- return 0, err896- }897- for _, sha := range expired {898- if err := s.Delete(ctx, sha); err != nil {899- return 0, err900- }901- }902- if _, err := s.store.DB().ExecContext(ctx, "DELETE FROM blob_claims WHERE expires>0 AND expires<=?", now); err != nil {903- return 0, fmt.Errorf("release expired blob claims: %w", err)904- }905- return len(expired), nil906-}907-908-func (s *Service) currentLimits() Limits {909- if s.config.Limits == nil {910- return Limits{}911- }912- limits := s.config.Limits()913- if limits.MaxFileBytes < 0 {914- limits.MaxFileBytes = 0915- }916- if limits.UserStorageBytes < 0 {917- limits.UserStorageBytes = 0918- }919- return limits920-}921-922-func (s *Service) hasClaim(ctx context.Context, sha, uploader string) bool {923- var found int924- err := s.store.DB().QueryRowContext(ctx, "SELECT 1 FROM blob_claims WHERE sha256=? AND uploader=?", sha, uploader).Scan(&found)925- return err == nil && found == 1926-}927-928-func (s *Service) lookup(ctx context.Context, sha string) (Blob, error) {929- var entry Blob930- err := s.store.DB().QueryRowContext(ctx, "SELECT sha256,size,type,uploader,uploaded FROM blobs WHERE sha256=?", sha).Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded)931- return entry, err932-}933-934-func (s *Service) blocked(ctx context.Context, sha string) bool {935- var found int936- err := s.store.DB().QueryRowContext(ctx, `SELECT 1 FROM hidden_events WHERE id=?937- UNION ALL SELECT 1 FROM blob_blocks WHERE sha256=? LIMIT 1`, sha, sha).Scan(&found)938- return err == nil && found == 1939-}940-941-func reconcile(ctx context.Context, store *storage.Store, root string) error {942- entries, err := os.ReadDir(root)943- if err != nil {944- return fmt.Errorf("scan blob directory: %w", err)945- }946- for _, entry := range entries {947- if entry.IsDir() {948- continue949- }950- name := entry.Name()951- if strings.HasPrefix(name, ".scoped-") {952- sha := strings.TrimPrefix(name, ".scoped-")953- if !shaPattern.MatchString(sha) {954- if err := os.Remove(filepath.Join(root, name)); err != nil {955- return fmt.Errorf("remove invalid staged blob: %w", err)956- }957- continue958- }959- if err := reconcileStaged(ctx, store, root, name, sha); err != nil {960- return err961- }962- continue963- }964- if strings.HasPrefix(name, ".upload-") || strings.HasPrefix(name, ".multipart-chunk-") {965- if err := os.Remove(filepath.Join(root, name)); err != nil {966- return fmt.Errorf("remove abandoned upload: %w", err)967- }968- continue969- }970- if !shaPattern.MatchString(name) {971- continue972- }973- var deleted int974- if err := store.DB().QueryRowContext(ctx, "SELECT 1 FROM blob_tombstones WHERE sha256=? UNION ALL SELECT 1 FROM blob_blocks WHERE sha256=? LIMIT 1", name, name).Scan(&deleted); err == nil {975- continue976- } else if !errors.Is(err, sql.ErrNoRows) {977- return fmt.Errorf("check deleted blob %q: %w", name, err)978- }979- var known int980- if err := store.DB().QueryRowContext(ctx, "SELECT 1 FROM blobs WHERE sha256=?", name).Scan(&known); err == nil {981- continue982- } else if !errors.Is(err, sql.ErrNoRows) {983- return fmt.Errorf("check orphan blob %q: %w", name, err)984- }985- if err := recoverBlob(ctx, store, root, name); err != nil {986- return err987- }988- }989- return nil990-}991-992-func reconcileStaged(ctx context.Context, store *storage.Store, root, name, sha string) error {993- var size int64994- if err := store.DB().QueryRowContext(ctx, "SELECT size FROM blobs WHERE sha256=?", sha).Scan(&size); errors.Is(err, sql.ErrNoRows) {995- if removeErr := os.Remove(filepath.Join(root, name)); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) {996- return removeErr997- }998- return nil999- } else if err != nil {1000- return fmt.Errorf("check staged blob %q: %w", sha, err)1001- }1002- file, err := os.Open(filepath.Join(root, name))1003- if err != nil {1004- return err1005- }1006- info, err := file.Stat()1007- if err != nil {1008- _ = file.Close()1009- return err1010- }1011- if info.Size() != size {1012- _ = file.Close()1013- return fmt.Errorf("staged blob %q has size %d, want %d", sha, info.Size(), size)1014- }1015- hasher := sha256.New()1016- if _, err := io.Copy(hasher, file); err != nil {1017- _ = file.Close()1018- return err1019- }1020- if err := file.Close(); err != nil {1021- return err1022- }1023- if hex.EncodeToString(hasher.Sum(nil)) != sha {1024- return fmt.Errorf("staged blob %q failed content hash", sha)1025- }1026- if _, err := os.Stat(filepath.Join(root, sha)); errors.Is(err, os.ErrNotExist) {1027- if err := os.Rename(filepath.Join(root, name), filepath.Join(root, sha)); err != nil {1028- return err1029- }1030- } else if err != nil {1031- return err1032- } else {1033- _ = os.Remove(filepath.Join(root, name))1034- }1035- return syncDirectory(root)1036-}1037-1038-func recoverBlob(ctx context.Context, store *storage.Store, root, name string) error {1039- file, err := os.Open(filepath.Join(root, name))1040- if err != nil {1041- return fmt.Errorf("open orphan blob %q: %w", name, err)1042- }1043- hash := sha256.New()1044- size, copyErr := io.Copy(hash, file)1045- closeErr := file.Close()1046- if copyErr != nil || closeErr != nil {1047- return fmt.Errorf("read orphan blob %q: %w", name, errors.Join(copyErr, closeErr))1048- }1049- if hex.EncodeToString(hash.Sum(nil)) != name {1050- return fmt.Errorf("orphan blob %q failed content hash", name)1051- }1052- info, err := os.Stat(filepath.Join(root, name))1053- if err != nil {1054- return fmt.Errorf("stat orphan blob %q: %w", name, err)1055- }1056- _, err = store.DB().ExecContext(ctx, "INSERT OR IGNORE INTO blobs(sha256,size,type,uploader,uploaded) VALUES(?,?,?,?,?)", name, size, "application/octet-stream", "", info.ModTime().Unix())1057- if err != nil {1058- return fmt.Errorf("recover orphan blob %q: %w", name, err)1059- }1060- return nil1061-}1062-1063-func syncDirectory(path string) error {1064- directory, err := os.Open(path)1065- if err != nil {1066- return err1067- }1068- defer directory.Close()1069- return directory.Sync()1070-}1071-1072-func (s *Service) list(w http.ResponseWriter, r *http.Request, pubkey string) {1073- if !shaPattern.MatchString(pubkey) {1074- s.fail(w, http.StatusBadRequest, errors.New("invalid: bad pubkey"))1075- return1076- }1077- if _, err := s.authorize(r, ActionList); err != nil {1078- s.fail(w, http.StatusUnauthorized, err)1079- return1080- }1081- limit, err := parseListLimit(r.URL.Query().Get("limit"))1082- if err != nil {1083- s.fail(w, http.StatusBadRequest, err)1084- return1085- }1086- entries, _, err := s.ListPage(r.Context(), pubkey, limit, strings.TrimSpace(r.URL.Query().Get("cursor")))1087- if err != nil {1088- if errors.Is(err, sql.ErrNoRows) {1089- s.fail(w, http.StatusBadRequest, errors.New("invalid: cursor does not identify a blob"))1090- return1091- }1092- s.fail(w, http.StatusInternalServerError, err)1093- return1094- }1095- descriptors := make([]map[string]any, 0, len(entries))1096- for _, b := range entries {1097- descriptors = append(descriptors, s.descriptor(r, b))1098- }1099- s.writeJSON(w, http.StatusOK, descriptors)1100-}1101-1102-func (s *Service) report(w http.ResponseWriter, r *http.Request) {1103- reporter, err := s.authorize(r, ActionReport)1104- if err != nil {1105- s.fail(w, http.StatusUnauthorized, err)1106- return1107- }1108- var input event.Event1109- if err := json.NewDecoder(r.Body).Decode(&input); err != nil || input.Kind != 1984 || input.PubKey != reporter {1110- s.fail(w, http.StatusBadRequest, errors.New("invalid: report must be a signed kind 1984 event"))1111- return1112- }1113- if err := event.Validate(input); err != nil || time.Since(time.Unix(input.CreatedAt, 0)) > time.Hour || time.Until(time.Unix(input.CreatedAt, 0)) > time.Hour {1114- s.fail(w, http.StatusBadRequest, errors.New("invalid: report is not a current signed kind 1984 event"))1115- return1116- }1117- filed := 01118- type reportTarget struct {1119- hash string1120- uploader string1121- }1122- targets := make([]reportTarget, 0, len(input.Tags))1123- for _, tag := range input.Tags {1124- if len(tag) < 2 || tag[0] != "x" || !shaPattern.MatchString(tag[1]) {1125- continue1126- }1127- entry, lookupErr := s.lookup(r.Context(), tag[1])1128- if lookupErr != nil {1129- continue1130- }1131- id := input.ID1132- if filed > 0 {1133- id += "." + strconv.Itoa(filed)1134- }1135- targets = append(targets, reportTarget{hash: tag[1], uploader: entry.Uploader})1136- }1137- if len(targets) == 0 {1138- s.fail(w, http.StatusNotFound, errors.New("not found: relay holds none of the reported blobs"))1139- return1140- }1141- err = s.store.WithTx(r.Context(), func(tx *sql.Tx) error {1142- for i, target := range targets {1143- id := input.ID1144- if i > 0 {1145- id += "." + strconv.Itoa(i)1146- }1147- result, insertErr := tx.ExecContext(r.Context(), `INSERT OR IGNORE INTO blob_reports(id,reporter,target_pubkey,target_blob,type,content,at) VALUES(?,?,?,?,?,?,?)`, id, reporter, target.uploader, target.hash, "blob", input.Content, input.CreatedAt)1148- if insertErr != nil {1149- return insertErr1150- }1151- changed, rowsErr := result.RowsAffected()1152- if rowsErr != nil {1153- return rowsErr1154- }1155- if changed == 0 {1156- continue1157- }1158- if s.config.RecordReport != nil {1159- if err := s.config.RecordReport(r.Context(), tx, reporter, target.hash, "blob", "blob", input.Content); err != nil {1160- return err1161- }1162- }1163- filed++1164- }1165- return nil1166- })1167- if err != nil {1168- s.fail(w, http.StatusInternalServerError, err)1169- return1170- }1171- if filed == 0 {1172- // Repeating an identical report is idempotent and still successful.1173- filed = len(targets)1174- }1175- s.writeJSON(w, http.StatusOK, map[string]any{"ok": true, "filed": filed})1176-}1177-1178-// Block permanently prevents a hash from returning through any upload door.1179-func (s *Service) Block(ctx context.Context, sha, reason string) error {1180- if !shaPattern.MatchString(sha) {1181- return errors.New("invalid: bad blob sha256")1182- }1183- _, err := s.store.DB().ExecContext(ctx, "INSERT INTO blob_blocks(sha256,reason,blocked_at) VALUES(?,?,?) ON CONFLICT(sha256) DO UPDATE SET reason=excluded.reason,blocked_at=excluded.blocked_at", sha, reason, time.Now().UTC().Unix())1184- if err != nil {1185- return fmt.Errorf("block blob: %w", err)1186- }1187- if _, err := s.store.DB().ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {1188- return fmt.Errorf("record blocked blob deletion: %w", err)1189- }1190- if _, err := s.store.DB().ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {1191- return fmt.Errorf("remove blocked blob metadata: %w", err)1192- }1193- if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {1194- return fmt.Errorf("remove blocked blob: %w", err)1195- }1196- return nil1197-}1198-1199-func (s *Service) mirror(w http.ResponseWriter, r *http.Request) {1200- pubkey, err := s.authorize(r, ActionMirror)1201- if err != nil {1202- s.fail(w, http.StatusUnauthorized, err)1203- return1204- }1205- var input struct {1206- URL string `json:"url"`1207- }1208- if err := json.NewDecoder(r.Body).Decode(&input); err != nil || input.URL == "" {1209- s.fail(w, http.StatusBadRequest, errors.New("invalid: body needs a url"))1210- return1211- }1212- requestedURL := input.URL1213- expectedHash := ""1214- extension := ""1215- uriServers := []string{}1216- authors := []string{}1217- if strings.HasPrefix(strings.ToLower(requestedURL), "blossom:") {1218- uri, err := ParseBlossomURI(requestedURL)1219- if err != nil {1220- s.fail(w, http.StatusBadRequest, err)1221- return1222- }1223- expectedHash, extension = uri.Hash, uri.Extension1224- uriServers = append(uriServers, uri.Servers...)1225- authors = append(authors, uri.Authors...)1226- requestedURL = ""1227- } else if hash := BlobHashFromURL(requestedURL); hash != "" {1228- expectedHash = hash1229- }1230- client := s.config.HTTPClient1231- if client == nil {1232- client = s.safeClient()1233- }1234- candidates := []string{}1235- if requestedURL != "" {1236- candidates = append(candidates, requestedURL)1237- }1238- candidates = append(candidates, FallbackURLs(uriServers, expectedHash, extension)...)1239- if s.config.ResolveServers != nil && expectedHash != "" {1240- if len(authors) == 0 && pubkey != "" {1241- authors = []string{pubkey}1242- }1243- for _, author := range uniqueStrings(authors) {1244- servers, resolveErr := s.config.ResolveServers(r.Context(), author)1245- if resolveErr != nil {1246- continue1247- }1248- candidates = append(candidates, FallbackURLs(servers, expectedHash, extension)...)1249- }1250- }1251- var lastErr error1252- for _, candidate := range uniqueStrings(candidates) {1253- if err := validateMirrorURL(candidate, s.resolve); err != nil {1254- lastErr = err1255- continue1256- }1257- response, fetchErr := client.Get(candidate)1258- if fetchErr != nil {1259- lastErr = fetchErr1260- continue1261- }1262- if response.StatusCode < 200 || response.StatusCode >= 300 {1263- lastErr = fmt.Errorf("error: origin answered %d", response.StatusCode)1264- _ = response.Body.Close()1265- continue1266- }1267- typ := contentType(response.Header.Get("content-type"))1268- if typ == "application/octet-stream" {1269- typ = typeForExtension(candidate)1270- }1271- entry, created, putErr := s.putValidated(r.Context(), response.Body, typ, pubkey, expectedHash, func(actual string) error {1272- if s.config.ValidateUpload != nil {1273- return s.config.ValidateUpload(r, actual)1274- }1275- return nil1276- })1277- _ = response.Body.Close()1278- if putErr == nil {1279- s.writeJSON(w, chooseStatus(created, http.StatusCreated), s.descriptor(r, entry))1280- return1281- }1282- lastErr = putErr1283- if !errors.Is(putErr, ErrHashMismatch) {1284- s.fail(w, statusFor(putErr), putErr)1285- return1286- }1287- }1288- if lastErr == nil {1289- lastErr = errors.New("error: no mirror origin available")1290- }1291- s.fail(w, statusFor(lastErr), lastErr)1292-}1293-1294-func (s *Service) nip96(w http.ResponseWriter, r *http.Request) {1295- if r.Method == http.MethodGet {1296- pubkey, err := s.authorize(r, ActionList)1297- if err != nil {1298- s.fail(w, http.StatusUnauthorized, err)1299- return1300- }1301- s.nip96List(w, r, pubkey)1302- return1303- }1304- if r.Method != http.MethodPost {1305- s.fail(w, http.StatusMethodNotAllowed, errors.New("method not allowed"))1306- return1307- }1308- pubkey, err := s.authorize(r, ActionUpload)1309- if err != nil {1310- s.fail(w, http.StatusUnauthorized, err)1311- return1312- }1313- if err := r.ParseMultipartForm(0); err != nil {1314- s.fail(w, http.StatusBadRequest, errors.New("invalid: body is not multipart form data"))1315- return1316- }1317- defer r.MultipartForm.RemoveAll()1318- file, header, err := r.FormFile("file")1319- if err != nil {1320- s.fail(w, http.StatusBadRequest, errors.New("invalid: form needs a file field"))1321- return1322- }1323- defer file.Close()1324- typ := contentType(header.Header.Get("Content-Type"))1325- if typ == "application/octet-stream" {1326- typ = contentType(r.FormValue("content_type"))1327- }1328- entry, created, err := s.put(r.Context(), file, typ, pubkey, "")1329- if err != nil {1330- s.fail(w, statusFor(err), err)1331- return1332- }1333- s.writeJSON(w, chooseStatus(created, http.StatusCreated), map[string]any{"status": "success", "message": "Upload successful.", "url": s.blobURL(r, entry), "nip94_event": map[string]any{"tags": s.nip94Tags(r, entry), "content": r.FormValue("caption")}})1334-}1335-1336-func (s *Service) nip96List(w http.ResponseWriter, r *http.Request, pubkey string) {1337- count := parsePageCount(r.URL.Query().Get("count"))1338- page := parsePage(r.URL.Query().Get("page"))1339- var total int1340- if err := s.store.DB().QueryRowContext(r.Context(), "SELECT count(*) FROM blob_claims WHERE uploader=?", pubkey).Scan(&total); err != nil {1341- s.fail(w, http.StatusInternalServerError, err)1342- return1343- }1344- rows, err := s.store.DB().QueryContext(r.Context(), `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded1345- FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?1346- ORDER BY b.uploaded DESC,b.sha256 DESC LIMIT ? OFFSET ?`, pubkey, count, page*count)1347- if err != nil {1348- s.fail(w, http.StatusInternalServerError, err)1349- return1350- }1351- defer rows.Close()1352- files := make([]map[string]any, 0)1353- for rows.Next() {1354- var b Blob1355- if err := rows.Scan(&b.SHA256, &b.Size, &b.Type, &b.Uploader, &b.Uploaded); err != nil {1356- s.fail(w, http.StatusInternalServerError, err)1357- return1358- }1359- files = append(files, map[string]any{"tags": s.nip94Tags(r, b), "content": "", "created_at": b.Uploaded})1360- }1361- s.writeJSON(w, http.StatusOK, map[string]any{"count": count, "total": total, "page": page, "files": files})1362-}1363-1364-func (s *Service) authorize(r *http.Request, action Action) (string, error) {1365- if s.config.Authorize == nil {1366- return "", errors.New("auth-required: authorization is required")1367- }1368- return s.config.Authorize(r, action)1369-}1370-func (s *Service) isOwner(pubkey string) bool {1371- return s.config.IsOwner != nil && s.config.IsOwner(pubkey)1372-}1373-func (s *Service) descriptor(r *http.Request, b Blob) map[string]any {1374- return map[string]any{"url": s.blobURL(r, b), "sha256": b.SHA256, "size": b.Size, "type": b.Type, "uploaded": b.Uploaded, "nip94": s.nip94Tags(r, b)}1375-}1376-func (s *Service) nip94Tags(r *http.Request, b Blob) [][]string {1377- return [][]string{{"url", s.blobURL(r, b)}, {"m", b.Type}, {"x", b.SHA256}, {"ox", b.SHA256}, {"size", strconv.FormatInt(b.Size, 10)}}1378-}1379-func (s *Service) blobURL(r *http.Request, b Blob) string {1380- return s.origin(r) + "/" + b.SHA256 + extensionSuffix(b.Type)1381-}1382-func (s *Service) nip96Info(r *http.Request) map[string]any {1383- return map[string]any{"api_url": s.origin(r) + "/nip96", "download_url": s.origin(r), "supported_nips": []int{94, 98}, "content_types": contentTypes(), "plans": map[string]any{"self-hosted": map[string]any{"name": "Self-hosted", "is_nip98_required": true, "max_byte_size": 0, "file_expiration": []int{0, 0}, "media_transformations": map[string]any{}}}}1384-}1385-func (s *Service) writeJSON(w http.ResponseWriter, status int, value any) {1386- w.Header().Set("content-type", "application/json")1387- w.WriteHeader(status)1388- _ = json.NewEncoder(w).Encode(value)1389-}1390-func (s *Service) fail(w http.ResponseWriter, status int, err error) {1391- w.Header().Set("x-reason", err.Error())1392- s.writeJSON(w, status, map[string]string{"error": err.Error()})1393-}1394-func origin(r *http.Request) string {1395- scheme := r.URL.Scheme1396- if scheme == "" {1397- scheme = "http"1398- if r.TLS != nil {1399- scheme = "https"1400- }1401- }1402- return scheme + "://" + r.Host1403-}1404-1405-func (s *Service) origin(r *http.Request) string {1406- if s.config.PublicURL != "" {1407- return strings.TrimRight(s.config.PublicURL, "/")1408- }1409- return origin(r)1410-}1411-1412-// DetectMediaType recognises images, video, audio and PDF from the first1413-// bytes of a blob. Anything else stays an octet stream so uploads can never1414-// be reinterpreted as a page or a script.1415-func DetectMediaType(head []byte) string {1416- detected := contentType(http.DetectContentType(head))1417- for _, prefix := range []string{"image/", "video/", "audio/"} {1418- if strings.HasPrefix(detected, prefix) {1419- return detected1420- }1421- }1422- if detected == "application/pdf" {1423- return detected1424- }1425- return "application/octet-stream"1426-}1427-1428-func detectFileType(name string) string {1429- file, err := os.Open(name)1430- if err != nil {1431- return "application/octet-stream"1432- }1433- defer file.Close()1434- head := make([]byte, 512)1435- n, _ := io.ReadFull(file, head)1436- return DetectMediaType(head[:n])1437-}1438-1439-func contentType(raw string) string {1440- raw = strings.ToLower(strings.TrimSpace(strings.Split(raw, ";")[0]))1441- if raw == "" {1442- return "application/octet-stream"1443- }1444- return raw1445-}1446-func extensionSuffix(typ string) string {1447- if ext := extensions[typ]; ext != "" {1448- return "." + ext1449- }1450- return ""1451-}1452-1453-func typeForExtension(raw string) string {1454- u, err := url.Parse(raw)1455- if err != nil {1456- return "application/octet-stream"1457- }1458- name := strings.ToLower(filepath.Ext(u.Path))1459- for typ, ext := range extensions {1460- if "."+ext == name {1461- return typ1462- }1463- }1464- return "application/octet-stream"1465-}1466-func contentTypes() []string {1467- out := make([]string, 0, len(extensions))1468- for typ := range extensions {1469- out = append(out, typ)1470- }1471- sort.Strings(out)1472- return out1473-}1474-1475-func parsePageCount(raw string) int {1476- value, err := strconv.Atoi(raw)1477- if err != nil || value < 1 {1478- return 201479- }1480- if value > 100 {1481- return 1001482- }1483- return value1484-}1485-1486-func parsePage(raw string) int {1487- value, err := strconv.Atoi(raw)1488- if err != nil || value < 0 {1489- return 01490- }1491- return value1492-}1493-1494-func parseListLimit(raw string) (int, error) {1495- if raw == "" {1496- return 20, nil1497- }1498- value, err := strconv.Atoi(raw)1499- if err != nil || value < 1 || value > 100 {1500- return 0, errors.New("invalid: limit must be between 1 and 100")1501- }1502- return value, nil1503-}1504-func chooseStatus(created bool, status int) int {1505- if created {1506- return status1507- }1508- return http.StatusOK1509-}1510-func statusFor(err error) int {1511- if errors.Is(err, ErrHashMismatch) {1512- return http.StatusConflict1513- }1514- if errors.Is(err, ErrMultipartConflict) {1515- return http.StatusConflict1516- }1517- if errors.Is(err, ErrFileTooLarge) {1518- return http.StatusRequestEntityTooLarge1519- }1520- if errors.Is(err, ErrQuotaExceeded) {1521- return http.StatusForbidden1522- }1523- if errors.Is(err, ErrBlocked) || errors.Is(err, ErrNotEncrypted) {1524- return http.StatusForbidden1525- }1526- if strings.HasPrefix(err.Error(), "auth-required:") {1527- return http.StatusUnauthorized1528- }1529- if strings.HasPrefix(err.Error(), "invalid:") {1530- return http.StatusBadRequest1531- }1532- if strings.HasPrefix(err.Error(), "restricted:") {1533- return http.StatusForbidden1534- }1535- return http.StatusInternalServerError1536-}1537-1538-func uniqueStrings(values []string) []string {1539- seen := make(map[string]struct{}, len(values))1540- result := make([]string, 0, len(values))1541- for _, value := range values {1542- if _, ok := seen[value]; ok || value == "" {1543- continue1544- }1545- seen[value] = struct{}{}1546- result = append(result, value)1547- }1548- return result1549-}1550-1551-func validateMirrorURL(raw string, resolve ResolveIP) error {1552- u, err := url.Parse(raw)1553- if err != nil || u.Scheme != "https" || u.Hostname() == "" {1554- return errors.New("invalid: only https urls can be mirrored")1555- }1556- ips, err := resolve(context.Background(), u.Hostname())1557- if err != nil || len(ips) == 0 {1558- return errors.New("invalid: origin hostname did not resolve")1559- }1560- for _, ip := range ips {1561- if !publicIP(ip) {1562- return errors.New("invalid: origin resolves to a private address")1563- }1564- }1565- return nil1566-}1567-1568-func publicIP(ip net.IP) bool {1569- if ip == nil || ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || ip.IsUnspecified() {1570- return false1571- }1572- if ip4 := ip.To4(); ip4 != nil {1573- if ip4[0] == 100 && ip4[1] >= 64 && ip4[1] <= 127 {1574- return false1575- }1576- if ip4[0] == 192 && ip4[1] == 0 && ip4[2] == 0 {1577- return false1578- }1579- if ip4[0] == 198 && (ip4[1] == 18 || ip4[1] == 19) {1580- return false1581- }1582- }1583- return true1584-}1585-func (s *Service) safeClient() *http.Client {1586- base, ok := http.DefaultTransport.(*http.Transport)1587- if !ok {1588- base = &http.Transport{}1589- }1590- transport := base.Clone()1591- dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}1592- transport.DialContext = func(ctx context.Context, network, address string) (net.Conn, error) {1593- host, port, err := net.SplitHostPort(address)1594- if err != nil {1595- return nil, err1596- }1597- ips, err := s.resolve(ctx, host)1598- if err != nil {1599- return nil, err1600- }1601- for _, ip := range ips {1602- if !publicIP(ip) {1603- continue1604- }1605- conn, err := dialer.DialContext(ctx, network, net.JoinHostPort(ip.String(), port))1606- if err == nil {1607- return conn, nil1608- }1609- }1610- return nil, errors.New("origin has no reachable public address")1611- }1612- return &http.Client{Timeout: 5 * time.Minute, Transport: transport, CheckRedirect: func(req *http.Request, _ []*http.Request) error {1613- return validateMirrorURL(req.URL.String(), s.resolve)1614- }}1615-}@@ -0,0 +1,221 @@
1+package blob2+3+import (4+ "context"5+ "errors"6+ "io"7+ "net"8+ "net/http"9+ "net/url"10+ "os"11+ "path/filepath"12+ "sort"13+ "strconv"14+ "strings"15+ "time"16+)17+18+// DetectMediaType recognises images, video, audio and PDF from the first19+// bytes of a blob. Anything else stays an octet stream so uploads can never20+// be reinterpreted as a page or a script.21+func DetectMediaType(head []byte) string {22+ detected := contentType(http.DetectContentType(head))23+ for _, prefix := range []string{"image/", "video/", "audio/"} {24+ if strings.HasPrefix(detected, prefix) {25+ return detected26+ }27+ }28+ if detected == "application/pdf" {29+ return detected30+ }31+ return "application/octet-stream"32+}33+34+func detectFileType(name string) string {35+ file, err := os.Open(name)36+ if err != nil {37+ return "application/octet-stream"38+ }39+ defer file.Close()40+ head := make([]byte, 512)41+ n, _ := io.ReadFull(file, head)42+ return DetectMediaType(head[:n])43+}44+45+func contentType(raw string) string {46+ raw = strings.ToLower(strings.TrimSpace(strings.Split(raw, ";")[0]))47+ if raw == "" {48+ return "application/octet-stream"49+ }50+ return raw51+}52+func extensionSuffix(typ string) string {53+ if ext := extensions[typ]; ext != "" {54+ return "." + ext55+ }56+ return ""57+}58+59+func typeForExtension(raw string) string {60+ u, err := url.Parse(raw)61+ if err != nil {62+ return "application/octet-stream"63+ }64+ name := strings.ToLower(filepath.Ext(u.Path))65+ for typ, ext := range extensions {66+ if "."+ext == name {67+ return typ68+ }69+ }70+ return "application/octet-stream"71+}72+func contentTypes() []string {73+ out := make([]string, 0, len(extensions))74+ for typ := range extensions {75+ out = append(out, typ)76+ }77+ sort.Strings(out)78+ return out79+}80+81+func parsePageCount(raw string) int {82+ value, err := strconv.Atoi(raw)83+ if err != nil || value < 1 {84+ return 2085+ }86+ if value > 100 {87+ return 10088+ }89+ return value90+}91+92+func parsePage(raw string) int {93+ value, err := strconv.Atoi(raw)94+ if err != nil || value < 0 {95+ return 096+ }97+ return value98+}99+100+func parseListLimit(raw string) (int, error) {101+ if raw == "" {102+ return 20, nil103+ }104+ value, err := strconv.Atoi(raw)105+ if err != nil || value < 1 || value > 100 {106+ return 0, errors.New("invalid: limit must be between 1 and 100")107+ }108+ return value, nil109+}110+func chooseStatus(created bool, status int) int {111+ if created {112+ return status113+ }114+ return http.StatusOK115+}116+func statusFor(err error) int {117+ if errors.Is(err, ErrHashMismatch) {118+ return http.StatusConflict119+ }120+ if errors.Is(err, ErrMultipartConflict) {121+ return http.StatusConflict122+ }123+ if errors.Is(err, ErrFileTooLarge) {124+ return http.StatusRequestEntityTooLarge125+ }126+ if errors.Is(err, ErrQuotaExceeded) {127+ return http.StatusForbidden128+ }129+ if errors.Is(err, ErrBlocked) || errors.Is(err, ErrNotEncrypted) {130+ return http.StatusForbidden131+ }132+ if strings.HasPrefix(err.Error(), "auth-required:") {133+ return http.StatusUnauthorized134+ }135+ if strings.HasPrefix(err.Error(), "invalid:") {136+ return http.StatusBadRequest137+ }138+ if strings.HasPrefix(err.Error(), "restricted:") {139+ return http.StatusForbidden140+ }141+ return http.StatusInternalServerError142+}143+144+func uniqueStrings(values []string) []string {145+ seen := make(map[string]struct{}, len(values))146+ result := make([]string, 0, len(values))147+ for _, value := range values {148+ if _, ok := seen[value]; ok || value == "" {149+ continue150+ }151+ seen[value] = struct{}{}152+ result = append(result, value)153+ }154+ return result155+}156+157+func validateMirrorURL(raw string, resolve ResolveIP) error {158+ u, err := url.Parse(raw)159+ if err != nil || u.Scheme != "https" || u.Hostname() == "" {160+ return errors.New("invalid: only https urls can be mirrored")161+ }162+ ips, err := resolve(context.Background(), u.Hostname())163+ if err != nil || len(ips) == 0 {164+ return errors.New("invalid: origin hostname did not resolve")165+ }166+ for _, ip := range ips {167+ if !publicIP(ip) {168+ return errors.New("invalid: origin resolves to a private address")169+ }170+ }171+ return nil172+}173+174+func publicIP(ip net.IP) bool {175+ if ip == nil || ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || ip.IsUnspecified() {176+ return false177+ }178+ if ip4 := ip.To4(); ip4 != nil {179+ if ip4[0] == 100 && ip4[1] >= 64 && ip4[1] <= 127 {180+ return false181+ }182+ if ip4[0] == 192 && ip4[1] == 0 && ip4[2] == 0 {183+ return false184+ }185+ if ip4[0] == 198 && (ip4[1] == 18 || ip4[1] == 19) {186+ return false187+ }188+ }189+ return true190+}191+func (s *Service) safeClient() *http.Client {192+ base, ok := http.DefaultTransport.(*http.Transport)193+ if !ok {194+ base = &http.Transport{}195+ }196+ transport := base.Clone()197+ dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}198+ transport.DialContext = func(ctx context.Context, network, address string) (net.Conn, error) {199+ host, port, err := net.SplitHostPort(address)200+ if err != nil {201+ return nil, err202+ }203+ ips, err := s.resolve(ctx, host)204+ if err != nil {205+ return nil, err206+ }207+ for _, ip := range ips {208+ if !publicIP(ip) {209+ continue210+ }211+ conn, err := dialer.DialContext(ctx, network, net.JoinHostPort(ip.String(), port))212+ if err == nil {213+ return conn, nil214+ }215+ }216+ return nil, errors.New("origin has no reachable public address")217+ }218+ return &http.Client{Timeout: 5 * time.Minute, Transport: transport, CheckRedirect: func(req *http.Request, _ []*http.Request) error {219+ return validateMirrorURL(req.URL.String(), s.resolve)220+ }}221+}@@ -0,0 +1,588 @@
1+package blob2+3+import (4+ "context"5+ "database/sql"6+ "encoding/json"7+ "errors"8+ "fmt"9+ "net/http"10+ "os"11+ "path/filepath"12+ "strconv"13+ "strings"14+ "time"15+16+ "github.com/FelineStateMachine/tinyrelay/internal/event"17+)18+19+func (s *Service) serveHTTP(w http.ResponseWriter, r *http.Request) {20+ w.Header().Set("access-control-allow-origin", "*")21+ w.Header().Set("access-control-allow-headers", "authorization, content-type, x-sha-256, x-content-length, x-content-type, upload-type, upload-length, upload-offset")22+ w.Header().Set("access-control-allow-methods", "GET, HEAD, PUT, PATCH, POST, DELETE, OPTIONS")23+ w.Header().Set("Access-Control-Expose-Headers", "Allow, X-Reason")24+ if r.Method == http.MethodOptions {25+ if blobPathPattern.MatchString(r.URL.Path) {26+ w.Header().Set("Allow", "GET, HEAD, PUT, PATCH, DELETE, OPTIONS")27+ }28+ w.WriteHeader(http.StatusNoContent)29+ return30+ }31+ if r.URL.Path == "/.well-known/nostr/nip96.json" {32+ s.writeJSON(w, http.StatusOK, s.nip96Info(r))33+ return34+ }35+ if r.URL.Path == "/upload" && r.Method == http.MethodHead {36+ s.uploadHead(w, r)37+ return38+ }39+ if r.URL.Path == "/upload" && r.Method == http.MethodPut {40+ s.upload(w, r)41+ return42+ }43+ if r.URL.Path == "/mirror" && r.Method == http.MethodPut {44+ s.mirror(w, r)45+ return46+ }47+ if match := blobPathPattern.FindStringSubmatch(r.URL.Path); match != nil && (r.Method == http.MethodPut || r.Method == http.MethodPatch || r.Method == http.MethodOptions) {48+ if r.Method == http.MethodPatch || r.Method == http.MethodOptions {49+ s.multipart(w, r, match[1])50+ return51+ }52+ s.pathUpload(w, r, match[1])53+ return54+ }55+ if r.URL.Path == "/report" && r.Method == http.MethodPut {56+ s.report(w, r)57+ return58+ }59+ if strings.HasPrefix(r.URL.Path, "/list/") && r.Method == http.MethodGet {60+ s.list(w, r, strings.TrimPrefix(r.URL.Path, "/list/"))61+ return62+ }63+ if r.URL.Path == "/nip96" {64+ s.nip96(w, r)65+ return66+ }67+ s.blob(w, r)68+}69+70+func (s *Service) upload(w http.ResponseWriter, r *http.Request) {71+ pubkey, err := s.authorize(r, ActionUpload)72+ if err != nil {73+ s.fail(w, http.StatusUnauthorized, err)74+ return75+ }76+ claimed := strings.ToLower(strings.TrimSpace(r.Header.Get("x-sha-256")))77+ if claimed != "" && !shaPattern.MatchString(claimed) {78+ s.fail(w, http.StatusBadRequest, errors.New("invalid: X-SHA-256 must be lowercase hex"))79+ return80+ }81+ blob, created, err := s.putValidated(r.Context(), r.Body, contentType(r.Header.Get("content-type")), pubkey, claimed, func(actual string) error {82+ if s.config.ValidateUpload != nil {83+ return s.config.ValidateUpload(r, actual)84+ }85+ return nil86+ })87+ if err != nil {88+ s.fail(w, statusFor(err), err)89+ return90+ }91+ _ = created92+ s.writeJSON(w, http.StatusOK, s.descriptor(r, blob))93+}94+95+func (s *Service) uploadHead(w http.ResponseWriter, r *http.Request) {96+ sha := strings.ToLower(strings.TrimSpace(r.Header.Get("x-sha-256")))97+ if !shaPattern.MatchString(sha) {98+ s.fail(w, http.StatusBadRequest, errors.New("invalid: X-SHA-256 must be lowercase hex"))99+ return100+ }101+ rawLength := strings.TrimSpace(r.Header.Get("x-content-length"))102+ if rawLength == "" {103+ s.fail(w, http.StatusLengthRequired, errors.New("invalid: X-Content-Length is required"))104+ return105+ }106+ length, err := strconv.ParseInt(rawLength, 10, 64)107+ if err != nil || length < 0 {108+ s.fail(w, http.StatusBadRequest, errors.New("invalid: X-Content-Length must be a nonnegative integer"))109+ return110+ }111+ pubkey, err := s.authorize(r, ActionUpload)112+ if err != nil {113+ s.fail(w, http.StatusUnauthorized, err)114+ return115+ }116+ typ := r.Header.Get("x-content-type")117+ if typ == "" {118+ typ = r.Header.Get("content-type")119+ }120+ if strings.TrimSpace(typ) == "" {121+ s.fail(w, http.StatusBadRequest, errors.New("invalid: X-Content-Type is required"))122+ return123+ }124+ if existing, lookupErr := s.lookup(r.Context(), sha); lookupErr == nil && existing.SHA256 != "" && s.hasClaim(r.Context(), sha, pubkey) {125+ w.WriteHeader(http.StatusOK)126+ return127+ }128+ limits := s.currentLimits()129+ if limits.MaxFileBytes > 0 && length > limits.MaxFileBytes {130+ s.fail(w, http.StatusRequestEntityTooLarge, ErrFileTooLarge)131+ return132+ }133+ // HEAD is advisory for deduplicated uploads, whose final physical size is134+ // already present, but it should still reflect the current quota budget.135+ if limits.UserStorageBytes > 0 {136+ if used, quotaErr := s.quotaUsage(r.Context(), pubkey); quotaErr != nil {137+ s.fail(w, http.StatusInternalServerError, quotaErr)138+ return139+ } else if length > limits.UserStorageBytes-used {140+ s.fail(w, http.StatusForbidden, ErrQuotaExceeded)141+ return142+ }143+ }144+ if existing, err := s.lookup(r.Context(), sha); err == nil && existing.SHA256 != "" {145+ w.WriteHeader(http.StatusOK)146+ return147+ }148+ w.WriteHeader(http.StatusOK)149+}150+151+func (s *Service) blob(w http.ResponseWriter, r *http.Request) {152+ path := r.URL.Path153+ if strings.HasPrefix(path, "/nip96/") {154+ path = strings.TrimPrefix(path, "/nip96")155+ }156+ match := blobPathPattern.FindStringSubmatch(path)157+ if match == nil {158+ s.fail(w, http.StatusNotFound, errors.New("not found"))159+ return160+ }161+ sha := match[1]162+ if r.Method == http.MethodDelete {163+ s.delete(w, r, sha)164+ return165+ }166+ if r.Method != http.MethodGet && r.Method != http.MethodHead {167+ s.fail(w, http.StatusMethodNotAllowed, errors.New("method not allowed"))168+ return169+ }170+ if s.config.CanRead != nil {171+ pubkey, err := s.authorize(r, ActionGet)172+ if err != nil {173+ s.fail(w, http.StatusUnauthorized, err)174+ return175+ }176+ if !s.config.CanRead(r.Context(), sha, []string{pubkey}) {177+ s.fail(w, http.StatusForbidden, errors.New("restricted: read access denied"))178+ return179+ }180+ }181+ entry, err := s.lookup(r.Context(), sha)182+ if errors.Is(err, sql.ErrNoRows) {183+ s.fail(w, http.StatusNotFound, errors.New("not found"))184+ return185+ }186+ if err != nil {187+ s.fail(w, http.StatusInternalServerError, err)188+ return189+ }190+ if s.blocked(r.Context(), sha) {191+ s.fail(w, http.StatusForbidden, ErrBlocked)192+ return193+ }194+ file, err := os.Open(filepath.Join(s.root, sha))195+ if err != nil {196+ s.fail(w, http.StatusNotFound, errors.New("not found"))197+ return198+ }199+ defer file.Close()200+ w.Header().Set("etag", `"`+sha+`"`)201+ w.Header().Set("Content-Type", entry.Type)202+ w.Header().Set("x-content-type-options", "nosniff")203+ w.Header().Set("accept-ranges", "bytes")204+ w.Header().Set("content-security-policy", "default-src 'none'")205+ if s.config.CanRead == nil {206+ w.Header().Set("cache-control", "public, max-age=31536000, immutable")207+ } else {208+ w.Header().Set("cache-control", "private, no-store")209+ }210+ http.ServeContent(w, r, sha, time.Unix(entry.Uploaded, 0), file)211+}212+213+func (s *Service) delete(w http.ResponseWriter, r *http.Request, sha string) {214+ pubkey, err := s.authorize(r, ActionDelete)215+ if err != nil {216+ s.fail(w, http.StatusUnauthorized, err)217+ return218+ }219+ _, err = s.lookup(r.Context(), sha)220+ if errors.Is(err, sql.ErrNoRows) {221+ s.fail(w, http.StatusNotFound, errors.New("not found"))222+ return223+ }224+ if err != nil {225+ s.fail(w, http.StatusInternalServerError, err)226+ return227+ }228+ if !s.hasClaim(r.Context(), sha, pubkey) && !s.isOwner(pubkey) {229+ s.fail(w, http.StatusForbidden, errors.New("restricted: not the uploader"))230+ return231+ }232+ if s.isOwner(pubkey) {233+ if err := s.Delete(r.Context(), sha); err != nil {234+ s.fail(w, http.StatusInternalServerError, err)235+ return236+ }237+ } else if err := s.DeleteForUploader(r.Context(), sha, pubkey); err != nil {238+ if errors.Is(err, sql.ErrNoRows) {239+ s.fail(w, http.StatusNotFound, errors.New("not found"))240+ return241+ }242+ s.fail(w, http.StatusInternalServerError, err)243+ return244+ }245+ // Owners and uploaders may remove their claim; a shared physical object is246+ // retained until its final claim disappears.247+ w.WriteHeader(http.StatusNoContent)248+}249+250+func (s *Service) list(w http.ResponseWriter, r *http.Request, pubkey string) {251+ if !shaPattern.MatchString(pubkey) {252+ s.fail(w, http.StatusBadRequest, errors.New("invalid: bad pubkey"))253+ return254+ }255+ if _, err := s.authorize(r, ActionList); err != nil {256+ s.fail(w, http.StatusUnauthorized, err)257+ return258+ }259+ limit, err := parseListLimit(r.URL.Query().Get("limit"))260+ if err != nil {261+ s.fail(w, http.StatusBadRequest, err)262+ return263+ }264+ entries, _, err := s.ListPage(r.Context(), pubkey, limit, strings.TrimSpace(r.URL.Query().Get("cursor")))265+ if err != nil {266+ if errors.Is(err, sql.ErrNoRows) {267+ s.fail(w, http.StatusBadRequest, errors.New("invalid: cursor does not identify a blob"))268+ return269+ }270+ s.fail(w, http.StatusInternalServerError, err)271+ return272+ }273+ descriptors := make([]map[string]any, 0, len(entries))274+ for _, b := range entries {275+ descriptors = append(descriptors, s.descriptor(r, b))276+ }277+ s.writeJSON(w, http.StatusOK, descriptors)278+}279+280+func (s *Service) report(w http.ResponseWriter, r *http.Request) {281+ reporter, err := s.authorize(r, ActionReport)282+ if err != nil {283+ s.fail(w, http.StatusUnauthorized, err)284+ return285+ }286+ var input event.Event287+ if err := json.NewDecoder(r.Body).Decode(&input); err != nil || input.Kind != 1984 || input.PubKey != reporter {288+ s.fail(w, http.StatusBadRequest, errors.New("invalid: report must be a signed kind 1984 event"))289+ return290+ }291+ if err := event.Validate(input); err != nil || time.Since(time.Unix(input.CreatedAt, 0)) > time.Hour || time.Until(time.Unix(input.CreatedAt, 0)) > time.Hour {292+ s.fail(w, http.StatusBadRequest, errors.New("invalid: report is not a current signed kind 1984 event"))293+ return294+ }295+ filed := 0296+ type reportTarget struct {297+ hash string298+ uploader string299+ }300+ targets := make([]reportTarget, 0, len(input.Tags))301+ for _, tag := range input.Tags {302+ if len(tag) < 2 || tag[0] != "x" || !shaPattern.MatchString(tag[1]) {303+ continue304+ }305+ entry, lookupErr := s.lookup(r.Context(), tag[1])306+ if lookupErr != nil {307+ continue308+ }309+ id := input.ID310+ if filed > 0 {311+ id += "." + strconv.Itoa(filed)312+ }313+ targets = append(targets, reportTarget{hash: tag[1], uploader: entry.Uploader})314+ }315+ if len(targets) == 0 {316+ s.fail(w, http.StatusNotFound, errors.New("not found: relay holds none of the reported blobs"))317+ return318+ }319+ err = s.store.WithTx(r.Context(), func(tx *sql.Tx) error {320+ for i, target := range targets {321+ id := input.ID322+ if i > 0 {323+ id += "." + strconv.Itoa(i)324+ }325+ result, insertErr := tx.ExecContext(r.Context(), `INSERT OR IGNORE INTO blob_reports(id,reporter,target_pubkey,target_blob,type,content,at) VALUES(?,?,?,?,?,?,?)`, id, reporter, target.uploader, target.hash, "blob", input.Content, input.CreatedAt)326+ if insertErr != nil {327+ return insertErr328+ }329+ changed, rowsErr := result.RowsAffected()330+ if rowsErr != nil {331+ return rowsErr332+ }333+ if changed == 0 {334+ continue335+ }336+ if s.config.RecordReport != nil {337+ if err := s.config.RecordReport(r.Context(), tx, reporter, target.hash, "blob", "blob", input.Content); err != nil {338+ return err339+ }340+ }341+ filed++342+ }343+ return nil344+ })345+ if err != nil {346+ s.fail(w, http.StatusInternalServerError, err)347+ return348+ }349+ if filed == 0 {350+ // Repeating an identical report is idempotent and still successful.351+ filed = len(targets)352+ }353+ s.writeJSON(w, http.StatusOK, map[string]any{"ok": true, "filed": filed})354+}355+356+// Block permanently prevents a hash from returning through any upload door.357+func (s *Service) Block(ctx context.Context, sha, reason string) error {358+ if !shaPattern.MatchString(sha) {359+ return errors.New("invalid: bad blob sha256")360+ }361+ _, err := s.store.DB().ExecContext(ctx, "INSERT INTO blob_blocks(sha256,reason,blocked_at) VALUES(?,?,?) ON CONFLICT(sha256) DO UPDATE SET reason=excluded.reason,blocked_at=excluded.blocked_at", sha, reason, time.Now().UTC().Unix())362+ if err != nil {363+ return fmt.Errorf("block blob: %w", err)364+ }365+ if _, err := s.store.DB().ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {366+ return fmt.Errorf("record blocked blob deletion: %w", err)367+ }368+ if _, err := s.store.DB().ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {369+ return fmt.Errorf("remove blocked blob metadata: %w", err)370+ }371+ if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {372+ return fmt.Errorf("remove blocked blob: %w", err)373+ }374+ return nil375+}376+377+func (s *Service) mirror(w http.ResponseWriter, r *http.Request) {378+ pubkey, err := s.authorize(r, ActionMirror)379+ if err != nil {380+ s.fail(w, http.StatusUnauthorized, err)381+ return382+ }383+ var input struct {384+ URL string `json:"url"`385+ }386+ if err := json.NewDecoder(r.Body).Decode(&input); err != nil || input.URL == "" {387+ s.fail(w, http.StatusBadRequest, errors.New("invalid: body needs a url"))388+ return389+ }390+ requestedURL := input.URL391+ expectedHash := ""392+ extension := ""393+ uriServers := []string{}394+ authors := []string{}395+ if strings.HasPrefix(strings.ToLower(requestedURL), "blossom:") {396+ uri, err := ParseBlossomURI(requestedURL)397+ if err != nil {398+ s.fail(w, http.StatusBadRequest, err)399+ return400+ }401+ expectedHash, extension = uri.Hash, uri.Extension402+ uriServers = append(uriServers, uri.Servers...)403+ authors = append(authors, uri.Authors...)404+ requestedURL = ""405+ } else if hash := BlobHashFromURL(requestedURL); hash != "" {406+ expectedHash = hash407+ }408+ client := s.config.HTTPClient409+ if client == nil {410+ client = s.safeClient()411+ }412+ candidates := []string{}413+ if requestedURL != "" {414+ candidates = append(candidates, requestedURL)415+ }416+ candidates = append(candidates, FallbackURLs(uriServers, expectedHash, extension)...)417+ if s.config.ResolveServers != nil && expectedHash != "" {418+ if len(authors) == 0 && pubkey != "" {419+ authors = []string{pubkey}420+ }421+ for _, author := range uniqueStrings(authors) {422+ servers, resolveErr := s.config.ResolveServers(r.Context(), author)423+ if resolveErr != nil {424+ continue425+ }426+ candidates = append(candidates, FallbackURLs(servers, expectedHash, extension)...)427+ }428+ }429+ var lastErr error430+ for _, candidate := range uniqueStrings(candidates) {431+ if err := validateMirrorURL(candidate, s.resolve); err != nil {432+ lastErr = err433+ continue434+ }435+ response, fetchErr := client.Get(candidate)436+ if fetchErr != nil {437+ lastErr = fetchErr438+ continue439+ }440+ if response.StatusCode < 200 || response.StatusCode >= 300 {441+ lastErr = fmt.Errorf("error: origin answered %d", response.StatusCode)442+ _ = response.Body.Close()443+ continue444+ }445+ typ := contentType(response.Header.Get("content-type"))446+ if typ == "application/octet-stream" {447+ typ = typeForExtension(candidate)448+ }449+ entry, created, putErr := s.putValidated(r.Context(), response.Body, typ, pubkey, expectedHash, func(actual string) error {450+ if s.config.ValidateUpload != nil {451+ return s.config.ValidateUpload(r, actual)452+ }453+ return nil454+ })455+ _ = response.Body.Close()456+ if putErr == nil {457+ s.writeJSON(w, chooseStatus(created, http.StatusCreated), s.descriptor(r, entry))458+ return459+ }460+ lastErr = putErr461+ if !errors.Is(putErr, ErrHashMismatch) {462+ s.fail(w, statusFor(putErr), putErr)463+ return464+ }465+ }466+ if lastErr == nil {467+ lastErr = errors.New("error: no mirror origin available")468+ }469+ s.fail(w, statusFor(lastErr), lastErr)470+}471+472+func (s *Service) nip96(w http.ResponseWriter, r *http.Request) {473+ if r.Method == http.MethodGet {474+ pubkey, err := s.authorize(r, ActionList)475+ if err != nil {476+ s.fail(w, http.StatusUnauthorized, err)477+ return478+ }479+ s.nip96List(w, r, pubkey)480+ return481+ }482+ if r.Method != http.MethodPost {483+ s.fail(w, http.StatusMethodNotAllowed, errors.New("method not allowed"))484+ return485+ }486+ pubkey, err := s.authorize(r, ActionUpload)487+ if err != nil {488+ s.fail(w, http.StatusUnauthorized, err)489+ return490+ }491+ if err := r.ParseMultipartForm(0); err != nil {492+ s.fail(w, http.StatusBadRequest, errors.New("invalid: body is not multipart form data"))493+ return494+ }495+ defer r.MultipartForm.RemoveAll()496+ file, header, err := r.FormFile("file")497+ if err != nil {498+ s.fail(w, http.StatusBadRequest, errors.New("invalid: form needs a file field"))499+ return500+ }501+ defer file.Close()502+ typ := contentType(header.Header.Get("Content-Type"))503+ if typ == "application/octet-stream" {504+ typ = contentType(r.FormValue("content_type"))505+ }506+ entry, created, err := s.put(r.Context(), file, typ, pubkey, "")507+ if err != nil {508+ s.fail(w, statusFor(err), err)509+ return510+ }511+ s.writeJSON(w, chooseStatus(created, http.StatusCreated), map[string]any{"status": "success", "message": "Upload successful.", "url": s.blobURL(r, entry), "nip94_event": map[string]any{"tags": s.nip94Tags(r, entry), "content": r.FormValue("caption")}})512+}513+514+func (s *Service) nip96List(w http.ResponseWriter, r *http.Request, pubkey string) {515+ count := parsePageCount(r.URL.Query().Get("count"))516+ page := parsePage(r.URL.Query().Get("page"))517+ var total int518+ if err := s.store.DB().QueryRowContext(r.Context(), "SELECT count(*) FROM blob_claims WHERE uploader=?", pubkey).Scan(&total); err != nil {519+ s.fail(w, http.StatusInternalServerError, err)520+ return521+ }522+ rows, err := s.store.DB().QueryContext(r.Context(), `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded523+ FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?524+ ORDER BY b.uploaded DESC,b.sha256 DESC LIMIT ? OFFSET ?`, pubkey, count, page*count)525+ if err != nil {526+ s.fail(w, http.StatusInternalServerError, err)527+ return528+ }529+ defer rows.Close()530+ files := make([]map[string]any, 0)531+ for rows.Next() {532+ var b Blob533+ if err := rows.Scan(&b.SHA256, &b.Size, &b.Type, &b.Uploader, &b.Uploaded); err != nil {534+ s.fail(w, http.StatusInternalServerError, err)535+ return536+ }537+ files = append(files, map[string]any{"tags": s.nip94Tags(r, b), "content": "", "created_at": b.Uploaded})538+ }539+ s.writeJSON(w, http.StatusOK, map[string]any{"count": count, "total": total, "page": page, "files": files})540+}541+542+func (s *Service) authorize(r *http.Request, action Action) (string, error) {543+ if s.config.Authorize == nil {544+ return "", errors.New("auth-required: authorization is required")545+ }546+ return s.config.Authorize(r, action)547+}548+func (s *Service) isOwner(pubkey string) bool {549+ return s.config.IsOwner != nil && s.config.IsOwner(pubkey)550+}551+func (s *Service) descriptor(r *http.Request, b Blob) map[string]any {552+ return map[string]any{"url": s.blobURL(r, b), "sha256": b.SHA256, "size": b.Size, "type": b.Type, "uploaded": b.Uploaded, "nip94": s.nip94Tags(r, b)}553+}554+func (s *Service) nip94Tags(r *http.Request, b Blob) [][]string {555+ return [][]string{{"url", s.blobURL(r, b)}, {"m", b.Type}, {"x", b.SHA256}, {"ox", b.SHA256}, {"size", strconv.FormatInt(b.Size, 10)}}556+}557+func (s *Service) blobURL(r *http.Request, b Blob) string {558+ return s.origin(r) + "/" + b.SHA256 + extensionSuffix(b.Type)559+}560+func (s *Service) nip96Info(r *http.Request) map[string]any {561+ return map[string]any{"api_url": s.origin(r) + "/nip96", "download_url": s.origin(r), "supported_nips": []int{94, 98}, "content_types": contentTypes(), "plans": map[string]any{"self-hosted": map[string]any{"name": "Self-hosted", "is_nip98_required": true, "max_byte_size": 0, "file_expiration": []int{0, 0}, "media_transformations": map[string]any{}}}}562+}563+func (s *Service) writeJSON(w http.ResponseWriter, status int, value any) {564+ w.Header().Set("content-type", "application/json")565+ w.WriteHeader(status)566+ _ = json.NewEncoder(w).Encode(value)567+}568+func (s *Service) fail(w http.ResponseWriter, status int, err error) {569+ w.Header().Set("x-reason", err.Error())570+ s.writeJSON(w, status, map[string]string{"error": err.Error()})571+}572+func origin(r *http.Request) string {573+ scheme := r.URL.Scheme574+ if scheme == "" {575+ scheme = "http"576+ if r.TLS != nil {577+ scheme = "https"578+ }579+ }580+ return scheme + "://" + r.Host581+}582+583+func (s *Service) origin(r *http.Request) string {584+ if s.config.PublicURL != "" {585+ return strings.TrimRight(s.config.PublicURL, "/")586+ }587+ return origin(r)588+}@@ -0,0 +1,197 @@
1+package blob2+3+import (4+ "context"5+ "database/sql"6+ "errors"7+ "fmt"8+ "github.com/FelineStateMachine/tinyrelay/internal/storage"9+ "io"10+ "net"11+ "net/http"12+ "os"13+ "path/filepath"14+ "regexp"15+ "strings"16+ "sync"17+)18+19+type Action string20+21+const (22+ ActionUpload Action = "upload"23+ ActionGet Action = "get"24+ ActionDelete Action = "delete"25+ ActionList Action = "list"26+ ActionMirror Action = "mirror"27+ ActionReport Action = "report"28+)29+30+type Authorize func(*http.Request, Action) (string, error)31+type CanRead func(context.Context, string, []string) bool32+type ResolveIP func(context.Context, string) ([]net.IP, error)33+type ReportRecorder func(context.Context, *sql.Tx, string, string, string, string, string) error34+35+type Config struct {36+ Root string37+ PublicURL string38+ Store *storage.Store39+ Authorize Authorize40+ CanRead CanRead41+ HTTPClient *http.Client42+ ResolveIP ResolveIP43+ IsOwner func(string) bool44+ ValidateUpload func(*http.Request, string) error45+ // ResolveServers returns the ordered BUD-03 servers for an authorized46+ // caller. It is used only for Blossom URI mirror fallback.47+ ResolveServers func(context.Context, string) ([]string, error)48+ // RecordReport joins native blob reporting to the host moderation ledger.49+ // It runs inside the blob report transaction.50+ RecordReport ReportRecorder51+ // Limits is read for each upload so policy changes apply without reopening52+ // the blob service. Zero values mean unlimited.53+ Limits func() Limits54+ // UploadTerms returns what an uploader's standing imposes on its uploads:55+ // an agent grant with a sites ttl gives its claims an expiry, and one with56+ // the encrypted flag refuses plain uploads. Zero values mean no terms.57+ UploadTerms func(context.Context, string) (UploadTerms, error)58+}59+60+type Limits struct {61+ MaxFileBytes int6462+ UserStorageBytes int6463+}64+65+// UploadTerms is what an uploader's grant imposes on the blobs it stores.66+// ExpiresAt is the unix time its claims lapse, 0 for never; Encrypted67+// requires the stored bytes to be an encrypted blob or manifest.68+type UploadTerms struct {69+ ExpiresAt int6470+ Encrypted bool71+}72+73+// encryptedTypes are the content types the browser uploader gives encrypted74+// objects: AES-GCM ciphertext for files and chunks, and the BUD-16 directory75+// type for encrypted manifests. Anything the relay recognizes as a page,76+// image, video, audio or document is plain.77+var encryptedTypes = map[string]bool{"application/octet-stream": true, "application/vnd.blossom.directory+msgpack": true}78+79+type Blob struct {80+ SHA256 string `json:"sha256"`81+ Size int64 `json:"size"`82+ Type string `json:"type"`83+ Uploader string `json:"uploader"`84+ Uploaded int64 `json:"uploaded"`85+}86+87+type PutOptions struct {88+ Reader io.Reader89+ Type string90+ Uploader string91+ Hash string92+ // Commit records ownership metadata in the blob transaction. The final93+ // argument reports whether the blob is new. An error rolls back all claims.94+ // New bytes remain unavailable until their ownership metadata is durable.95+ Commit func(context.Context, *sql.Tx, Blob, bool) error96+}97+98+type Service struct {99+ config Config100+ root string101+ store *storage.Store102+ resolve ResolveIP103+ quotaMu sync.Mutex104+ // multipartMu protects partial files and their range metadata. Lock it105+ // before quotaMu; network bodies are never read while either is held.106+ multipartMu sync.Mutex107+ activeParts map[string]int108+ // multipartFinalizing prevents another chunk from changing a complete109+ // session while its assembled file is being hashed. Hashing is deliberately110+ // outside multipartMu because the upload may be as large as 1 GiB.111+ multipartFinalizing map[string]bool112+ // verifyMultipartHook is used by package tests to pause finalization113+ // without allocating a GiB fixture. It is nil in production.114+ verifyMultipartHook func(context.Context, string, string, int64) error115+}116+117+var ErrBlocked = errors.New("blocked: this blob was removed by a moderator")118+var ErrHashMismatch = errors.New("conflict: mirrored blob hash does not match the requested hash")119+var ErrFileTooLarge = errors.New("invalid: blob exceeds the per-file size limit")120+var ErrQuotaExceeded = errors.New("restricted: uploader storage quota exceeded")121+var ErrMultipartConflict = errors.New("conflict: upload metadata differs from existing session")122+var ErrNotEncrypted = errors.New("restricted: agent grant requires encrypted uploads")123+124+var shaPattern = regexp.MustCompile(`^[0-9a-f]{64}$`)125+var blobPathPattern = regexp.MustCompile(`^/([0-9a-f]{64})(?:\.[a-z0-9]{1,8})?$`)126+127+// HandlesPath keeps the daemon mount in sync with every supported file door.128+func HandlesPath(path string) bool {129+ return path == "/upload" || path == "/mirror" || path == "/report" ||130+ path == "/nip96" || path == "/.well-known/nostr/nip96.json" ||131+ strings.HasPrefix(path, "/nip96/") || strings.HasPrefix(path, "/list/") ||132+ blobPathPattern.MatchString(path)133+}134+135+var extensions = map[string]string{136+ "image/png": "png", "image/jpeg": "jpg", "image/gif": "gif", "image/webp": "webp", "image/avif": "avif", "image/svg+xml": "svg",137+ "video/mp4": "mp4", "video/webm": "webm", "video/quicktime": "mov", "audio/mpeg": "mp3", "audio/ogg": "ogg", "audio/wav": "wav", "audio/mp4": "m4a",138+ "application/pdf": "pdf", "application/json": "json", "text/plain": "txt", "text/markdown": "md",139+ "application/vnd.blossom.directory+msgpack": "bdir",140+}141+142+func New(ctx context.Context, config Config) (*Service, error) {143+ if config.Root == "" || config.Store == nil {144+ return nil, errors.New("blob root and store are required")145+ }146+ if err := os.MkdirAll(filepath.Join(config.Root, "blobs"), 0o700); err != nil {147+ return nil, fmt.Errorf("create blob root: %w", err)148+ }149+ if _, err := config.Store.DB().ExecContext(ctx, `CREATE TABLE IF NOT EXISTS blobs (150+ sha256 TEXT PRIMARY KEY, size INTEGER NOT NULL, type TEXT NOT NULL,151+ uploader TEXT NOT NULL, uploaded INTEGER NOT NULL152+ ); CREATE TABLE IF NOT EXISTS blob_claims (153+ sha256 TEXT NOT NULL, uploader TEXT NOT NULL, claimed_at INTEGER NOT NULL,154+ PRIMARY KEY (sha256, uploader), FOREIGN KEY (sha256) REFERENCES blobs(sha256) ON DELETE CASCADE155+ ); CREATE INDEX IF NOT EXISTS blobs_uploader ON blobs(uploader, uploaded DESC);156+ CREATE INDEX IF NOT EXISTS blob_claims_uploader ON blob_claims(uploader, claimed_at DESC, sha256 DESC);157+ CREATE TABLE IF NOT EXISTS blob_reports (158+ id TEXT PRIMARY KEY, reporter TEXT NOT NULL, target_pubkey TEXT NOT NULL DEFAULT '',159+ target_blob TEXT NOT NULL, type TEXT NOT NULL, content TEXT NOT NULL, at INTEGER NOT NULL,160+ status TEXT NOT NULL DEFAULT 'open', resolved_by TEXT NOT NULL DEFAULT '',161+ resolved_at INTEGER NOT NULL DEFAULT 0, action TEXT NOT NULL DEFAULT '',162+ UNIQUE(reporter,target_blob));163+ CREATE TRIGGER IF NOT EXISTS blob_claim_on_insert AFTER INSERT ON blobs164+ WHEN NEW.uploader != '' BEGIN INSERT OR IGNORE INTO blob_claims(sha256,uploader,claimed_at) VALUES(NEW.sha256,NEW.uploader,NEW.uploaded); END;165+ CREATE TABLE IF NOT EXISTS blob_blocks (sha256 TEXT PRIMARY KEY, reason TEXT NOT NULL DEFAULT '', blocked_at INTEGER NOT NULL DEFAULT 0);166+ CREATE TABLE IF NOT EXISTS blob_tombstones (sha256 TEXT PRIMARY KEY, deleted_at INTEGER NOT NULL);167+ CREATE TABLE IF NOT EXISTS multipart_uploads (id TEXT PRIMARY KEY, sha256 TEXT NOT NULL, uploader TEXT NOT NULL, length INTEGER NOT NULL, type TEXT NOT NULL, created INTEGER NOT NULL, last_seen INTEGER NOT NULL);168+ CREATE TABLE IF NOT EXISTS multipart_parts (upload_id TEXT NOT NULL, offset INTEGER NOT NULL, length INTEGER NOT NULL, PRIMARY KEY(upload_id, offset), FOREIGN KEY(upload_id) REFERENCES multipart_uploads(id) ON DELETE CASCADE);169+ INSERT OR IGNORE INTO blob_claims(sha256,uploader,claimed_at)170+ SELECT sha256,uploader,uploaded FROM blobs WHERE uploader != ''`); err != nil {171+ return nil, fmt.Errorf("initialize blob metadata: %w", err)172+ }173+ // Claims made under an agent grant with a ttl carry an expiry; every174+ // earlier claim keeps 0, which never lapses.175+ if _, err := config.Store.DB().ExecContext(ctx, "ALTER TABLE blob_claims ADD COLUMN expires INTEGER NOT NULL DEFAULT 0"); err != nil && !strings.Contains(err.Error(), "duplicate column") {176+ return nil, fmt.Errorf("add blob claim expiry: %w", err)177+ }178+ if _, err := config.Store.DB().ExecContext(ctx, "CREATE INDEX IF NOT EXISTS blob_claims_expires ON blob_claims(expires) WHERE expires>0"); err != nil {179+ return nil, fmt.Errorf("index blob claim expiry: %w", err)180+ }181+ if err := reconcile(ctx, config.Store, filepath.Join(config.Root, "blobs")); err != nil {182+ return nil, err183+ }184+ resolve := config.ResolveIP185+ if resolve == nil {186+ resolve = func(ctx context.Context, host string) ([]net.IP, error) {187+ return net.DefaultResolver.LookupIP(ctx, "ip", host)188+ }189+ }190+ s := &Service{config: config, root: filepath.Join(config.Root, "blobs"), store: config.Store, resolve: resolve, activeParts: make(map[string]int), multipartFinalizing: make(map[string]bool)}191+ if err := s.reconcileMultipart(ctx); err != nil {192+ return nil, err193+ }194+ return s, nil195+}196+197+func (s *Service) Handler() http.HandlerFunc { return s.serveHTTP }@@ -0,0 +1,649 @@
1+package blob2+3+import (4+ "context"5+ "crypto/sha256"6+ "database/sql"7+ "encoding/hex"8+ "errors"9+ "fmt"10+ "io"11+ "os"12+ "path/filepath"13+ "strconv"14+ "strings"15+ "time"16+17+ "github.com/FelineStateMachine/tinyrelay/internal/storage"18+)19+20+// Put stores bytes using their SHA-256 digest as the durable filename.21+func (s *Service) Put(ctx context.Context, options PutOptions) (Blob, error) {22+ if options.Reader == nil {23+ return Blob{}, errors.New("blob reader is required")24+ }25+ entry, _, err := s.putOptions(ctx, options)26+ return entry, err27+}28+29+func (s *Service) putOptions(ctx context.Context, options PutOptions) (Blob, bool, error) {30+ return s.putValidatedWithCommit(ctx, options.Reader, options.Type, options.Uploader, options.Hash, nil, options.Commit)31+}32+33+// Get opens a verified metadata-backed blob for streaming to another service.34+func (s *Service) Get(ctx context.Context, sha string) (Blob, io.ReadCloser, error) {35+ if !shaPattern.MatchString(sha) {36+ return Blob{}, nil, errors.New("invalid: bad blob sha256")37+ }38+ entry, err := s.lookup(ctx, sha)39+ if err != nil {40+ return Blob{}, nil, err41+ }42+ if s.blocked(ctx, sha) {43+ return Blob{}, nil, ErrBlocked44+ }45+ file, err := os.Open(filepath.Join(s.root, sha))46+ if err != nil {47+ return Blob{}, nil, err48+ }49+ return entry, file, nil50+}51+52+// ListMetadata returns all blobs uploaded by the supplied pubkey.53+func (s *Service) ListMetadata(ctx context.Context, uploader string) ([]Blob, error) {54+ rows, err := s.store.DB().QueryContext(ctx, `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded55+ FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?56+ ORDER BY b.uploaded DESC,b.sha256 DESC`, uploader)57+ if err != nil {58+ return nil, fmt.Errorf("list blob metadata: %w", err)59+ }60+ defer rows.Close()61+ entries := make([]Blob, 0)62+ for rows.Next() {63+ var entry Blob64+ if err := rows.Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded); err != nil {65+ return nil, fmt.Errorf("scan blob metadata: %w", err)66+ }67+ entries = append(entries, entry)68+ }69+ return entries, rows.Err()70+}71+72+// ListPage returns a BUD-12 cursor page. The cursor is the last blob hash73+// returned by the previous page and is intentionally opaque to callers.74+func (s *Service) ListPage(ctx context.Context, uploader string, limit int, cursor string) ([]Blob, string, error) {75+ if limit < 1 {76+ limit = 2077+ }78+ if limit > 100 {79+ limit = 10080+ }81+ query := `SELECT b.sha256,b.size,b.type,c.uploader,b.uploaded FROM blobs b82+ JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?`83+ args := []any{uploader}84+ if cursor != "" {85+ var uploaded int6486+ if err := s.store.DB().QueryRowContext(ctx, `SELECT b.uploaded FROM blobs b87+ JOIN blob_claims c ON c.sha256=b.sha256 WHERE b.sha256=? AND c.uploader=?`, cursor, uploader).Scan(&uploaded); err != nil {88+ return nil, "", fmt.Errorf("resolve blob cursor: %w", err)89+ }90+ query += " AND (b.uploaded < ? OR (b.uploaded = ? AND b.sha256 < ?))"91+ args = append(args, uploaded, uploaded, cursor)92+ }93+ query += " ORDER BY b.uploaded DESC,b.sha256 DESC LIMIT ?"94+ args = append(args, limit+1)95+ rows, err := s.store.DB().QueryContext(ctx, query, args...)96+ if err != nil {97+ return nil, "", fmt.Errorf("list blob page: %w", err)98+ }99+ defer rows.Close()100+ entries := make([]Blob, 0, limit)101+ for rows.Next() {102+ var entry Blob103+ if err := rows.Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded); err != nil {104+ return nil, "", fmt.Errorf("scan blob page: %w", err)105+ }106+ entries = append(entries, entry)107+ }108+ if err := rows.Err(); err != nil {109+ return nil, "", fmt.Errorf("read blob page: %w", err)110+ }111+ next := ""112+ if len(entries) > limit {113+ next = entries[limit-1].SHA256114+ entries = entries[:limit]115+ }116+ return entries, next, nil117+}118+119+// Delete removes a blob's metadata and content. Authorization is the caller's responsibility.120+func (s *Service) Delete(ctx context.Context, sha string) error {121+ if !shaPattern.MatchString(sha) {122+ return errors.New("invalid: bad blob sha256")123+ }124+ if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {125+ if _, err := tx.ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {126+ return fmt.Errorf("record blob deletion: %w", err)127+ }128+ if _, err := tx.ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {129+ return fmt.Errorf("delete blob metadata: %w", err)130+ }131+ return nil132+ }); err != nil {133+ return err134+ }135+ if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {136+ return fmt.Errorf("delete blob content: %w", err)137+ }138+ return nil139+}140+141+// DeleteForUploader removes one owner's claim. The physical object remains142+// while another owner still references it, which makes deduplicated uploads143+// safe to manage independently.144+func (s *Service) DeleteForUploader(ctx context.Context, sha, uploader string) error {145+ if !shaPattern.MatchString(sha) {146+ return errors.New("invalid: bad blob sha256")147+ }148+ removed := false149+ if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {150+ result, err := tx.ExecContext(ctx, "DELETE FROM blob_claims WHERE sha256=? AND uploader=?", sha, uploader)151+ if err != nil {152+ return fmt.Errorf("delete blob claim: %w", err)153+ }154+ changed, err := result.RowsAffected()155+ if err != nil {156+ return fmt.Errorf("check blob claim deletion: %w", err)157+ }158+ if changed == 0 {159+ return sql.ErrNoRows160+ }161+ var remaining int162+ if err := tx.QueryRowContext(ctx, "SELECT count(*) FROM blob_claims WHERE sha256=?", sha).Scan(&remaining); err != nil {163+ return fmt.Errorf("count blob claims: %w", err)164+ }165+ if remaining == 0 {166+ removed = true167+ if _, err := tx.ExecContext(ctx, "INSERT INTO blob_tombstones(sha256,deleted_at) VALUES(?,?) ON CONFLICT(sha256) DO UPDATE SET deleted_at=excluded.deleted_at", sha, time.Now().UTC().Unix()); err != nil {168+ return fmt.Errorf("record blob deletion: %w", err)169+ }170+ if _, err := tx.ExecContext(ctx, "DELETE FROM blobs WHERE sha256=?", sha); err != nil {171+ return fmt.Errorf("delete blob metadata: %w", err)172+ }173+ }174+ return nil175+ }); err != nil {176+ return err177+ }178+ if removed {179+ if err := os.Remove(filepath.Join(s.root, sha)); err != nil && !errors.Is(err, os.ErrNotExist) {180+ return fmt.Errorf("delete blob content: %w", err)181+ }182+ }183+ return nil184+}185+186+// Descriptor returns the Blossom descriptor and NIP-94 tags for a blob.187+func (s *Service) Descriptor(baseURL string, entry Blob) map[string]any {188+ return map[string]any{189+ "url": baseURL + "/" + entry.SHA256 + extensionSuffix(entry.Type),190+ "sha256": entry.SHA256,191+ "size": entry.Size,192+ "type": entry.Type,193+ "uploaded": entry.Uploaded,194+ "nip94": [][]string{{"url", baseURL + "/" + entry.SHA256 + extensionSuffix(entry.Type)}, {"m", entry.Type}, {"x", entry.SHA256}, {"ox", entry.SHA256}, {"size", strconv.FormatInt(entry.Size, 10)}},195+ }196+}197+func (s *Service) put(ctx context.Context, source io.Reader, typ, uploader, claimed string) (Blob, bool, error) {198+ return s.putValidatedWithCommit(ctx, source, typ, uploader, claimed, nil, nil)199+}200+201+func (s *Service) putValidated(ctx context.Context, source io.Reader, typ, uploader, claimed string, validate func(string) error) (Blob, bool, error) {202+ return s.putValidatedWithCommit(ctx, source, typ, uploader, claimed, validate, nil)203+}204+205+func (s *Service) putValidatedWithCommit(ctx context.Context, source io.Reader, typ, uploader, claimed string, validate func(string) error, commit func(context.Context, *sql.Tx, Blob, bool) error) (Blob, bool, error) {206+ limits := s.currentLimits()207+ streamLimit := limits.MaxFileBytes208+ quotaStream := false209+ if streamLimit == 0 && limits.UserStorageBytes > 0 && uploader != "" {210+ streamLimit = limits.UserStorageBytes211+ quotaStream = true212+ }213+ if streamLimit > 0 && streamLimit < int64(^uint64(0)>>1) {214+ source = io.LimitReader(source, streamLimit+1)215+ }216+ tmp, err := os.CreateTemp(s.root, ".upload-")217+ if err != nil {218+ return Blob{}, false, fmt.Errorf("create upload temporary file: %w", err)219+ }220+ tmpName := tmp.Name()221+ defer os.Remove(tmpName)222+ hasher := sha256.New()223+ count, err := io.Copy(io.MultiWriter(tmp, hasher), source)224+ if err != nil {225+ _ = tmp.Close()226+ return Blob{}, false, fmt.Errorf("read upload: %w", err)227+ }228+ if limits.MaxFileBytes > 0 && count > limits.MaxFileBytes {229+ _ = tmp.Close()230+ return Blob{}, false, ErrFileTooLarge231+ }232+ if quotaStream && count > streamLimit {233+ _ = tmp.Close()234+ return Blob{}, false, ErrQuotaExceeded235+ }236+ if err := tmp.Sync(); err != nil {237+ _ = tmp.Close()238+ return Blob{}, false, fmt.Errorf("sync upload: %w", err)239+ }240+ if err := tmp.Close(); err != nil {241+ return Blob{}, false, fmt.Errorf("close upload: %w", err)242+ }243+ sha := hex.EncodeToString(hasher.Sum(nil))244+ if claimed != "" && claimed != sha {245+ return Blob{}, false, ErrHashMismatch246+ }247+ if validate != nil {248+ if err := validate(sha); err != nil {249+ return Blob{}, false, err250+ }251+ }252+ return s.installUpload(ctx, uploadCandidate{path: tmpName, hash: sha, size: count, typ: typ, uploader: uploader, commit: commit})253+}254+255+type uploadCandidate struct {256+ path, hash, typ, uploader, reservation string257+ size int64258+ commit func(context.Context, *sql.Tx, Blob, bool) error259+}260+261+func (s *Service) installUpload(ctx context.Context, candidate uploadCandidate) (Blob, bool, error) {262+ sha, count, typ, uploader := candidate.hash, candidate.size, candidate.typ, candidate.uploader263+ // Serialize the lookup, quota check, installation and metadata transaction.264+ // This closes the race where concurrent uploads could each observe the same265+ // remaining quota before one of them installs a new physical object.266+ s.quotaMu.Lock()267+ defer s.quotaMu.Unlock()268+ if s.blocked(ctx, sha) {269+ return Blob{}, false, ErrBlocked270+ }271+ terms, err := s.uploadTerms(ctx, uploader)272+ if err != nil {273+ return Blob{}, false, err274+ }275+ limits := s.currentLimits()276+ if limits.MaxFileBytes > 0 && count > limits.MaxFileBytes {277+ return Blob{}, false, ErrFileTooLarge278+ }279+ if limits.UserStorageBytes > 0 && uploader != "" && count > limits.UserStorageBytes {280+ return Blob{}, false, ErrQuotaExceeded281+ }282+ if existing, err := s.lookup(ctx, sha); err == nil {283+ if terms.Encrypted && !encryptedTypes[existing.Type] {284+ return Blob{}, false, ErrNotEncrypted285+ }286+ if uploader != "" {287+ if limits.UserStorageBytes > 0 && !s.hasClaim(ctx, sha, uploader) {288+ used, usageErr := s.quotaUsageExcept(ctx, uploader, candidate.reservation)289+ if usageErr != nil {290+ return Blob{}, false, fmt.Errorf("check uploader storage quota: %w", usageErr)291+ }292+ if existing.Size > limits.UserStorageBytes-used {293+ return Blob{}, false, ErrQuotaExceeded294+ }295+ }296+ }297+ if err := s.store.WithTx(ctx, func(tx *sql.Tx) error {298+ if candidate.commit != nil {299+ if err := candidate.commit(ctx, tx, existing, false); err != nil {300+ return err301+ }302+ }303+ if uploader != "" {304+ if _, err := tx.ExecContext(ctx, claimSQL, sha, uploader, time.Now().Unix(), terms.ExpiresAt); err != nil {305+ return err306+ }307+ }308+ if candidate.reservation != "" {309+ if _, err := tx.ExecContext(ctx, "DELETE FROM multipart_uploads WHERE id=?", candidate.reservation); err != nil {310+ return err311+ }312+ }313+ return nil314+ }); err != nil {315+ return Blob{}, false, err316+ }317+ // A crash after a scoped metadata commit may leave the bytes missing.318+ // A retry supplies those same verified bytes without losing the scope.319+ if _, err := os.Stat(filepath.Join(s.root, sha)); errors.Is(err, os.ErrNotExist) {320+ if err := s.installBlobFile(candidate.path, sha); err != nil {321+ return Blob{}, false, err322+ }323+ } else if err != nil {324+ return Blob{}, false, err325+ }326+ return existing, false, nil327+ } else if !errors.Is(err, sql.ErrNoRows) {328+ return Blob{}, false, err329+ }330+ if limits.UserStorageBytes > 0 && uploader != "" {331+ used, err := s.quotaUsageExcept(ctx, uploader, candidate.reservation)332+ if err != nil {333+ return Blob{}, false, fmt.Errorf("check uploader storage quota: %w", err)334+ }335+ if count > limits.UserStorageBytes-used {336+ return Blob{}, false, ErrQuotaExceeded337+ }338+ }339+ // The type is settled before the bytes are installed so a plain upload340+ // under an encrypted-only grant leaves nothing behind. A declared octet341+ // stream is sniffed: encrypted bytes stay one, a page or an image does not.342+ typ = contentType(typ)343+ if typ == "application/octet-stream" {344+ typ = detectFileType(candidate.path)345+ }346+ if terms.Encrypted && !encryptedTypes[typ] {347+ return Blob{}, false, ErrNotEncrypted348+ }349+ stage := ""350+ if candidate.commit != nil {351+ stage = filepath.Join(s.root, ".scoped-"+sha)352+ if err := os.Rename(candidate.path, stage); err != nil {353+ return Blob{}, false, fmt.Errorf("stage scoped blob: %w", err)354+ }355+ if err := syncDirectory(s.root); err != nil {356+ return Blob{}, false, fmt.Errorf("sync staged blob directory: %w", err)357+ }358+ }359+ if candidate.commit == nil {360+ if err := s.installBlobFile(candidate.path, sha); err != nil {361+ return Blob{}, false, err362+ }363+ }364+ now := time.Now().UTC().Unix()365+ entry := Blob{SHA256: sha, Size: count, Type: typ, Uploader: uploader, Uploaded: now}366+ err = s.store.WithTx(ctx, func(tx *sql.Tx) error {367+ if _, err := tx.ExecContext(ctx, "DELETE FROM blob_tombstones WHERE sha256=?", entry.SHA256); err != nil {368+ return fmt.Errorf("clear blob deletion tombstone: %w", err)369+ }370+ if _, err := tx.ExecContext(ctx, "INSERT OR IGNORE INTO blobs(sha256,size,type,uploader,uploaded) VALUES(?,?,?,?,?)", entry.SHA256, entry.Size, entry.Type, entry.Uploader, entry.Uploaded); err != nil {371+ return fmt.Errorf("record blob: %w", err)372+ }373+ if entry.Uploader != "" {374+ if _, err := tx.ExecContext(ctx, claimSQL, entry.SHA256, entry.Uploader, entry.Uploaded, terms.ExpiresAt); err != nil {375+ return fmt.Errorf("record blob claim: %w", err)376+ }377+ }378+ if candidate.reservation != "" {379+ if _, err := tx.ExecContext(ctx, "DELETE FROM multipart_uploads WHERE id=?", candidate.reservation); err != nil {380+ return fmt.Errorf("release upload reservation: %w", err)381+ }382+ }383+ if candidate.commit != nil {384+ if err := candidate.commit(ctx, tx, entry, true); err != nil {385+ return err386+ }387+ }388+ return nil389+ })390+ if err != nil {391+ if stage != "" {392+ _ = os.Remove(stage)393+ }394+ return Blob{}, false, err395+ }396+ if candidate.commit != nil {397+ if err := s.installBlobFile(stage, sha); err != nil {398+ return Blob{}, false, err399+ }400+ }401+ if saved, err := s.lookup(ctx, sha); err == nil {402+ return saved, saved.Uploader == uploader && saved.Uploaded == now, nil403+ }404+ return entry, true, nil405+}406+407+func (s *Service) installBlobFile(source, hash string) error {408+ if err := os.Rename(source, filepath.Join(s.root, hash)); err != nil {409+ return fmt.Errorf("install blob: %w", err)410+ }411+ if err := syncDirectory(s.root); err != nil {412+ return fmt.Errorf("sync blob directory: %w", err)413+ }414+ return nil415+}416+417+func (s *Service) quotaUsage(ctx context.Context, uploader string) (int64, error) {418+ return s.quotaUsageExcept(ctx, uploader, "")419+}420+421+func (s *Service) quotaUsageExcept(ctx context.Context, uploader, reservation string) (int64, error) {422+ var used sql.NullInt64423+ err := s.store.DB().QueryRowContext(ctx, `SELECT COALESCE((SELECT SUM(b.size)424+ FROM blobs b JOIN blob_claims c ON c.sha256=b.sha256 WHERE c.uploader=?),0) +425+ COALESCE((SELECT SUM(length) FROM multipart_uploads WHERE uploader=? AND id != ?),0)`, uploader, uploader, reservation).Scan(&used)426+ if err != nil {427+ return 0, err428+ }429+ return used.Int64, nil430+}431+432+// claimSQL records a claim. The insert trigger on blobs has usually made the433+// row already, without an expiry, so the newest upload's terms replace it.434+const claimSQL = "INSERT INTO blob_claims(sha256,uploader,claimed_at,expires) VALUES(?,?,?,?) ON CONFLICT(sha256,uploader) DO UPDATE SET expires=excluded.expires"435+436+func (s *Service) claim(ctx context.Context, sha, uploader string, expires int64) error {437+ _, err := s.store.DB().ExecContext(ctx, claimSQL, sha, uploader, time.Now().UTC().Unix(), expires)438+ if err != nil {439+ return fmt.Errorf("record blob claim: %w", err)440+ }441+ return nil442+}443+444+func (s *Service) uploadTerms(ctx context.Context, uploader string) (UploadTerms, error) {445+ if s.config.UploadTerms == nil || uploader == "" {446+ return UploadTerms{}, nil447+ }448+ // A refusal keeps its reason so the door answers with the right status.449+ return s.config.UploadTerms(ctx, uploader)450+}451+452+// SweepExpired removes the blobs whose every claim has lapsed and releases453+// lapsed claims on blobs someone else still holds, so an agent's files leave454+// with its site while a person's claim keeps them. It returns how many blobs455+// were removed.456+func (s *Service) SweepExpired(ctx context.Context, now int64) (int, error) {457+ s.quotaMu.Lock()458+ defer s.quotaMu.Unlock()459+ rows, err := s.store.DB().QueryContext(ctx, `SELECT DISTINCT sha256 FROM blob_claims c WHERE c.expires>0 AND c.expires<=?460+ AND NOT EXISTS(SELECT 1 FROM blob_claims live WHERE live.sha256=c.sha256 AND (live.expires=0 OR live.expires>?))`, now, now)461+ if err != nil {462+ return 0, fmt.Errorf("list expired blobs: %w", err)463+ }464+ var expired []string465+ for rows.Next() {466+ var sha string467+ if err := rows.Scan(&sha); err != nil {468+ _ = rows.Close()469+ return 0, err470+ }471+ expired = append(expired, sha)472+ }473+ if err := errors.Join(rows.Err(), rows.Close()); err != nil {474+ return 0, err475+ }476+ for _, sha := range expired {477+ if err := s.Delete(ctx, sha); err != nil {478+ return 0, err479+ }480+ }481+ if _, err := s.store.DB().ExecContext(ctx, "DELETE FROM blob_claims WHERE expires>0 AND expires<=?", now); err != nil {482+ return 0, fmt.Errorf("release expired blob claims: %w", err)483+ }484+ return len(expired), nil485+}486+487+func (s *Service) currentLimits() Limits {488+ if s.config.Limits == nil {489+ return Limits{}490+ }491+ limits := s.config.Limits()492+ if limits.MaxFileBytes < 0 {493+ limits.MaxFileBytes = 0494+ }495+ if limits.UserStorageBytes < 0 {496+ limits.UserStorageBytes = 0497+ }498+ return limits499+}500+501+func (s *Service) hasClaim(ctx context.Context, sha, uploader string) bool {502+ var found int503+ err := s.store.DB().QueryRowContext(ctx, "SELECT 1 FROM blob_claims WHERE sha256=? AND uploader=?", sha, uploader).Scan(&found)504+ return err == nil && found == 1505+}506+507+func (s *Service) lookup(ctx context.Context, sha string) (Blob, error) {508+ var entry Blob509+ err := s.store.DB().QueryRowContext(ctx, "SELECT sha256,size,type,uploader,uploaded FROM blobs WHERE sha256=?", sha).Scan(&entry.SHA256, &entry.Size, &entry.Type, &entry.Uploader, &entry.Uploaded)510+ return entry, err511+}512+513+func (s *Service) blocked(ctx context.Context, sha string) bool {514+ var found int515+ err := s.store.DB().QueryRowContext(ctx, `SELECT 1 FROM hidden_events WHERE id=?516+ UNION ALL SELECT 1 FROM blob_blocks WHERE sha256=? LIMIT 1`, sha, sha).Scan(&found)517+ return err == nil && found == 1518+}519+520+func reconcile(ctx context.Context, store *storage.Store, root string) error {521+ entries, err := os.ReadDir(root)522+ if err != nil {523+ return fmt.Errorf("scan blob directory: %w", err)524+ }525+ for _, entry := range entries {526+ if entry.IsDir() {527+ continue528+ }529+ name := entry.Name()530+ if strings.HasPrefix(name, ".scoped-") {531+ sha := strings.TrimPrefix(name, ".scoped-")532+ if !shaPattern.MatchString(sha) {533+ if err := os.Remove(filepath.Join(root, name)); err != nil {534+ return fmt.Errorf("remove invalid staged blob: %w", err)535+ }536+ continue537+ }538+ if err := reconcileStaged(ctx, store, root, name, sha); err != nil {539+ return err540+ }541+ continue542+ }543+ if strings.HasPrefix(name, ".upload-") || strings.HasPrefix(name, ".multipart-chunk-") {544+ if err := os.Remove(filepath.Join(root, name)); err != nil {545+ return fmt.Errorf("remove abandoned upload: %w", err)546+ }547+ continue548+ }549+ if !shaPattern.MatchString(name) {550+ continue551+ }552+ var deleted int553+ if err := store.DB().QueryRowContext(ctx, "SELECT 1 FROM blob_tombstones WHERE sha256=? UNION ALL SELECT 1 FROM blob_blocks WHERE sha256=? LIMIT 1", name, name).Scan(&deleted); err == nil {554+ continue555+ } else if !errors.Is(err, sql.ErrNoRows) {556+ return fmt.Errorf("check deleted blob %q: %w", name, err)557+ }558+ var known int559+ if err := store.DB().QueryRowContext(ctx, "SELECT 1 FROM blobs WHERE sha256=?", name).Scan(&known); err == nil {560+ continue561+ } else if !errors.Is(err, sql.ErrNoRows) {562+ return fmt.Errorf("check orphan blob %q: %w", name, err)563+ }564+ if err := recoverBlob(ctx, store, root, name); err != nil {565+ return err566+ }567+ }568+ return nil569+}570+571+func reconcileStaged(ctx context.Context, store *storage.Store, root, name, sha string) error {572+ var size int64573+ if err := store.DB().QueryRowContext(ctx, "SELECT size FROM blobs WHERE sha256=?", sha).Scan(&size); errors.Is(err, sql.ErrNoRows) {574+ if removeErr := os.Remove(filepath.Join(root, name)); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) {575+ return removeErr576+ }577+ return nil578+ } else if err != nil {579+ return fmt.Errorf("check staged blob %q: %w", sha, err)580+ }581+ file, err := os.Open(filepath.Join(root, name))582+ if err != nil {583+ return err584+ }585+ info, err := file.Stat()586+ if err != nil {587+ _ = file.Close()588+ return err589+ }590+ if info.Size() != size {591+ _ = file.Close()592+ return fmt.Errorf("staged blob %q has size %d, want %d", sha, info.Size(), size)593+ }594+ hasher := sha256.New()595+ if _, err := io.Copy(hasher, file); err != nil {596+ _ = file.Close()597+ return err598+ }599+ if err := file.Close(); err != nil {600+ return err601+ }602+ if hex.EncodeToString(hasher.Sum(nil)) != sha {603+ return fmt.Errorf("staged blob %q failed content hash", sha)604+ }605+ if _, err := os.Stat(filepath.Join(root, sha)); errors.Is(err, os.ErrNotExist) {606+ if err := os.Rename(filepath.Join(root, name), filepath.Join(root, sha)); err != nil {607+ return err608+ }609+ } else if err != nil {610+ return err611+ } else {612+ _ = os.Remove(filepath.Join(root, name))613+ }614+ return syncDirectory(root)615+}616+617+func recoverBlob(ctx context.Context, store *storage.Store, root, name string) error {618+ file, err := os.Open(filepath.Join(root, name))619+ if err != nil {620+ return fmt.Errorf("open orphan blob %q: %w", name, err)621+ }622+ hash := sha256.New()623+ size, copyErr := io.Copy(hash, file)624+ closeErr := file.Close()625+ if copyErr != nil || closeErr != nil {626+ return fmt.Errorf("read orphan blob %q: %w", name, errors.Join(copyErr, closeErr))627+ }628+ if hex.EncodeToString(hash.Sum(nil)) != name {629+ return fmt.Errorf("orphan blob %q failed content hash", name)630+ }631+ info, err := os.Stat(filepath.Join(root, name))632+ if err != nil {633+ return fmt.Errorf("stat orphan blob %q: %w", name, err)634+ }635+ _, err = store.DB().ExecContext(ctx, "INSERT OR IGNORE INTO blobs(sha256,size,type,uploader,uploaded) VALUES(?,?,?,?,?)", name, size, "application/octet-stream", "", info.ModTime().Unix())636+ if err != nil {637+ return fmt.Errorf("recover orphan blob %q: %w", name, err)638+ }639+ return nil640+}641+642+func syncDirectory(path string) error {643+ directory, err := os.Open(path)644+ if err != nil {645+ return err646+ }647+ defer directory.Close()648+ return directory.Sync()649+}@@ -0,0 +1,253 @@
1+package community2+3+import (4+ "context"5+ "database/sql"6+ "time"7+)8+9+// ConfigSnapshot is the community-owned portion of relay configuration.10+// The type deliberately contains no configport types so the domain owns its11+// storage contract.12+type ConfigSnapshot struct {13+ Members []Member14+ Bans []ConfigBan15+ IPBlocks []ConfigIPBlock16+ EventBans []ConfigEventBan17+ KindRules []ConfigKindRule18+ Retention []ConfigRetention19+}20+21+type ConfigBan struct {22+ PubKey string23+ Reason string24+}25+26+type ConfigIPBlock struct {27+ IP string28+ Reason string29+}30+31+type ConfigEventBan struct {32+ ID string33+ Reason string34+}35+36+type ConfigKindRule struct {37+ Kind int38+ Rule string39+}40+41+type ConfigRetention struct {42+ Kind *int43+ Days int44+}45+46+type ConfigMemberRetention struct {47+ PubKey string48+ Days int49+}50+51+// RetentionRules returns the community-owned retention settings.52+func (s *Service) RetentionRules(ctx context.Context) ([]ConfigRetention, error) {53+ rows, err := s.store.DB().QueryContext(ctx, `SELECT kind,days FROM community_retention ORDER BY kind`)54+ if err != nil {55+ return nil, err56+ }57+ defer rows.Close()58+ var out []ConfigRetention59+ for rows.Next() {60+ var kind, days int61+ if err := rows.Scan(&kind, &days); err != nil {62+ return nil, err63+ }64+ out = append(out, ConfigRetention{Kind: &kind, Days: days})65+ }66+ return out, rows.Err()67+}68+69+// MembersForRetention returns members with an explicit retention override.70+func (s *Service) MembersForRetention(ctx context.Context) ([]ConfigMemberRetention, error) {71+ rows, err := s.store.DB().QueryContext(ctx, `SELECT pubkey,keep_days FROM community_members WHERE keep_days>0 ORDER BY pubkey`)72+ if err != nil {73+ return nil, err74+ }75+ defer rows.Close()76+ var out []ConfigMemberRetention77+ for rows.Next() {78+ var member ConfigMemberRetention79+ if err := rows.Scan(&member.PubKey, &member.Days); err != nil {80+ return nil, err81+ }82+ out = append(out, member)83+ }84+ return out, rows.Err()85+}86+87+// ExportConfig reads the community-owned configuration tables.88+func ExportConfig(ctx context.Context, store interface {89+ DB() *sql.DB90+}) (ConfigSnapshot, error) {91+ var out ConfigSnapshot92+ rows, err := store.DB().QueryContext(ctx, `SELECT pubkey,name,note,role,invited_by,via,created_at,keep_days FROM community_members WHERE role<>'owner' ORDER BY pubkey`)93+ if err != nil {94+ return out, err95+ }96+ defer rows.Close()97+ for rows.Next() {98+ var member Member99+ if err := rows.Scan(&member.PubKey, &member.Name, &member.Note, &member.Role, &member.InvitedBy, &member.Via, &member.CreatedAt, &member.KeepDays); err != nil {100+ return out, err101+ }102+ out.Members = append(out.Members, member)103+ }104+ if err := rows.Err(); err != nil {105+ return out, err106+ }107+ if err := readConfigRows(ctx, store.DB(), `SELECT pubkey,reason FROM community_bans ORDER BY pubkey`, func(rows *sql.Rows) error {108+ var item ConfigBan109+ if err := rows.Scan(&item.PubKey, &item.Reason); err != nil {110+ return err111+ }112+ out.Bans = append(out.Bans, item)113+ return nil114+ }); err != nil {115+ return out, err116+ }117+ if err := readConfigRows(ctx, store.DB(), `SELECT ip,reason FROM community_ip_blocks ORDER BY ip`, func(rows *sql.Rows) error {118+ var item ConfigIPBlock119+ if err := rows.Scan(&item.IP, &item.Reason); err != nil {120+ return err121+ }122+ out.IPBlocks = append(out.IPBlocks, item)123+ return nil124+ }); err != nil {125+ return out, err126+ }127+ if err := readConfigRows(ctx, store.DB(), `SELECT id,reason FROM community_event_bans ORDER BY id`, func(rows *sql.Rows) error {128+ var item ConfigEventBan129+ if err := rows.Scan(&item.ID, &item.Reason); err != nil {130+ return err131+ }132+ out.EventBans = append(out.EventBans, item)133+ return nil134+ }); err != nil {135+ return out, err136+ }137+ if err := readConfigRows(ctx, store.DB(), `SELECT kind,rule FROM community_kind_rules ORDER BY kind`, func(rows *sql.Rows) error {138+ var item ConfigKindRule139+ if err := rows.Scan(&item.Kind, &item.Rule); err != nil {140+ return err141+ }142+ out.KindRules = append(out.KindRules, item)143+ return nil144+ }); err != nil {145+ return out, err146+ }147+ if err := readConfigRows(ctx, store.DB(), `SELECT kind,days FROM community_retention ORDER BY kind`, func(rows *sql.Rows) error {148+ var kind, days int149+ if err := rows.Scan(&kind, &days); err != nil {150+ return err151+ }152+ if kind < 0 {153+ out.Retention = append(out.Retention, ConfigRetention{Days: days})154+ return nil155+ }156+ out.Retention = append(out.Retention, ConfigRetention{Kind: &kind, Days: days})157+ return nil158+ }); err != nil {159+ return out, err160+ }161+ return out, nil162+}163+164+func readConfigRows(ctx context.Context, db *sql.DB, query string, scan func(*sql.Rows) error) error {165+ rows, err := db.QueryContext(ctx, query)166+ if err != nil {167+ return err168+ }169+ defer rows.Close()170+ for rows.Next() {171+ if err := scan(rows); err != nil {172+ return err173+ }174+ }175+ return rows.Err()176+}177+178+// ApplyConfigTx replaces selected community configuration sections inside a179+// caller-owned transaction. Empty sections are meaningful replacements.180+func ApplyConfigTx(ctx context.Context, tx *sql.Tx, c ConfigSnapshot, members, bans, addresses, eventBans, kinds, retention bool) error {181+ now := time.Now().Unix()182+ if members {183+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_members WHERE role<>'owner'`); err != nil {184+ return err185+ }186+ for _, member := range c.Members {187+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_members(pubkey,name,note,role,invited_by,via,created_at,keep_days) VALUES(?,?,?,?,?,?,?,?)`, member.PubKey, member.Name, member.Note, member.Role, member.InvitedBy, member.Via, now, member.KeepDays); err != nil {188+ return err189+ }190+ }191+ }192+ if bans {193+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_bans`); err != nil {194+ return err195+ }196+ for _, item := range c.Bans {197+ if item.PubKey != "" {198+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_bans(pubkey,reason,at) VALUES(?,?,?)`, item.PubKey, item.Reason, now); err != nil {199+ return err200+ }201+ }202+ }203+ }204+ if addresses {205+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_ip_blocks`); err != nil {206+ return err207+ }208+ for _, item := range c.IPBlocks {209+ if item.IP != "" {210+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_ip_blocks(ip,reason,at) VALUES(?,?,?)`, item.IP, item.Reason, now); err != nil {211+ return err212+ }213+ }214+ }215+ }216+ if eventBans {217+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_event_bans`); err != nil {218+ return err219+ }220+ for _, item := range c.EventBans {221+ if item.ID != "" {222+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_event_bans(id,reason,at) VALUES(?,?,?)`, item.ID, item.Reason, now); err != nil {223+ return err224+ }225+ }226+ }227+ }228+ if kinds {229+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_kind_rules`); err != nil {230+ return err231+ }232+ for _, item := range c.KindRules {233+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_kind_rules(kind,rule) VALUES(?,?)`, item.Kind, item.Rule); err != nil {234+ return err235+ }236+ }237+ }238+ if retention {239+ if _, err := tx.ExecContext(ctx, `DELETE FROM community_retention`); err != nil {240+ return err241+ }242+ for _, item := range c.Retention {243+ kind := -1244+ if item.Kind != nil {245+ kind = *item.Kind246+ }247+ if _, err := tx.ExecContext(ctx, `INSERT INTO community_retention(kind,days) VALUES(?,?)`, kind, item.Days); err != nil {248+ return err249+ }250+ }251+ }252+ return nil253+}@@ -0,0 +1,72 @@
1+package community2+3+import (4+ "context"5+ "errors"6+ "strings"7+8+ "github.com/FelineStateMachine/tinyrelay/internal/communityread"9+)10+11+// Members exposes the community-owned membership projection to records. The12+// records package consumes this narrow read model and never relies on table13+// names or columns.14+func (s *Service) Members(ctx context.Context) ([]communityread.Member, error) {15+ rows, err := s.store.DB().QueryContext(ctx, `SELECT pubkey,name,role FROM community_members ORDER BY pubkey`)16+ if err != nil {17+ if strings.Contains(err.Error(), "no such table") {18+ return nil, nil19+ }20+ return nil, err21+ }22+ defer rows.Close()23+ var out []communityread.Member24+ for rows.Next() {25+ var member communityread.Member26+ if err := rows.Scan(&member.PubKey, &member.Name, &member.Role); err != nil {27+ return nil, err28+ }29+ out = append(out, member)30+ }31+ return out, rows.Err()32+}33+34+// MemberStatus reports heir membership and the current member count in one35+// call so succession checks use a consistent community snapshot.36+func (s *Service) MemberStatus(ctx context.Context, pubkey string) (bool, int, error) {37+ var count int38+ var member int39+ if err := s.store.DB().QueryRowContext(ctx, `SELECT count(*), coalesce(max(pubkey=?),0) FROM community_members`, pubkey).Scan(&count, &member); err != nil {40+ if strings.Contains(err.Error(), "no such table") {41+ return false, 0, nil42+ }43+ return false, 0, err44+ }45+ return member != 0, count, nil46+}47+48+// ModerationCounts returns the counters needed by the moderation view.49+func (s *Service) ModerationCounts(ctx context.Context, start int64) (communityread.ModerationCounts, error) {50+ var out communityread.ModerationCounts51+ queries := []struct {52+ table string53+ to *int54+ where string55+ }{56+ {"community_bans", &out.Bans, "at>=?"},57+ {"community_reports", &out.Reports, "at>=?"},58+ {"community_reports", &out.Resolved, "resolved_at>=?"},59+ {"community_event_bans", &out.Hidden, "at>=?"},60+ {"community_ip_blocks", &out.BlockedAddresses, "at>=?"},61+ }62+ var failures error63+ for _, query := range queries {64+ if err := s.store.DB().QueryRowContext(ctx, "SELECT count(*) FROM "+query.table+" WHERE "+query.where, start).Scan(query.to); err != nil {65+ if strings.Contains(err.Error(), "no such table") {66+ continue67+ }68+ failures = errors.Join(failures, err)69+ }70+ }71+ return out, failures72+}@@ -0,0 +1,13 @@
1+// Package communityread contains the small read models shared by community2+// and records. It has no storage or service dependencies.3+package communityread4+5+type Member struct {6+ PubKey string7+ Name string8+ Role string9+}10+11+type ModerationCounts struct {12+ Bans, Reports, Resolved, Hidden, BlockedAddresses int13+}@@ -14,6 +14,7 @@ import (
1414 1515 "github.com/FelineStateMachine/tinyrelay/internal/community"1616 "github.com/FelineStateMachine/tinyrelay/internal/policy"17+ "github.com/FelineStateMachine/tinyrelay/internal/replication"1718 "github.com/FelineStateMachine/tinyrelay/internal/storage"1819 "github.com/FelineStateMachine/tinyrelay/internal/templates"1920 )@@ -78,7 +79,7 @@ func New(c ConfigStore) *ConfigStore {
7879 if c.Store != nil {7980 _, _ = c.Store.DB().Exec(`CREATE TABLE IF NOT EXISTS config_connections(position INTEGER PRIMARY KEY, value TEXT NOT NULL)`)8081 _, _ = c.Store.DB().Exec(`CREATE TABLE IF NOT EXISTS config_jobs(id TEXT PRIMARY KEY, recipe TEXT NOT NULL)`)81- _, _ = c.Store.DB().Exec(`CREATE TABLE IF NOT EXISTS replication_jobs (id TEXT PRIMARY KEY, payload TEXT NOT NULL, state TEXT NOT NULL, created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL, phase TEXT NOT NULL DEFAULT 'pending', error TEXT NOT NULL DEFAULT '', stored INTEGER NOT NULL DEFAULT 0, sent INTEGER NOT NULL DEFAULT 0, skipped INTEGER NOT NULL DEFAULT 0, refused INTEGER NOT NULL DEFAULT 0, cursor INTEGER NOT NULL DEFAULT 0, rounds INTEGER NOT NULL DEFAULT 0, started_at INTEGER NOT NULL DEFAULT 0, finished_at INTEGER NOT NULL DEFAULT 0)`)82+ _ = replication.EnsureJobSchema(c.Store.DB())8283 }8384 return &c8485 }@@ -179,55 +180,33 @@ func (s *ConfigStore) Export(ctx context.Context) (Config, error) {
179180 if s.Store == nil {180181 return c, errors.New("config: missing store")181182 }182- rows, err := s.Store.DB().QueryContext(ctx, `SELECT pubkey,name,note,role FROM community_members WHERE role<>'owner' ORDER BY pubkey`)183+ communityConfig, err := community.ExportConfig(ctx, s.Store)183184 if err != nil {184185 return c, err185186 }186- defer rows.Close()187- for rows.Next() {188- var m Member189- if err := rows.Scan(&m.PubKey, &m.Name, &m.Note, &m.Role); err != nil {190- return c, err191- }192- c.Members = append(c.Members, m)187+ for _, m := range communityConfig.Members {188+ c.Members = append(c.Members, Member{PubKey: m.PubKey, Name: m.Name, Note: m.Note, Role: m.Role})193189 }194- if err := rows.Err(); err != nil {195- return c, err190+ for _, item := range communityConfig.Bans {191+ c.Bans = append(c.Bans, map[string]any{"pubkey": item.PubKey, "reason": item.Reason})196192 }197- for _, q := range []struct {198- query string199- dst *[]map[string]any200- fields []any201- }{{`SELECT pubkey,reason FROM community_bans ORDER BY pubkey`, &c.Bans, nil}, {`SELECT ip,reason FROM community_ip_blocks ORDER BY ip`, &c.Addresses, nil}, {`SELECT id,reason FROM community_event_bans ORDER BY id`, &c.BannedEvents, nil}} {202- rs, e := s.Store.DB().QueryContext(ctx, q.query)203- if e != nil {204- return c, e205- }206- for rs.Next() {207- if q.dst == &c.Bans {208- var k, v string209- if e = rs.Scan(&k, &v); e == nil {210- *q.dst = append(*q.dst, map[string]any{"pubkey": k, "reason": v})211- }212- } else if q.dst == &c.Addresses {213- var k, v string214- if e = rs.Scan(&k, &v); e == nil {215- *q.dst = append(*q.dst, map[string]any{"ip": k, "reason": v})216- }217- } else {218- var k, v string219- if e = rs.Scan(&k, &v); e == nil {220- *q.dst = append(*q.dst, map[string]any{"id": k, "reason": v})221- }222- }223- if e != nil {224- rs.Close()225- return c, e226- }193+ for _, item := range communityConfig.IPBlocks {194+ c.Addresses = append(c.Addresses, map[string]any{"ip": item.IP, "reason": item.Reason})195+ }196+ for _, item := range communityConfig.EventBans {197+ c.BannedEvents = append(c.BannedEvents, map[string]any{"id": item.ID, "reason": item.Reason})198+ }199+ for _, item := range communityConfig.KindRules {200+ if item.Rule == "allow" {201+ c.Kinds.Allow = append(c.Kinds.Allow, item.Kind)202+ } else if item.Rule == "block" {203+ c.Kinds.Block = append(c.Kinds.Block, item.Kind)227204 }228- rs.Close()229205 }230- rows, err = s.Store.DB().QueryContext(ctx, `SELECT value FROM config_connections ORDER BY position`)206+ for _, item := range communityConfig.Retention {207+ c.Retention = append(c.Retention, Rule{Kind: item.Kind, Days: item.Days})208+ }209+ rows, err := s.Store.DB().QueryContext(ctx, `SELECT value FROM config_connections ORDER BY position`)231210 if err == nil {232211 defer rows.Close()233212 for rows.Next() {@@ -362,91 +341,41 @@ func (s *ConfigStore) ApplyWithOptions(ctx context.Context, c Config, opts Apply
362341 return err363342 }364343 }365- if has(c, "members") {366- if _, err := tx.ExecContext(ctx, `DELETE FROM community_members WHERE role<>'owner'`); err != nil {367- return err368- }369- for _, m := range c.Members {370- if _, err := tx.ExecContext(ctx, `INSERT INTO community_members(pubkey,name,note,role,created_at) VALUES(?,?,?,?,?)`, m.PubKey, m.Name, m.Note, m.Role, time.Now().Unix()); err != nil {371- return err372- }373- }344+ communityConfig := community.ConfigSnapshot{}345+ for _, m := range c.Members {346+ communityConfig.Members = append(communityConfig.Members, community.Member{PubKey: m.PubKey, Name: m.Name, Note: m.Note, Role: m.Role})374347 }375- if s.Community != nil && ch.FinalPolicy.Owner != "" && ch.FinalPolicy.Owner != cur.Owner {376- if err := s.Community.ApplyOwnerTx(ctx, tx, cur.Owner, ch.FinalPolicy.Owner); err != nil {377- return err378- }348+ for _, b := range c.Bans {349+ pk, _ := b["pubkey"].(string)350+ reason, _ := b["reason"].(string)351+ communityConfig.Bans = append(communityConfig.Bans, community.ConfigBan{PubKey: pk, Reason: reason})379352 }380- if has(c, "bans") {381- if _, err := tx.ExecContext(ctx, `DELETE FROM community_bans`); err != nil {382- return err383- }384- for _, b := range c.Bans {385- pk, _ := b["pubkey"].(string)386- reason, _ := b["reason"].(string)387- if pk != "" {388- if _, err := tx.ExecContext(ctx, `INSERT INTO community_bans(pubkey,reason,at) VALUES(?,?,?)`, pk, reason, time.Now().Unix()); err != nil {389- return err390- }391- }392- }353+ for _, b := range c.Addresses {354+ ip, _ := b["ip"].(string)355+ reason, _ := b["reason"].(string)356+ communityConfig.IPBlocks = append(communityConfig.IPBlocks, community.ConfigIPBlock{IP: ip, Reason: reason})393357 }394- if has(c, "addresses") {395- if _, err := tx.ExecContext(ctx, `DELETE FROM community_ip_blocks`); err != nil {396- return err397- }398- for _, b := range c.Addresses {399- ip, _ := b["ip"].(string)400- reason, _ := b["reason"].(string)401- if ip != "" {402- if _, err := tx.ExecContext(ctx, `INSERT INTO community_ip_blocks(ip,reason,at) VALUES(?,?,?)`, ip, reason, time.Now().Unix()); err != nil {403- return err404- }405- }406- }358+ for _, b := range c.BannedEvents {359+ id, _ := b["id"].(string)360+ reason, _ := b["reason"].(string)361+ communityConfig.EventBans = append(communityConfig.EventBans, community.ConfigEventBan{ID: id, Reason: reason})407362 }408- if has(c, "banned_events") {409- if _, err := tx.ExecContext(ctx, `DELETE FROM community_event_bans`); err != nil {410- return err411- }412- for _, b := range c.BannedEvents {413- id, _ := b["id"].(string)414- reason, _ := b["reason"].(string)415- if id != "" {416- if _, err := tx.ExecContext(ctx, `INSERT INTO community_event_bans(id,reason,at) VALUES(?,?,?)`, id, reason, time.Now().Unix()); err != nil {417- return err418- }419- }420- }363+ for _, k := range c.Kinds.Allow {364+ communityConfig.KindRules = append(communityConfig.KindRules, community.ConfigKindRule{Kind: k, Rule: "allow"})421365 }422- if has(c, "kinds") {423- if _, err := tx.ExecContext(ctx, `DELETE FROM community_kind_rules`); err != nil {424- return err425- }426- for _, k := range c.Kinds.Allow {427- if _, err := tx.ExecContext(ctx, `INSERT INTO community_kind_rules(kind,rule) VALUES(?,?)`, k, "allow"); err != nil {428- return err429- }430- }431- for _, k := range c.Kinds.Block {432- if _, err := tx.ExecContext(ctx, `INSERT INTO community_kind_rules(kind,rule) VALUES(?,?)`, k, "block"); err != nil {433- return err434- }435- }366+ for _, k := range c.Kinds.Block {367+ communityConfig.KindRules = append(communityConfig.KindRules, community.ConfigKindRule{Kind: k, Rule: "block"})436368 }437- if has(c, "retention") {438- if _, err := tx.ExecContext(ctx, `DELETE FROM community_retention`); err != nil {369+ for _, r := range c.Retention {370+ communityConfig.Retention = append(communityConfig.Retention, community.ConfigRetention{Kind: r.Kind, Days: r.Days})371+ }372+ if err := community.ApplyConfigTx(ctx, tx, communityConfig, has(c, "members"), has(c, "bans"), has(c, "addresses"), has(c, "banned_events"), has(c, "kinds"), has(c, "retention")); err != nil {373+ return err374+ }375+ if s.Community != nil && ch.FinalPolicy.Owner != "" && ch.FinalPolicy.Owner != cur.Owner {376+ if err := s.Community.ApplyOwnerTx(ctx, tx, cur.Owner, ch.FinalPolicy.Owner); err != nil {439377 return err440378 }441- for _, r := range c.Retention {442- kind := -1443- if r.Kind != nil {444- kind = *r.Kind445- }446- if _, err := tx.ExecContext(ctx, `INSERT INTO community_retention(kind,days) VALUES(?,?)`, kind, r.Days); err != nil {447- return err448- }449- }450379 }451380 if has(c, "connections") {452381 if _, err := tx.ExecContext(ctx, `DELETE FROM config_connections`); err != nil {@@ -463,9 +392,7 @@ func (s *ConfigStore) ApplyWithOptions(ctx context.Context, c Config, opts Apply
463392 if _, err := tx.ExecContext(ctx, `DELETE FROM config_jobs`); err != nil {464393 return err465394 }466- if _, err := tx.ExecContext(ctx, `DELETE FROM replication_jobs`); err != nil {467- return err468- }395+ var replicationJobs []replication.ConfigJob469396 for _, job := range c.Jobs {470397 id, _ := job["id"].(string)471398 if id == "" {@@ -489,9 +416,10 @@ func (s *ConfigStore) ApplyWithOptions(ctx context.Context, c Config, opts Apply
489416 if err != nil {490417 return err491418 }492- if _, err := tx.ExecContext(ctx, `INSERT INTO replication_jobs(id,payload,state,created_at,updated_at) VALUES(?,?, 'pending',strftime('%s','now'),strftime('%s','now')) ON CONFLICT(id) DO UPDATE SET payload=excluded.payload,updated_at=excluded.updated_at,state='pending'`, id, string(payload)); err != nil {493- return err494- }419+ replicationJobs = append(replicationJobs, replication.ConfigJob{ID: id, Payload: payload})420+ }421+ if err := replication.ReplaceJobsTx(ctx, tx, replicationJobs); err != nil {422+ return err495423 }496424 }497425 return nil@@ -0,0 +1,213 @@
1+package daemon2+3+import (4+ "context"5+ "database/sql"6+ "encoding/json"7+ "errors"8+ "strconv"9+ "strings"10+ "testing"11+ "time"12+13+ "github.com/FelineStateMachine/tinyrelay/internal/event"14+ "github.com/FelineStateMachine/tinyrelay/internal/relay"15+ "github.com/FelineStateMachine/tinyrelay/internal/replication"16+ "github.com/FelineStateMachine/tinyrelay/internal/storage"17+)18+19+func TestCallbackFollowupPlanningFailureDoesNotRejectEvents(t *testing.T) {20+ for _, source := range []string{"client", "import"} {21+ t.Run(source, func(t *testing.T) {22+ _, tenant := loopbackTenant(t)23+ ctx := context.Background()24+ member, _ := event.PublicKey(testMemberSecret)25+ setRole(t, tenant, member, "member")26+ addCallback(t, tenant, member, "https://127.0.0.1:1/wake", `{"kinds":[1]}`)27+ if _, err := tenant.store.DB().ExecContext(ctx, `CREATE TRIGGER reject_callback_work BEFORE INSERT ON work_intents WHEN NEW.kind='callback-delivery' BEGIN SELECT RAISE(ABORT,'callback work unavailable'); END`); err != nil {28+ t.Fatal(err)29+ }30+ e := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), [][]string{{"p", member}}, "callback planning")31+ var err error32+ if source == "import" {33+ err = tenant.commitImported(ctx, e, replication.OriginImport)34+ } else {35+ _, err = tenant.Publish(ctx, e, relay.Session{PubKeys: []string{e.PubKey}})36+ }37+ if err != nil {38+ t.Fatalf("%s event rejected after optional planning failure: %v", source, err)39+ }40+ var stored int41+ if err := tenant.store.DB().QueryRowContext(ctx, `SELECT count(*) FROM events WHERE id=?`, e.ID).Scan(&stored); err != nil {42+ t.Fatal(err)43+ }44+ if stored != 1 {45+ t.Fatalf("stored events=%d, want 1", stored)46+ }47+ var pending int48+ if err := tenant.store.DB().QueryRowContext(ctx, `SELECT count(*) FROM work_intents WHERE kind=? AND event_id=? AND state='pending'`, eventFollowupKind, e.ID).Scan(&pending); err != nil {49+ t.Fatal(err)50+ }51+ if pending != 1 {52+ t.Fatalf("pending follow-up jobs=%d, want 1", pending)53+ }54+ })55+ }56+}57+58+func TestCallbackFollowupRecoveryPreservesCompletedDelivery(t *testing.T) {59+ _, tenant := loopbackTenant(t)60+ ctx := context.Background()61+ member, _ := event.PublicKey(testMemberSecret)62+ setRole(t, tenant, member, "member")63+ first := addCallback(t, tenant, member, "https://127.0.0.1:1/first", `{"kinds":[1]}`)64+ second := addCallback(t, tenant, member, "https://127.0.0.1:1/second", `{"kinds":[1]}`)65+ e := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), [][]string{{"p", member}}, "recover callback work")66+ followup := tenant.followups.prepare(ctx, e, time.Now().Unix(), false)67+ if followup == nil {68+ t.Fatal("missing callback follow-up")69+ }70+ if _, err := tenant.store.Save(ctx, e, storage.SaveOptions{Now: time.Now().Unix(), Intents: []storage.Intent{*followup}}); err != nil {71+ t.Fatal(err)72+ }73+ job := followupJob(t, tenant, e.ID)74+ if err := tenant.followups.handle(ctx, job); err != nil {75+ t.Fatal(err)76+ }77+ firstID := first["id"].(string)78+ secondID := second["id"].(string)79+ markCallbackDeliveryCompleted(t, tenant, firstID, e.ID)80+ deleteCallbackDelivery(t, tenant, secondID, e.ID)81+ if err := tenant.followups.handle(ctx, job); err != nil {82+ t.Fatal(err)83+ }84+ if state := callbackDeliveryState(t, tenant, firstID, e.ID); state != "completed" {85+ t.Fatalf("completed delivery state=%q, want completed", state)86+ }87+ if state := callbackDeliveryState(t, tenant, secondID, e.ID); state != "pending" {88+ t.Fatalf("recovered delivery state=%q, want pending", state)89+ }90+}91+92+func TestCallbackFollowupSkipsPausedRemovedAndLaterRegistrations(t *testing.T) {93+ for _, action := range []string{"pausecallback", "removecallback"} {94+ t.Run(action, func(t *testing.T) {95+ _, tenant := loopbackTenant(t)96+ ctx := context.Background()97+ member, _ := event.PublicKey(testMemberSecret)98+ setRole(t, tenant, member, "member")99+ captured := addCallback(t, tenant, member, "https://127.0.0.1:1/captured", `{"kinds":[1]}`)100+ e := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), [][]string{{"p", member}}, "captured callback")101+ followup := tenant.followups.prepare(ctx, e, time.Now().Unix(), false)102+ if followup == nil {103+ t.Fatal("missing callback follow-up")104+ }105+ id := captured["id"].(string)106+ if _, err := tenant.Execute(ctx, member, action, []json.RawMessage{json.RawMessage(strconv.Quote(id))}); err != nil {107+ t.Fatal(err)108+ }109+ later := addCallback(t, tenant, member, "https://127.0.0.1:1/later", `{"kinds":[1]}`)110+ if _, err := tenant.store.Save(ctx, e, storage.SaveOptions{Now: time.Now().Unix(), Intents: []storage.Intent{*followup}}); err != nil {111+ t.Fatal(err)112+ }113+ if err := tenant.followups.handle(ctx, followupJob(t, tenant, e.ID)); err != nil {114+ t.Fatal(err)115+ }116+ if got := callbackDeliveryState(t, tenant, id, e.ID); got != "" {117+ t.Fatalf("%s callback state=%q, want absent", action, got)118+ }119+ if got := callbackDeliveryState(t, tenant, later["id"].(string), e.ID); got != "" {120+ t.Fatalf("later callback state=%q, want absent", got)121+ }122+ })123+ }124+}125+126+func TestCallbackFollowupPlanningExhaustionStops(t *testing.T) {127+ _, tenant := loopbackTenant(t)128+ ctx := context.Background()129+ member, _ := event.PublicKey(testMemberSecret)130+ setRole(t, tenant, member, "member")131+ addCallback(t, tenant, member, "https://127.0.0.1:1/wake", `{"kinds":[1]}`)132+ e := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), [][]string{{"p", member}}, "exhaust planning")133+ followup := tenant.followups.prepare(ctx, e, time.Now().Unix(), false)134+ if followup == nil {135+ t.Fatal("missing callback follow-up")136+ }137+ if _, err := tenant.store.Save(ctx, e, storage.SaveOptions{Now: time.Now().Unix(), Intents: []storage.Intent{*followup}}); err != nil {138+ t.Fatal(err)139+ }140+ if _, err := tenant.store.DB().ExecContext(ctx, `CREATE TRIGGER reject_callback_work BEFORE INSERT ON work_intents WHEN NEW.kind='callback-delivery' BEGIN SELECT RAISE(ABORT,'callback work unavailable'); END`); err != nil {141+ t.Fatal(err)142+ }143+ job := followupJob(t, tenant, e.ID)144+ job.Attempts = followupMaxAttempts145+ err := tenant.followups.handle(ctx, job)146+ if err == nil || !strings.Contains(err.Error(), "callback work unavailable") || errors.Unwrap(err) == nil {147+ t.Fatalf("exhausted planning error=%v, want terminal wrapped error", err)148+ }149+}150+151+func TestCallbackFollowupRecoversWhenViewLookupIsUnavailable(t *testing.T) {152+ tenant, _, _ := viewTenant(t, []int{1}, nil)153+ ctx := context.Background()154+ member, _ := event.PublicKey(testMemberSecret)155+ setRole(t, tenant, member, "member")156+ callback := addCallback(t, tenant, member, "https://127.0.0.1:1/wake", `{"kinds":[1]}`)157+ e := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), [][]string{{"p", member}}, "```mermaid\ngraph LR; A-->B\n```")158+ if _, err := tenant.store.DB().ExecContext(ctx, `ALTER TABLE custom_views RENAME TO custom_views_unavailable`); err != nil {159+ t.Fatal(err)160+ }161+ if _, err := tenant.Publish(ctx, e, relay.Session{PubKeys: []string{e.PubKey}}); err != nil {162+ t.Fatalf("publish with unavailable optional view lookup: %v", err)163+ }164+ job := followupJob(t, tenant, e.ID)165+ callbackID := callback["id"].(string)166+ if state := callbackDeliveryState(t, tenant, callbackID, e.ID); state != "pending" {167+ t.Fatalf("callback state after partial planning=%q, want pending", state)168+ }169+ markCallbackDeliveryCompleted(t, tenant, callbackID, e.ID)170+ if _, err := tenant.store.DB().ExecContext(ctx, `ALTER TABLE custom_views_unavailable RENAME TO custom_views`); err != nil {171+ t.Fatal(err)172+ }173+ if err := tenant.followups.handle(ctx, job); err != nil {174+ t.Fatalf("recover follow-up planning: %v", err)175+ }176+ if state := callbackDeliveryState(t, tenant, callbackID, e.ID); state != "completed" {177+ t.Fatalf("callback state after recovery=%q, want completed", state)178+ }179+ var callbackCount int180+ if err := tenant.store.DB().QueryRowContext(ctx, `SELECT count(*) FROM work_intents WHERE kind=? AND target=? AND event_id=?`, callbackDelivery, callbackID, e.ID).Scan(&callbackCount); err != nil {181+ t.Fatal(err)182+ }183+ if callbackCount != 1 {184+ t.Fatalf("callback intent count=%d, want 1", callbackCount)185+ }186+}187+188+func callbackDeliveryState(t *testing.T, tenant *Tenant, target, eventID string) string {189+ t.Helper()190+ var state string191+ err := tenant.store.DB().QueryRow(`SELECT state FROM work_intents WHERE kind=? AND target=? AND event_id=?`, callbackDelivery, target, eventID).Scan(&state)192+ if errors.Is(err, sql.ErrNoRows) {193+ return ""194+ }195+ if err != nil {196+ t.Fatal(err)197+ }198+ return state199+}200+201+func markCallbackDeliveryCompleted(t *testing.T, tenant *Tenant, target, eventID string) {202+ t.Helper()203+ if _, err := tenant.store.DB().Exec(`UPDATE work_intents SET state='completed' WHERE kind=? AND target=? AND event_id=?`, callbackDelivery, target, eventID); err != nil {204+ t.Fatal(err)205+ }206+}207+208+func deleteCallbackDelivery(t *testing.T, tenant *Tenant, target, eventID string) {209+ t.Helper()210+ if _, err := tenant.store.DB().Exec(`DELETE FROM work_intents WHERE kind=? AND target=? AND event_id=?`, callbackDelivery, target, eventID); err != nil {211+ t.Fatal(err)212+ }213+}@@ -117,7 +117,7 @@ func TestCallbackIntentSurvivesRestartAndHonorsRevocation(t *testing.T) {
117117 }118118 runCtx, cancel := context.WithCancel(ctx)119119 done := make(chan error, 1)120- go func() { done <- reloaded.replication.Run(runCtx) }()120+ go func() { done <- reloaded.runWork(runCtx) }()121121 if !waitFor(t, 3*time.Second, func() bool {122122 items, listErr := reloaded.replication.Queue().Pending(ctx, "callback")123123 return listErr == nil && len(items) == 0@@ -185,15 +185,15 @@ func callbackHost(raw string) string {
185185 return parsed.Host186186 }187187 188-func (t *Tenant) initCallbacks(ctx context.Context) error {189- _, err := t.store.DB().ExecContext(ctx, callbackSchema)188+func (s *callbackService) initCallbacks(ctx context.Context) error {189+ _, err := s.store.DB().ExecContext(ctx, callbackSchema)190190 return err191191 }192192 193-// checkCallbackURL accepts https URLs to public hosts. When the relay itself194-// runs on a loopback address, as it does in tests, private targets are193+// validateWebhookURL accepts https URLs to public hosts. When the relay194+// itself runs on a loopback address, as it does in tests, private targets are195195 // allowed so a local receiver can be exercised.196-func (t *Tenant) checkCallbackURL(raw string) error {196+func validateWebhookURL(raw, publicURL string) error {197197 if len(raw) > callbackURLMax {198198 return errors.New("invalid: url is too long")199199 }@@ -201,7 +201,7 @@ func (t *Tenant) checkCallbackURL(raw string) error {
201201 if err != nil || parsed.Scheme != "https" || parsed.Hostname() == "" || parsed.User != nil || parsed.Fragment != "" {202202 return errors.New("invalid: url must be https without credentials or a fragment")203203 }204- if t.loopbackRelay() {204+ if loopbackPublicURL(publicURL) {205205 return nil206206 }207207 host := strings.ToLower(parsed.Hostname())@@ -225,10 +225,10 @@ func privateAddress(ip net.IP) bool {
225225 return shared.Contains(ip)226226 }227227 228-// loopbackRelay reports whether the relay's own public URL is a loopback or229-// private address, which only happens in tests and local trials.230-func (t *Tenant) loopbackRelay() bool {231- parsed, err := url.Parse(t.publicURL)228+// loopbackPublicURL reports whether the relay's own public URL is a loopback229+// or private address, which only happens in tests and local trials.230+func loopbackPublicURL(publicURL string) bool {231+ parsed, err := url.Parse(publicURL)232232 if err != nil {233233 return false234234 }@@ -247,8 +247,8 @@ func callbackMethod(method string) bool {
247247 // callbackExecute serves the callback management methods. Members and agents248248 // manage callbacks for their own key; the owner and moderators see and249249 // control every callback. Each call runs under the callback operation.250-func (t *Tenant) callbackExecute(ctx context.Context, actor, method string, params []json.RawMessage) (result any, err error) {251- ctx, finish := t.app.telemetry.Start(ctx, "callback")250+func (s *callbackService) callbackExecute(ctx context.Context, actor, method string, params []json.RawMessage) (result any, err error) {251+ ctx, finish := s.telemetry.Start(ctx, "callback")252252 outcome := "error"253253 defer func() {254254 if err != nil {@@ -256,7 +256,7 @@ func (t *Tenant) callbackExecute(ctx context.Context, actor, method string, para
256256 }257257 finish(outcome)258258 }()259- role, err := t.community.Role(ctx, actor)259+ role, err := s.community.Role(ctx, actor)260260 if err != nil {261261 return nil, err262262 }@@ -268,9 +268,9 @@ func (t *Tenant) callbackExecute(ctx context.Context, actor, method string, para
268268 case "listcallbacks":269269 var rows []callbackRecord270270 if operator {271- rows, err = t.callbackRows(ctx, "")271+ rows, err = s.callbackRows(ctx, "")272272 } else {273- rows, err = t.callbackRows(ctx, actor)273+ rows, err = s.callbackRows(ctx, actor)274274 }275275 if err != nil {276276 return nil, err@@ -288,9 +288,9 @@ func (t *Tenant) callbackExecute(ctx context.Context, actor, method string, para
288288 outcome = "ok"289289 return out, nil290290 case "addcallback":291- result, err = t.addCallback(ctx, actor, role, params)291+ result, err = s.addCallback(ctx, actor, role, params)292292 default:293- result, err = t.changeCallback(ctx, actor, operator, method, params)293+ result, err = s.changeCallback(ctx, actor, operator, method, params)294294 }295295 if err == nil {296296 outcome = "ok"@@ -309,7 +309,7 @@ func callbackOutcome(err error) string {
309309 }310310 }311311 312-func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []json.RawMessage) (any, error) {312+func (s *callbackService) addCallback(ctx context.Context, actor, role string, params []json.RawMessage) (any, error) {313313 if len(params) != 1 {314314 return nil, errors.New("invalid: addcallback expects one object with url, filter and an optional secret")315315 }@@ -322,7 +322,7 @@ func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []j
322322 return nil, errors.New("invalid: addcallback expects one object with url, filter and an optional secret")323323 }324324 options.URL = strings.TrimSpace(options.URL)325- if err := t.checkCallbackURL(options.URL); err != nil {325+ if err := validateWebhookURL(options.URL, s.publicURL); err != nil {326326 return nil, err327327 }328328 filter, err := parseCallbackFilter(options.Filter)@@ -330,7 +330,7 @@ func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []j
330330 return nil, err331331 }332332 if role == "agent" {333- if err := t.checkCallbackScope(ctx, actor, filter); err != nil {333+ if err := s.checkCallbackScope(ctx, actor, filter); err != nil {334334 return nil, err335335 }336336 }@@ -353,9 +353,9 @@ func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []j
353353 if err != nil {354354 return nil, err355355 }356- limit := t.Policy().Callbacks356+ limit := s.policy().Callbacks357357 now := time.Now().Unix()358- err = t.store.WithTx(ctx, func(tx *sql.Tx) error {358+ err = s.store.WithTx(ctx, func(tx *sql.Tx) error {359359 if role != "owner" {360360 var count int361361 if err := tx.QueryRowContext(ctx, `SELECT count(*) FROM callbacks WHERE owner=?`, actor).Scan(&count); err != nil {@@ -371,12 +371,12 @@ func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []j
371371 if err != nil {372372 return nil, err373373 }374- t.invalidateCallbacks()375- if err := t.community.Record(ctx, actor, "addcallback", id, ""); err != nil {374+ s.invalidateCallbacks()375+ if err := s.community.Record(ctx, actor, "addcallback", id, ""); err != nil {376376 return nil, err377377 }378378 // Counts only: the host, path and secret stay out of the log.379- t.app.telemetry.Logger().Info("callback registered", "tenant", t.meta.Name, "kinds", len(filter.Kinds), "generated", generated)379+ s.telemetry.Logger().Info("callback registered", "tenant", s.tenant.Name, "kinds", len(filter.Kinds), "generated", generated)380380 record := callbackRecord{ID: id, Owner: actor, URL: options.URL, Filter: filter, CreatedAt: now}381381 out := record.summary()382382 out["secret"] = secret@@ -387,8 +387,8 @@ func (t *Tenant) addCallback(ctx context.Context, actor, role string, params []j
387387 // names must be one it may post in and a repository it names must be one388388 // the grant covers. Kinds are not restricted, so an agent can listen for389389 // answers it may not publish itself.390-func (t *Tenant) checkCallbackScope(ctx context.Context, actor string, filter callbackFilter) error {391- grant, ok, err := t.community.AgentGrant(ctx, actor)390+func (s *callbackService) checkCallbackScope(ctx context.Context, actor string, filter callbackFilter) error {391+ grant, ok, err := s.community.AgentGrant(ctx, actor)392392 if err != nil {393393 return err394394 }@@ -412,12 +412,12 @@ func (t *Tenant) checkCallbackScope(ctx context.Context, actor string, filter ca
412412 return nil413413 }414414 415-func (t *Tenant) changeCallback(ctx context.Context, actor string, operator bool, method string, params []json.RawMessage) (any, error) {415+func (s *callbackService) changeCallback(ctx context.Context, actor string, operator bool, method string, params []json.RawMessage) (any, error) {416416 id := callbackIDParam(params)417417 if id == "" {418418 return nil, errors.New("invalid: callback id required")419419 }420- record, err := t.callback(ctx, id)420+ record, err := s.callback(ctx, id)421421 if err != nil {422422 return nil, err423423 }@@ -426,12 +426,12 @@ func (t *Tenant) changeCallback(ctx context.Context, actor string, operator bool
426426 }427427 switch method {428428 case "removecallback":429- _, err = t.store.DB().ExecContext(ctx, `DELETE FROM callbacks WHERE id=?`, id)429+ _, err = s.store.DB().ExecContext(ctx, `DELETE FROM callbacks WHERE id=?`, id)430430 case "pausecallback":431- _, err = t.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=1, last_status='paused' WHERE id=?`, id)431+ _, err = s.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=1, last_status='paused' WHERE id=?`, id)432432 record.Paused, record.LastStatus = true, "paused"433433 case "resumecallback":434- _, err = t.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=0, failures=0, last_status='' WHERE id=?`, id)434+ _, err = s.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=0, failures=0, last_status='' WHERE id=?`, id)435435 record.Paused, record.Failures, record.LastStatus = false, 0, ""436436 default:437437 return nil, fmt.Errorf("unsupported: management method %q", method)@@ -439,8 +439,8 @@ func (t *Tenant) changeCallback(ctx context.Context, actor string, operator bool
439439 if err != nil {440440 return nil, err441441 }442- t.invalidateCallbacks()443- if err := t.community.Record(ctx, actor, method, id, ""); err != nil {442+ s.invalidateCallbacks()443+ if err := s.community.Record(ctx, actor, method, id, ""); err != nil {444444 return nil, err445445 }446446 if method == "removecallback" {@@ -464,8 +464,8 @@ func callbackIDParam(params []json.RawMessage) string {
464464 return strings.TrimSpace(options.ID)465465 }466466 467-func (t *Tenant) callback(ctx context.Context, id string) (callbackRecord, error) {468- rows, err := t.callbackRows(ctx, "", id)467+func (s *callbackService) callback(ctx context.Context, id string) (callbackRecord, error) {468+ rows, err := s.callbackRows(ctx, "", id)469469 if err != nil {470470 return callbackRecord{}, err471471 }@@ -476,7 +476,7 @@ func (t *Tenant) callback(ctx context.Context, id string) (callbackRecord, error
476476 }477477 478478 // callbackRows lists callbacks, narrowed to an owner or to one id.479-func (t *Tenant) callbackRows(ctx context.Context, owner string, ids ...string) ([]callbackRecord, error) {479+func (s *callbackService) callbackRows(ctx context.Context, owner string, ids ...string) ([]callbackRecord, error) {480480 query := `SELECT id,owner,url,filter,secret,created_at,last_delivery_at,last_status,failures,paused FROM callbacks`481481 args := []any{}482482 switch {@@ -488,7 +488,7 @@ func (t *Tenant) callbackRows(ctx context.Context, owner string, ids ...string)
488488 args = append(args, owner)489489 }490490 query += ` ORDER BY created_at,id`491- rows, err := t.store.DB().QueryContext(ctx, query, args...)491+ rows, err := s.store.DB().QueryContext(ctx, query, args...)492492 if err != nil {493493 return nil, fmt.Errorf("list callbacks: %w", err)494494 }@@ -513,16 +513,16 @@ func (t *Tenant) callbackRows(ctx context.Context, owner string, ids ...string)
513513 // callbackRules returns the active callbacks indexed by kind. The index is514514 // built from the table on first use and dropped whenever a callback515515 // changes, so matching an event costs one map lookup.516-func (t *Tenant) callbackRules(ctx context.Context) map[int][]callbackRecord {517- t.callbackMu.Lock()518- defer t.callbackMu.Unlock()519- if t.callbackIndex != nil {520- return t.callbackIndex516+func (s *callbackService) callbackRules(ctx context.Context) map[int][]callbackRecord {517+ s.callbackMu.Lock()518+ defer s.callbackMu.Unlock()519+ if s.callbackIndex != nil {520+ return s.callbackIndex521521 }522522 index := map[int][]callbackRecord{}523- rows, err := t.callbackRows(ctx, "")523+ rows, err := s.callbackRows(ctx, "")524524 if err != nil {525- t.app.telemetry.Logger().Error("load callbacks", "tenant", t.meta.Name, "error", err)525+ s.telemetry.Logger().Error("load callbacks", "tenant", s.tenant.Name, "error", err)526526 return index527527 }528528 for _, row := range rows {@@ -533,52 +533,106 @@ func (t *Tenant) callbackRules(ctx context.Context) map[int][]callbackRecord {
533533 index[kind] = append(index[kind], row)534534 }535535 }536- t.callbackIndex = index536+ s.callbackIndex = index537537 return index538538 }539539 540-func (t *Tenant) invalidateCallbacks() {541- t.callbackMu.Lock()542- t.callbackIndex = nil543- t.callbackMu.Unlock()540+func (s *callbackService) invalidateCallbacks() {541+ s.callbackMu.Lock()542+ s.callbackIndex = nil543+ s.callbackMu.Unlock()544544 }545545 546-// notifyCallbacks runs after an event is stored. Each active callback whose547-// filter matches and whose key may see the event gets one delivery intent.548-// It never fails the publish; a callback is best effort on top of the549-// durable event.550-func (t *Tenant) notifyCallbacks(ctx context.Context, e event.Event) {551- candidates := t.callbackRules(ctx)[e.Kind]552- if len(candidates) == 0 {553- return546+// CandidateIDs captures the active callback registrations that match an547+// event. The caller must persist these IDs with the event so later planning548+// cannot include callbacks registered after the event was accepted.549+func (s *callbackService) CandidateIDs(ctx context.Context, e event.Event) ([]string, error) {550+ rows, err := s.callbackRows(ctx, "")551+ if err != nil {552+ return nil, err554553 }555- var intents []storage.Intent556- unauthorized := 0557- for _, candidate := range candidates {558- if !event.Matches(candidate.Filter.eventFilter(), e) {554+ ids := make([]string, 0, len(rows))555+ for _, candidate := range rows {556+ if candidate.Paused || !event.Matches(candidate.Filter.eventFilter(), e) {559557 continue560558 }561- if !t.gate.CanSee(ctx, e, relay.Session{PubKeys: []string{candidate.Owner}, RelayURL: t.RelayURL()}, nil) {562- unauthorized++559+ ids = append(ids, candidate.ID)560+ }561+ return ids, nil562+}563+564+// PrepareFor creates delivery intents for captured registration IDs. Each565+// registration is re-read so deletion, pausing, membership changes and566+// visibility changes take effect before an intent is created.567+func (s *callbackService) PrepareFor(ctx context.Context, e event.Event, ids []string) ([]storage.Intent, error) {568+ if len(ids) == 0 {569+ return nil, nil570+ }571+ intents := make([]storage.Intent, 0, len(ids))572+ seen := make(map[string]struct{}, len(ids))573+ for _, id := range ids {574+ if id == "" {563575 continue564576 }565- payload, err := json.Marshal(callbackPayload{Event: e, Attempt: 1})577+ if _, ok := seen[id]; ok {578+ continue579+ }580+ seen[id] = struct{}{}581+ candidate, err := s.callback(ctx, id)566582 if err != nil {583+ if strings.HasPrefix(err.Error(), "not found:") {584+ continue585+ }586+ return nil, err587+ }588+ if candidate.Paused || !event.Matches(candidate.Filter.eventFilter(), e) {589+ continue590+ }591+ if !s.callbackOwnerCanSee(ctx, candidate.Owner, e) {567592 continue568593 }594+ payload, err := json.Marshal(callbackPayload{Event: e, Attempt: 1})595+ if err != nil {596+ return nil, fmt.Errorf("encode callback delivery: %w", err)597+ }569598 intents = append(intents, storage.Intent{Kind: callbackDelivery, EventID: e.ID, Target: candidate.ID, Payload: string(payload)})570599 }600+ return intents, nil601+}602+603+func (s *callbackService) callbackOwnerCanSee(ctx context.Context, owner string, e event.Event) bool {604+ return s.gate.CanSee(ctx, e, relay.Session{PubKeys: []string{owner}, RelayURL: s.relayURL}, nil)605+}606+607+// Prepare preserves the original planning API for callers that do not yet608+// persist captured candidate IDs alongside the event.609+func (s *callbackService) Prepare(ctx context.Context, e event.Event) ([]storage.Intent, error) {610+ ids, err := s.CandidateIDs(ctx, e)611+ if err != nil {612+ return nil, err613+ }614+ return s.PrepareFor(ctx, e, ids)615+}616+617+// notifyCallbacks runs after an event is stored. It preserves the legacy618+// best-effort queueing path for callers that do not share the event tx.619+func (s *callbackService) notifyCallbacks(ctx context.Context, e event.Event) {620+ intents, err := s.Prepare(ctx, e)621+ if err != nil {622+ s.telemetry.Logger().Error("callback deliveries not prepared", "tenant", s.tenant.Name, "error", err)623+ return624+ }571625 if len(intents) == 0 {572626 return573627 }574- err := t.store.WithTx(ctx, func(tx *sql.Tx) error {628+ err = s.store.WithTx(ctx, func(tx *sql.Tx) error {575629 return storage.AddIntents(ctx, tx, intents, time.Now().Unix())576630 })577631 if err != nil {578- t.app.telemetry.Logger().Error("callback deliveries not queued", "tenant", t.meta.Name, "error", err)632+ s.telemetry.Logger().Error("callback deliveries not queued", "tenant", s.tenant.Name, "error", err)579633 return580634 }581- t.app.telemetry.Logger().Debug("callback deliveries queued", "tenant", t.meta.Name, "queued", len(intents), "unauthorized", unauthorized)635+ s.telemetry.Logger().Debug("callback deliveries queued", "tenant", s.tenant.Name, "queued", len(intents))582636 }583637 584638 func randomHex(size int) (string, error) {@@ -588,3 +642,17 @@ func randomHex(size int) (string, error) {
588642 }589643 return hex.EncodeToString(buffer), nil590644 }645+646+// Tenant wrappers keep management and work dispatch independent of the647+// callback service's storage and delivery state.648+func (t *Tenant) initCallbacks(ctx context.Context) error {649+ return t.callbacks.initCallbacks(ctx)650+}651+652+func (t *Tenant) callbackExecute(ctx context.Context, actor, method string, params []json.RawMessage) (any, error) {653+ return t.callbacks.callbackExecute(ctx, actor, method, params)654+}655+656+func (t *Tenant) notifyCallbacks(ctx context.Context, e event.Event) {657+ t.callbacks.notifyCallbacks(ctx, e)658+}@@ -44,27 +44,27 @@ func callbackSignature(secret string, body []byte) string {
4444 4545 // callbackHTTPClient refuses private addresses and redirects unless the4646 // relay itself runs on a loopback address.47-func (t *Tenant) callbackHTTPClient() *http.Client {48- if t.callbackClient != nil {49- return t.callbackClient47+func (s *callbackService) callbackHTTPClient() *http.Client {48+ if s.callbackClient != nil {49+ return s.callbackClient5050 }51- if t.loopbackRelay() {51+ if loopbackPublicURL(s.publicURL) {5252 return &http.Client{Timeout: callbackTimeout, Transport: &http.Transport{TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12}}, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}5353 }5454 return replication.NewPinnedClient(callbackTimeout)5555 }5656 5757 // callbackLock bounds delivery to one POST per callback at a time.58-func (t *Tenant) callbackLock(id string) *sync.Mutex {59- t.callbackMu.Lock()60- defer t.callbackMu.Unlock()61- if t.callbackLocks == nil {62- t.callbackLocks = map[string]*sync.Mutex{}58+func (s *callbackService) callbackLock(id string) *sync.Mutex {59+ s.callbackMu.Lock()60+ defer s.callbackMu.Unlock()61+ if s.callbackLocks == nil {62+ s.callbackLocks = map[string]*sync.Mutex{}6363 }64- lock := t.callbackLocks[id]64+ lock := s.callbackLocks[id]6565 if lock == nil {6666 lock = &sync.Mutex{}67- t.callbackLocks[id] = lock67+ s.callbackLocks[id] = lock6868 }6969 return lock7070 }@@ -72,8 +72,8 @@ func (t *Tenant) callbackLock(id string) *sync.Mutex {
7272 // handleCallbackDelivery serves one callback-delivery intent. It rechecks7373 // the callback and the key's access right before the POST, so a callback7474 // removed or paused after the event arrived delivers nothing.75-func (t *Tenant) handleCallbackDelivery(ctx context.Context, intent work.Intent) (err error) {76- ctx, finish := t.app.telemetry.Start(ctx, "callback")75+func (s *callbackService) handleCallbackDelivery(ctx context.Context, intent work.Intent) (err error) {76+ ctx, finish := s.telemetry.Start(ctx, "callback")7777 outcome := "error"7878 defer func() { finish(outcome) }()7979 var payload callbackPayload@@ -84,10 +84,10 @@ func (t *Tenant) handleCallbackDelivery(ctx context.Context, intent work.Intent)
8484 if payload.Attempt < 1 {8585 payload.Attempt = 18686 }87- lock := t.callbackLock(intent.Target)87+ lock := s.callbackLock(intent.Target)8888 lock.Lock()8989 defer lock.Unlock()90- record, err := t.callback(ctx, intent.Target)90+ record, err := s.callback(ctx, intent.Target)9191 if err != nil {9292 if strings.HasPrefix(err.Error(), "not found:") {9393 outcome = "paused"@@ -99,26 +99,26 @@ func (t *Tenant) handleCallbackDelivery(ctx context.Context, intent work.Intent)
9999 outcome = "paused"100100 return nil101101 }102- role, err := t.community.Role(ctx, record.Owner)102+ role, err := s.community.Role(ctx, record.Owner)103103 if err != nil {104104 return err105105 }106- if role == "" && record.Owner != t.Policy().Owner {106+ if role == "" && record.Owner != s.policy().Owner {107107 // The key lost its standing since it registered; stop until a108108 // person resumes the callback.109109 outcome = "unauthorized"110- return t.pauseCallback(ctx, record.ID, record.Failures, "paused: the key is no longer a member")110+ return s.pauseCallback(ctx, record.ID, record.Failures, "paused: the key is no longer a member")111111 }112- if !t.gate.CanSee(ctx, payload.Event, relay.Session{PubKeys: []string{record.Owner}, RelayURL: t.RelayURL()}, nil) {112+ if !s.gate.CanSee(ctx, payload.Event, relay.Session{PubKeys: []string{record.Owner}, RelayURL: s.relayURL}, nil) {113113 outcome = "unauthorized"114114 return nil115115 }116- status, postErr := t.postCallback(ctx, record, payload.Event)116+ status, postErr := s.postCallback(ctx, record, payload.Event)117117 now := time.Now().Unix()118118 if postErr == nil {119119 outcome = "ok"120- _, err = t.store.DB().ExecContext(ctx, `UPDATE callbacks SET last_delivery_at=?, last_status='ok', failures=0 WHERE id=?`, now, record.ID)121- t.app.telemetry.Logger().Info("callback delivered", "tenant", t.meta.Name, "attempt", payload.Attempt, "status", status)120+ _, err = s.store.DB().ExecContext(ctx, `UPDATE callbacks SET last_delivery_at=?, last_status='ok', failures=0 WHERE id=?`, now, record.ID)121+ s.telemetry.Logger().Info("callback delivered", "tenant", s.tenant.Name, "attempt", payload.Attempt, "status", status)122122 return err123123 }124124 // Counts only: the reason names the HTTP status or the failure class,@@ -128,32 +128,32 @@ func (t *Tenant) handleCallbackDelivery(ctx context.Context, intent work.Intent)
128128 paused := failures >= callbackPauseFailures129129 if paused {130130 outcome = "paused"131- if err := t.pauseCallback(ctx, record.ID, failures, fmt.Sprintf("paused after %d failures: %s", failures, reason)); err != nil {131+ if err := s.pauseCallback(ctx, record.ID, failures, fmt.Sprintf("paused after %d failures: %s", failures, reason)); err != nil {132132 return err133133 }134- } else if _, err := t.store.DB().ExecContext(ctx, `UPDATE callbacks SET failures=?, last_status=? WHERE id=?`, failures, reason, record.ID); err != nil {134+ } else if _, err := s.store.DB().ExecContext(ctx, `UPDATE callbacks SET failures=?, last_status=? WHERE id=?`, failures, reason, record.ID); err != nil {135135 return err136136 }137- t.app.telemetry.Logger().Info("callback delivery failed", "tenant", t.meta.Name, "attempt", payload.Attempt, "status", status, "failures", failures, "paused", paused)137+ s.telemetry.Logger().Info("callback delivery failed", "tenant", s.tenant.Name, "attempt", payload.Attempt, "status", status, "failures", failures, "paused", paused)138138 if paused || payload.Attempt >= callbackMaxAttempts {139139 return nil140140 }141- return t.retryCallback(ctx, intent, payload)141+ return s.retryCallback(ctx, intent, payload)142142 }143143 144144 // pauseCallback stops deliveries and records the count and reason, then145145 // drops the index so no new intents are queued for the callback.146-func (t *Tenant) pauseCallback(ctx context.Context, id string, failures int, status string) error {147- if _, err := t.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=1, failures=?, last_status=? WHERE id=?`, failures, status, id); err != nil {146+func (s *callbackService) pauseCallback(ctx context.Context, id string, failures int, status string) error {147+ if _, err := s.store.DB().ExecContext(ctx, `UPDATE callbacks SET paused=1, failures=?, last_status=? WHERE id=?`, failures, status, id); err != nil {148148 return err149149 }150- t.invalidateCallbacks()150+ s.invalidateCallbacks()151151 return nil152152 }153153 154154 // retryCallback queues the next attempt after its backoff. Each attempt is155155 // its own intent so the durable queue's ordering and fencing still apply.156-func (t *Tenant) retryCallback(ctx context.Context, intent work.Intent, payload callbackPayload) error {156+func (s *callbackService) retryCallback(ctx context.Context, intent work.Intent, payload callbackPayload) error {157157 delay := callbackBackoff[len(callbackBackoff)-1]158158 if payload.Attempt-1 < len(callbackBackoff) {159159 delay = callbackBackoff[payload.Attempt-1]@@ -165,7 +165,7 @@ func (t *Tenant) retryCallback(ctx context.Context, intent work.Intent, payload
165165 }166166 now := time.Now().Unix()167167 key := sha256.Sum256([]byte(callbackDelivery + "\x00" + payload.Event.ID + "\x00" + intent.Target + "\x00" + fmt.Sprint(payload.Attempt)))168- return t.store.WithTx(ctx, func(tx *sql.Tx) error {168+ return s.store.WithTx(ctx, func(tx *sql.Tx) error {169169 _, err := tx.ExecContext(ctx, `INSERT OR IGNORE INTO work_intents(id,kind,event_id,target,payload,next_at,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?)`, hex.EncodeToString(key[:]), callbackDelivery, payload.Event.ID+"#"+fmt.Sprint(payload.Attempt), intent.Target, string(encoded), now+int64(delay/time.Second), now, now)170170 return err171171 })@@ -173,7 +173,7 @@ func (t *Tenant) retryCallback(ctx context.Context, intent work.Intent, payload
173173 174174 // postCallback sends the event and returns the HTTP status with a bounded175175 // reason on failure.176-func (t *Tenant) postCallback(ctx context.Context, record callbackRecord, e event.Event) (int, error) {176+func (s *callbackService) postCallback(ctx context.Context, record callbackRecord, e event.Event) (int, error) {177177 body, err := json.Marshal(e)178178 if err != nil {179179 return 0, errors.New("encode event")@@ -187,9 +187,9 @@ func (t *Tenant) postCallback(ctx context.Context, record callbackRecord, e even
187187 request.Header.Set("Content-Type", "application/json")188188 request.Header.Set("X-Tiny-Callback", record.ID)189189 request.Header.Set("X-Tiny-Signature", callbackSignature(record.secret, body))190- request.Header.Set("X-Tiny-Relay", t.publicURL)190+ request.Header.Set("X-Tiny-Relay", s.publicURL)191191 request.Header.Set("User-Agent", "tinyrelay")192- response, err := t.callbackHTTPClient().Do(request)192+ response, err := s.callbackHTTPClient().Do(request)193193 if err != nil {194194 if errors.Is(err, context.DeadlineExceeded) || ctx.Err() != nil {195195 return 0, errors.New("timeout")@@ -203,3 +203,7 @@ func (t *Tenant) postCallback(ctx context.Context, record callbackRecord, e even
203203 }204204 return response.StatusCode, nil205205 }206+207+func (t *Tenant) handleCallbackDelivery(ctx context.Context, intent work.Intent) error {208+ return t.callbacks.handleCallbackDelivery(ctx, intent)209+}@@ -0,0 +1,50 @@
1+package daemon2+3+import (4+ "net/http"5+ "sync"6+7+ "github.com/FelineStateMachine/tinyrelay/internal/catalog"8+ "github.com/FelineStateMachine/tinyrelay/internal/community"9+ "github.com/FelineStateMachine/tinyrelay/internal/gates"10+ "github.com/FelineStateMachine/tinyrelay/internal/policy"11+ "github.com/FelineStateMachine/tinyrelay/internal/storage"12+ "github.com/FelineStateMachine/tinyrelay/internal/telemetry"13+)14+15+// callbackService owns callback registration, matching and durable delivery.16+// Its dependencies are explicit so callback state does not enlarge Tenant.17+type callbackService struct {18+ store *storage.Store19+ community *community.Service20+ gate *gates.Gate21+ policy func() policy.Policy22+ publicURL string23+ relayURL string24+ tenant catalog.Tenant25+ telemetry *telemetry.Telemetry26+27+ callbackMu sync.Mutex28+ callbackIndex map[int][]callbackRecord29+ callbackLocks map[string]*sync.Mutex30+ callbackClient *http.Client31+}32+33+type callbackServiceConfig struct {34+ Store *storage.Store35+ Community *community.Service36+ Gate *gates.Gate37+ Policy func() policy.Policy38+ PublicURL string39+ RelayURL string40+ Tenant catalog.Tenant41+ Telemetry *telemetry.Telemetry42+}43+44+func newCallbackService(cfg callbackServiceConfig) *callbackService {45+ return &callbackService{46+ store: cfg.Store, community: cfg.Community, gate: cfg.Gate,47+ policy: cfg.Policy, publicURL: cfg.PublicURL, relayURL: cfg.RelayURL,48+ tenant: cfg.Tenant, telemetry: cfg.Telemetry,49+ }50+}@@ -412,7 +412,7 @@ func TestCallbackDeliverySignsBodyRetriesAndPauses(t *testing.T) {
412412 receiver := &callbackReceiver{status: http.StatusAccepted}413413 service := httptest.NewTLSServer(receiver)414414 defer service.Close()415- tenant.callbackClient = service.Client()415+ tenant.callbacks.callbackClient = service.Client()416416 registered := addCallback(t, tenant, member, service.URL+"/wake?token=t0k3n", `{"kinds":[1621]}`)417417 id, secret := registered["id"].(string), registered["secret"].(string)418418 now := time.Now().Unix()@@ -561,7 +561,7 @@ func TestCallbackDeliveryRechecksTheGateAndMembership(t *testing.T) {
561561 receiver := &callbackReceiver{status: http.StatusOK}562562 service := httptest.NewTLSServer(receiver)563563 defer service.Close()564- tenant.callbackClient = service.Client()564+ tenant.callbacks.callbackClient = service.Client()565565 registered := addCallback(t, tenant, member, service.URL+"/wake", `{"kinds":[1]}`)566566 id := registered["id"].(string)567567 now := time.Now().Unix()@@ -0,0 +1,146 @@
1+package daemon2+3+import (4+ "context"5+ "database/sql"6+ "errors"7+ "strings"8+9+ "github.com/FelineStateMachine/tinyrelay/internal/community"10+ "github.com/FelineStateMachine/tinyrelay/internal/event"11+ "github.com/FelineStateMachine/tinyrelay/internal/policy"12+ "github.com/FelineStateMachine/tinyrelay/internal/relay"13+ "github.com/FelineStateMachine/tinyrelay/internal/replication"14+ "github.com/FelineStateMachine/tinyrelay/internal/storage"15+)16+17+type metadataProjection struct {18+ next policy.Policy19+ changed bool20+}21+22+func (t *Tenant) publishReport(ctx context.Context, e event.Event) (string, error) {23+ target := ""24+ targetType := ""25+ reportType := ""26+ for _, tag := range e.Tags {27+ if len(tag) > 1 && tag[0] == "e" && target == "" {28+ target = tag[1]29+ targetType = "event"30+ if len(tag) > 2 {31+ reportType = tag[2]32+ }33+ }34+ if len(tag) > 1 && tag[0] == "p" && target == "" {35+ target = tag[1]36+ targetType = "pubkey"37+ if len(tag) > 2 {38+ reportType = tag[2]39+ }40+ }41+ if len(tag) > 1 && tag[0] == "x" && target == "" {42+ target = tag[1]43+ targetType = "blob"44+ if len(tag) > 2 {45+ reportType = tag[2]46+ }47+ }48+ }49+ if target == "" {50+ return "", errors.New("invalid: report needs an e or p tag")51+ }52+ if err := t.community.SubmitReportTarget(ctx, e.PubKey, target, targetType, reportType, e.Content, t.Policy().ReportThreshold); err != nil {53+ return "", err54+ }55+ return "info: report received", nil56+}57+58+func (t *Tenant) prepareClientEvent(ctx context.Context, e event.Event, s relay.Session, now int64) (eventCommitPlan, *metadataProjection, error) {59+ metadata := &metadataProjection{}60+ plan, err := t.prepareStoredEvent(ctx, e, replication.OriginClient, now)61+ if err != nil {62+ return plan, metadata, err63+ }64+ opts := plan.options65+ if (e.Kind == event.KIND_EDIT_METADATA || e.Kind == event.KIND_PINS) && t.roomScope(e) == "" {66+ role, roleErr := t.community.Role(ctx, e.PubKey)67+ if roleErr != nil {68+ return plan, metadata, roleErr69+ }70+ if role != "owner" && role != "moderator" {71+ return plan, metadata, errors.New("restricted: not a group admin")72+ }73+ opts.AddBeforeCommit(func(txCtx context.Context, tx *sql.Tx) error {74+ return t.community.HandleProjectionEventTx(txCtx, tx, e, func(sideTx *sql.Tx) error {75+ if e.Kind == event.KIND_EDIT_METADATA {76+ metadata.next = t.Policy()77+ for _, tag := range e.Tags {78+ if len(tag) < 2 {79+ continue80+ }81+ switch tag[0] {82+ case "name":83+ metadata.next.Name = tag[1][:min(200, len(tag[1]))]84+ case "about":85+ metadata.next.Description = tag[1][:min(2000, len(tag[1]))]86+ case "picture":87+ metadata.next.Icon = tag[1][:min(2000, len(tag[1]))]88+ }89+ }90+ metadata.changed = true91+ return storage.PutSetting(txCtx, sideTx, "policy", metadata.next)92+ }93+ pins, err := parsePinTags(e.Tags)94+ if err != nil {95+ return err96+ }97+ return t.records.ReplacePinsTx(txCtx, sideTx, pins)98+ })99+ })100+ }101+ opts.Intents = append(opts.Intents, t.replication.Prepare(e, replication.OriginClient)...)102+ if e.Kind == event.KIND_MARMOT_GROUP {103+ principal, principalErr := t.gate.Principal(ctx, e, s)104+ if principalErr != nil {105+ return plan, metadata, principalErr106+ }107+ opts.AddBeforeCommit(func(txCtx context.Context, tx *sql.Tx) error {108+ _, err := tx.ExecContext(txCtx, `INSERT OR REPLACE INTO marmot_principals(event_id,pubkey) VALUES(?,?)`, e.ID, principal)109+ return err110+ })111+ }112+ plan.options = opts113+ return plan, metadata, nil114+}115+116+func (t *Tenant) persistClientEvent(ctx context.Context, e event.Event, opts storage.SaveOptions) (string, error) {117+ var err error118+ persist := func(tx *sql.Tx) error {119+ _, saveErr := storage.SaveTx(ctx, tx, e, opts)120+ return saveErr121+ }122+ // reason is the OK message; an access request tells the asker it waits,123+ // and a member asking again is told so.124+ reason := ""125+ room := t.roomScope(e)126+ switch {127+ case room != "" && community.RoomAdminKind(e.Kind):128+ _, err = t.community.HandleRoomEventTx(ctx, e, persist)129+ case room != "" && !event.IsEphemeral(e.Kind):130+ err = t.community.HandleRoomMessageTx(ctx, e, persist)131+ case e.Kind == event.KIND_JOIN, e.Kind == event.KIND_LEAVE, e.Kind == event.KIND_NIP43_JOIN, e.Kind == event.KIND_NIP43_LEAVE:132+ var membership community.MembershipResult133+ membership, err = t.community.HandleMembershipEventTx(ctx, e, persist)134+ if err == nil && (membership.AccessRequest || strings.HasPrefix(membership.Message, "duplicate:")) {135+ reason = membership.Message136+ }137+ if err == nil && membership.NewRequest {138+ t.notifyJoinRequest(ctx, e)139+ }140+ case e.Kind == event.KIND_PUT_USER, e.Kind == event.KIND_REMOVE_USER, e.Kind == event.KIND_DELETE_EVENT, e.Kind == event.KIND_CREATE_INVITE:141+ _, err = t.community.HandleModerationEventTx(ctx, e, persist)142+ default:143+ _, err = t.store.Save(ctx, e, opts)144+ }145+ return reason, err146+}@@ -8,20 +8,76 @@ package daemon
88 99 import (1010 "context"11+ "crypto/sha256"1112 "database/sql"13+ "encoding/hex"1214 "encoding/json"1315 "errors"1416 "fmt"17+ "net/http"1518 "sort"1619 "strconv"1720 "strings"21+ "sync"1822 "time"1923 24+ "github.com/FelineStateMachine/tinyrelay/internal/community"2025 "github.com/FelineStateMachine/tinyrelay/internal/event"26+ "github.com/FelineStateMachine/tinyrelay/internal/gitrelay"27+ "github.com/FelineStateMachine/tinyrelay/internal/records"2128 "github.com/FelineStateMachine/tinyrelay/internal/storage"29+ "github.com/FelineStateMachine/tinyrelay/internal/telemetry"2230 "github.com/FelineStateMachine/tinyrelay/internal/views"2331 )2432 33+// customViewService owns custom view definitions, transform execution and34+// artifact storage. Tenant supplies the narrow lifecycle callbacks when it35+// constructs the service.36+type customViewService struct {37+ store *storage.Store38+ community *community.Service39+ records *records.Service40+ git *gitrelay.GitRelay41+ telemetry *telemetry.Telemetry42+ tenantName string43+ publicURL string44+ loopback func() bool45+ resolveActor func(*http.Request) (string, error)46+ browseRead func(context.Context, string) error47+ viewClient *http.Client48+ viewMu sync.Mutex49+ viewIndex map[int][]customView50+ viewLocks map[string]*sync.Mutex51+}52+53+type customViewServiceConfig struct {54+ Store *storage.Store55+ Community *community.Service56+ Records *records.Service57+ Git *gitrelay.GitRelay58+ Telemetry *telemetry.Telemetry59+ TenantName string60+ PublicURL string61+ Loopback func() bool62+ ResolveActor func(*http.Request) (string, error)63+ BrowseRead func(context.Context, string) error64+}65+66+func newCustomViewService(cfg customViewServiceConfig) *customViewService {67+ return &customViewService{68+ store: cfg.Store,69+ community: cfg.Community,70+ records: cfg.Records,71+ git: cfg.Git,72+ telemetry: cfg.Telemetry,73+ tenantName: cfg.TenantName,74+ publicURL: cfg.PublicURL,75+ loopback: cfg.Loopback,76+ resolveActor: cfg.ResolveActor,77+ browseRead: cfg.BrowseRead,78+ }79+}80+2581 const (2682 viewTransform = "view-transform"2783 viewKindMax = 16@@ -40,7 +96,7 @@ const (
4096 // viewBackoff is the wait before the second and third attempt.4197 var viewBackoff = []time.Duration{time.Minute, 5 * time.Minute}4298 43-const customViewSchema = `CREATE TABLE IF NOT EXISTS custom_views(name TEXT PRIMARY KEY, kinds TEXT NOT NULL, transform TEXT NOT NULL, trigger TEXT NOT NULL, audience TEXT NOT NULL, languages TEXT NOT NULL, max_bytes INTEGER NOT NULL, secret TEXT NOT NULL, enabled INTEGER NOT NULL DEFAULT 1, created_at INTEGER NOT NULL, last_run_at INTEGER NOT NULL DEFAULT 0, last_status TEXT NOT NULL DEFAULT '', failures INTEGER NOT NULL DEFAULT 0);99+const customViewSchema = `CREATE TABLE IF NOT EXISTS custom_views(name TEXT PRIMARY KEY, kinds TEXT NOT NULL, transform TEXT NOT NULL, trigger TEXT NOT NULL, audience TEXT NOT NULL, languages TEXT NOT NULL, max_bytes INTEGER NOT NULL, secret TEXT NOT NULL, generation TEXT NOT NULL DEFAULT '', enabled INTEGER NOT NULL DEFAULT 1, created_at INTEGER NOT NULL, last_run_at INTEGER NOT NULL DEFAULT 0, last_status TEXT NOT NULL DEFAULT '', failures INTEGER NOT NULL DEFAULT 0);44100 CREATE TABLE IF NOT EXISTS custom_view_artifacts(view TEXT NOT NULL, hash TEXT NOT NULL, event_id TEXT NOT NULL, type TEXT NOT NULL, engine TEXT NOT NULL DEFAULT '', audience TEXT NOT NULL, content TEXT NOT NULL, raw TEXT NOT NULL, created_at INTEGER NOT NULL, PRIMARY KEY(view,hash));45101 CREATE TABLE IF NOT EXISTS custom_view_sources(view TEXT NOT NULL, hash TEXT NOT NULL, source TEXT NOT NULL, block INTEGER NOT NULL, expires INTEGER NOT NULL DEFAULT 0, PRIMARY KEY(view,hash,source));46102 CREATE INDEX IF NOT EXISTS custom_view_sources_source ON custom_view_sources(view,source);`@@ -61,6 +117,30 @@ type customView struct {
61117 LastStatus string `json:"lastStatus"`62118 Failures int `json:"failures"`63119 secret string120+ generation string121+}122+123+type customViewTarget struct {124+ Name string125+ Fingerprint string126+}127+128+func customViewFingerprint(view customView) string {129+ canonical := struct {130+ Name string131+ Kinds []int132+ Transform string133+ Trigger string134+ Audience string135+ Languages []string136+ MaxBytes int137+ Secret string138+ CreatedAt int64139+ Generation string140+ }{view.Name, view.Kinds, view.Transform, view.Trigger, view.Audience, view.Languages, view.MaxBytes, view.secret, view.CreatedAt, view.generation}141+ raw, _ := json.Marshal(canonical)142+ sum := sha256.Sum256(raw)143+ return hex.EncodeToString(sum[:])64144 }65145 66146 func (v customView) summary() map[string]any {@@ -80,9 +160,40 @@ func (v customView) watches(kind int) bool {
80160 return false81161 }82162 83-func (t *Tenant) initCustomViews(ctx context.Context) error {84- _, err := t.store.DB().ExecContext(ctx, customViewSchema)85- return err163+func (t *customViewService) initCustomViews(ctx context.Context) error {164+ if _, err := t.store.DB().ExecContext(ctx, customViewSchema); err != nil {165+ return err166+ }167+ _, err := t.store.DB().ExecContext(ctx, `ALTER TABLE custom_views ADD COLUMN generation TEXT NOT NULL DEFAULT ''`)168+ if err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {169+ return err170+ }171+ rows, err := t.store.DB().QueryContext(ctx, `SELECT name FROM custom_views WHERE generation=''`)172+ if err != nil {173+ return err174+ }175+ defer rows.Close()176+ var names []string177+ for rows.Next() {178+ var name string179+ if err := rows.Scan(&name); err != nil {180+ return err181+ }182+ names = append(names, name)183+ }184+ if err := rows.Err(); err != nil {185+ return err186+ }187+ for _, name := range names {188+ generation, err := randomHex(16)189+ if err != nil {190+ return err191+ }192+ if _, err := t.store.DB().ExecContext(ctx, `UPDATE custom_views SET generation=? WHERE name=? AND generation=''`, generation, name); err != nil {193+ return err194+ }195+ }196+ return nil86197 }87198 88199 func customViewMethod(method string) bool {@@ -91,8 +202,8 @@ func customViewMethod(method string) bool {
91202 92203 // customViewExecute serves the owner's custom view methods under the view93204 // operation. Every change is recorded in the audit log.94-func (t *Tenant) customViewExecute(ctx context.Context, actor, method string, params []json.RawMessage) (result any, err error) {95- ctx, finish := t.app.telemetry.Start(ctx, "view")205+func (t *customViewService) customViewExecute(ctx context.Context, actor, method string, params []json.RawMessage) (result any, err error) {206+ ctx, finish := t.telemetry.Start(ctx, "view")96207 outcome := "error"97208 defer func() {98209 if err != nil {@@ -143,7 +254,7 @@ type customViewOptions struct {
143254 Secret string `json:"secret"`144255 }145256 146-func (t *Tenant) addCustomView(ctx context.Context, actor string, params []json.RawMessage) (any, error) {257+func (t *customViewService) addCustomView(ctx context.Context, actor string, params []json.RawMessage) (any, error) {147258 if len(params) != 1 {148259 return nil, errors.New("invalid: addcustomview expects one object with name, kinds, transform, trigger, audience, languages and optional max_bytes and secret")149260 }@@ -167,11 +278,15 @@ func (t *Tenant) addCustomView(ctx context.Context, actor string, params []json.
167278 return nil, fmt.Errorf("invalid: secret must be %d to %d printable ASCII characters", callbackSecretMin, callbackSecretMax)168279 }169280 view.secret = secret281+ view.generation, err = randomHex(16)282+ if err != nil {283+ return nil, err284+ }170285 view.Enabled = true171286 view.CreatedAt = time.Now().Unix()172287 kinds, _ := json.Marshal(view.Kinds)173288 languages, _ := json.Marshal(view.Languages)174- _, err = t.store.DB().ExecContext(ctx, `INSERT INTO custom_views(name,kinds,transform,trigger,audience,languages,max_bytes,secret,enabled,created_at) VALUES(?,?,?,?,?,?,?,?,1,?)`, view.Name, string(kinds), view.Transform, view.Trigger, view.Audience, string(languages), view.MaxBytes, secret, view.CreatedAt)289+ _, err = t.store.DB().ExecContext(ctx, `INSERT INTO custom_views(name,kinds,transform,trigger,audience,languages,max_bytes,secret,generation,enabled,created_at) VALUES(?,?,?,?,?,?,?,?,?,1,?)`, view.Name, string(kinds), view.Transform, view.Trigger, view.Audience, string(languages), view.MaxBytes, secret, view.generation, view.CreatedAt)175290 if err != nil {176291 if strings.Contains(err.Error(), "UNIQUE") {177292 return nil, errors.New("invalid: a view with that name exists")@@ -183,7 +298,7 @@ func (t *Tenant) addCustomView(ctx context.Context, actor string, params []json.
183298 return nil, err184299 }185300 // Counts only: the transform host, path and secret stay out of the log.186- t.app.telemetry.Logger().Info("custom view added", "tenant", t.meta.Name, "kinds", len(view.Kinds), "languages", len(view.Languages), "generated", generated)301+ t.telemetry.Logger().Info("custom view added", "tenant", t.tenantName, "kinds", len(view.Kinds), "languages", len(view.Languages), "generated", generated)187302 out := view.summary()188303 out["secret"] = secret189304 return out, nil@@ -192,7 +307,7 @@ func (t *Tenant) addCustomView(ctx context.Context, actor string, params []json.
192307 // parseCustomView validates a definition: the name, the kinds it watches,193308 // the https transform, the trigger, the audience, the languages and the194309 // artifact size limit.195-func (t *Tenant) parseCustomView(options customViewOptions) (customView, error) {310+func (t *customViewService) parseCustomView(options customViewOptions) (customView, error) {196311 view := customView{Name: strings.TrimSpace(options.Name), Transform: strings.TrimSpace(options.Transform), Trigger: strings.TrimSpace(options.Trigger), Audience: strings.TrimSpace(options.Audience), MaxBytes: viewDefaultMaxBytes}197312 if !views.NamePattern.MatchString(view.Name) {198313 return customView{}, errors.New("invalid: name must be 1 to 32 lowercase letters, digits or hyphens")@@ -208,7 +323,7 @@ func (t *Tenant) parseCustomView(options customViewOptions) (customView, error)
208323 }209324 sort.Ints(kinds)210325 view.Kinds = dedupeInts(kinds)211- if err := t.checkCallbackURL(view.Transform); err != nil {326+ if err := validateWebhookURL(view.Transform, t.publicURL); err != nil {212327 return customView{}, fmt.Errorf("invalid: transform %s", strings.TrimPrefix(err.Error(), "invalid: "))213328 }214329 if view.Trigger == "" {@@ -313,7 +428,7 @@ func dedupeInts(values []int) []int {
313428 return out314429 }315430 316-func (t *Tenant) changeCustomView(ctx context.Context, actor, method string, params []json.RawMessage) (any, error) {431+func (t *customViewService) changeCustomView(ctx context.Context, actor, method string, params []json.RawMessage) (any, error) {317432 name := customViewNameParam(params)318433 if name == "" {319434 return nil, errors.New("invalid: view name required")@@ -345,7 +460,7 @@ func (t *Tenant) changeCustomView(ctx context.Context, actor, method string, par
345460 if err := t.community.Record(ctx, actor, method, name, strconv.Itoa(queued)); err != nil {346461 return nil, err347462 }348- t.app.telemetry.Logger().Info("custom view run queued", "tenant", t.meta.Name, "queued", queued)463+ t.telemetry.Logger().Info("custom view run queued", "tenant", t.tenantName, "queued", queued)349464 out := view.summary()350465 out["queued"] = queued351466 return out, nil@@ -367,7 +482,7 @@ func (t *Tenant) changeCustomView(ctx context.Context, actor, method string, par
367482 368483 // removeCustomView deletes the definition and every artifact it produced,369484 // including the stored public records.370-func (t *Tenant) removeCustomView(ctx context.Context, view customView) error {485+func (t *customViewService) removeCustomView(ctx context.Context, view customView) error {371486 ids, err := t.customViewArtifactIDs(ctx, `SELECT event_id FROM custom_view_artifacts WHERE view=?`, view.Name)372487 if err != nil {373488 return err@@ -417,7 +532,7 @@ func customViewRefreshParam(params []json.RawMessage) bool {
417532 return options.Refresh418533 }419534 420-func (t *Tenant) customViewByName(ctx context.Context, name string) (customView, error) {535+func (t *customViewService) customViewByName(ctx context.Context, name string) (customView, error) {421536 rows, err := t.customViewRows(ctx, name)422537 if err != nil {423538 return customView{}, err@@ -429,8 +544,8 @@ func (t *Tenant) customViewByName(ctx context.Context, name string) (customView,
429544 }430545 431546 // customViewRows lists definitions, narrowed to one name.432-func (t *Tenant) customViewRows(ctx context.Context, names ...string) ([]customView, error) {433- query := `SELECT name,kinds,transform,trigger,audience,languages,max_bytes,secret,enabled,created_at,last_run_at,last_status,failures FROM custom_views`547+func (t *customViewService) customViewRows(ctx context.Context, names ...string) ([]customView, error) {548+ query := `SELECT name,kinds,transform,trigger,audience,languages,max_bytes,secret,generation,enabled,created_at,last_run_at,last_status,failures FROM custom_views`434549 args := []any{}435550 if len(names) > 0 {436551 query += ` WHERE name=?`@@ -447,7 +562,7 @@ func (t *Tenant) customViewRows(ctx context.Context, names ...string) ([]customV
447562 var item customView448563 var kinds, languages string449564 var enabled int450- if err := rows.Scan(&item.Name, &kinds, &item.Transform, &item.Trigger, &item.Audience, &languages, &item.MaxBytes, &item.secret, &enabled, &item.CreatedAt, &item.LastRunAt, &item.LastStatus, &item.Failures); err != nil {565+ if err := rows.Scan(&item.Name, &kinds, &item.Transform, &item.Trigger, &item.Audience, &languages, &item.MaxBytes, &item.secret, &item.generation, &enabled, &item.CreatedAt, &item.LastRunAt, &item.LastStatus, &item.Failures); err != nil {451566 return nil, err452567 }453568 item.Enabled = enabled != 0@@ -464,7 +579,7 @@ func (t *Tenant) customViewRows(ctx context.Context, names ...string) ([]customV
464579 465580 // customViewIndex returns the enabled views indexed by kind, built on first466581 // use and dropped whenever a view changes.467-func (t *Tenant) customViewIndex(ctx context.Context) map[int][]customView {582+func (t *customViewService) customViewIndex(ctx context.Context) map[int][]customView {468583 t.viewMu.Lock()469584 defer t.viewMu.Unlock()470585 if t.viewIndex != nil {@@ -473,7 +588,7 @@ func (t *Tenant) customViewIndex(ctx context.Context) map[int][]customView {
473588 index := map[int][]customView{}474589 rows, err := t.customViewRows(ctx)475590 if err != nil {476- t.app.telemetry.Logger().Error("load custom views", "tenant", t.meta.Name, "error", err)591+ t.telemetry.Logger().Error("load custom views", "tenant", t.tenantName, "error", err)477592 return index478593 }479594 for _, row := range rows {@@ -488,7 +603,7 @@ func (t *Tenant) customViewIndex(ctx context.Context) map[int][]customView {
488603 return index489604 }490605 491-func (t *Tenant) invalidateCustomViews() {606+func (t *customViewService) invalidateCustomViews() {492607 t.viewMu.Lock()493608 t.viewIndex = nil494609 t.viewMu.Unlock()@@ -496,7 +611,7 @@ func (t *Tenant) invalidateCustomViews() {
496611 497612 // CustomViews is the summary the web UI renders with: the enabled views'498613 // names and languages, nothing else.499-func (t *Tenant) CustomViews() []views.View {614+func (t *customViewService) CustomViews() []views.View {500615 seen := map[string]bool{}501616 var out []views.View502617 for _, list := range t.customViewIndex(context.Background()) {@@ -514,45 +629,135 @@ func (t *Tenant) CustomViews() []views.View {
514629 // queueCustomViews runs after an event is stored or released. Each enabled515630 // write-triggered view that watches the kind and finds a block it renders516631 // gets one transform intent. It never fails the publish.517-func (t *Tenant) queueCustomViews(ctx context.Context, e event.Event) {518- candidates := t.customViewIndex(ctx)[e.Kind]519- if len(candidates) == 0 {632+func (t *customViewService) queueCustomViews(ctx context.Context, e event.Event) {633+ intents := t.PrepareReleased(ctx, e)634+ if len(intents) == 0 {520635 return521636 }522- if e.Kind == event.KIND_REPO_STATE && (t.git == nil || t.statePending(ctx, e.ID)) {523- // The README is read from the objects, which have not arrived yet;524- // releaseGit queues the state once they have.637+ err := t.store.WithTx(ctx, func(tx *sql.Tx) error {638+ return storage.AddIntents(ctx, tx, intents, time.Now().Unix())639+ })640+ if err != nil {641+ t.telemetry.Logger().Error("view transforms not queued", "tenant", t.tenantName, "error", err)525642 return526643 }527- var intents []storage.Intent528- for _, view := range candidates {529- if view.Trigger != "write" {644+ t.telemetry.Logger().Debug("view transforms queued", "tenant", t.tenantName, "queued", len(intents))645+}646+647+// Prepare returns write-triggered transform intents for an event. The caller648+// owns the transaction that persists the returned intents, allowing event and649+// follow-up work to share one commit boundary.650+func (t *customViewService) Prepare(ctx context.Context, e event.Event) []storage.Intent {651+ targets, err := t.CandidateTargets(ctx, e)652+ if err != nil {653+ t.telemetry.Logger().Error("plan custom view transforms", "tenant", t.tenantName, "error", err)654+ return nil655+ }656+ intents, err := t.PrepareTargets(ctx, e, targets, false)657+ if err != nil {658+ t.telemetry.Logger().Error("prepare custom view transforms", "tenant", t.tenantName, "error", err)659+ return nil660+ }661+ return intents662+}663+664+// PrepareReleased returns intents after repository promotion has made Git665+// objects available. Repository state events are intentionally accepted only666+// through this path.667+func (t *customViewService) PrepareReleased(ctx context.Context, e event.Event) []storage.Intent {668+ targets, err := t.CandidateTargets(ctx, e)669+ if err != nil {670+ t.telemetry.Logger().Error("plan released custom view transforms", "tenant", t.tenantName, "error", err)671+ return nil672+ }673+ intents, err := t.PrepareTargets(ctx, e, targets, true)674+ if err != nil {675+ t.telemetry.Logger().Error("prepare released custom view transforms", "tenant", t.tenantName, "error", err)676+ return nil677+ }678+ return intents679+}680+681+// CandidateTargets captures names and registration identities for an event.682+// The identity prevents a later registration with the same name from683+// receiving work planned for an earlier registration.684+func (t *customViewService) CandidateTargets(ctx context.Context, e event.Event) ([]customViewTarget, error) {685+ if eventExpired(e, time.Now().Unix()) {686+ return nil, nil687+ }688+ rows, err := t.customViewRows(ctx)689+ if err != nil {690+ return nil, err691+ }692+ targets := make([]customViewTarget, 0, len(rows))693+ for _, view := range rows {694+ if !view.Enabled || view.Trigger != "write" || !view.watches(e.Kind) {530695 continue531696 }532697 if e.Kind != event.KIND_REPO_STATE && len(views.Matching(views.Blocks(e.Content), view.Languages)) == 0 {533698 continue534699 }535- intents = append(intents, t.viewIntent(view, e, "", false))700+ targets = append(targets, customViewTarget{Name: view.Name, Fingerprint: customViewFingerprint(view)})536701 }537- if len(intents) == 0 {538- return702+ return targets, nil703+}704+705+// PrepareTargets rechecks captured registration identities before creating706+// intents. A removed and recreated view with the same name is rejected.707+func (t *customViewService) PrepareTargets(ctx context.Context, e event.Event, targets []customViewTarget, released bool) ([]storage.Intent, error) {708+ if len(targets) == 0 || eventExpired(e, time.Now().Unix()) || (!released && e.Kind == event.KIND_REPO_STATE) {709+ return nil, nil539710 }540- err := t.store.WithTx(ctx, func(tx *sql.Tx) error {541- return storage.AddIntents(ctx, tx, intents, time.Now().Unix())542- })711+ if e.Kind != event.KIND_REPO_STATE {712+ current, err := t.sourceCurrent(ctx, e)713+ if err != nil {714+ return nil, err715+ }716+ if !current {717+ return nil, nil718+ }719+ }720+ rows, err := t.customViewRows(ctx)543721 if err != nil {544- t.app.telemetry.Logger().Error("view transforms not queued", "tenant", t.meta.Name, "error", err)545- return722+ return nil, err723+ }724+ candidates := make(map[string]customView, len(rows))725+ for _, view := range rows {726+ candidates[view.Name] = view727+ }728+ if e.Kind == event.KIND_REPO_STATE {729+ if t.git == nil || t.statePending(ctx, e.ID) {730+ return nil, nil731+ }546732 }547- t.app.telemetry.Logger().Debug("view transforms queued", "tenant", t.meta.Name, "queued", len(intents))733+ var intents []storage.Intent734+ for _, target := range targets {735+ view, ok := candidates[target.Name]736+ if !ok || !view.Enabled || view.Trigger != "write" || !view.watches(e.Kind) {737+ continue738+ }739+ if customViewFingerprint(view) != target.Fingerprint {740+ continue741+ }742+ if e.Kind != event.KIND_REPO_STATE && len(views.Matching(views.Blocks(e.Content), view.Languages)) == 0 {743+ continue744+ }745+ intents = append(intents, t.viewIntent(view, e, "", false))746+ }747+ return intents, nil748+}749+750+func eventExpired(e event.Event, now int64) bool {751+ expires := event.Expiration(e)752+ return expires > 0 && expires <= now548753 }549754 550755 // viewIntent is the transform intent for one source. The event travels in551756 // the payload; run makes a backfill or an hourly pass distinct from the552757 // write-time intent for the same event.553-func (t *Tenant) viewIntent(view customView, e event.Event, run string, refresh ...bool) storage.Intent {758+func (t *customViewService) viewIntent(view customView, e event.Event, run string, refresh ...bool) storage.Intent {554759 force := len(refresh) > 0 && refresh[0]555- payload, _ := json.Marshal(viewPayload{Event: e, Attempt: 1, Run: run, Refresh: force})760+ payload, _ := json.Marshal(viewPayload{Event: e, Attempt: 1, Run: run, Refresh: force, Generation: view.generation})556761 id := e.ID557762 if run != "" {558763 id += "@" + run@@ -562,7 +767,7 @@ func (t *Tenant) viewIntent(view customView, e event.Event, run string, refresh
562767 563768 // statePending reports whether a repository state still waits for its564769 // objects.565-func (t *Tenant) statePending(ctx context.Context, id string) bool {770+func (t *customViewService) statePending(ctx context.Context, id string) bool {566771 var one int567772 err := t.store.DB().QueryRowContext(ctx, `SELECT 1 FROM pending_events WHERE id=?`, id).Scan(&one)568773 return err == nil || t.git.IsPending(id)@@ -571,7 +776,7 @@ func (t *Tenant) statePending(ctx context.Context, id string) bool {
571776 // queueCustomViewBackfill queues the newest sources of a view's kinds, at572777 // most 500, written since the given time. Repository states are queued573778 // whether or not their README has blocks; the handler reads it.574-func (t *Tenant) queueCustomViewBackfill(ctx context.Context, view customView, since int64, run string, refresh ...bool) (int, error) {779+func (t *customViewService) queueCustomViewBackfill(ctx context.Context, view customView, since int64, run string, refresh ...bool) (int, error) {575780 force := len(refresh) > 0 && refresh[0]576781 rows, err := t.store.Query(ctx, event.Filter{Kinds: view.Kinds, Tags: map[string][]string{}}, storage.QueryOptions{Now: time.Now().Unix(), Access: storage.Access{All: true}, Limit: viewBackfillLimit})577782 if err != nil {@@ -602,7 +807,7 @@ func (t *Tenant) queueCustomViewBackfill(ctx context.Context, view customView, s
602807 603808 // tickCustomViews queues the hourly views that are due. It is called from604809 // the tenant scheduler on its own timer.605-func (t *Tenant) tickCustomViews(ctx context.Context, now int64) error {810+func (t *customViewService) tickCustomViews(ctx context.Context, now int64) error {606811 rows, err := t.customViewRows(ctx)607812 if err != nil {608813 return err@@ -623,7 +828,7 @@ func (t *Tenant) tickCustomViews(ctx context.Context, now int64) error {
623828 return err624829 }625830 if queued > 0 {626- t.app.telemetry.Logger().Info("custom view hourly run queued", "tenant", t.meta.Name, "queued", queued)831+ t.telemetry.Logger().Info("custom view hourly run queued", "tenant", t.tenantName, "queued", queued)627832 }628833 }629834 return nil@@ -18,8 +18,8 @@ func isViewArtifactPath(path string) bool {
1818 return strings.HasPrefix(path, "/views/")1919 }2020 21-func (t *Tenant) viewArtifactHTTP(w http.ResponseWriter, r *http.Request) {22- ctx, finish := t.app.telemetry.Start(r.Context(), "view")21+func (t *customViewService) viewArtifactHTTP(w http.ResponseWriter, r *http.Request) {22+ ctx, finish := t.telemetry.Start(r.Context(), "view")2323 r = r.WithContext(ctx)2424 outcome := "error"2525 defer func() { finish(outcome) }()@@ -41,7 +41,7 @@ func (t *Tenant) viewArtifactHTTP(w http.ResponseWriter, r *http.Request) {
4141 outcome = "invalid"4242 return4343 }44- actor, err := t.resolveUIActor(r)44+ actor, err := t.resolveActor(r)4545 if err != nil {4646 http.Error(w, err.Error(), http.StatusUnauthorized)4747 outcome = "unauthorized"@@ -0,0 +1,91 @@
1+package daemon2+3+import (4+ "context"5+ "testing"6+ "time"7+8+ "github.com/FelineStateMachine/tinyrelay/internal/work"9+)10+11+func TestCustomViewTargetsRejectRecreatedRegistration(t *testing.T) {12+ tenant, server, _ := viewTenant(t, []int{1}, nil)13+ ctx := context.Background()14+ note := signedEvent(t, testOwnerSecret, 1, time.Now().Unix(), nil, "```mermaid\ngraph TD; a-->b;\n```")15+ if err := publishAs(t, tenant, note); err != nil {16+ t.Fatal(err)17+ }18+ targets, err := tenant.customViews.CandidateTargets(ctx, note)19+ if err != nil || len(targets) != 1 {20+ t.Fatalf("candidate targets = %+v, %v", targets, err)21+ }22+ oldIntents, err := tenant.customViews.PrepareTargets(ctx, note, targets, false)23+ if err != nil || len(oldIntents) != 1 {24+ t.Fatalf("planned intents = %+v, %v", oldIntents, err)25+ }26+ if _, err := tenant.Execute(ctx, tenant.Policy().Owner, "removecustomview", viewParams(map[string]any{"name": "diagrams"})); err != nil {27+ t.Fatal(err)28+ }29+ addView(t, tenant, map[string]any{30+ "name": "diagrams",31+ "kinds": []int{1},32+ "transform": "https://render.example/tiny",33+ "languages": []string{"mermaid"},34+ "secret": "same-secret-value-1234",35+ })36+ targets, err = tenant.customViews.CandidateTargets(ctx, note)37+ if err != nil || len(targets) != 1 {38+ t.Fatalf("replacement candidate targets = %+v, %v", targets, err)39+ }40+ if _, err := tenant.Execute(ctx, tenant.Policy().Owner, "removecustomview", viewParams(map[string]any{"name": "diagrams"})); err != nil {41+ t.Fatal(err)42+ }43+ addView(t, tenant, map[string]any{44+ "name": "diagrams",45+ "kinds": []int{1},46+ "transform": "https://render.example/tiny",47+ "languages": []string{"mermaid"},48+ "secret": "same-secret-value-1234",49+ })50+ intents, err := tenant.customViews.PrepareTargets(ctx, note, targets, false)51+ if err != nil {52+ t.Fatal(err)53+ }54+ if len(intents) != 0 {55+ t.Fatalf("recreated view received old targets: %+v", intents)56+ }57+ if err := tenant.customViews.handleViewTransform(ctx, work.Intent{ID: "stale", Kind: oldIntents[0].Kind, EventID: oldIntents[0].EventID, Target: oldIntents[0].Target, Payload: oldIntents[0].Payload}); err != nil {58+ t.Fatal(err)59+ }60+ if server.count() != 0 {61+ t.Fatalf("stale generation contacted transform: %d requests", server.count())62+ }63+}64+65+func TestCustomViewGenerationBackfillIsStable(t *testing.T) {66+ tenant, _, _ := viewTenant(t, []int{1}, nil)67+ ctx := context.Background()68+ if _, err := tenant.store.DB().ExecContext(ctx, `ALTER TABLE custom_views DROP COLUMN generation`); err != nil {69+ t.Fatal(err)70+ }71+ if err := tenant.customViews.initCustomViews(ctx); err != nil {72+ t.Fatal(err)73+ }74+ var first string75+ if err := tenant.store.DB().QueryRowContext(ctx, `SELECT generation FROM custom_views WHERE name='diagrams'`).Scan(&first); err != nil {76+ t.Fatal(err)77+ }78+ if first == "" {79+ t.Fatal("legacy view generation was not backfilled")80+ }81+ if err := tenant.customViews.initCustomViews(ctx); err != nil {82+ t.Fatal(err)83+ }84+ var second string85+ if err := tenant.store.DB().QueryRowContext(ctx, `SELECT generation FROM custom_views WHERE name='diagrams'`).Scan(&second); err != nil {86+ t.Fatal(err)87+ }88+ if first != second {89+ t.Fatalf("generation changed on reopen: %q != %q", first, second)90+ }91+}@@ -92,7 +92,7 @@ func viewTenant(t *testing.T, kinds []int, extra map[string]any) (*Tenant, *tran
9292 server.answer(http.StatusOK)9393 service := httptest.NewTLSServer(server)9494 t.Cleanup(service.Close)95- tenant.viewClient = service.Client()95+ tenant.customViews.viewClient = service.Client()9696 values := map[string]any{"name": "diagrams", "kinds": kinds, "transform": service.URL + "/render?token=t0k3n", "languages": []string{"mermaid", "dot"}}9797 for key, value := range extra {9898 values[key] = value@@ -126,7 +126,7 @@ func pendingViewIntents(t *testing.T, tenant *Tenant, name, eventID string) []wo
126126 // complete.127127 func runViewIntent(t *testing.T, tenant *Tenant, intent work.Intent) {128128 t.Helper()129- if err := tenant.handleViewTransform(context.Background(), intent); err != nil {129+ if err := tenant.customViews.handleViewTransform(context.Background(), intent); err != nil {130130 t.Fatalf("handleViewTransform: %v", err)131131 }132132 if _, err := tenant.store.DB().ExecContext(context.Background(), "UPDATE work_intents SET state='completed' WHERE id=?", intent.ID); err != nil {@@ -301,7 +301,7 @@ func TestCustomViewManagementIsOwnerOnlyAndValidated(t *testing.T) {
301301 if _, err := tenant.Execute(ctx, owner, "runcustomview", []json.RawMessage{json.RawMessage(`"diagrams"`)}); err == nil || !strings.Contains(err.Error(), "resume") {302302 t.Fatalf("run while paused: %v", err)303303 }304- for _, view := range tenant.customViewIndex(ctx)[1] {304+ for _, view := range tenant.customViews.customViewIndex(ctx)[1] {305305 if view.Name == "diagrams" {306306 t.Fatal("paused view stayed in the kind index")307307 }@@ -557,8 +557,8 @@ func TestCustomViewReplacementReusesAndDropsBlocks(t *testing.T) {
557557 t.Fatal("B survived the replacement")558558 }559559 // A stale intent for the replaced version renders nothing.560- stale := tenant.viewIntent(customView{Name: "diagrams"}, first, "old")561- if err := tenant.handleViewTransform(ctx, work.Intent{ID: "stale", Kind: stale.Kind, EventID: stale.EventID, Target: stale.Target, Payload: stale.Payload}); err != nil {560+ stale := tenant.customViews.viewIntent(customView{Name: "diagrams"}, first, "old")561+ if err := tenant.customViews.handleViewTransform(ctx, work.Intent{ID: "stale", Kind: stale.Kind, EventID: stale.EventID, Target: stale.Target, Payload: stale.Payload}); err != nil {562562 t.Fatal(err)563563 }564564 if server.count() != 1 {@@ -832,7 +832,7 @@ func TestCustomViewHourlyTickQueuesRecentSources(t *testing.T) {
832832 if queued := pendingViewIntents(t, tenant, "diagrams", note.ID); len(queued) != 0 {833833 t.Fatal("an hourly view queued at write time")834834 }835- if err := tenant.tickCustomViews(ctx, now); err != nil {835+ if err := tenant.customViews.tickCustomViews(ctx, now); err != nil {836836 t.Fatal(err)837837 }838838 queued := pendingViewIntents(t, tenant, "diagrams", note.ID)@@ -844,7 +844,7 @@ func TestCustomViewHourlyTickQueuesRecentSources(t *testing.T) {
844844 t.Fatal("hourly artifact missing")845845 }846846 // Not due again for an hour; a backfill through runcustomview queues now.847- if err := tenant.tickCustomViews(ctx, now+60); err != nil {847+ if err := tenant.customViews.tickCustomViews(ctx, now+60); err != nil {848848 t.Fatal(err)849849 }850850 if queued := pendingViewIntents(t, tenant, "diagrams", note.ID); len(queued) != 0 {@@ -900,7 +900,7 @@ func TestCustomViewRendersTheReadmeAtTheStateHead(t *testing.T) {
900900 relay := httptest.NewServer(tenant)901901 defer relay.Close()902902 gitTest(t, "-C", work, "push", relay.URL+"/npub10xlxvlhemja6c4dqv22uapctqupfhlxm9h8z3k2e72q4k9hcz7vqpkge6d/docs.git", "refs/heads/main:refs/heads/main")903- blocks, err := tenant.sourceBlocks(ctx, state)903+ blocks, err := tenant.customViews.sourceBlocks(ctx, state)904904 if err != nil {905905 t.Fatal(err)906906 }@@ -37,10 +37,11 @@ import (
3737 // not depend on the event still being stored, the attempt number and the3838 // run that queued it.3939 type viewPayload struct {40- Event event.Event `json:"event"`41- Attempt int `json:"attempt"`42- Run string `json:"run,omitempty"`43- Refresh bool `json:"refresh,omitempty"`40+ Event event.Event `json:"event"`41+ Attempt int `json:"attempt"`42+ Run string `json:"run,omitempty"`43+ Refresh bool `json:"refresh,omitempty"`44+ Generation string `json:"generation,omitempty"`4445 }4546 4647 // viewRequest is the body posted to a transform: only the blocks, with the@@ -95,18 +96,18 @@ func viewSourceTag(source string) []string {
9596 9697 // viewHTTPClient refuses private addresses and redirects unless the relay9798 // itself runs on a loopback address.98-func (t *Tenant) viewHTTPClient() *http.Client {99+func (t *customViewService) viewHTTPClient() *http.Client {99100 if t.viewClient != nil {100101 return t.viewClient101102 }102- if t.loopbackRelay() {103+ if t.loopback != nil && t.loopback() {103104 return &http.Client{Timeout: viewTimeout, Transport: &http.Transport{TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12}}, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}104105 }105106 return replication.NewPinnedClient(viewTimeout)106107 }107108 108109 // viewLock bounds work to one transform run per view at a time.109-func (t *Tenant) viewLock(name string) *sync.Mutex {110+func (t *customViewService) viewLock(name string) *sync.Mutex {110111 t.viewMu.Lock()111112 defer t.viewMu.Unlock()112113 if t.viewLocks == nil {@@ -122,7 +123,7 @@ func (t *Tenant) viewLock(name string) *sync.Mutex {
122123 123124 // sourceBlocks finds the blocks of a source: the fenced blocks of its124125 // content, or of the README at the head of a repository state.125-func (t *Tenant) sourceBlocks(ctx context.Context, e event.Event) ([]views.Block, error) {126+func (t *customViewService) sourceBlocks(ctx context.Context, e event.Event) ([]views.Block, error) {126127 if e.Kind != event.KIND_REPO_STATE {127128 return views.Blocks(e.Content), nil128129 }@@ -135,7 +136,7 @@ func (t *Tenant) sourceBlocks(ctx context.Context, e event.Event) ([]views.Block
135136 136137 // stateReadme reads README.md at the head named by a repository state. A137138 // missing README yields no content.138-func (t *Tenant) stateReadme(ctx context.Context, e event.Event) (string, error) {139+func (t *customViewService) stateReadme(ctx context.Context, e event.Event) (string, error) {139140 if t.git == nil {140141 return "", nil141142 }@@ -168,7 +169,7 @@ func (t *Tenant) stateReadme(ctx context.Context, e event.Event) (string, error)
168169 // sourceCurrent reports whether the source is still the stored event: an169170 // event that was deleted, or a state that a newer state replaced, renders170171 // nothing.171-func (t *Tenant) sourceCurrent(ctx context.Context, e event.Event) (bool, error) {172+func (t *customViewService) sourceCurrent(ctx context.Context, e event.Event) (bool, error) {172173 var id string173174 var err error174175 if e.Kind == event.KIND_REPO_STATE {@@ -186,8 +187,8 @@ func (t *Tenant) sourceCurrent(ctx context.Context, e event.Event) (bool, error)
186187 // view and the source right before the POST, attaches artifacts that187188 // already exist for a block's hash and posts only the blocks that need188189 // rendering.189-func (t *Tenant) handleViewTransform(ctx context.Context, intent work.Intent) (err error) {190- ctx, finish := t.app.telemetry.Start(ctx, "view")190+func (t *customViewService) handleViewTransform(ctx context.Context, intent work.Intent) (err error) {191+ ctx, finish := t.telemetry.Start(ctx, "view")191192 outcome := "error"192193 defer func() { finish(outcome) }()193194 var payload viewPayload@@ -213,6 +214,10 @@ func (t *Tenant) handleViewTransform(ctx context.Context, intent work.Intent) (e
213214 outcome = "paused"214215 return nil215216 }217+ if payload.Generation != "" && payload.Generation != view.generation {218+ outcome = "invalid"219+ return nil220+ }216221 current, err := t.sourceCurrent(ctx, payload.Event)217222 if err != nil {218223 return err@@ -286,7 +291,7 @@ func (t *Tenant) handleViewTransform(ctx context.Context, intent work.Intent) (e
286291 return err287292 }288293 // Counts only: no source, block or body leaves the relay in a log.289- t.app.telemetry.Logger().Info("view transform delivered", "tenant", t.meta.Name, "attempt", payload.Attempt, "status", status, "blocks", len(pending), "artifacts", stored, "refused", refused, "errors", len(response.Errors))294+ t.telemetry.Logger().Info("view transform delivered", "tenant", t.tenantName, "attempt", payload.Attempt, "status", status, "blocks", len(pending), "artifacts", stored, "refused", refused, "errors", len(response.Errors))290295 return t.pruneViewArtifacts(ctx, view.Name, intent.ID)291296 }292297 reason := postErr.Error()@@ -300,7 +305,7 @@ func (t *Tenant) handleViewTransform(ctx context.Context, intent work.Intent) (e
300305 } else if _, err := t.store.DB().ExecContext(ctx, `UPDATE custom_views SET last_run_at=?, failures=?, last_status=? WHERE name=?`, now, failures, reason, view.Name); err != nil {301306 return err302307 }303- t.app.telemetry.Logger().Info("view transform failed", "tenant", t.meta.Name, "attempt", payload.Attempt, "status", status, "failures", failures, "paused", paused)308+ t.telemetry.Logger().Info("view transform failed", "tenant", t.tenantName, "attempt", payload.Attempt, "status", status, "failures", failures, "paused", paused)304309 if paused || payload.Attempt >= viewMaxAttempts {305310 return nil306311 }@@ -309,7 +314,7 @@ func (t *Tenant) handleViewTransform(ctx context.Context, intent work.Intent) (e
309314 310315 // pauseCustomView stops the view and records the count and reason, then311316 // drops the index so no new intents are queued for it.312-func (t *Tenant) pauseCustomView(ctx context.Context, name string, failures int, status string) error {317+func (t *customViewService) pauseCustomView(ctx context.Context, name string, failures int, status string) error {313318 if _, err := t.store.DB().ExecContext(ctx, `UPDATE custom_views SET enabled=0, failures=?, last_status=? WHERE name=?`, failures, status, name); err != nil {314319 return err315320 }@@ -319,7 +324,7 @@ func (t *Tenant) pauseCustomView(ctx context.Context, name string, failures int,
319324 320325 // retryViewTransform queues the next attempt after its backoff as its own321326 // intent so the durable queue's ordering and fencing still apply.322-func (t *Tenant) retryViewTransform(ctx context.Context, intent work.Intent, payload viewPayload) error {327+func (t *customViewService) retryViewTransform(ctx context.Context, intent work.Intent, payload viewPayload) error {323328 delay := viewBackoff[len(viewBackoff)-1]324329 if payload.Attempt-1 < len(viewBackoff) {325330 delay = viewBackoff[payload.Attempt-1]@@ -342,7 +347,7 @@ func (t *Tenant) retryViewTransform(ctx context.Context, intent work.Intent, pay
342347 // postViewTransform sends the blocks and reads the answer. It returns the343348 // HTTP status with a bounded reason on failure; a response that is not344349 // JSON counts as a failure.345-func (t *Tenant) postViewTransform(ctx context.Context, view customView, e event.Event, blocks []views.Block) (int, viewResponse, error) {350+func (t *customViewService) postViewTransform(ctx context.Context, view customView, e event.Event, blocks []views.Block) (int, viewResponse, error) {346351 body, err := json.Marshal(viewRequest{Relay: t.publicURL, View: view.Name, Source: viewSource{ID: viewSourceKey(e), Kind: e.Kind}, Blocks: blocks})347352 if err != nil {348353 return 0, viewResponse{}, errors.New("encode request")@@ -455,7 +460,7 @@ func checkSVG(body string) error {
455460 456461 // storeArtifact checks one artifact and keeps it as a signed record457462 // attached to its source.458-func (t *Tenant) storeArtifact(ctx context.Context, view customView, block views.Block, artifact viewArtifact, source string, expires int64, now int64) error {463+func (t *customViewService) storeArtifact(ctx context.Context, view customView, block views.Block, artifact viewArtifact, source string, expires int64, now int64) error {459464 content, err := checkArtifact(artifact, view.MaxBytes)460465 if err != nil {461466 return err@@ -480,7 +485,7 @@ func (t *Tenant) storeArtifact(ctx context.Context, view customView, block views
480485 481486 // attachArtifact reports whether an artifact exists for the hash and, when482487 // it does, attaches the source to it through a fresh signed record.483-func (t *Tenant) attachArtifact(ctx context.Context, view customView, hash, source string, block int, expires int64) (bool, error) {488+func (t *customViewService) attachArtifact(ctx context.Context, view customView, hash, source string, block int, expires int64) (bool, error) {484489 var exists int485490 err := t.store.DB().QueryRowContext(ctx, `SELECT 1 FROM custom_view_artifacts WHERE view=? AND hash=?`, view.Name, hash).Scan(&exists)486491 if errors.Is(err, sql.ErrNoRows) {@@ -507,7 +512,7 @@ func (t *Tenant) attachArtifact(ctx context.Context, view customView, hash, sour
507512 508513 // signArtifact signs the artifact with its current sources as a kind 30078509514 // record and stores the record when the view is public.510-func (t *Tenant) signArtifact(ctx context.Context, view customView, hash string, now int64) error {515+func (t *customViewService) signArtifact(ctx context.Context, view customView, hash string, now int64) error {511516 var typ, engine, content string512517 if err := t.store.DB().QueryRowContext(ctx, `SELECT type,engine,content FROM custom_view_artifacts WHERE view=? AND hash=?`, view.Name, hash).Scan(&typ, &engine, &content); err != nil {513518 return err@@ -566,7 +571,7 @@ func (t *Tenant) signArtifact(ctx context.Context, view customView, hash string,
566571 567572 // detachStaleArtifacts drops the source from artifacts whose block it no568573 // longer carries, then removes artifacts left without a source.569-func (t *Tenant) detachStaleArtifacts(ctx context.Context, name, source string, hashes []string) error {574+func (t *customViewService) detachStaleArtifacts(ctx context.Context, name, source string, hashes []string) error {570575 rows, err := t.store.DB().QueryContext(ctx, `SELECT hash FROM custom_view_sources WHERE view=? AND source=?`, name, source)571576 if err != nil {572577 return err@@ -600,7 +605,7 @@ func (t *Tenant) detachStaleArtifacts(ctx context.Context, name, source string,
600605 // pruneViewArtifacts detaches sources that no longer exist and removes the601606 // artifacts that lose their last source. A view with transform work still602607 // queued is left alone so a replaced source can reattach its blocks first.603-func (t *Tenant) pruneViewArtifacts(ctx context.Context, name, except string) error {608+func (t *customViewService) pruneViewArtifacts(ctx context.Context, name, except string) error {604609 var pending int605610 if err := t.store.DB().QueryRowContext(ctx, `SELECT count(*) FROM work_intents WHERE kind=? AND target=? AND state IN ('pending','running') AND id<>?`, viewTransform, name, except).Scan(&pending); err != nil {606611 return err@@ -646,7 +651,7 @@ func (t *Tenant) pruneViewArtifacts(ctx context.Context, name, except string) er
646651 647652 // pruneAllViewArtifacts runs the prune for every view; the maintenance648653 // sweep calls it after expired events are removed.649-func (t *Tenant) pruneAllViewArtifacts(ctx context.Context) error {654+func (t *customViewService) pruneAllViewArtifacts(ctx context.Context) error {650655 rows, err := t.customViewRows(ctx)651656 if err != nil {652657 return err@@ -659,7 +664,7 @@ func (t *Tenant) pruneAllViewArtifacts(ctx context.Context) error {
659664 return nil660665 }661666 662-func (t *Tenant) sourcePresent(ctx context.Context, source string) (bool, error) {667+func (t *customViewService) sourcePresent(ctx context.Context, source string) (bool, error) {663668 var one int664669 var err error665670 if tag := viewSourceTag(source); tag[0] == "a" {@@ -679,7 +684,7 @@ func (t *Tenant) sourcePresent(ctx context.Context, source string) (bool, error)
679684 680685 // dropOrphanArtifacts removes artifacts without a source, and their stored681686 // records, then re-signs the artifacts whose source list shrank.682-func (t *Tenant) dropOrphanArtifacts(ctx context.Context, name string) error {687+func (t *customViewService) dropOrphanArtifacts(ctx context.Context, name string) error {683688 ids, err := t.customViewArtifactIDs(ctx, `SELECT event_id FROM custom_view_artifacts WHERE view=? AND NOT EXISTS(SELECT 1 FROM custom_view_sources WHERE custom_view_sources.view=custom_view_artifacts.view AND custom_view_sources.hash=custom_view_artifacts.hash)`, name)684689 if err != nil {685690 return err@@ -691,7 +696,7 @@ func (t *Tenant) dropOrphanArtifacts(ctx context.Context, name string) error {
691696 return err692697 }693698 if len(ids) > 0 {694- t.app.telemetry.Logger().Info("view artifacts removed", "tenant", t.meta.Name, "removed", len(ids))699+ t.telemetry.Logger().Info("view artifacts removed", "tenant", t.tenantName, "removed", len(ids))695700 }696701 // Artifacts that kept a source but lost one carry a tag for it; a fresh697702 // record drops it.@@ -744,7 +749,7 @@ func (t *Tenant) dropOrphanArtifacts(ctx context.Context, name string) error {
744749 return nil745750 }746751 747-func (t *Tenant) artifactSources(ctx context.Context, name, hash string) ([]string, error) {752+func (t *customViewService) artifactSources(ctx context.Context, name, hash string) ([]string, error) {748753 rows, err := t.store.DB().QueryContext(ctx, `SELECT source FROM custom_view_sources WHERE view=? AND hash=?`, name, hash)749754 if err != nil {750755 return nil, err@@ -761,7 +766,7 @@ func (t *Tenant) artifactSources(ctx context.Context, name, hash string) ([]stri
761766 return out, rows.Err()762767 }763768 764-func (t *Tenant) customViewArtifactIDs(ctx context.Context, query string, args ...any) ([]string, error) {769+func (t *customViewService) customViewArtifactIDs(ctx context.Context, query string, args ...any) ([]string, error) {765770 rows, err := t.store.DB().QueryContext(ctx, query, args...)766771 if err != nil {767772 return nil, err@@ -781,7 +786,7 @@ func (t *Tenant) customViewArtifactIDs(ctx context.Context, query string, args .
781786 }782787 783788 // deleteStoredArtifacts removes the public records of artifacts by id.784-func (t *Tenant) deleteStoredArtifacts(ctx context.Context, ids []string) error {789+func (t *customViewService) deleteStoredArtifacts(ctx context.Context, ids []string) error {785790 for _, id := range ids {786791 if _, err := t.store.DeleteEvent(ctx, id); err != nil {787792 return err@@ -792,7 +797,7 @@ func (t *Tenant) deleteStoredArtifacts(ctx context.Context, ids []string) error
792797 793798 // cascadeViewDeletion runs after a kind 5 deletion is stored: artifacts794799 // attached to the deleted events or addresses lose that source at once.795-func (t *Tenant) cascadeViewDeletion(ctx context.Context, e event.Event) {800+func (t *customViewService) cascadeViewDeletion(ctx context.Context, e event.Event) {796801 if e.Kind != event.KIND_DELETION {797802 return798803 }@@ -802,7 +807,7 @@ func (t *Tenant) cascadeViewDeletion(ctx context.Context, e event.Event) {
802807 }803808 for _, view := range rows {804809 if err := t.pruneViewArtifacts(ctx, view.Name, ""); err != nil {805- t.app.telemetry.Logger().Error("view artifacts not pruned", "tenant", t.meta.Name, "error", err)810+ t.telemetry.Logger().Error("view artifacts not pruned", "tenant", t.tenantName, "error", err)806811 }807812 }808813 }@@ -815,7 +820,7 @@ type artifactRecord struct {
815820 EventID string816821 }817822 818-func (t *Tenant) customViewArtifact(ctx context.Context, name, hash string) (artifactRecord, error) {823+func (t *customViewService) customViewArtifact(ctx context.Context, name, hash string) (artifactRecord, error) {819824 var record artifactRecord820825 err := t.store.DB().QueryRowContext(ctx, `SELECT type,audience,content,event_id FROM custom_view_artifacts WHERE view=? AND hash=?`, name, hash).Scan(&record.Type, &record.Audience, &record.Content, &record.EventID)821826 if errors.Is(err, sql.ErrNoRows) {@@ -186,20 +186,21 @@ func initializeTenant(ctx context.Context, s *storage.Store, p policy.Policy, na
186186 if err := storage.PutSetting(ctx, tx, "policy", p); err != nil {187187 return err188188 }189+ communityConfig := community.ConfigSnapshot{}189190 for _, rule := range []struct {190191 kinds []int191192 name string192193 }{{p.AllowedKinds, "allow"}, {p.BlockedKinds, "block"}} {193194 for _, kind := range rule.kinds {194- if _, err := tx.ExecContext(ctx, "INSERT INTO community_kind_rules(kind,rule) VALUES(?,?)", kind, rule.name); err != nil {195- return err196- }195+ communityConfig.KindRules = append(communityConfig.KindRules, community.ConfigKindRule{Kind: kind, Rule: rule.name})197196 }198197 }199198 for _, r := range p.Retention {200- if _, err := tx.ExecContext(ctx, "INSERT INTO community_retention(kind,days) VALUES(?,?)", r.Kind, r.Days); err != nil {201- return err202- }199+ kind := r.Kind200+ communityConfig.Retention = append(communityConfig.Retention, community.ConfigRetention{Kind: &kind, Days: r.Days})201+ }202+ if err := community.ApplyConfigTx(ctx, tx, communityConfig, false, false, false, false, true, true); err != nil {203+ return err203204 }204205 if _, err := tx.ExecContext(ctx, "CREATE TABLE IF NOT EXISTS config_connections(position INTEGER PRIMARY KEY,value TEXT NOT NULL)"); err != nil {205206 return err@@ -0,0 +1,122 @@
1+package daemon2+3+import (4+ "context"5+ "database/sql"6+ "encoding/json"7+8+ "github.com/FelineStateMachine/tinyrelay/internal/community"9+ "github.com/FelineStateMachine/tinyrelay/internal/event"10+ "github.com/FelineStateMachine/tinyrelay/internal/gitrelay"11+ "github.com/FelineStateMachine/tinyrelay/internal/policy"12+ "github.com/FelineStateMachine/tinyrelay/internal/replication"13+ "github.com/FelineStateMachine/tinyrelay/internal/sites"14+ "github.com/FelineStateMachine/tinyrelay/internal/storage"15+)16+17+// eventCommitter prepares the projections common to client publication and18+// replication. Admission and client-only protocol actions remain with their19+// entry points; every projection here participates in the event transaction.20+type eventCommitter struct {21+ policy policy.Policy22+ community *community.Service23+ sites *sites.Service24+ git *gitrelay.GitRelay25+}26+27+type eventCommitPlan struct {28+ options storage.SaveOptions29+ repository gitrelay.Repository30+ gitMetadata bool31+ followups *storage.Intent32+}33+34+func (c eventCommitter) prepare(ctx context.Context, e event.Event, origin replication.Origin, now int64) (eventCommitPlan, error) {35+ plan := eventCommitPlan{options: storage.SaveOptions{Now: now}}36+ if c.sites != nil {37+ plan.options = c.sites.SaveOptions(e, now)38+ }39+ plan.options.SearchMode = c.policy.Features.Search40+ if e.Kind == 30023 {41+ plan.options.Intents = append(plan.options.Intents, storage.Intent{Kind: "view-publish", EventID: e.ID, Target: "articles", Payload: "{}"})42+ }43+ if e.Kind == event.KIND_AGENT_GRANT || e.Kind == event.KIND_DELETION {44+ plan.options.AddBeforeCommit(func(ctx context.Context, tx *sql.Tx) error {45+ return c.community.ApplyAgentEventTx(ctx, tx, e, now)46+ })47+ }48+ if c.policy.Features.Grasp && (e.Kind == event.KIND_REPO || e.Kind == event.KIND_REPO_STATE) {49+ if err := c.prepareRepository(ctx, e, origin, &plan); err != nil {50+ return eventCommitPlan{}, err51+ }52+ }53+ return plan, nil54+}55+56+func (c eventCommitter) prepareRepository(ctx context.Context, e event.Event, origin replication.Origin, plan *eventCommitPlan) error {57+ validate := c.git.ValidateImported58+ if origin == replication.OriginClient {59+ validate = c.git.Validate60+ }61+ repo, err := validate(ctx, e)62+ if err != nil {63+ return err64+ }65+ raw, err := json.Marshal(repo)66+ if err != nil {67+ return err68+ }69+ plan.repository, plan.gitMetadata = repo, true70+ plan.options.Intents = append(plan.options.Intents, storage.Intent{Kind: "git-metadata", EventID: e.ID, Target: repo.Owner + ":" + repo.Identifier, Payload: string(raw)})71+ if e.Kind == event.KIND_REPO_STATE {72+ plan.options.AddBeforeCommit(func(ctx context.Context, tx *sql.Tx) error {73+ _, err := tx.ExecContext(ctx, "INSERT OR REPLACE INTO pending_events(id,reason) VALUES(?,'git objects')", e.ID)74+ return err75+ })76+ }77+ return nil78+}79+80+func (t *Tenant) validateStoredEvent(ctx context.Context, e event.Event) error {81+ if err := t.validatePrivateRepositoryPlacement(ctx, e); err != nil {82+ return err83+ }84+ if err := validateReviewAnchor(e); err != nil {85+ return err86+ }87+ if t.sites != nil {88+ return sites.ValidateManifest(e)89+ }90+ return nil91+}92+93+func (t *Tenant) prepareStoredEvent(ctx context.Context, e event.Event, origin replication.Origin, now int64) (eventCommitPlan, error) {94+ committer := eventCommitter{policy: t.Policy(), community: t.community, sites: t.sites, git: t.git}95+ plan, err := committer.prepare(ctx, e, origin, now)96+ if err != nil {97+ return plan, err98+ }99+ var push *replicationPushFollowupPayload100+ if origin == replication.OriginClient && t.Policy().Features.Push {101+ payload, _ := t.prepareReplicationPushFollowup(ctx, e, now)102+ if len(payload.RegistrationIDs) != 0 || payload.Discover {103+ push = &payload104+ }105+ }106+ plan.followups = t.followups.prepareWithPush(ctx, e, now, false, push)107+ if plan.followups != nil {108+ plan.options.Intents = append(plan.options.Intents, *plan.followups)109+ }110+ return plan, nil111+}112+113+func (t *Tenant) afterStoredEvent(ctx context.Context, e event.Event, followups *storage.Intent) {114+ t.notifyDevices(ctx, e)115+ t.followups.try(ctx, followups)116+ if event.IsEphemeral(e.Kind) {117+ t.notifyCallbacks(ctx, e)118+ }119+ t.customViews.cascadeViewDeletion(ctx, e)120+ t.notifyWikiMerge(ctx, e)121+ t.notifyWikiProposal(ctx, e)122+}Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.
Not shown: this page already shows 6000 lines of diff.