How it works — a file's journey to a page

Adapted from the repo's docs/how-it-works.md — Part of Internals: the platform's design set, published as content on the platform it describes.

Engineer-to-engineer walkthrough of how an article travels from a file on disk to a rendered page in the reader. Grounded in the shipped code and the design set; it links to the ADRs rather than re-arguing them. Read architecture.md and master-blueprint.md first for the tier model and the product design this implements. Reflects contract v0.8.1 and ADRs 0005–0017.

Thesis

The filesystem is the CMS (ADR-0005). Content lives as a recursively-scanned tree of sections (a directory + optional metadata.json) and articles (a *.md file: YAML front-matter + Markdown body), rooted at CONTENT_ROOT (storage/app/content). Path = identity = URL: an article's slug path relative to the root (minus .md) is its canonical id and its front-end URL. There is no admin panel — authoring is out-of-band, via an editor and git (ADR-0006). PostgreSQL holds a rebuildable derived index (one row per article/section: parsed metadata, resolved publication state, go_live_at, tags, author slugs, a content_hash, and an FTS vector) — never the source of truth, always reconstructable from disk. Redis does two jobs: it caches rendered+sanitised HTML keyed by renderer-version + content-hash, and it carries the content event streams. The backend exposes a read-only, unauthenticated REST API under /v1 on a dedicated api. host (ADR-0010); the Next.js reader lives on the bare host and consumes the API only through the client generated from contract/openapi.yaml — the contract is the single coupling point between the tiers.

The two stores are strictly derived, in one direction:

Architecture

flowchart TB
    subgraph authoring["Authoring — out of band"]
        editor["Author's editor / git"]
        tree["Content tree<br/>storage/app/content/**<br/>metadata.json · *.md · images"]
        editor -->|"write files"| tree
    end

    subgraph backend["backend/ — Laravel API (api. host, base /v1)"]
        sched["Scheduler<br/>schedule:run every minute"]
        reindexer["Reindexer<br/>content:reindex"]
        relay["Event relay<br/>content:relay-events"]
        http["Http controllers<br/>Domain actions → ContentIndex port"]
        pg[("PostgreSQL<br/>derived index")]
        rcache[("Redis<br/>render cache")]
        streams[("Redis<br/>event streams")]

        sched -->|"tick"| reindexer
        sched -->|"tick (seconds later)"| relay
        reindexer -->|"read + parse"| tree
        reindexer -->|"upsert / prune"| pg
        reindexer -->|"warm by content-hash"| rcache
        reindexer -->|"emit events post-commit"| streams
        relay -->|"read past cursor"| streams
        http -->|"query PUBLISHED"| pg
        http -->|"rendered HTML by hash"| rcache
        http -->|"stream asset bytes"| tree
    end

    subgraph contract["contract/ — the tier boundary"]
        spec["OpenAPI 3.1 · AsyncAPI 3.1<br/>v0.8.1"]
    end

    subgraph frontend["frontend/ — Next.js reader (bare host)"]
        reval["POST /api/revalidate<br/>signed webhook receiver"]
        pages["Server components — ISR<br/>tagged fetches, generated client"]
        feed["feed.xml · sitemap.xml<br/>derived routes"]
    end

    browser["Reader's browser"]

    relay -->|"signed webhook (HMAC-SHA256)"| reval
    reval -->|"revalidateTag content / site"| pages
    browser -->|"HTTPS page request"| pages
    pages -->|"generated client → GET /v1/*"| http
    feed -->|"listArticles + getArticle + getSiteMeta"| http
    browser -.->|"img src → GET /v1/assets/*"| http

    http -.->|"conforms to"| spec
    pages -.->|"generated from"| spec

The write path flows top-down: editor → content tree → scheduler → Reindexer → Postgres index + Redis render cache + Redis event streams → relay → signed webhook → Next /api/revalidate → tag bust. The read path flows bottom-up: browser → Next (ISR, tagged fetches) → /v1 API → index/cache → filesystem assets. frontend/ and backend/ never touch each other's code — every solid up-arrow across the middle is mediated by contract/. The API lives on the api. host; the reader on the bare host; SSR/build fetches actually go container-internal (http://web/v1) because the public cert is untrusted inside the frontend container (ADR-0010).

Write → live: the end-to-end sequence

sequenceDiagram
    autonumber
    actor Author
    participant FS as Content tree
    participant Sched as Scheduler
    participant RI as Reindexer
    participant PG as Postgres index
    participant RC as Redis render cache
    participant ES as Redis event streams
    participant Relay as Event relay
    participant Next as Next.js frontend
    participant HTTP as API /v1
    actor Reader

    Author->>FS: save article.md (front-matter + Markdown body)
    Note over Sched,RI: schedule:run every minute → content:reindex
    Sched->>RI: run reindex tick
    RI->>FS: enumerate tree + parse front-matter
    RI->>RI: PublicationPolicy gates on go-live instant
    RI->>RC: render + sanitise (CommonMark, h1→h2), cache by content-hash
    RI->>PG: upsert changed / prune deleted (hash-guarded, one tx)
    RI->>ES: emit article-published + reindexed (after commit)
    Note over Sched,Relay: same minute → content:relay-events
    Sched->>Relay: run relay
    Relay->>ES: read entries beyond the persisted cursor
    Relay->>Relay: change-detection rule (deltas > 0 or new content-version?)
    alt run changed content
        Relay->>Next: POST /api/revalidate (HMAC-signed raw body)
        Next->>Next: verify signature, revalidateTag content / site (expire 0)
        Next-->>Relay: 200 → advance cursor
    else no-op run
        Relay->>Relay: advance cursor, send nothing
    end
    Reader->>Next: GET the article page
    Next->>HTTP: getArticle via generated client (internal /v1)
    HTTP->>PG: query PUBLISHED where go_live <= now
    HTTP->>RC: fetch rendered HTML by content-hash
    HTTP-->>Next: Article JSON (sanitised HTML body)
    Next-->>Reader: server-rendered page

Step by step, against the real classes:

1. Author saves a file. No process is notified; the tree is just newer than the index.
2. Scheduler tick (routes/console.php): Schedule::command('content:reindex') ->everyMinute()->withoutOverlapping(), then content:relay-events immediately after, so one schedule:run does both within seconds.
3. Reindex (App\Infrastructure\Index\Reindexer): enumerate via ContentSource, parse front-matter, apply PublicationPolicy::state(frontMatter, Clock) to gate on the go-live instant, render + sanitise through the CommonMark pipeline (below) memoised by content hash, then upsert changed rows and prune deleted ones inside a DB transaction. It is idempotent and hash-guarded: a row is rewritten only when its content_hash changed or its publication state changed (a SCHEDULED article crossing its go-live instant — clock-driven, no hash change). Unchanged content writes nothing.
4. Events, post-commit (RedisEventPublisher → Redis streams): content.article-published for each article that became public, and content.reindexed once per run (carrying the articlesIndexed/sectionsIndexed totals, the four per-run delta counters, skipped, durationMs, and contentVersion). Shapes live in contract/asyncapi.yaml.
5. Relay (App\Domain\Content\Action\RelayContentEvents, behind content:relay-events): reads both channels from a persisted per-channel cursor, applies the change-detection rule (any article-published, or a reindexed whose deltas sum above zero, or a changed contentVersion), and only on real change POSTs one aggregated contentChanged webhook to the frontend, HMAC-SHA256-signed over the raw body. No-op runs send nothing. The cursor advances past a changed batch only after a 200 — at-least-once delivery; the receiver is idempotent by contract. Delivery failure is logged, never fatal, and never delays reindexing (ADR-0016).
6. Revalidate (frontend/src/app/api/revalidate/route.ts): recompute the HMAC over the raw bytes, compare in constant time, reject missing/mismatching with 401 and no invalidation; on success revalidateTag(tag, { expire: 0 }) for the payload's tags, constrained to the content/site allowlist. The next request re-renders fresh.
7. A visitor reads. A Next server component fetches through the generated client (frontend/src/lib/api/client.ts) with { next: { revalidate, tags } }; on a tag-fresh cache it renders immediately, otherwise it hits /v1, which serves from the Postgres index + Redis render cache and returns the sanitised Article JSON.

Measured freshness (ADR-0016 acceptance): a publish is visible in ~21 s, a prune in ~51 s, worst case within ~90 s — the event loop collapses the ISR window to roughly one scheduler tick plus one render. ISR is the safety net, not the primary mechanism: content fetches default to a 300 s revalidate window (3600 s for feed/sitemap), so even if a webhook is missed, pages self-heal on the next window. Correctness never depends on the event loop — read-time gating already makes every query correct (below).

How the pieces work

Rendering pipeline (ADR-0007 + one-h1 invariant)

App\Infrastructure\Rendering\MarkdownRenderer runs league/commonmark → the shared TrustedHtmlSanitizer allowlist (the XSS boundary; scripts, iframes, on* handlers, javascript:/data: URLs are stripped) → stable heading anchor ids → a tableOfContents. Any # h1 authored in the body is demoted to <h2> on the parsed AST (a DocumentParsedEvent listener) so the front-matter title is the page's only h1 (ADR-0015). The whole pipeline is a pure function of the input string, so its output is memoised in Redis by content hash; a renderer behaviour change bumps MarkdownRenderer::VERSION (currently 2), which changes the cache key so old HTML misses cleanly instead of going stale.

Publication gating (ADR-0008)

Visibility is a pure function of front-matter and the current time: go-live = scheduled_at ?? published_at; DRAFT if draft:true or no go-live instant, SCHEDULED if go-live is in the future, PUBLISHED once it passes. Read-time gating is the source of truth — every list/fetch query filters state = PUBLISHED AND go_live_at <= now(), so the API is correct even if the scheduler never runs. Not-yet-public and unknown paths both return 404, never 403 — existence is never disclosed (ADR-0006).

Search (Track 4a — Postgres FTS)

GET /v1/search?q= runs full-text search over published articles via a weighted tsvector (WeightedSearchVector, GIN-indexed) in PostgresContentIndex; SearchArticles returns a paginated SearchResultList, most-relevant-first (ordering conveys relevance — no numeric score), with an optional sanitised <mark>-highlighted snippet. q is trimmed and bounded 2–128 chars (400 otherwise); a valid query with no hits is a 200 empty page.

Assets (ADR-0011)

GET /v1/assets/{path} streams cover and in-body images from the content root, originals as-is, behind a fail-closed media-type allowlist (png/jpg/jpeg/gif/webp/avif); anything else — including the Markdown/JSON source — returns an identical 404, so the endpoint can't be used to read unpublished source. Responses carry a strong ETag + long-lived immutable Cache-Control and honour If-None-Match (304). Cover URLs and in-body img srcs are rewritten to absolute asset URLs (InBodyAssetRewriter); the asset route has its own 300 rpm limiter, exempt from the article limiter. (ADR-0012: Mermaid fences render client-side in the browser, and degrade to a visible code block anywhere the island can't run — e.g. feed readers.)

Authors (ADR-0013)

author front-matter is a free display label, not an account. The server derives one canonical kebab-case slug per label and publishes it on every byline (AuthorRef), in the /v1/authors catalogue, and as the ?author= / /authors/{author} filter key — so a byline's slug always equals its catalogue item's slug. Clients never guess slugs.

Site identity & navigation (ADR-0014 / ADR-0015)

GET /v1/site resolves the masthead title, tagline, homepage thesis, footer memo/copyright, and curated nav from the optional site block of the root metadata.json — the file-first replacement for env-var branding and hardcoded chrome. Resolution falls back to the root section's title/description for identity fields; footer/nav fields have no fallback. nav entries (NavItem) point at content paths — typically unlisted pages.

Unlisted pages (ADR-0015)

An article with unlisted: true is excluded from every listing surface — /articles and all its filters, tag/author catalogues and pages, search, prev/next chains, and derived surfaces like the feed and sitemap — while remaining fetchable at GET /v1/articles/{path}. This is how a curated nav page (e.g. the About page, a root-level /about-me) reaches the header without appearing in any list.

Feeds (ADR-0017 — full-content RSS)

frontend/src/app/feed.xml/route.ts builds an RSS 2.0 feed entirely from published operations (listArticles + getArticle + getSiteMeta) — a frontend-derived artifact, no contract change. Each item carries full sanitised HTML in content:encoded (CDATA, split on any literal ]]>), an escaped <description>, absolute <link> and guid isPermaLink="true", RFC-822 pubDate, per-author dc:creator, per-tag <category>; the channel adds an atom:link rel="self" and <lastBuildDate>. A failed per-item body fetch degrades that one item to summary-only rather than failing the feed. Rendering ~20 full items fans out to ~20 getArticle calls, which is why ADR-0017 also raised the api limiter 60 → 120 rpm per IP (the assets limiter stays 300 rpm).

The /shell terminal

bflo.sh/shell is a client-island browser terminal — a purely frontend feature layered on the public read-only API; it introduces no new backend or contract surface.

Verify gates (Definition of Done, per tier)

Properties that hold (the invariants)

• One <h1> per rendered page. The front-matter title is the sole h1; body h1s are demoted to h2 before render/sanitise (ADR-0015).
• Unlisted means hidden-but-reachable. Absent from every listing/derived surface, still retrievable by path (ADR-0015).
• No forbidden tier edges. frontend → contract ← backend; inside backend Http → Domain → Data (ports). Domain/Data never import Http; controllers never touch Eloquent directly (architecture.md).
• The contract is the only coupling point. The frontend consumes only the generated client; neither tier reads the other's source, types, or database.
• The index is always rebuildable. Filesystem is the sole source of truth; Postgres + Redis are derived and reconstructable with migrate:fresh + content:reindex (ADR-0005).
• Read-time gating is authoritative. Every query filters PUBLISHED AND go_live_at <= now(); the scheduler/relay are optimisation + notification, never correctness (ADR-0008).
• Unpublished existence is never disclosed. SCHEDULED/DRAFT/unknown paths, and non-servable assets, all return an identical 404 (ADR-0006, ADR-0011).
• Rendering is a trusted, sanitised, deterministic function of the input string, safe to cache by content hash; the sanitiser allowlist is the XSS boundary (ADR-0007).
• Webhook delivery is at-least-once and idempotent. Signed over raw bytes, verified in constant time; the cursor advances only after acknowledgement; no-op runs notify nothing (ADR-0016).

Sources

• Design set: architecture.md, master-blueprint.md (content model §3, stores §4, backend §5, contract §6, sequencing §11).
• ADRs: 0005, 0006, 0007, 0008, 0010, 0011, 0012, 0013, 0014, 0015, 0016, 0017.
• Contract: ../contract/openapi.yaml, ../contract/asyncapi.yaml (both v0.8.1).
• Code: backend/app/Domain/Content/**, backend/app/Infrastructure/{Index,Rendering,Cache,Events,Content}/**, backend/routes/{api.php,console.php}, frontend/src/lib/api/client.ts, frontend/src/app/{feed.xml,api/revalidate}/route.ts.