Backend architecture

This is the canonical map of LumenClip's server-side architecture as it exists in the repository. Domain object shapes are in Data structures, the HTTP surface is in backend-endpoints.md, and the queue lifecycle is in Backend scheduling.

The additive Railway-to-Railway replacement is tracked in Railway migration. Railway remains the runtime default until the documented cutover gates pass.

Maintained backend foundations

The migration now has concrete compatibility boundaries rather than parallel ad-hoc implementations:

• lib/railway/schema.ts describes the provisioned Railway tables with Drizzle; getRailwayOrm() shares the existing pooled postgres connection.
• lib/railway/job-queue.ts provides a pg-boss adapter with queue creation, exponential retries, single-job workers, and graceful shutdown. It is not the active queue until worker-handler parity is complete.
• lib/server-env.ts validates the new optional server variables. Missing Railway or integration configuration disables those features and does not stop local Railway development.
• withHandler() emits structured Pino completion/failure records and returns an x-request-id on both success and failure.
• providerFetch() retries only transient network, timeout, throttling, and 5xx failures. The TikTok/Apify importer is the first migrated provider path.
• /api/v1/openapi.json is generated from Hono/Zod contracts and /api-reference renders it with Scalar. Existing routes remain compatible while domains move into the versioned router incrementally.

Runtime topology

flowchart LR
    Browser["Next.js browser client"] --> Clerk["Clerk session"]
    Clerk --> Proxy["proxy.ts request boundary"]
    Proxy --> Pages["App Router pages"]
    Proxy --> Routes["app/api route handlers"]

    Routes --> Domain["lib domain modules"]
    Pages --> Domain
    Scheduler["automation-scheduler"] --> Jobs["jobs table"]
    Jobs --> Worker["job-worker / local worker"]
    Worker --> Domain

    Domain --> JsonStore["lib/json-store.ts"]
    Domain --> DirectStores["direct Railway modules"]
    JsonStore --> Tables["Railway TablesDB"]
    DirectStores --> Tables
    Domain --> Assets["lib/asset-storage.ts"]
    Assets --> Storage["Railway Storage"]

    Domain --> Providers["OpenRouter / Rendi / PostFast / KIE / Pexels / Pinterest / DeepL"]

The HTTP layer is an adapter, not a separate backend. Most route handlers call modules under lib/; scheduled work calls the same domain modules where possible. The scheduled slideshow worker still contains a parallel JavaScript pipeline that must be kept aligned with the main generation path.

Request and ownership boundary

proxy.ts initializes Clerk for all application and API requests, protects /app/**, and returns 401 for unauthenticated private APIs. Clerk owns sign-in, sign-up, verification, recovery, session cookies, and session revocation. Domain stores call getCurrentUser() before reading or writing private data, so the proxy is not the only authorization check. The adapter maps a Clerk user to the stable owner ID used by existing application records; preferences live in Clerk private metadata. Railway is not an authentication provider.

Ownership rules:

• Private rows have an owner_id Railway column.
• Serialized domain records normally also contain ownerId after persistence.
• Deterministic private row IDs hash physical table, source_key where applicable, owner ID, and domain record ID.
• Worker requests use systemOwnerId() so queued work remains attributed to the user who owns the automation.
• Shareable output categories may be read by accepted workspace collaborators; automations and mutable reference collections remain owner-only.
• Public reference categories are rows with a public store route, not separate globally public tables.

Persistence layers

1. Compatibility JSON-store API

Most domain modules still present a historical rootDir + fileName + key interface through lib/json-store.ts. Despite the filesystem-looking API, mapped mutable stores are Railway-only. There is no JSON-file fallback.

The mapping in lib/railway-stores.ts resolves each logical store to:

type StoreRoute = {
  table: string
  sourceKey: string
  public: boolean
  shareable?: boolean
}

sourceKey is required because multiple logical record types now share the same physical table. Reads always filter it for consolidated tables.

2. Consolidated physical tables

Reusable inputs and generated outputs are polymorphic:

Both consolidated parent tables retain the full serialized domain record in data while projecting commonly queried fields into columns. The projected columns are indexes/search aids; the TypeScript object serialized in data is the compatibility source for domain hydration.

Common consolidated row fields:

type ConsolidatedRow = {
  rid: string
  owner_id?: string
  source_key: string
  name?: string
  status?: string
  created_raw?: string
  data: string
  ord: number
  // permanent_assets and outputs add category-specific projected columns
  // outputs project source ids, publication status, kind, and has_video
  // for targeted reads and aggregate counts
}

output_media is deleted and recreated when its parent output is updated. The JSON-store hydrates normalized media rows back into the domain object before its normalizer runs. Rows contain only the parent/owner IDs, media kind and role, position, storage reference, URL, and creation time. File metadata belongs to Storage and is not duplicated in this join table.

3. Dedicated physical tables

High-churn or operational records keep dedicated tables:

Pre-consolidation tables are not part of the maintained schema. Run pnpm railway:prune-schema -- --env=<environment file> to audit them and add --apply to delete only tables that Railway confirms are empty. Current results and generated videos use outputs; PostFast publication records are embedded in an output's publications field.

Logical-to-physical store map

This table mirrors STORE_ROUTES in lib/railway-stores.ts.

Dedicated tables do not carry source_key; their table identity is already the record discriminator. Snapshot and usage tables store query fields plus the serialized domain record without unused generic name or status columns.

Automation template definitions and curated example runs live in local Railway as public reference categories. Creating a user automation writes a separate owner-scoped row to automations.

Output and publication model

Generated content and its social publication state are related but not the same record lifecycle:

flowchart LR
    Automation --> Run["AutomationRunRecord"]
    Run --> Result["ResultRecord in outputs"]
    Result --> Media["output_media rows"]
    Result --> Publications["PostFastPostRecord[] in outputs.publications"]
    Publications --> PostFast["PostFast social post"]

    Manual["Manual/external post"] --> Wrapper["publication_wrapper output"]
    Wrapper --> Publications

• ResultRecord.status describes generation: succeeded | failed.
• Slideshow render status is exported | failed.
• PostFastPostRecord.status describes distribution: draft, awaiting manual posting, review, scheduled, published, or failed.
• Marking something published writes publication evidence; it does not rename a generation status to "completed".
• A publication that does not match an existing output receives a small publication_wrapper output so publication history still has an owner and a stable parent.

Binary storage

lib/asset-storage.ts persists files to Railway Storage. Some generation and render paths also require local working files for ffmpeg/sharp before mirroring or after downloading provider output.

/api/local-assets/** is a compatibility URL namespace, not proof that the bytes live only on local disk. The route derives a deterministic Storage bucket and file ID from the data-relative path and streams the Railway object, with range support for video/audio.

Removed path categories such as characters/, knowledge-base files, and benchmark images now fall through to misc if an old URL is requested; their former dedicated mappings are no longer part of bucketForPath().

File IDs use sha256(relativePath).slice(0, 36). Do not add a second lookup table for path-derived files unless the storage contract itself changes.

External providers

Provider credentials stay server-side. API responses return provider IDs, status, and safe media references, never API keys or Railway credentials.

Source-of-truth rules

1. lib/railway-stores.ts is authoritative for logical store routing.
2. Type definitions in lib/ are authoritative for serialized domain shapes.
3. app/api/**/route.ts is authoritative for the internal HTTP contract.
4. Provisioning scripts define physical columns and indexes.
5. Runtime brand configuration lives in lib/realfarm-data.ts; persisted workspace data lives in Railway.
6. Roadmap documents describe intended changes and must not be read as current behavior.