Skip to content
arbe docs
View as .md

Durable streams

Append-only JSON logs addressed by URL, hosted by Electric SQL. Not Cloudflare Durable Objects — they share a word and nothing else (a durable stream is byte storage; a Durable Object is an isolate with state). This page is protocol substrate; arbe’s envelope, client boundary, and event contract live in streams.

PUT  /stream/{path}                     # create (JSON mode by default)
POST /stream/{path}                     # append (returns Stream-Next-Offset header)
GET  /stream/{path}?offset=X            # read range (JSON array)
GET  /stream/{path}?offset=X&live=long-poll | live=sse   # tail

Offsets are opaque, monotonic, lexicographically sortable strings — store them, never parse or construct. Sentinels: -1 (full replay), now (tail). Historical reads are immutable byte ranges, CDN-cacheable. Live SSE cycles ~60s for CDN connection collapsing — clients reconnect from the last control event’s streamNextOffset.

arbe-thread-{threadId}     # arbe's only stream type today — one per thread (threadStreamId() in @arbe/core/schemas/thread.ts)

Each item is an ArbeStreamEntry<ArbeThreadPayload>; that envelope and payload contract live in streams. @arbe/streams/client provides the two transport clients (raw service + scoped proxy); @arbe/core/entries is the thread-aware consumer (ensureThreadStream, postThreadEntry, readThreadEntries) used from CLI, www, and dispatch. The web chat UI reads and writes via @arbe/core/client, decoded by apps/www/src/lib/chat-stream.ts and rendered by Chat.svelte.

Worth knowing:

  • Idempotent producers — Producer-Id / Producer-Epoch / Producer-Seq headers. Epoch fences zombies with 403; a retry of an accepted (id, epoch, seq) returns a dedup’d success. IdempotentProducer in the TS client handles batching and pipelining.
  • Forking — PUT with Stream-Forked-From: <path> branches a new independent stream at Stream-Fork-Offset (defaults to source tail) without copying history. Deleting the source soft-deletes it until the last fork is gone. Fits conversation branching, deterministic replay, and producer handoff.
  • Closure — Stream-Closed: true on a final POST seals a stream. Once closed it stays closed; reads still work.
  • TTL — Stream-TTL (relative seconds) or Stream-Expires-At (RFC 3339), mutually exclusive at create.
  • Retention — servers may drop old data. On 410 Gone, reset to -1 or now.
  • Live modes — ?live=sse for JSON/text streams, ?live=long-poll for binary or simple request-response.

Delete is hard. deleteThread (called by arbe thread delete and DELETE /api/threads/:id) drops the durable stream first, then the row — so a stream-side failure leaves the row in place for retry rather than orphaning a stream. deleteHouse snapshots child thread ids, lets the row delete cascade (FK on house), then sweeps each thread’s stream best-effort — a stream-side hiccup is logged, never strands the row.

The wider durable-streams ecosystem (Durable State, StreamDB, StreamFS, Yjs provider, TanStack AI / Vercel AI SDK transports, Durable Proxy, multi-language clients) lives upstream. arbe uses only the protocol + @durable-streams/client + the TanStack AI transport for chat. Backend today: Electric Cloud (electric-sql.cloud).

@arbe/streams also ships a self-host backend as an alternative to Electric Cloud: gateway.ts is a single-token auth proxy (DS_SECRET, file-backed storage, make deploy to a sprite) in front of the embedded server.ts stream server, which also backs the package’s tests and demo/ scripts.

Code: @arbe/streams/client, packages/streams/{gateway,server}.ts, @arbe/core/entries, packages/core/schemas/stream-events/thread.ts, packages/core/schemas/thread.ts (threadStreamId). Upstream docs: durable-streams.com, protocol spec.
See streams, threads, system/dispatch.