Durable Streams operations for in-process callers, including the HTTP layer
and Yjs. Each stream has one server process on the shard selected by its
placement key (see placement_key/1). Writes are acknowledged only once durable.
Offsets are integers here (Slap.Streams.Offset converts to and from the wire
format), and results are protocol outcomes, not HTTP responses.
Pass cluster: to use a module defined with use Slap.Streams.Cluster, otp_app: :my_app. Run one Streams cluster per VM.
PROTOCOL.md sections below refer to the
Durable Streams specification.
Errors shared by all calls
Invalid stream names, offsets, bodies and request fields return
{:error, {:bad_request, reason}}. Invalid control options (:timeout,
:cluster, :placement_key, read limits and waits) and unknown option
names raise ArgumentError. The placement_key/1 path helper also raises
for a non-binary argument.
{:error, :not_found}- no such stream (404).{:error, :gone}- soft-deleted (410).{:error, :unavailable}- the shard is stopping or moved, or a write failed; retry (503).{:error, :timeout}- no reply within:timeout(default 30 s). The request may still complete.{:error, {:bad_request, reason}}- 400.
reason is one of:
:invalid_path,:invalid_offset,:invalid_body,:invalid_content_type,:invalid_ttl,:invalid_expires_at,:invalid_closed,:invalid_close,:invalid_stream_seq,:invalid_producer;:invalid_fork_source,:invalid_fork_offset,:invalid_fork_sub_offset,:fork_of_itself,:fork_offset_beyond_source,:fork_offset_trimmed,:max_fork_depth_exceeded;:empty_body,:empty_array,:invalid_json,:ttl_and_expires_at,:new_epoch_must_start_at_zero,:trim_offset_beyond_tail.
Summary
Functions
Appends body (PROTOCOL.md §5.2). Options
Waits up to timeout ms for the wake of a wait from wait/4: :data,
:closed, :deleted, :unavailable (the stream's server or its node
went down) or :timeout (the wait is then cancelled).
Cancels a wait from wait/4.
Closes the stream without appending: append(path, "", close: true).
Copies the source's messages before :at into a new stream with its own storage.
Creates a stream (PROTOCOL.md §5.1). Options: :content_type, :ttl_s,
:expires_at_ms, :closed, :body (initial content).
Deletes the stream (PROTOCOL.md §5.4). Returns :ok once durable.
The stream's durable metadata and tail (PROTOCOL.md §5.5).
Lists active streams in prefix's placement group (placement_key/1) whose
paths start with prefix, in path order. Pending creations and deleted
streams are excluded. An expired stream stays listed until it is removed.
This operation is available only through the Elixir API.
The key that places path on a shard: the path up to its first segment
that starts with ., or the whole path. Streams that differ only from
such a segment are one group on one shard, whether they are reached here
or over HTTP: a Yjs document's .updates, .index and
.snapshots/<offset>_snapshot (slap_yjs), for example. The
:placement_key option of the calls here overrides it.
Reads from offset (an integer, :start for the first retained message,
or :now) up to the durable tail (PROTOCOL.md §5.6), at most about
:max_bytes (default 1 MiB; at least one message). The data is read in
the calling process.
Permanently prevents new streams in the placement group group
(see placement_key/1). New streams, forks and copies return
{:error, :sealed} (409). A fork or copy that is still pending is removed
when it tries to become active, and that call returns {:error, :sealed}.
Existing streams are unchanged, and a retry of a creation that had already
completed returns {:ok, :exists, info}.
This operation is available only through the Elixir API.
Makes reads below offset return {:error, :trimmed} (410). A background
job deletes the messages below both offset and the inherited ranges of
this stream's forks and pending copies. Trimming a fork does not shrink its
inherited range on its source. An offset past the tail is a bad request.
Waits for data after offset (an integer or :now), for long-poll and
SSE. Returns {:ok, :data} or {:ok, :closed} at once if there is
something to report, else {:ok, {:waiting, wait}}: pid then gets
{:slap_streams_wake, ref, reason} once, where ref is
Slap.Streams.Wait.ref(wait) and reason is :data, :closed,
:deleted or :unavailable. Cancel with cancel_wait/2.
Types
@type append_result() :: %{ result: :appended | :duplicate | :closed, next_offset: non_neg_integer(), closed: boolean(), producer: {non_neg_integer(), non_neg_integer()} | nil }
@type bad_request_reason() ::
:empty_array
| :empty_body
| :fork_of_itself
| :fork_offset_beyond_source
| :fork_offset_trimmed
| :max_fork_depth_exceeded
| :invalid_body
| :invalid_close
| :invalid_closed
| :invalid_content_type
| :invalid_expires_at
| :invalid_fork_offset
| :invalid_fork_source
| :invalid_fork_sub_offset
| :invalid_json
| :invalid_offset
| :invalid_path
| :invalid_producer
| :invalid_stream_seq
| :invalid_ttl
| :new_epoch_must_start_at_zero
| :trim_offset_beyond_tail
| :ttl_and_expires_at
@type error() :: :bad_offset | :conflict | :content_type_mismatch | :deleted | :gone | :not_found | :overloaded | :offset_beyond_tail | :payload_too_large | :sealed | :source_gone | :source_not_found | :stream_seq_conflict | :timeout | :trimmed | :unavailable | {:bad_request, bad_request_reason()} | {:closed, non_neg_integer()} | {:producer_seq_gap, non_neg_integer(), non_neg_integer()} | {:stale_epoch, non_neg_integer()}
@type info() :: %{ next_offset: non_neg_integer(), closed: boolean(), content_type: binary(), ttl_s: non_neg_integer() | nil, expires_at_ms: integer() | nil }
@type path() :: binary()
@type read_result() :: %{ messages: [{non_neg_integer(), binary()}], next_offset: non_neg_integer(), up_to_date: boolean(), closed: boolean(), content_type: binary() }
Functions
@spec append(path(), binary(), keyword()) :: {:ok, append_result()} | {:error, error()}
Appends body (PROTOCOL.md §5.2). Options:
:content_type- checked against the stream's, if given.:close- close the stream with this append (an empty body closes without appending, §5.3).:stream_seq- theStream-Seqheader.:producer-{producer_id, epoch, seq}(§5.2.1).
Returns {:ok, %{result: r, next_offset: o, closed: c, producer: p}} where
r is :appended (200 with a producer, else 204), :duplicate (204) or
:closed (a close-only request, 204), and p is {epoch, seq} to echo.
Errors besides the shared ones: {:closed, next_offset} (409 with
Stream-Closed), :content_type_mismatch (409), :stream_seq_conflict
(409), {:stale_epoch, epoch} (403), {:producer_seq_gap, expected, received} (409).
@spec await_wake(path(), Slap.Streams.Wait.t(), timeout()) :: :data | :closed | :deleted | :unavailable | :timeout
Waits up to timeout ms for the wake of a wait from wait/4: :data,
:closed, :deleted, :unavailable (the stream's server or its node
went down) or :timeout (the wait is then cancelled).
@spec cancel_wait(path(), Slap.Streams.Wait.t(), keyword()) :: :ok | {:error, error()}
Cancels a wait from wait/4.
@spec close(path(), keyword()) :: {:ok, append_result()} | {:error, error()}
Closes the stream without appending: append(path, "", close: true).
Copies the source's messages before :at into a new stream with its own storage.
Options: :at (:latest, the default, or an integer offset), :cluster,
:timeout, :source_placement_key and target :placement_key. The copy
preserves offsets, trim, content type, closed state and expiry configuration.
Relative TTL starts when the copy becomes active; an absolute expiry that
passes before then cancels it. Producer and Stream-Seq state start empty.
Reference depth resets to zero.
Time and storage grow with the copied data. A completed copy does not depend on its source. Until then, the target is hidden and the source keeps the messages being copied.
The copy runs in the calling process and stops at :timeout (default
30 s), keeping the pages copied so far. Call copy/3 again with the same
source path, :source_placement_key and :at until it returns
{:ok, :created, info}, or {:ok, :exists, info} if an earlier call
finished after returning. A larger :timeout needs fewer calls. Retries
copy the range the first call selected, even if the source changes or is
deleted. Delete the target to cancel the copy; repair cancels it once no
page has been copied for :creation_grace.
Creates a stream (PROTOCOL.md §5.1). Options: :content_type, :ttl_s,
:expires_at_ms, :closed, :body (initial content).
A fork (§4.2) also takes :forked_from (the source's path), and optionally
:fork_offset (default: the source's tail) and :fork_sub_offset. The
fork reads the source's durable messages before the fork offset, its
inherited range, from the source's storage; creation does not copy them.
Returns {:ok, :created, info} (201), {:ok, :exists, info} when a stream
with the same configuration exists (200), or {:error, :conflict} when one
with a different configuration does, or the path is soft-deleted (409).
Fork errors: :source_not_found (404), :source_gone (409),
:content_type_mismatch (409), and
{:bad_request, :max_fork_depth_exceeded} (400, default depth limit 8).
Reads of the inherited range need the source's shard, and those of the
source's own sources, to be available. The source keeps the inherited range
until the fork and every fork made from it are removed; trimming the fork
does not shrink it. Use :source_placement_key to select the source's shard
and :placement_key for the target.
A fork that is still being created returns :unavailable for reads and
appends. After {:error, :timeout} or another error, the fork can stay
pending: call create/2 again with the same options to finish it (it
returns {:ok, :exists, info} if the earlier call finished after
returning), or delete it to cancel. Repair cancels it after
:creation_grace.
Deletes the stream (PROTOCOL.md §5.4). Returns :ok once durable.
The stream's durable metadata and tail (PROTOCOL.md §5.5).
Lists active streams in prefix's placement group (placement_key/1) whose
paths start with prefix, in path order. Pending creations and deleted
streams are excluded. An expired stream stays listed until it is removed.
This operation is available only through the Elixir API.
The key that places path on a shard: the path up to its first segment
that starts with ., or the whole path. Streams that differ only from
such a segment are one group on one shard, whether they are reached here
or over HTTP: a Yjs document's .updates, .index and
.snapshots/<offset>_snapshot (slap_yjs), for example. The
:placement_key option of the calls here overrides it.
@spec read(path(), non_neg_integer() | :start | :now, keyword()) :: {:ok, read_result()} | {:error, error()}
Reads from offset (an integer, :start for the first retained message,
or :now) up to the durable tail (PROTOCOL.md §5.6), at most about
:max_bytes (default 1 MiB; at least one message). The data is read in
the calling process.
Returns {:ok, %{messages: [{offset, bytes}], next_offset, up_to_date, closed, content_type}}. For a JSON stream each message is one JSON value;
the HTTP layer joins them into a response array. closed is true only when the
read reached the end of a closed stream. An offset past the durable tail is
{:error, :offset_beyond_tail}.
With wait: ms, a read that finds no messages at the tail of an open
stream waits up to ms for data (long-poll), then reads again from the
tail; that read's result is returned, empty if nothing came. It reads
again after a timeout too, so that an empty result's next_offset and
closed come from one view of the stream. A stream deleted during the
wait is {:error, :deleted}; one whose server or node went down without
saying so is {:error, :unavailable}.
An explicit :timeout bounds the read, wait, and final read together. A
long-poll reserves time for the final read. If that read takes longer than
the remaining budget, the call returns {:error, :timeout} even when data
arrived; the caller can read again. Without :timeout, each request uses
its 30-second default and :wait bounds the wait.
Permanently prevents new streams in the placement group group
(see placement_key/1). New streams, forks and copies return
{:error, :sealed} (409). A fork or copy that is still pending is removed
when it tries to become active, and that call returns {:error, :sealed}.
Existing streams are unchanged, and a retry of a creation that had already
completed returns {:ok, :exists, info}.
This operation is available only through the Elixir API.
@spec trim(path(), non_neg_integer(), keyword()) :: :ok | {:error, error()}
Makes reads below offset return {:error, :trimmed} (410). A background
job deletes the messages below both offset and the inherited ranges of
this stream's forks and pending copies. Trimming a fork does not shrink its
inherited range on its source. An offset past the tail is a bad request.
@spec wait(path(), non_neg_integer() | :now) :: {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}
Waits for data after offset (an integer or :now), for long-poll and
SSE. Returns {:ok, :data} or {:ok, :closed} at once if there is
something to report, else {:ok, {:waiting, wait}}: pid then gets
{:slap_streams_wake, ref, reason} once, where ref is
Slap.Streams.Wait.ref(wait) and reason is :data, :closed,
:deleted or :unavailable. Cancel with cancel_wait/2.
The third argument can be an options list; the waiting pid then defaults to the caller.
When pid is the caller, the stream's server (perhaps on another node)
is monitored too, so that await_wake/3 reports :unavailable if it or
its node goes down without saying so.
@spec wait(path(), non_neg_integer() | :now, pid() | keyword()) :: {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}
@spec wait(path(), non_neg_integer() | :now, pid(), keyword()) :: {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}