Slap.Streams (slap_streams v0.6.1)

Copy Markdown View Source

Durable Streams operations for in-process callers, including the HTTP layer and Yjs. Each stream has one server process on the shard selected by its placement key (see placement_key/1). Writes are acknowledged only once durable.

Offsets are integers here (Slap.Streams.Offset converts to and from the wire format), and results are protocol outcomes, not HTTP responses. Pass cluster: to use a module defined with use Slap.Streams.Cluster, otp_app: :my_app. Run one Streams cluster per VM.

PROTOCOL.md sections below refer to the Durable Streams specification.

Errors shared by all calls

Invalid stream names, offsets, bodies and request fields return {:error, {:bad_request, reason}}. Invalid control options (:timeout, :cluster, :placement_key, read limits and waits) and unknown option names raise ArgumentError. The placement_key/1 path helper also raises for a non-binary argument.

  • {:error, :not_found} - no such stream (404).
  • {:error, :gone} - soft-deleted (410).
  • {:error, :unavailable} - the shard is stopping or moved, or a write failed; retry (503).
  • {:error, :timeout} - no reply within :timeout (default 30 s). The request may still complete.
  • {:error, {:bad_request, reason}} - 400.

reason is one of:

  • :invalid_path, :invalid_offset, :invalid_body, :invalid_content_type, :invalid_ttl, :invalid_expires_at, :invalid_closed, :invalid_close, :invalid_stream_seq, :invalid_producer;
  • :invalid_fork_source, :invalid_fork_offset, :invalid_fork_sub_offset, :fork_of_itself, :fork_offset_beyond_source, :fork_offset_trimmed, :max_fork_depth_exceeded;
  • :empty_body, :empty_array, :invalid_json, :ttl_and_expires_at, :new_epoch_must_start_at_zero, :trim_offset_beyond_tail.

Summary

Functions

Appends body (PROTOCOL.md §5.2). Options

Waits up to timeout ms for the wake of a wait from wait/4: :data, :closed, :deleted, :unavailable (the stream's server or its node went down) or :timeout (the wait is then cancelled).

Closes the stream without appending: append(path, "", close: true).

Copies the source's messages before :at into a new stream with its own storage.

Creates a stream (PROTOCOL.md §5.1). Options: :content_type, :ttl_s, :expires_at_ms, :closed, :body (initial content).

Deletes the stream (PROTOCOL.md §5.4). Returns :ok once durable.

The stream's durable metadata and tail (PROTOCOL.md §5.5).

Lists active streams in prefix's placement group (placement_key/1) whose paths start with prefix, in path order. Pending creations and deleted streams are excluded. An expired stream stays listed until it is removed. This operation is available only through the Elixir API.

The key that places path on a shard: the path up to its first segment that starts with ., or the whole path. Streams that differ only from such a segment are one group on one shard, whether they are reached here or over HTTP: a Yjs document's .updates, .index and .snapshots/<offset>_snapshot (slap_yjs), for example. The :placement_key option of the calls here overrides it.

Reads from offset (an integer, :start for the first retained message, or :now) up to the durable tail (PROTOCOL.md §5.6), at most about :max_bytes (default 1 MiB; at least one message). The data is read in the calling process.

Permanently prevents new streams in the placement group group (see placement_key/1). New streams, forks and copies return {:error, :sealed} (409). A fork or copy that is still pending is removed when it tries to become active, and that call returns {:error, :sealed}. Existing streams are unchanged, and a retry of a creation that had already completed returns {:ok, :exists, info}. This operation is available only through the Elixir API.

Makes reads below offset return {:error, :trimmed} (410). A background job deletes the messages below both offset and the inherited ranges of this stream's forks and pending copies. Trimming a fork does not shrink its inherited range on its source. An offset past the tail is a bad request.

Waits for data after offset (an integer or :now), for long-poll and SSE. Returns {:ok, :data} or {:ok, :closed} at once if there is something to report, else {:ok, {:waiting, wait}}: pid then gets {:slap_streams_wake, ref, reason} once, where ref is Slap.Streams.Wait.ref(wait) and reason is :data, :closed, :deleted or :unavailable. Cancel with cancel_wait/2.

Types

append_result()

@type append_result() :: %{
  result: :appended | :duplicate | :closed,
  next_offset: non_neg_integer(),
  closed: boolean(),
  producer: {non_neg_integer(), non_neg_integer()} | nil
}

bad_request_reason()

@type bad_request_reason() ::
  :empty_array
  | :empty_body
  | :fork_of_itself
  | :fork_offset_beyond_source
  | :fork_offset_trimmed
  | :max_fork_depth_exceeded
  | :invalid_body
  | :invalid_close
  | :invalid_closed
  | :invalid_content_type
  | :invalid_expires_at
  | :invalid_fork_offset
  | :invalid_fork_source
  | :invalid_fork_sub_offset
  | :invalid_json
  | :invalid_offset
  | :invalid_path
  | :invalid_producer
  | :invalid_stream_seq
  | :invalid_ttl
  | :new_epoch_must_start_at_zero
  | :trim_offset_beyond_tail
  | :ttl_and_expires_at

error()

@type error() ::
  :bad_offset
  | :conflict
  | :content_type_mismatch
  | :deleted
  | :gone
  | :not_found
  | :overloaded
  | :offset_beyond_tail
  | :payload_too_large
  | :sealed
  | :source_gone
  | :source_not_found
  | :stream_seq_conflict
  | :timeout
  | :trimmed
  | :unavailable
  | {:bad_request, bad_request_reason()}
  | {:closed, non_neg_integer()}
  | {:producer_seq_gap, non_neg_integer(), non_neg_integer()}
  | {:stale_epoch, non_neg_integer()}

info()

@type info() :: %{
  next_offset: non_neg_integer(),
  closed: boolean(),
  content_type: binary(),
  ttl_s: non_neg_integer() | nil,
  expires_at_ms: integer() | nil
}

path()

@type path() :: binary()

read_result()

@type read_result() :: %{
  messages: [{non_neg_integer(), binary()}],
  next_offset: non_neg_integer(),
  up_to_date: boolean(),
  closed: boolean(),
  content_type: binary()
}

Functions

append(path, body, opts \\ [])

@spec append(path(), binary(), keyword()) ::
  {:ok, append_result()} | {:error, error()}

Appends body (PROTOCOL.md §5.2). Options:

  • :content_type - checked against the stream's, if given.
  • :close - close the stream with this append (an empty body closes without appending, §5.3).
  • :stream_seq - the Stream-Seq header.
  • :producer - {producer_id, epoch, seq} (§5.2.1).

Returns {:ok, %{result: r, next_offset: o, closed: c, producer: p}} where r is :appended (200 with a producer, else 204), :duplicate (204) or :closed (a close-only request, 204), and p is {epoch, seq} to echo.

Errors besides the shared ones: {:closed, next_offset} (409 with Stream-Closed), :content_type_mismatch (409), :stream_seq_conflict (409), {:stale_epoch, epoch} (403), {:producer_seq_gap, expected, received} (409).

await_wake(path, pending, timeout)

@spec await_wake(path(), Slap.Streams.Wait.t(), timeout()) ::
  :data | :closed | :deleted | :unavailable | :timeout

Waits up to timeout ms for the wake of a wait from wait/4: :data, :closed, :deleted, :unavailable (the stream's server or its node went down) or :timeout (the wait is then cancelled).

cancel_wait(path, pending, opts \\ [])

@spec cancel_wait(path(), Slap.Streams.Wait.t(), keyword()) :: :ok | {:error, error()}

Cancels a wait from wait/4.

close(path, opts \\ [])

@spec close(path(), keyword()) :: {:ok, append_result()} | {:error, error()}

Closes the stream without appending: append(path, "", close: true).

copy(source_path, target_path, opts \\ [])

@spec copy(path(), path(), keyword()) ::
  {:ok, :created | :exists, info()} | {:error, error()}

Copies the source's messages before :at into a new stream with its own storage.

Options: :at (:latest, the default, or an integer offset), :cluster, :timeout, :source_placement_key and target :placement_key. The copy preserves offsets, trim, content type, closed state and expiry configuration. Relative TTL starts when the copy becomes active; an absolute expiry that passes before then cancels it. Producer and Stream-Seq state start empty. Reference depth resets to zero.

Time and storage grow with the copied data. A completed copy does not depend on its source. Until then, the target is hidden and the source keeps the messages being copied.

The copy runs in the calling process and stops at :timeout (default 30 s), keeping the pages copied so far. Call copy/3 again with the same source path, :source_placement_key and :at until it returns {:ok, :created, info}, or {:ok, :exists, info} if an earlier call finished after returning. A larger :timeout needs fewer calls. Retries copy the range the first call selected, even if the source changes or is deleted. Delete the target to cancel the copy; repair cancels it once no page has been copied for :creation_grace.

create(path, opts \\ [])

@spec create(path(), keyword()) ::
  {:ok, :created | :exists, info()} | {:error, error()}

Creates a stream (PROTOCOL.md §5.1). Options: :content_type, :ttl_s, :expires_at_ms, :closed, :body (initial content).

A fork (§4.2) also takes :forked_from (the source's path), and optionally :fork_offset (default: the source's tail) and :fork_sub_offset. The fork reads the source's durable messages before the fork offset, its inherited range, from the source's storage; creation does not copy them.

Returns {:ok, :created, info} (201), {:ok, :exists, info} when a stream with the same configuration exists (200), or {:error, :conflict} when one with a different configuration does, or the path is soft-deleted (409).

Fork errors: :source_not_found (404), :source_gone (409), :content_type_mismatch (409), and {:bad_request, :max_fork_depth_exceeded} (400, default depth limit 8).

Reads of the inherited range need the source's shard, and those of the source's own sources, to be available. The source keeps the inherited range until the fork and every fork made from it are removed; trimming the fork does not shrink it. Use :source_placement_key to select the source's shard and :placement_key for the target.

A fork that is still being created returns :unavailable for reads and appends. After {:error, :timeout} or another error, the fork can stay pending: call create/2 again with the same options to finish it (it returns {:ok, :exists, info} if the earlier call finished after returning), or delete it to cancel. Repair cancels it after :creation_grace.

delete(path, opts \\ [])

@spec delete(path(), keyword()) :: :ok | {:error, error()}

Deletes the stream (PROTOCOL.md §5.4). Returns :ok once durable.

head(path, opts \\ [])

@spec head(path(), keyword()) :: {:ok, info()} | {:error, error()}

The stream's durable metadata and tail (PROTOCOL.md §5.5).

list(prefix, opts \\ [])

@spec list(path(), keyword()) :: {:ok, [path()]} | {:error, error()}

Lists active streams in prefix's placement group (placement_key/1) whose paths start with prefix, in path order. Pending creations and deleted streams are excluded. An expired stream stays listed until it is removed. This operation is available only through the Elixir API.

placement_key(path)

@spec placement_key(path()) :: String.t()

The key that places path on a shard: the path up to its first segment that starts with ., or the whole path. Streams that differ only from such a segment are one group on one shard, whether they are reached here or over HTTP: a Yjs document's .updates, .index and .snapshots/<offset>_snapshot (slap_yjs), for example. The :placement_key option of the calls here overrides it.

read(path, offset, opts \\ [])

@spec read(path(), non_neg_integer() | :start | :now, keyword()) ::
  {:ok, read_result()} | {:error, error()}

Reads from offset (an integer, :start for the first retained message, or :now) up to the durable tail (PROTOCOL.md §5.6), at most about :max_bytes (default 1 MiB; at least one message). The data is read in the calling process.

Returns {:ok, %{messages: [{offset, bytes}], next_offset, up_to_date, closed, content_type}}. For a JSON stream each message is one JSON value; the HTTP layer joins them into a response array. closed is true only when the read reached the end of a closed stream. An offset past the durable tail is {:error, :offset_beyond_tail}.

With wait: ms, a read that finds no messages at the tail of an open stream waits up to ms for data (long-poll), then reads again from the tail; that read's result is returned, empty if nothing came. It reads again after a timeout too, so that an empty result's next_offset and closed come from one view of the stream. A stream deleted during the wait is {:error, :deleted}; one whose server or node went down without saying so is {:error, :unavailable}.

An explicit :timeout bounds the read, wait, and final read together. A long-poll reserves time for the final read. If that read takes longer than the remaining budget, the call returns {:error, :timeout} even when data arrived; the caller can read again. Without :timeout, each request uses its 30-second default and :wait bounds the wait.

seal(group, opts \\ [])

@spec seal(binary(), keyword()) :: :ok | {:error, error()}

Permanently prevents new streams in the placement group group (see placement_key/1). New streams, forks and copies return {:error, :sealed} (409). A fork or copy that is still pending is removed when it tries to become active, and that call returns {:error, :sealed}. Existing streams are unchanged, and a retry of a creation that had already completed returns {:ok, :exists, info}. This operation is available only through the Elixir API.

trim(path, offset, opts \\ [])

@spec trim(path(), non_neg_integer(), keyword()) :: :ok | {:error, error()}

Makes reads below offset return {:error, :trimmed} (410). A background job deletes the messages below both offset and the inherited ranges of this stream's forks and pending copies. Trimming a fork does not shrink its inherited range on its source. An offset past the tail is a bad request.

wait(path, offset)

@spec wait(path(), non_neg_integer() | :now) ::
  {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}

Waits for data after offset (an integer or :now), for long-poll and SSE. Returns {:ok, :data} or {:ok, :closed} at once if there is something to report, else {:ok, {:waiting, wait}}: pid then gets {:slap_streams_wake, ref, reason} once, where ref is Slap.Streams.Wait.ref(wait) and reason is :data, :closed, :deleted or :unavailable. Cancel with cancel_wait/2.

The third argument can be an options list; the waiting pid then defaults to the caller.

When pid is the caller, the stream's server (perhaps on another node) is monitored too, so that await_wake/3 reports :unavailable if it or its node goes down without saying so.

wait(path, offset, opts)

@spec wait(path(), non_neg_integer() | :now, pid() | keyword()) ::
  {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}

wait(path, offset, pid, opts)

@spec wait(path(), non_neg_integer() | :now, pid(), keyword()) ::
  {:ok, :data | :closed | {:waiting, Slap.Streams.Wait.t()}} | {:error, error()}