# Laser SDK Laser SDK lets agents and services exchange messages, share state and memory, and keep a durable, replayable record of every session, on one log. It has clients for Rust, Python, and TypeScript that behave the same way. It runs on [Apache Iggy](https://iggy.apache.org), locally with [Laser Stack](/laser-sdk/laser-stack) or in production on LaserData Cloud. ## An agent and a caller The `triage` agent advertises the `resolve-ticket` capability and replies to each request. The caller sends a request to whichever agent has that capability and waits for the reply. Both run on plain Apache Iggy. The connection string is the Laser Stack default. ```ts import { Agent, AgentId, AgentTopic, Laser, TopicRetention, agentMessageBody, routeToCapable } from "@laserdata/laser-sdk" await using laser = await Laser.connectWithStream("iggy:laser@127.0.0.1:8090", "support") await laser.bootstrap(1, TopicRetention.expireAfter(86_400_000)) await using triage = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .capabilities([{ skillId: "resolve-ticket" }]) .handler({ handle: (_message, ctx) => ctx.respond(new TextEncoder().encode("on it")) }) .build() .spawn(laser) await triage.ready() const result = await laser .contract(routeToCapable("resolve-ticket", { kind: "any" })) .from(AgentId.new("support-desk")) .payload(new TextEncoder().encode("ticket #42 is stuck")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .send() if (result.kind === "completed") console.log(new TextDecoder().decode(agentMessageBody(result.reply))) ``` ```rust use laser_sdk::prelude::full::*; use laser_sdk::wire::agent::CapabilityDescriptor; use std::time::Duration; struct Triage; impl AgentHandler for Triage { async fn handle(&self, _message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { ctx.respond("on it").await } } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_with_stream("iggy:laser@127.0.0.1:8090", "support").await?; laser.bootstrap(1, TopicRetention::expire_after(Duration::from_secs(86_400))).await?; let mut triage = Agent::builder() .id("triage".parse()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .capabilities(vec![CapabilityDescriptor { skill_id: "resolve-ticket".into(), ..Default::default() }]) .handler(Triage) .build() .spawn(laser.clone()); triage.ready().await?; let result = laser .contract(Router::to_capable("resolve-ticket", RoutePolicy::Any)) .from("support-desk".parse()?) .payload("ticket #42 is stuck") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .send() .await?; if let Contract::Completed(reply) = result { println!("{}", String::from_utf8_lossy(reply.body())); } triage.shutdown().await } ``` ```python import asyncio import laser_sdk as ls SESSIONS = ls.AgentTopic.Sessions async def handle(ctx, message): await ctx.respond(b"on it") async def main(): laser = await ls.Laser.connect_with_stream("iggy:laser@127.0.0.1:8090", "support") await laser.bootstrap(1, ls.TopicRetention.expire_after(86_400_000)) triage = laser.spawn_agent("triage", SESSIONS, handle, respond_on=SESSIONS, capabilities=["resolve-ticket"]) await triage.ready() result = await laser.contract("resolve-ticket", b"ticket #42 is stuck", source="support-desk", fixed_inbox=SESSIONS) if isinstance(result, ls.Contract.Completed): print(bytes(result[0].body()).decode()) await triage.shutdown() await laser.close() asyncio.run(main()) ``` The request and the reply are messages on the `agent.sessions` topic, so they stay on the log after the program exits. The [runnable agent example](https://github.com/laserdata/laser-sdk/tree/main/examples) adds a deadline and pickup tracking. The [Quickstart](/laser-sdk/quickstart) walks through a first program step by step. ## What you can build | You need | Use | Needs Laser Stack or Cloud | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------- | | Agents that find each other by capability, send requests, and get replies | [Agents](/laser-sdk/fabric) | No | | A recorded unit of work with model calls, tool calls, state, and a final status | [Sessions](/laser-sdk/session) | Only for the session index and timelines | | Facts an agent remembers and recalls across conversations | [Memory](/laser-sdk/memory) | Only for the managed memory view | | Keyed state with compare-and-swap, expiry, and branches | [Key-value state](/laser-sdk/state) | Yes | | Queries over your messages that stay up to date, and a feed of what changed | [Queries and views](/laser-sdk/views), [Changes](/laser-sdk/change-feed) | Yes | | A stream where each consumer receives only the records it asked for | [Filters](/laser-sdk/consumer-filters) | Yes | ## Where to go next ## Install Laser SDK needs Node.js 22.14 or later. ```bash npm install @laserdata/laser-sdk ``` The `agent` feature adds agents, sessions, and memory. The `managed` feature adds queries, key-value state, graph, forks, filters, and changes. Streaming is on by default. ```bash cargo add laser-sdk --features agent cargo add tokio --features macros,rt-multi-thread ``` Laser SDK needs Python 3.10 or later. ```bash pip install laser-sdk ``` Packages are on [npm](https://www.npmjs.com/package/@laserdata/laser-sdk), [crates.io](https://crates.io/crates/laser-sdk), and [PyPI](https://pypi.org/project/laser-sdk). Source and issues are at [github.com/laserdata/laser-sdk](https://github.com/laserdata/laser-sdk). Source: https://docs.laserdata.com/laser-sdk --- # Quickstart This page gets two agents talking. One agent handles a request, a caller sends it one, and the caller prints the reply. Then the same exchange runs inside a recorded session. Pick Rust, Python, or TypeScript in the tabs. Your choice carries across the SDK docs. ## 1. Start a target Every step on this page works on any of these targets: * Laser Stack runs Apache Iggy and the managed plane on your machine. Use it for local development. * LaserData Cloud gives you a connection string on the Credentials tab of your deployment. Make sure an [access rule](/networking/access-rules) allows your client IP. * Plain Apache Iggy runs messaging, agents, and sessions. Managed features such as queries and key-value state return `Unsupported`. To start Laser Stack: ```bash git clone https://github.com/laserdata/laser-stack cd laser-stack ./scripts/up ``` The script waits for Iggy and the plane, then prints a `LASER_CONNECTION_STRING` export. Copy it into your shell. For Cloud or your own Iggy server, export that server's connection string the same way: ```bash export LASER_CONNECTION_STRING='iggy:laser@127.0.0.1:8090' ``` [Laser Stack](/laser-sdk/laser-stack) covers requirements, persistence, and troubleshooting. [Connect](/laser-sdk/connect) covers connection strings, tokens, and TLS. ## 2. Install the SDK ```bash npm install @laserdata/laser-sdk ``` Requires Node.js 22.14 or later. ```bash cargo add laser-sdk --features agent cargo add tokio --features macros,rt-multi-thread ``` Requires Rust 1.98.0 or later. The `agent` feature adds agents and sessions. Add `managed` for queries, key-value state, and the other managed features. ```bash pip install laser-sdk ``` Requires Python 3.10 or later. With `uv`, run `uv pip install laser-sdk` instead. ## 3. Have one agent answer another The program connects, creates the agent topics on a stream named `quickstart`, and starts an agent called `triage`. Then a caller called `intake` sends `triage` a request and prints the reply. ```ts import { Agent, AgentId, AgentTopic, Laser, TopicRetention, agentMessageBody, routeTo } from "@laserdata/laser-sdk" const text = new TextEncoder() const decode = (bytes: Uint8Array) => new TextDecoder().decode(bytes) await using root = await Laser.connectEnv() const laser = root.withDefaultStream("quickstart") await laser.bootstrap(1, TopicRetention.expireAfter(86_400_000)) await using triage = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .handler({ handle: (message, ctx) => ctx.respond(text.encode(`triaged: ${decode(agentMessageBody(message))}`)) }) .build() .spawn(laser) await triage.ready() const outcome = await laser .contract(routeTo(AgentId.new("triage"))) .from(AgentId.new("intake")) .payload(text.encode("disk full on node-7")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .send() if (outcome.kind === "completed") { console.log(decode(agentMessageBody(outcome.reply))) } else { console.log(`no reply: ${outcome.kind}`) } ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; struct Triage; impl AgentHandler for Triage { async fn handle(&self, message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { let ticket = String::from_utf8_lossy(message.body()); ctx.respond(format!("triaged: {ticket}")).await } } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?.with_default_stream("quickstart"); let retention = TopicRetention::expire_after(Duration::from_secs(86_400)); laser.bootstrap(1, retention).await?; let mut triage = Agent::builder() .id("triage".parse()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .handler(Triage) .build() .spawn(laser.clone()); triage.ready().await?; let outcome = laser .contract(Router::to("triage".parse()?)) .from("intake".parse()?) .payload("disk full on node-7") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .send() .await?; match outcome { Contract::Completed(reply) => println!("{}", String::from_utf8_lossy(reply.body())), other => println!("no reply: {other:?}"), } triage.shutdown().await } ``` ```python import asyncio import laser_sdk as ls async def handle(ctx, message): ticket = bytes(message.body()).decode() await ctx.respond(f"triaged: {ticket}".encode()) async def main(): async with await ls.Laser.connect_env() as root: laser = root.with_default_stream("quickstart") await laser.bootstrap(1, ls.TopicRetention.expire_after(86_400_000)) async with laser.spawn_agent( "triage", ls.AgentTopic.Sessions, handle, respond_on=ls.AgentTopic.Sessions, ): outcome = await laser.contract( None, b"disk full on node-7", agent="triage", source="intake", fixed_inbox=ls.AgentTopic.Sessions, ) if isinstance(outcome, ls.Contract.Completed): print(bytes(outcome[0].body()).decode()) else: print(f"no reply: {outcome!r}") asyncio.run(main()) ``` The program prints: ```text triaged: disk full on node-7 ``` What each part does: * `bootstrap` creates the agent topics on the stream, one partition each, and keeps `agent.sessions` records for one day. It is safe to run again. * `ready()` waits until the agent reads its topic, so the request cannot arrive before it listens. Python's `async with` waits for you. * `contract` sends one request to one agent and waits for the answer, 30 seconds by default. It ends as completed, failed, not consumed, or timed out. * The fixed inbox sends the request to `agent.sessions`, where `triage` listens. Without it, the SDK looks up the agent's inbox from live presence, which needs an agent that advertises capabilities and a server that serves presence. ## 4. Record the exchange as a session A session is one unit of work with a start, an end, and every record in between, kept on the log. Open a session as `intake`, send the request inside it, end the session, and read it back. Replace the request in step 3 with this code. ```ts import { sessionTurnText } from "@laserdata/laser-sdk" const { session } = await laser.sessions().start().agent(AgentId.new("intake")).begin() await laser .contract(routeTo(AgentId.new("triage"))) .from(AgentId.new("intake")) .payload(text.encode("disk full on node-7")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .conversation(session.conversation) .send() await session.end() for (const turn of await session.context()) { console.log(turn.display, turn.display === "agent.message" ? sessionTurnText(turn) : "") } ``` ```rust let (session, _lease) = laser .sessions() .start() .agent("intake".parse::()?) .begin() .await?; laser .contract(Router::to("triage".parse()?)) .from("intake".parse()?) .payload("disk full on node-7") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .conversation(session.conversation()) .send() .await?; session.end().await?; for turn in session.context().await? { let kind: &str = turn.display.into(); match kind { "agent.message" => println!("{kind}: {}", turn.text()), _ => println!("{kind}"), } } ``` ```python session, _lease = await laser.sessions().start().agent("intake").begin() await laser.contract( None, b"disk full on node-7", agent="triage", source="intake", fixed_inbox=ls.AgentTopic.Sessions, conversation=session.conversation, ) await session.end() for turn in await session.context(): print(turn.display, turn.text() if turn.display == "agent.message" else "") ``` The program prints the session as it was recorded: ```text session.started agent.message disk full on node-7 agent.message triaged: disk full on node-7 session.completed ``` The request and the reply carry the session ID as their conversation, so `context()` returns them with the session's start and end. This works on plain Apache Iggy, because the SDK reads the records back from `agent.sessions`. Laser Stack and LaserData Cloud also index sessions, which adds listing, status, and timeline reads across the stream. [Sessions](/laser-sdk/session) shows both. ## 5. Publish and read your own messages Agents use ordinary topics underneath. To publish your own records, such as telemetry or support tickets, and read them back by offset, follow [Messages](/laser-sdk/log). ## Next Source: https://docs.laserdata.com/laser-sdk/quickstart --- # Concepts Laser SDK keeps everything on one log. Messages, agent requests and replies, session records, and memory are all records on topics in Apache Iggy. Read this page once and the guides will make sense. ## Stream A stream is the isolation boundary. It groups topics and owns their permissions. A session and all of its child sessions live in one stream, and no read, link, or derived id crosses streams. Use one stream per application or tenant. Most agent calls need a default stream. Set it when you connect, for example with `connect_with_stream` (TypeScript `connectWithStream`). Managed names such as key-value namespaces and graphs are scoped to that stream as `stream:/`. See [Connect](/laser-sdk/connect). ## Topic A topic is a named log inside a stream. Producers append records to it and consumers read them back. The call `bootstrap(partitions, retention)` creates the agent topics: `agent.sessions` for requests, replies, and session records, plus `agent.heartbeats`, `agent.streams`, `agent.memory`, `agent.dlq`, `agent.audit`, and `agent.workflow_journal`. You create your own topics for your own data. See [Messages](/laser-sdk/log). ## Partition A topic is split into partitions. Order holds only within one partition. Records with the same partition key land on the same partition, so they stay in order. Records without a key are spread across partitions. In the default layout, the SDK keys every agent record by its session. All records of one session land on one partition of `agent.sessions`, in the order they were written. That ordered slice is the session lane. Lifecycle and state records stay on the lane in every layout. ## Message and offset A message is one record: a payload, headers, and a timestamp. Its offset is its position in the partition. Offsets start at 0 and only grow. A partition id plus an offset names one message exactly, and the SDK calls that pair a message id. Records stay on the log until the topic's retention removes them. Agent topics that `bootstrap` creates never expire, except `agent.sessions`, which uses the retention you pass, and `agent.heartbeats`, which expires after one hour. ## Consumer group A consumer group is a set of readers that share a topic. Each partition goes to one member at a time. The group stores how far it has read, its committed offset, so a restarted reader continues where the group stopped. ## Agent An agent is an id plus a handler. It joins its own consumer group, named after the agent id, and runs the handler for each record addressed to it. Replies, status records, and records for other agents are skipped. The runtime commits a record only after the handler succeeds. If the process crashes, the record is delivered again. Failed handlers are retried, by default up to 5 attempts starting at a 200 ms delay. A record that still fails goes to `agent.dlq`. An agent can advertise capabilities, so a caller can route work to "any agent that can resolve tickets" instead of a fixed name. See [Agents](/laser-sdk/fabric). ## Session A session is one unit of work with a recorded lifecycle. It records the requests, replies, model calls, tool calls, and state changes of that work on its lane, so you can read it back or replay it later. Its id is the conversation id that every record of the session carries. A session has one of these statuses. The SDK and the session index spell them in lowercase, such as `canceled`. | Status | Meaning | | --------- | ----------------------------------------------------------------------------------------------- | | Submitted | Work was handed to an agent that has not picked it up yet | | Active | An agent is working on it | | Paused | An operator paused it, and agents hold new work for it | | Completed | The owning agent ended it successfully | | Failed | It ended with an error or a rejection, or an agent or workflow stopped it for going over budget | | Canceled | It was canceled | Work with its own lifecycle runs as a child session. A workflow run is a root session, and each step is a child. A contract can run as a child of the session that sent it. Child sessions stay in the parent's stream. See [Sessions](/laser-sdk/session). ## Managed plane Plain Apache Iggy runs everything that only needs the log: messages, agents, contracts, workflows, sessions, and memory folded from the log. Other features need the LaserData Apache Iggy fork and `laser-plane`, a service that reads the log and builds views from it. [Laser Stack](/laser-sdk/laser-stack) runs both locally, and LaserData Cloud runs them in production. | Feature | Plain Apache Iggy | Laser Stack or Cloud | | -------------------------------------- | ----------------- | -------------------- | | Messages, agents, contracts, workflows | Yes | Yes | | Session records, lane reads, replay | Yes | Yes | | Session index, lists, and timelines | No | Yes | | Memory folded from the log | Yes | Yes | | Managed memory view | No | Yes | | Server-side filters | No | Yes | | Queries and views, changes | No | Yes | | Key-value state, forks, graph | No | Yes | | Managed permissions | No | Yes | ## Capabilities At connect, the client asks the server what it serves and keeps the answer as capabilities. Read them with `laser.capabilities()`. When the answer has no managed plane, a read at least one second after the last probe asks again, so a backend that comes up later is picked up. A managed call that the server does not serve fails with an unsupported error (Rust `LaserError::Unsupported`, Python and TypeScript `UnsupportedError`). Nothing falls back silently. Check a capability first when your code must run on both plain Apache Iggy and a managed deployment. Source: https://docs.laserdata.com/laser-sdk/concepts --- # AI agents This page is for coding agents, LLMs, and the people who set them up. It lists the machine-readable entry points to these docs and the SDK, and plain criteria for when Laser SDK fits a task. ## Read the docs as text * [`/llms.txt`](https://docs.laserdata.com/llms.txt) lists every page on this site with its URL and a one-line description. Laser SDK pages come first. * [`/llms-full.txt`](https://docs.laserdata.com/llms-full.txt) holds the full text of every page as markdown in one file. Load it when you need the whole SDK in context. * [`/sitemap.xml`](https://docs.laserdata.com/sitemap.xml) lists every page URL. Code samples come in TypeScript, Rust, and Python tabs. In the text files each tab appears in order. The three clients behave the same, so pick the tab for the language you write and ignore the others. Start with [Concepts](/laser-sdk/concepts) for the model, then the guide for the feature you need. Each guide links to an Advanced page that lists every option, default, limit, and error for that feature. ## Use the SDK repository The source is at [github.com/laserdata/laser-sdk](https://github.com/laserdata/laser-sdk). These files help an agent write correct code: * The `examples/rust`, `examples/python`, and `examples/typescript` folders hold runnable programs for each feature. They are the best shape to copy. * The guides `docs/building-agents.md` and `docs/tutorial.md` walk through a full agent application. * The file `docs/client-behavior.md` lists the behavior of the current release and the changes from earlier ones. * The parity matrix `docs/parity.md` maps each public Rust API to its Python and TypeScript name. * The files `foreign/python/laser_sdk.pyi` and `foreign/typescript/api/laser-sdk.api.md` list every Python and TypeScript signature. The repository also has `AGENTS.md` and Claude Code skills under `.claude/skills`. They are written for agents that change the SDK itself. Load `laser-sdk-overview` first, then the skill for the area you touch. ## Use the laser CLI The [`laser` CLI](/cli) manages LaserData Cloud deployments with JSON output and exit codes. An agent can use it to create a deployment and read its Iggy credentials with `laser deployment credentials`. The SDK connects with those credentials and the deployment domain. See [Connect](/laser-sdk/connect). The [Claude Code skills](/cli/claude-skills) turn requests into `laser` commands. The CLI does not replace the SDK. Use the SDK for messages, agents, sessions, and memory. ## Run code against a server [Laser Stack](/laser-sdk/laser-stack) starts Apache Iggy and `laser-plane` in Docker and prints a connection string. Every SDK feature works against it. Agents, sessions, and messages also work on plain Apache Iggy. Call `laser.capabilities()` before you use a managed feature, because a call the server does not serve fails with an unsupported error. ## When to choose Laser SDK Choose Laser SDK when one or more of these is true: * Agents or services need to send each other requests and get replies, or route work by capability. * You need a durable record of what each agent did, with model calls, tool calls, and state, that you can read back later. * You need to replay a session or a stream from an earlier point. * Agents need shared memory or shared key-value state. * The system mixes Rust, Python, and TypeScript and needs the same behavior in each. * Consumers should receive only the records they need, filtered on the server. It is a poor fit when you only need an in-process function call between two prompts, or a single request to a model with no record of it. Source: https://docs.laserdata.com/laser-sdk/ai-agents --- # Agents An agent is an ID plus a handler that reads one topic. Other agents and services send it work through the log, and it replies through the log. Use agents when several services or models must hand work to each other and you need a record of every request and reply. Agents run on plain Apache Iggy, Laser Stack, and LaserData Cloud. ## Quick example The `triage` agent advertises the `triage-ticket` capability. The caller asks for that capability, not for a named agent, and prints the reply. ```ts import { Agent, AgentId, AgentTopic, Laser, TopicRetention, agentMessageBody, routeToCapable } from "@laserdata/laser-sdk" const text = new TextEncoder() const decode = (bytes: Uint8Array) => new TextDecoder().decode(bytes) await using root = await Laser.connectEnv() const laser = root.withDefaultStream("support") await laser.bootstrap(1, TopicRetention.expireAfter(86_400_000)) await using triage = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .capabilities([{ skillId: "triage-ticket" }]) .ackOnPickup() .handler({ handle: (message, ctx) => ctx.respond(text.encode(`triaged: ${decode(agentMessageBody(message))}`)) }) .build() .spawn(laser) await triage.ready() const outcome = await laser .contract(routeToCapable("triage-ticket", { kind: "any" })) .from(AgentId.new("intake")) .payload(text.encode("disk full on node-7")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .send() if (outcome.kind === "completed") { console.log(decode(agentMessageBody(outcome.reply))) } ``` ```rust use laser_sdk::prelude::full::*; use laser_sdk::wire::agent::CapabilityDescriptor; use std::time::Duration; struct Triage; impl AgentHandler for Triage { async fn handle(&self, message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { let ticket = String::from_utf8_lossy(message.body()); ctx.respond(format!("triaged: {ticket}")).await } } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?.with_default_stream("support"); laser .bootstrap(1, TopicRetention::expire_after(Duration::from_secs(86_400))) .await?; let mut triage = Agent::builder() .id("triage".parse()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .capabilities(vec![CapabilityDescriptor { skill_id: "triage-ticket".to_owned(), ..Default::default() }]) .ack_on_pickup(true) .handler(Triage) .build() .spawn(laser.clone()); triage.ready().await?; let outcome = laser .contract(Router::to_capable("triage-ticket", RoutePolicy::Any)) .from("intake".parse()?) .payload("disk full on node-7") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .send() .await?; if let Contract::Completed(reply) = outcome { println!("{}", String::from_utf8_lossy(reply.body())); } triage.shutdown().await } ``` ```python import asyncio import laser_sdk as ls async def triage(ctx, message): await ctx.respond(b"triaged: " + bytes(message.body())) async def main(): async with await ls.Laser.connect_env() as root: laser = root.with_default_stream("support") await laser.bootstrap(1, ls.TopicRetention.expire_after(86_400_000)) async with laser.spawn_agent( "triage", ls.AgentTopic.Sessions, triage, respond_on=ls.AgentTopic.Sessions, capabilities=["triage-ticket"], ack_on_pickup=True, ): outcome = await laser.contract( "triage-ticket", b"disk full on node-7", source="intake", fixed_inbox=ls.AgentTopic.Sessions, ) if isinstance(outcome, ls.Contract.Completed): print(bytes(outcome[0].body()).decode()) asyncio.run(main()) ``` An agent with capabilities publishes a card to the registry when it starts, so callers can find it. `ack_on_pickup` makes the agent report `Working` as soon as it takes a request. The fixed inbox sends the request to `agent.sessions`. Without it, the SDK sends each request to the inbox the agent advertises in its live presence, which needs a server that serves presence. ## Send a request to one agent Name the agent with `Router::to` in Rust, `routeTo` in TypeScript, or `agent=` in Python. Set `deadline` to change how long the caller waits, 30 seconds by default. Set `expire_if_not_consumed` to learn that no agent picked the request up. Handle all four outcomes. ```ts const outcome = await laser .contract(routeTo(AgentId.new("triage"))) .from(AgentId.new("intake")) .payload(text.encode("node-7 unreachable")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .expireIfNotConsumed(5_000) .deadline(60_000) .send() switch (outcome.kind) { case "completed": console.log(`done: ${decode(agentMessageBody(outcome.reply))}`) break case "failed": console.log(`failed: ${decode(agentMessageBody(outcome.reply))}`) break case "notConsumed": console.log("no agent picked it up") break case "timedOut": console.log("picked up, no answer in time") break } ``` ```rust let outcome = laser .contract(Router::to("triage".parse()?)) .from("intake".parse()?) .payload("node-7 unreachable") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .expire_if_not_consumed(Duration::from_secs(5)) .deadline(Duration::from_secs(60)) .send() .await?; match outcome { Contract::Completed(reply) => println!("done: {}", String::from_utf8_lossy(reply.body())), Contract::Failed(reply) => println!("failed: {}", String::from_utf8_lossy(reply.body())), Contract::NotConsumed => println!("no agent picked it up"), Contract::TimedOut => println!("picked up, no answer in time"), } ``` ```python outcome = await laser.contract( None, b"node-7 unreachable", agent="triage", source="intake", fixed_inbox=ls.AgentTopic.Sessions, expire_if_not_consumed_ms=5_000, deadline_ms=60_000, ) match outcome: case ls.Contract.Completed(reply): print(f"done: {bytes(reply.body()).decode()}") case ls.Contract.Failed(reply): print(f"failed: {bytes(reply.body()).decode()}") case ls.Contract.NotConsumed(): print("no agent picked it up") case ls.Contract.TimedOut(): print("picked up, no answer in time") ``` `NotConsumed` means that neither a pickup report nor a reply arrived before the consumption expiry. Trust it only when the agent runs with `ack_on_pickup`, because without pickup reports a slow handler also reads as not consumed. Without an expiry, an unanswered request ends as `TimedOut`. ## Hand off work without waiting `submit` starts a [session](/laser-sdk/session) for an agent and puts the input in front of it. The call returns once both records are written. The agent's handler reaches the session through `ctx.session()` and ends it when the work is done. ```ts const summarizer = { handle: async (message: AgentMessage, ctx: AgentCtx) => { console.log(`summarizing ${decode(agentMessageBody(message))}`) await ctx.session().end() } } const submitted = await laser .sessions() .submit(AgentId.new("summarizer"), text.encode("summarize incident 7")) .from(AgentId.new("intake")) .send() ``` ```rust struct Summarizer; impl AgentHandler for Summarizer { async fn handle(&self, message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { println!("summarizing {}", String::from_utf8_lossy(message.body())); ctx.session().end().await } } let submitted = laser .sessions() .submit("summarizer".parse::()?, "summarize incident 7") .from("intake".parse::()?) .send() .await?; ``` ```python async def summarize(ctx, message): print(f"summarizing {bytes(message.body()).decode()}") await ctx.session().end() submitted = await ( laser.sessions() .submit("summarizer", b"summarize incident 7") .from_("intake") .send() ) ``` `submitted.session` is the session ID. Read the session back with `laser.sessions().open(id).context()`. ## Find agents by capability The registry holds the cards that agents publish. `resolve` returns the cards that advertise a skill, are fresh, do not mark the skill unavailable, and are not quarantined. ```ts import { SystemClock } from "@laserdata/laser-sdk" const registry = await laser.agentRegistry() const now = new SystemClock().nowMicros() await registry.refresh(now) for (const card of registry.resolve("triage-ticket", now)) { console.log(`${card.agent.asStr()} serves triage-ticket`) } ``` ```rust use laser_sdk::agent::{Clock, SystemClock}; let mut registry = laser.agent_registry()?; let now = SystemClock.now_micros(); registry.refresh(now).await?; for card in registry.resolve("triage-ticket", now) { println!("{} serves triage-ticket", card.agent); } ``` ```python registry = laser.agent_registry() await registry.refresh() for card in registry.resolve("triage-ticket"): print(f"{card.agent} serves triage-ticket") ``` A capability route picks one of these agents. The default policy takes any of them. Other policies pick the cheapest, the fastest, the least loaded, or a preferred agent, or run your own ranking. [Advanced agents](/laser-sdk/advanced/agents#route-policies) lists them. ## Stop an agent The handle returned by spawn owns the running agent. Keep it until you stop the agent. * `shutdown()` stops new reads, finishes the message in flight, and reports any consumer error. It waits up to the shutdown grace, 30 seconds by default. * `join()` waits for the agent to stop on its own. * `abort()` stops it at once. TypeScript also stops the agent at the end of an `await using` block. Python stops it at the end of an `async with` block. Dropping the Rust handle, or letting Python collect it, also stops the agent, but you will not see an error from the final drain. ## Good to know * Run `bootstrap(partitions, retention)` once per stream before agents start. It creates `agent.sessions` and the other agent topics. * Delivery is at least once. The agent commits a message only after the handler succeeds, so a crash before that means the agent sees the message again. Make external effects safe to repeat. * A handler that fails with a retryable error gets 5 attempts in total. The first retry waits 200 ms and each later wait doubles. After the last attempt, or after a non-retryable error, the message moves to `agent.dlq` and the agent moves on. * Each agent ID reads through its own consumer group and handles only commands addressed to it or to every agent. Replies and other agents' work are skipped. Run more copies of the same agent to share its load, and never give two different agent IDs the same group. * Live presence belongs to a connection. On a server that serves presence, give each agent that advertises capabilities its own connection. A second advertising agent on the same connection fails to start with a presence conflict error. * Exclusive workflow steps need the managed plane. Session listing and status reads need the managed session index. Everything else on this page works on plain Apache Iggy. ## Related Source: https://docs.laserdata.com/laser-sdk/fabric --- # Sessions A session is one unit of agent work in a stream, with a lifecycle: it starts, runs, and ends as completed, failed, or canceled. Every model call, tool call, state change, and handoff inside it is recorded on the log, so you can read back exactly what an agent did. Use a session for each ticket, incident, request, or task an agent handles. The SDK never calls a model. Your code calls its provider, and the session records the call. ## Quick example Bootstrap the stream once, start a session, record a model call and a tool call, keep some state, and read the session back. ```ts import { AgentId, LastN, Laser, ModelRequest, SessionConfig, TopicRetention, sessionTurnText } from "@laserdata/laser-sdk" const utf8 = (text: string) => new TextEncoder().encode(text) await using laser = await Laser.connectEnv() const sessions = laser.sessions(new SessionConfig().stream("ops")) await sessions.bootstrap(4, TopicRetention.expireAfter(7 * 86_400_000)) const { session, lease } = await sessions.create("incident-42").agent(AgentId.new("triage")).begin() await session.run(lease, async (session) => { const assembled = await session.assemble(new LastN(20)) const call = await session.model(new ModelRequest("gpt-4o", utf8(assembled.text())), assembled) // Call your model provider here, then record its answer. await call.complete({ body: utf8("Check the cache on node-7"), usage: { inputTokens: 812n, outputTokens: 64n } }) const tool = await session.tool("read_metrics", { host: "node-7" }) await tool.complete(utf8("cache hit rate 41%")) await session.state().set("status", "triaged") }) for (const turn of await sessions.open(session.conversation).context()) { console.log(turn.display, sessionTurnText(turn)) } ``` ```rust use laser_sdk::agent::{ModelRequest, ModelResponse}; use laser_sdk::prelude::full::*; use laser_sdk::wire::agent::TokenUsage; use serde_json::json; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?; let sessions = laser.sessions_with(SessionConfig::new().stream("ops")); sessions .bootstrap(4, TopicRetention::expire_after(Duration::from_secs(7 * 86_400))) .await?; let (session, lease) = sessions.create("incident-42").agent("triage".parse::()?).begin().await?; let id = session.conversation(); session .run(lease, |session| async move { let assembled = session.assemble(Box::new(LastN(20))).await?; let call = session .model(ModelRequest::new("gpt-4o", assembled.text()), Some(&assembled)) .await?; // Call your model provider here, then record its answer. call.complete(ModelResponse { body: b"Check the cache on node-7".to_vec(), usage: Some(TokenUsage { input_tokens: 812, output_tokens: 64, ..Default::default() }), ..Default::default() }) .await?; let tool = session.tool("read_metrics", json!({ "host": "node-7" })).await?; tool.complete(b"cache hit rate 41%".to_vec()).await?; session.state().set("status", json!("triaged")).await?; Ok(()) }) .await?; for turn in sessions.open(id).context().await? { println!("{} {}", turn.display, turn.text()); } Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main(): async with await ls.Laser.connect_env() as laser: sessions = laser.sessions(stream="ops") await sessions.bootstrap(4, ls.TopicRetention.expire_after(7 * 86_400_000)) async with sessions.create("incident-42").agent("triage") as session: assembled = await session.assemble(ls.LastN(20)) call = await session.model(ls.ModelRequest("gpt-4o", assembled.text()), assembled) # Call your model provider here, then record its answer. await call.complete( ls.ModelResponse("Check the cache on node-7", usage={"input_tokens": 812, "output_tokens": 64}) ) tool = await session.tool("read_metrics", {"host": "node-7"}) await tool.complete("cache hit rate 41%") await session.state().set("status", "triaged") for turn in await sessions.open(session.conversation).context(): print(turn.display, turn.text()) asyncio.run(main()) ``` The read prints `session.started`, `model.request`, `context.assembled`, `model.response`, `tool.call`, `tool.result`, `state.updated` twice (the change and the snapshot written at the end), and `session.completed`. In Rust, sessions need the `agent` feature. ## Bootstrap once `bootstrap(partitions, retention)` creates the session topic `agent.sessions` and the other agent topics in the stream. Run it once per stream before the first session. The retention is required, because `agent.sessions` holds every session's records. On Laser Stack and LaserData Cloud, bootstrap also registers the stream so the managed session reads below can see it. ## Start a session `create(label)` derives the session ID from the stream, the namespace, and the label, so the same label always reaches the same session. `start()` gives a fresh ID. Both need `.agent(id)` for the agent that owns the session, then `begin()` writes the start record and returns the session with a lease. The lease keeps the session listed in the client's heartbeat while your process works on it. ## End a session or mark it failed The guard in the quick example ends the session as completed when the work returns and as failed when it throws: `run(lease, work)` in Rust and TypeScript, `async with` or `run(lease, work)` in Python. To control the outcome yourself, call `end()`, `fail(error)`, or `cancel()`. The first terminal call wins, and a different one afterwards fails. ```ts const { session, lease } = await sessions.start().agent(AgentId.new("triage")).begin() try { await triage(session) await session.end() } catch (error) { await session.fail({ code: { kind: "known", name: "Internal" }, message: String(error), retryable: false }) throw error } finally { lease.release() } ``` ```rust let (session, lease) = sessions.start().agent("triage".parse::()?).begin().await?; match triage(&session).await { Ok(()) => session.end().await?, Err(error) => { let body = AgentErrorBody { code: AgentErrorCode::Internal, message: Some(error.to_string()), retryable: false, detail: None, }; session.fail(body).await?; } } drop(lease); ``` ```python session, lease = await sessions.start().agent("triage").begin() try: await triage(session) await session.end() except Exception as error: await session.fail({"code": 7, "message": str(error), "retryable": False}) raise finally: lease.release() ``` In Rust, `AgentErrorBody` and `AgentErrorCode` are part of `laser_sdk::prelude::full`. Python passes the same body as a dict whose `code` is the wire number, 7 for `Internal`. ## Delegate work to a child session Give delegated work its own lifecycle as a child session. Pass the parent and the root, which is the parent itself for a top-level session. The child lives in the same stream, and its history stays in the child. ```ts const { session: child, lease } = await sessions .start() .agent(AgentId.new("specialist")) .parent(session.conversation, session.conversation) .begin() await child.run(lease, async (child) => { const tool = await child.tool("inspect_service", { service: "api" }) await tool.complete(utf8("cache saturation")) }) ``` ```rust let (child, lease) = sessions .start() .agent("specialist".parse::()?) .parent(id, id) .begin() .await?; child .run(lease, |child| async move { let tool = child.tool("inspect_service", json!({ "service": "api" })).await?; tool.complete(b"cache saturation".to_vec()).await?; Ok(()) }) .await?; ``` ```python child_builder = sessions.start().agent("specialist").parent(session.conversation, session.conversation) async with child_builder as child: tool = await child.tool("inspect_service", {"service": "api"}) await tool.complete("cache saturation") ``` Workflow steps become child sessions on their own. So do contracts, A2A tasks, and MCP tool calls that you start with a parent. ## Read a session back `open(id).context()` returns the session's recent records with their display types, as in the quick example. It reads the log directly, so it works everywhere. `checkpoint()` with `turns_since(checkpoint)` or `replay(..)` reads what happened after a saved point. On Laser Stack or LaserData Cloud, the managed session index adds a searchable list, summaries with counts and tokens, and a timeline per session: ```ts const page = await sessions.list().status("active").limit(20).fetch() for (const info of page.items) { console.log(info.label, info.status, info.idle, info.toolCalls, info.tokensIn) } const timeline = await sessions.events(session.conversation).limit(100).fetch() ``` ```rust let page = sessions.list().status(SessionStatus::Active).limit(20).fetch().await?; for info in &page.items { println!("{:?} {:?} {} {} {}", info.label, info.status, info.idle, info.tool_calls, info.tokens_in); } let timeline = sessions.events(id).limit(100).fetch().await?; ``` ```python page = await sessions.list(status="active", limit=20) for info in page["items"]: print(info.get("label"), info["status"], info["idle"], info["tool_calls"], info["tokens_in"]) timeline = await sessions.events(session.conversation, limit=100) ``` In Rust, `SessionStatus` is part of `laser_sdk::prelude::full`. ## Good to know * The stream is the isolation boundary. A session and its child sessions never leave their stream. Give each application and environment its own stream. * By default every session's records ride one `agent.sessions` topic keyed by session, so one session's lane is one partition with exact order. * A session whose process stops without ending it shows as idle, never as failed. The defaults are a 60-second heartbeat and a 5-minute idle timeout. * A dead-lettered record never fails its session by itself. Failing on a dead letter is an opt-in setting. * Labels always derive the same ID within a namespace. Use `start()` or a per-parent namespace for child work that repeats a label. * Recording, state, context, replay, budget checks with `over_budget()`, operator control, and pause work on plain Apache Iggy. Listing sessions, summaries, timelines, links, the change feed, and runtime budget enforcement need the managed session index on [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. Source: https://docs.laserdata.com/laser-sdk/session --- # Messages A topic is a durable, ordered log of messages. Producers append to it, and any number of readers can read it live, resume from a saved position, or replay it from the start. Use it for agent events, tool-call records, telemetry, and any feed that more than one service needs. ## Quick example An agent records each tool call it makes. An auditor reads every record back as a typed value. ```ts import { Json, Laser } from "@laserdata/laser-sdk" interface ToolCall { readonly agent: string readonly tool: string readonly ms: number } // Types do not exist at runtime, so the codec checks each decoded value. const TOOL_CALL = new Json((value) => { const { agent, tool, ms } = value as Record if (typeof agent !== "string" || typeof tool !== "string" || typeof ms !== "number") { throw new TypeError("not a tool call") } return { agent, tool, ms } }) await using laser = await Laser.connectEnv() const topic = laser.stream("agents").topic("tool-calls") await topic.ensure(2) const calls = topic.json(TOOL_CALL) await calls.publish({ agent: "researcher", tool: "search", ms: 120 }) await calls.publish({ agent: "researcher", tool: "summarize", ms: 840 }) const reader = await calls.records("audit") for (let result = await reader.next(); result !== undefined; result = await reader.next()) { if (result.kind === "error") throw result.error const call = result.record.value console.log(call.agent, call.tool, call.ms) } ``` ```rust use laser_sdk::prelude::*; use serde::{Deserialize, Serialize}; #[derive(Debug, Serialize, Deserialize)] struct ToolCall { agent: String, tool: String, ms: u32, } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?; let topic = laser.stream("agents").topic("tool-calls"); topic.ensure(2).await?; let calls = topic.json::(); for (tool, ms) in [("search", 120), ("summarize", 840)] { let call = ToolCall { agent: "researcher".into(), tool: tool.into(), ms }; calls.publish(&call)?.send().await?; } let mut reader = calls.records("audit")?; while let Some(next) = reader.next().await { let call = next?.value; println!("{} {} {}", call.agent, call.tool, call.ms); } Ok(()) } ``` ```python import asyncio from dataclasses import dataclass import laser_sdk as ls @dataclass class ToolCall: agent: str tool: str ms: int async def main() -> None: laser = await ls.Laser.connect_env() topic = laser.stream("agents").topic("tool-calls") await topic.ensure(2) calls = topic.json(ToolCall) for tool, ms in (("search", 120), ("summarize", 840)): await calls.publish(ToolCall("researcher", tool, ms)).send() reader = calls.records("audit") while (record := await reader.next()) is not None: print(record.value.agent, record.value.tool, record.value.ms) await laser.close() asyncio.run(main()) ``` `topic.json(..)` returns a typed handle. Its `publish` encodes the value and its `records` reader decodes it. The reader starts at offset 0. When it reaches the end of the topic, `next()` returns `None` in Rust and Python and `undefined` in TypeScript. ## Process messages live in a consumer group A consumer group shares a topic's partitions across processes. Each partition goes to one member at a time. The server stores the group's progress, so a restarted member resumes where the group left off. Commit a message after you finish handling it. ```ts await using consumer = await topic.consumerGroup("auditors").consumer({ commitPolicy: { kind: "disabled" }, startAt: { kind: "first" } }) for await (const message of consumer) { const call = TOOL_CALL.decode(message.payload) console.log(message.position.offset, call.tool) await consumer.commit(message) } ``` ```rust let mut consumer = topic .consumer_group("auditors") .consumer() .commit_policy(CommitPolicy::Disabled) .start_at(ConsumerStart::First) .build() .await?; while let Some(message) = consumer.next().await { let message = message?; let call: ToolCall = message.json()?; println!("{} {}", message.position.offset, call.tool); consumer.commit(&message).await?; } consumer.shutdown().await?; ``` ```python consumer = topic.consumer_group("auditors").consumer( auto_commit="disabled", polling="first", ) async for message in consumer: call = message.json() print(message.position.offset, call["tool"]) await consumer.commit(message) await consumer.shutdown() ``` The loop waits for new messages until you stop it. To commit automatically, pick another commit policy. See [Consumers and commit policies](/laser-sdk/advanced/log#consumers-offsets-and-commit-policies). ## Keep related messages in order Order holds only within one partition. Give messages that must stay in order the same key, such as an agent or session ID. The key picks the partition. ```ts const key = new TextEncoder().encode("researcher") await calls.publish({ agent: "researcher", tool: "search", ms: 120 }, { key }) ``` ```rust let call = ToolCall { agent: "researcher".into(), tool: "search".into(), ms: 120 }; calls.publish(&call)?.partition_key("researcher").send().await?; ``` ```python call = ToolCall("researcher", "search", 120) await calls.publish(call).partition_key("researcher").send() ``` ## Resume a reader from saved offsets A typed reader tracks the next offset of each partition. Save the offsets, then hand them back to resume after a restart. Without saved offsets, a reader starts at offset 0. ```ts const saved = reader.offsets // Map of partition to bigint offset const resumed = (await calls.records("audit")).fromOffsets(saved) ``` ```rust let saved = reader.offsets().to_vec(); // one offset per partition let resumed = calls.records("audit")?.from_offsets(saved); ``` ```python saved = reader.offsets # one offset per partition resumed = calls.records("audit", from_offsets=saved) ``` A typed reader is a cursor that you own. It is not a consumer group, and the server does not store its offsets. Use a consumer group for production processing. ## Publish at volume `publish` sends one message and waits for the server. For throughput, use a long-lived producer, a background producer, or a batching producer. See [Producing at volume](/laser-sdk/advanced/log#producing-at-volume). ## Good to know * Delivery is at least once. A crash between handling and commit delivers the message again, so handlers must be safe to repeat. * There is no order across partitions. More partitions give more parallel consumers. One key keeps its messages in order. * When a group member joins or leaves, the server moves partitions between members. Each partition has one member at a time. * A message that does not decode fails with a decode error that names its position, and the next read moves past it. In Python, an `async for` loop over a typed reader ends on that error, so use `next()` in a loop to skip it. * Payloads are bytes. JSON, CBOR, MessagePack, BSON, and Avro helpers encode values into those bytes. * Messages work on plain Apache Iggy, Laser Stack, and LaserData Cloud. Source: https://docs.laserdata.com/laser-sdk/log --- # Memory Memory stores facts your agents learn, such as a user's preferences or the fix that closed an incident. Any agent that opens the same namespace can recall them. Every change is a record on the log, so you can see who wrote a fact, when, and from which message. ## Quick example This program remembers a fact about a user, recalls it, and forgets it. It reads the memory topic in process with `folded`, so it runs against plain Apache Iggy. ```ts import { ConversationId, Laser, TopicRetention, memoryItemText } from "@laserdata/laser-sdk" await using laser = await Laser.connectWithStream("iggy:iggy@127.0.0.1:8090", "support") await laser.bootstrap(1, TopicRetention.expireAfter(86_400_000)) const memory = laser.memory("support") const conversation = ConversationId.new() const fact = await memory .remember(new TextEncoder().encode("Prefers email, works in UTC+2")) .scope(conversation) .user("user-42") .send() const hits = await memory.recall(conversation).user("user-42").recent().limit(5).folded().fetch() for (const hit of hits) console.log(memoryItemText(hit)) await memory.forget({ conversation }, fact) ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_with_stream("iggy:iggy@127.0.0.1:8090", "support").await?; laser .bootstrap(1, TopicRetention::expire_after(Duration::from_secs(86_400))) .await?; let memory = laser.memory("support"); let conversation = ConversationId::new(); let fact = memory .remember("Prefers email, works in UTC+2".as_bytes()) .scope(conversation) .user("user-42") .send() .await?; let hits = memory .recall(conversation) .user("user-42") .recent() .limit(5) .folded() .fetch() .await?; for hit in &hits { println!("{}", hit.text()); } let scope = MemoryScope::builder().conversation(conversation).build(); memory.forget(&scope, fact).await?; Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main(): async with await ls.Laser.connect("iggy:iggy@127.0.0.1:8090", stream="support") as laser: await laser.bootstrap(1, retention=ls.TopicRetention.expire_after(86_400_000)) memory = laser.memory("support") conversation = ls.new_conversation_id() fact = await memory.remember( "Prefers email, works in UTC+2", conversation=conversation, user="user-42", ) hits = await memory.recall( limit=5, conversation=conversation, user="user-42", strategy="recent", folded=True, ) for hit in hits: print(hit.text()) await memory.forget(fact, conversation=conversation) asyncio.run(main()) ``` `bootstrap` creates the `agent.memory` topic once per stream. Drop `folded` to read the managed view on [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud, which scales to large histories. ## Share facts across agents Agents share memory by opening the same namespace on the same stream, in any of the three languages. Each item records the conversation, agent, user, and application it belongs to. Set them when you remember and narrow by them when you recall. A field you leave out widens the read, so a recall without a user returns every user's items. [Agents](/laser-sdk/fabric) shows `MemoryHandler`, which remembers every message an agent handles. ```ts import { AgentId } from "@laserdata/laser-sdk" const triage = AgentId.new("triage") await memory.remember(new TextEncoder().encode("Disk alerts on node-7 are noise")).scope(conversation).agent(triage).send() const notes = await memory.recall(conversation).agent(triage).fetch() ``` ```rust let triage: AgentId = "triage".parse()?; memory .remember("Disk alerts on node-7 are noise".as_bytes()) .scope(conversation) .agent(triage.clone()) .send() .await?; let notes = memory.recall(conversation).agent(triage).fetch().await?; ``` ```python await memory.remember( "Disk alerts on node-7 are noise", conversation=conversation, agent="triage", ) notes = await memory.recall(conversation=conversation, agent="triage") ``` ## Recall by meaning The vector backend ranks items by how close they are to a question. You supply the embedding function, usually a call to an embedding model. Vector items live in process memory, so use this for working memory inside one agent. ```ts import { MemoryBackend } from "@laserdata/laser-sdk" const runbooks = laser.memoryWith("runbooks", MemoryBackend.Vector, embedder) await runbooks.remember(new TextEncoder().encode("Gateway latency spikes mean the db pool is exhausted")).scope(conversation).send() const hits = await runbooks.recall(conversation).semantic("gateway is slow").limit(3).fetch() ``` ```rust let runbooks = laser.memory_with("runbooks", MemoryBackend::Vector).embedder(embedder); runbooks .remember("Gateway latency spikes mean the db pool is exhausted".as_bytes()) .scope(conversation) .send() .await?; let hits = runbooks.recall(conversation).semantic("gateway is slow").limit(3).fetch().await?; ``` ```python runbooks = laser.memory_with("runbooks", "vector", embedder=embed) await runbooks.remember( "Gateway latency spikes mean the db pool is exhausted", conversation=conversation, ) hits = await runbooks.recall(conversation=conversation, semantic="gateway is slow", limit=3) ``` ## Trace a fact back to the log Each item recalled from log memory carries `source`, the stream, topic, partition, and offset of the record it came from. Inside a [session](/laser-sdk/session), `linked_memory()` (TypeScript: `linkedMemory`) also stamps each new item with the record the session was handling and the agent that wrote it. ## Reweight or forget `improve` adds a feedback weight that moves an item up or down in later recalls. `forget` writes a tombstone, so the item stops appearing while the topic keeps the history. For a value you look up by name, such as the current plan, use `set(key, body)` and `fetch(key)` instead of a recall. ## Good to know * Default recall and `fetch` read the managed view, so they need Laser Stack or LaserData Cloud. `folded` recall and `fetch_folded` work with Iggy alone. Keep one handle for folded reads, because it reads only new records on each call. * The managed view catches up in the background, so a fresh write can take a moment to show in default recall. * Only the vector backend ranks by meaning. Log memory uses query text as a filter and returns the newest matches. * Namespaces are scoped to the default stream, so `laser.memory("support")` addresses `stream:/support`. * In Rust, memory needs the `agent` feature, and default recall also needs `kv`, which `managed` includes. Source: https://docs.laserdata.com/laser-sdk/memory --- # Context Context reads one conversation's messages back from the log and trims them to fit a prompt. The conversation ID ties messages, memory, and state together, and a [session](/laser-sdk/session) ID is a conversation ID, so the same reads work on a session. Use Context when you build a prompt from earlier turns or rebuild state from what a conversation recorded. ## Quick example Append two turns to a conversation, then read them back as the last 20 messages trimmed to about 4,000 tokens. ```ts import { AgentTopic, Chain, ConversationId, LastN, Laser, TokenBudget, TopicRetention } from "@laserdata/laser-sdk" const utf8 = (text: string) => new TextEncoder().encode(text) await using connection = await Laser.connectEnv() const laser = connection.withDefaultStream("ops") await laser.bootstrap(4, TopicRetention.expireAfter(7 * 86_400_000)) const ctx = laser.context(ConversationId.new()) await ctx.append(AgentTopic.Sessions, utf8("drain node-7")) await ctx.append(AgentTopic.Sessions, utf8("drained, 0 connections left")) const turns = await ctx.fetchWith([AgentTopic.Sessions], new Chain([new LastN(20), new TokenBudget(4_000)])) for (const turn of turns) { console.log(new TextDecoder().decode(turn.payload)) } ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?.with_default_stream("ops"); laser .bootstrap(4, TopicRetention::expire_after(Duration::from_secs(7 * 86_400))) .await?; let scope = laser.context(ConversationId::new()); scope.append(AgentTopic::Sessions, "drain node-7".as_bytes()).await?; scope.append(AgentTopic::Sessions, "drained, 0 connections left".as_bytes()).await?; let policy = Chain(vec![Box::new(LastN(20)), Box::new(TokenBudget::new(4_000))]); let turns = scope.fetch_with(vec![AgentTopic::Sessions], Box::new(policy)).await?; for turn in &turns { println!("{}", String::from_utf8_lossy(&turn.payload)); } Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main(): async with await ls.Laser.connect_env() as connection: laser = connection.with_default_stream("ops") await laser.bootstrap(4, ls.TopicRetention.expire_after(7 * 86_400_000)) ctx = laser.context(ls.new_conversation_id()) await ctx.append(ls.AgentTopic.Sessions, b"drain node-7") await ctx.append(ls.AgentTopic.Sessions, b"drained, 0 connections left") turns = await ctx.fetch_with([ls.AgentTopic.Sessions], ls.Chain([ls.LastN(20), ls.TokenBudget(4_000)])) for turn in turns: print(bytes(turn.payload).decode()) asyncio.run(main()) ``` `laser.context(id)` does no I/O. `append` keys each message by the conversation ID, so one conversation's messages on a topic share a partition. In Rust, context needs the `agent` feature. ## Trim history with a policy A policy decides which messages reach the prompt. Combine them with `Chain`, which applies each policy to the output of the one before. | Policy | Keeps | | -------------------- | -------------------------------------------------------------------------- | | `LastN(n)` | The newest `n` messages | | `TokenBudget(n)` | The newest messages that fit `n` estimated tokens, and always at least one | | `RoleFilter(agents)` | Messages written by the given agents | | `Chain([...])` | Each policy applied in turn | The token estimate is the payload size in bytes divided by four, rounded up. Pass your own estimator to `TokenBudget` to use a real tokenizer. You can also write your own policy, see [Context in depth](/laser-sdk/advanced/context#write-your-own-policy). ## Get one prompt string `block(topics, n, token_budget)` returns the last `n` messages joined by newlines, ready to paste into a prompt. The token budget is optional and trims after `n`. Rust takes it as an `Option`, TypeScript as an optional third argument, and Python as keywords: `block(topics=, n=, token_budget=)`, where `topics` defaults to `agent.sessions` and `n` to 50. ## Assemble context for a session Inside a session, `session.assemble(policy)` applies a policy to the session's records and returns them with a manifest of what the model received. Pass the result to `session.model(request, assembled)` and the manifest is recorded with the model call. `session.context()` returns the last 50 records trimmed to about 4,000 tokens, each with its timeline type. See [Sessions](/laser-sdk/session). ## Keep memory for the conversation `memory(namespace)` on a scope returns memory with the conversation already applied, so `remember` and `recall` need no conversation argument. ```ts const notes = ctx.memory("notes") await notes.remember(utf8("drain node-7 before the upgrade")).send() const hits = await notes.recall().limit(5).folded().fetch() ``` ```rust let notes = scope.memory("notes"); notes.remember("drain node-7 before the upgrade".as_bytes()).send().await?; let hits = notes.recall().limit(5).folded().fetch().await?; ``` ```python notes = ctx.memory("notes") await notes.remember("drain node-7 before the upgrade") hits = await notes.recall(limit=5, folded=True) ``` Folded recall rebuilds memory from the log, so it works on plain Apache Iggy. Default recall reads the managed key-value view on Laser Stack or LaserData Cloud. See [Memory](/laser-sdk/memory). ## Rebuild state from the conversation `state(topics, bound, init, fold)` folds the conversation's messages into your own state. The bound says how much to read: the last N messages, everything after saved offsets or a checkpoint, everything up to a checkpoint, or the whole partition. `checkpoint(topics)` records where the topics end now, so a later fold reads only what came after it. Snapshot stores save folded state so a restart replays only the tail. See [Context in depth](/laser-sdk/advanced/context#fold-state-from-the-log). ## Good to know * Session reads use the `agent.sessions` topic, and so do Python's `fetch`, `block`, and `checkpoint` when you pass no topics. Rust and TypeScript always take the topics. Run `bootstrap(partitions, retention)` once per stream to create it. * A context read examines at most the newest 10,000 records of each partition. Many conversations share a partition, so older messages of a quiet conversation on a busy partition can fall outside the read. * Messages from several topics are ordered by the broker's append time, because each topic has its own offsets. * `graph(name)` on a scope returns the shared graph. It is not narrowed to the conversation. * Context needs no managed backend. Only default memory recall and the key-value snapshot store need Laser Stack or LaserData Cloud. Source: https://docs.laserdata.com/laser-sdk/context --- # Key-value state Key-value state holds small values your agents read and change by key, such as a service config, a feature flag, or the owner of an incident. Every write goes through the log first, so each value points back to the record that wrote it. Use it when several agents or services need the same current value. ## Quick example This program writes a config value with a one-day expiry, reads it with its version, and updates it with compare-and-swap. It needs [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. ```ts import { Laser } from "@laserdata/laser-sdk" await using laser = await Laser.connectWithStream("iggy:iggy@127.0.0.1:8090", "ops") const kv = laser.kv("config") const key = new TextEncoder().encode("service:auth") await kv.set(key).json({ log_level: "info" }).ttl(86_400_000).send() const entry = await kv.getEntry(key) if (entry === undefined) throw new Error("service:auth is missing") const version = await kv.set(key).json({ log_level: "debug" }).expectVersion(entry.version).commit() console.log(`service:auth is at version ${String(version)}`) ``` ```rust use laser_sdk::prelude::*; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_with_stream("iggy:iggy@127.0.0.1:8090", "ops").await?; let kv = laser.kv("config"); kv.set("service:auth") .json(&serde_json::json!({ "log_level": "info" }))? .ttl(Duration::from_secs(86_400)) .send() .await?; let entry = kv .get_entry("service:auth") .await? .ok_or_else(|| LaserError::Invalid("service:auth is missing".to_owned()))?; let version = kv .set("service:auth") .json(&serde_json::json!({ "log_level": "debug" }))? .expect_version(entry.version) .commit() .await?; println!("service:auth is at version {version}"); Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main(): async with await ls.Laser.connect("iggy:iggy@127.0.0.1:8090", stream="ops") as laser: store = laser.kv("config") await store.set("service:auth").json({"log_level": "info"}).ttl(86_400_000).send() entry = await store.get_entry("service:auth") if entry is None: raise RuntimeError("service:auth is missing") version = await ( store.set("service:auth") .json({"log_level": "debug"}) .expect_version(entry.version) .commit() ) print(f"service:auth is at version {version}") asyncio.run(main()) ``` If another writer changed the key after your read, `commit()` fails with a version conflict that carries the current version. Read again and retry. ## Read a value `get(key)` returns the raw bytes, `get_typed(key)` decodes JSON, and `get_entry(key)` adds the version and expiry. A missing or expired key reads as empty. TypeScript's `getTyped(key, decode)` takes a function that checks the decoded value. ```ts const config = await kv.getTyped(key, (value) => value as { log_level: string }) ``` ```rust let config = kv.get_typed::("service:auth").await?; ``` ```python config = await store.get_typed("service:auth") ``` ## Create a key only once `expect_absent()` (TypeScript: `expectAbsent`) writes only when the key does not exist yet. Two agents that race to claim an incident both call it, and the second one gets a version conflict. ```ts const owner = new TextEncoder().encode("incident:4711:owner") await kv.set(owner).json({ agent: "triage" }).expectAbsent().commit() ``` ```rust kv.set("incident:4711:owner") .json(&serde_json::json!({ "agent": "triage" }))? .expect_absent() .commit() .await?; ``` ```python await store.set("incident:4711:owner").json({"agent": "triage"}).expect_absent().commit() ``` ## Expire values `.ttl(..)` on a write sets how long the value lives, counted from now. Rust takes a `Duration`, and TypeScript and Python take milliseconds. `expire(key, ttl)` changes the expiry of an existing value without rewriting it. Leave out the TTL (Rust: pass `None`) to clear the expiry. ## Share state between agents Agents share state by opening the same namespace on the same stream, in any of the three languages. Use compare-and-swap when several agents update one key. When one worker must own a task for a while, use a lease and fenced writes, covered in [Locks and fenced writes](/laser-sdk/advanced/state#locks-and-fenced-writes). Inside a [session](/laser-sdk/session), `session.kv(namespace)` links every write to that session. To try a change on a copy of your view data before you apply it, use a [fork](/laser-sdk/advanced/state#forks). ## Good to know * Key-value state needs Laser Stack or LaserData Cloud. Check `capabilities.kv.available`, and `capabilities.kv.cas` before you call `commit()`. Without the capability, the call fails with an unsupported error. * Finish a guarded write with `.commit()` and a plain write with `.send()`. Mixing them fails with an invalid error, so a precondition is never dropped. * Keys are at most 512 bytes, values at most 8 MiB, and namespaces at most 128 bytes. TypeScript keys are `Uint8Array` values. * Namespaces are scoped to the default stream, so `laser.kv("config")` addresses `stream:/config`. * Each entry from `get_entry` carries `source`, the log record that wrote it. Source: https://docs.laserdata.com/laser-sdk/state --- # Queries and views A view is a table that stays up to date with a topic. You say which fields to extract, and the managed plane writes a row for every new message. Use views when agents or services need to look up, filter, or count what happened without reading the whole log. Views need [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. ## Quick example Register a projection with the fields to extract, bind the `readings` topic to it, publish a reading, and query the view. ```ts import { ContentType, Laser, ProjectionBindingBuilder, ProjectionBuilder, queryResultValueText } from "@laserdata/laser-sdk" const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) await using laser = await Laser.connectEnv() const readings = laser.stream("telemetry").topic("readings") await readings.ensure(1) await laser.projections().register( new ProjectionBuilder("readings_v1.v1") .name("readings_v1") .contentType(ContentType.Json) .fields(["host", "cpu", "status"]) .build() ) await laser.bindings().apply( new ProjectionBindingBuilder() .source("telemetry", "readings") .allow("readings_v1.v1") .defaultProjection("readings_v1.v1") .index("readings_v1") .build() ) // The binding is applied asynchronously. Queries fail until the index exists. const indexReady = () => laser.query("readings_v1").fetch().then(() => true, () => false) while (!(await indexReady())) await sleep(200) await readings.publish().json({ host: "node-2", cpu: 91, status: "degraded" }).send() // The row lands shortly after the publish. const findDegraded = () => laser.query("readings_v1").whereEq("status", "degraded").fetch() let degraded = await findDegraded() while (degraded.rows.length === 0) degraded = await sleep(200).then(findDegraded) for (const row of degraded.rows) console.log(queryResultValueText(degraded, row, "host")) ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?; let readings = laser.stream("telemetry").topic("readings"); readings.ensure(1).await?; let projection = Projection::builder("readings_v1.v1") .name("readings_v1") .content_type(ContentType::Json) .fields(["host", "cpu", "status"]) .build(); laser.projections().register(projection).await?; let binding = ProjectionBinding::builder() .source("telemetry", "readings") .allow("readings_v1.v1") .default_projection("readings_v1.v1") .index("readings_v1") .build(); laser.bindings().apply(binding).await?; // The binding is applied asynchronously. Queries fail until the index exists. while laser.query("readings_v1").fetch().await.is_err() { tokio::time::sleep(Duration::from_millis(200)).await; } let reading = serde_json::json!({"host": "node-2", "cpu": 91, "status": "degraded"}); readings.publish().json(&reading)?.send().await?; // The row lands shortly after the publish. loop { let degraded = laser .query("readings_v1") .where_eq("status", "degraded") .fetch() .await?; if !degraded.rows.is_empty() { for row in °raded.rows { println!("{:?}", degraded.value_text(row, "host")); } return Ok(()); } tokio::time::sleep(Duration::from_millis(200)).await; } } ``` ```python import asyncio import laser_sdk as ls async def main() -> None: async with await ls.Laser.connect_env() as laser: readings = laser.stream("telemetry").topic("readings") await readings.ensure(1) await laser.projections().register( ls.ProjectionBuilder("readings_v1.v1") .name("readings_v1") .content_type("json") .fields(["host", "cpu", "status"]) .build() ) await laser.bindings().apply( ls.ProjectionBindingBuilder() .source("telemetry", "readings") .allow("readings_v1.v1") .default_projection("readings_v1.v1") .index("readings_v1") .build() ) # The binding is applied asynchronously. Queries fail until the index exists. while True: try: await laser.query("readings_v1").fetch() break except ls.LaserError: await asyncio.sleep(0.2) await readings.publish({"host": "node-2", "cpu": 91, "status": "degraded"}).send() # The row lands shortly after the publish. degraded = await laser.query("readings_v1").where_eq("status", "degraded").fetch() while not degraded.rows: await asyncio.sleep(0.2) degraded = await laser.query("readings_v1").where_eq("status", "degraded").fetch() for row in degraded.rows: print(degraded.value_text(row, "host")) asyncio.run(main()) ``` The index name `readings_v1` is separate from the topic name, so you can version the view without renaming the topic. ## Filter and sort `where_eq` matches an indexed field exactly. The `filter_*` family compares any indexed field, and chained filters combine with AND. TypeScript comparison filters take a typed value object. ```ts const hot = await laser .query("readings_v1") .filterGte("cpu", { kind: "long", value: 80n }) .orderDesc("cpu") .limit(20) .fetch() console.log(`${hot.rows.length} hot hosts, more pages: ${hot.page.hasMore}`) ``` ```rust let hot = laser .query("readings_v1") .filter_gte("cpu", 80_i64) .order_desc("cpu") .limit(20) .fetch() .await?; println!("{} hot hosts, more pages: {}", hot.rows.len(), hot.page.has_more); ``` ```python hot = await laser.query("readings_v1").filter_gte("cpu", 80).order_desc("cpu").limit(20).fetch() print(f"{len(hot.rows)} hot hosts, more pages: {hot.page.has_more}") ``` ## Page through rows A page holds 50 rows by default and at most 1,000. To read more, chain `max_rows(n)` (TypeScript: `maxRows(n)`) and walk the rows with `rows()`, which fetches pages as you go and stops at the ceiling. `rows()` without a ceiling fails with an invalid error. [Walk many rows](/laser-sdk/advanced/views#walk-many-rows) has the code in all three languages. ## Count and group Call `count()`, `sum(field)`, or `avg(field)` and add `group_by([...])` to get one row per group instead of a list of rows. `window(field, every_micros)` buckets counts by time. [Queries and views in depth](/laser-sdk/advanced/views#aggregate) has the full list. ## Know when new data landed Bind with `notify()` and read the [change feed](/laser-sdk/change-feed) to query only when the view moved. When one query must see your own earlier writes, chain `read_your_writes()` (TypeScript: `readYourWrites()`) on a deployment that supports it. ## Good to know * Registration and binding apply asynchronously. Register the projection before the binding, and wait for a query to succeed before you publish records you expect to see. * Rows appear shortly after a publish, not at once. Queries are eventually consistent by default. * Only declared fields are queryable. The projection builder also keeps a copy of each payload with its row, which typed reads such as `fetch_typed` need. `index_only()` turns that off. * A client with a default stream scopes index and projection names to that stream. Use the same client setup to bind and to query. * Row retention is set on the binding and is independent of topic retention. By default rows go when the log drops their messages. * On standalone Apache Iggy, queries return an unsupported error. Source: https://docs.laserdata.com/laser-sdk/views --- # Changes The change feed tells you when a [view](/laser-sdk/views) advanced. Instead of re-running a query on a timer, you poll a small feed and query only when the view moved. Use it for live dashboards, cache invalidation, and agents that react to new data. It needs [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. ## Quick example This assumes the `readings_v1` binding from [Queries and views](/laser-sdk/views) with `notify()` added to the binding builder. The program opens a reader, skips old records, publishes a reading, waits for the view to move, then queries it. ```ts import { Laser } from "@laserdata/laser-sdk" const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) await using laser = await Laser.connectEnv() const feed = await laser.watch().index("readings_v1").records() while ((await feed.poll()).length > 0) { // Skip batches that earlier runs left behind. } await laser .stream("telemetry") .topic("readings") .publish() .json({ host: "node-7", cpu: 82, status: "degraded" }) .send() let changes = await feed.poll() while (changes.length === 0) changes = await sleep(200).then(() => feed.poll()) for (const change of changes) { console.log(`${change.rows} row(s), offsets ${change.fromOffset}..${change.toOffset}`) } const latest = await laser.query("readings_v1").whereEq("status", "degraded").fetch() console.log(`${latest.rows.length} degraded readings`) ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?; let mut feed = laser.watch().index("readings_v1").records()?; while !feed.poll().await?.is_empty() { // Skip batches that earlier runs left behind. } let reading = serde_json::json!({"host": "node-7", "cpu": 82, "status": "degraded"}); laser .stream("telemetry") .topic("readings") .publish() .json(&reading)? .send() .await?; let mut changes = feed.poll().await?; while changes.is_empty() { tokio::time::sleep(Duration::from_millis(200)).await; changes = feed.poll().await?; } for change in &changes { println!("{} row(s), offsets {}..{}", change.rows, change.from_offset, change.to_offset); } let latest = laser .query("readings_v1") .where_eq("status", "degraded") .fetch() .await?; println!("{} degraded readings", latest.rows.len()); Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main() -> None: async with await ls.Laser.connect_env() as laser: feed = laser.watch(index="readings_v1") while await feed.poll(): pass # Skip batches that earlier runs left behind. await laser.stream("telemetry").topic("readings").publish( {"host": "node-7", "cpu": 82, "status": "degraded"} ).send() while not (changes := await feed.poll()): await asyncio.sleep(0.2) for change in changes: print(f"{change.rows} row(s), offsets {change.from_offset}..{change.to_offset}") latest = await laser.query("readings_v1").where_eq("status", "degraded").fetch() print(f"{len(latest.rows)} degraded readings") asyncio.run(main()) ``` A change record says which index moved, over which source offsets, and how many rows landed. It does not carry the rows. Query the index to read them. ## Watch every view Leave out the index to read changes for every notifying binding on one feed: `laser.watch().records()` in Rust and TypeScript, `laser.watch()` in Python. Each record's `index` field says which view moved. ## Resume after a restart Save the reader's offsets while it runs. After a restart, open a new reader and restore them. Without restored offsets, a new reader starts at the beginning of the feed. ```ts const saved = feed.offsets // ReadonlyMap // After a restart const resumed = (await laser.watch().index("readings_v1").records()).fromOffsets(saved) ``` ```rust let saved: Vec = feed.offsets().to_vec(); // After a restart let mut resumed = laser .watch() .index("readings_v1") .records()? .from_offsets(saved); ``` ```python saved = feed.offsets # list[int] # After a restart resumed = laser.watch(index="readings_v1", from_offsets=saved) ``` Store the saved offsets wherever your application keeps state, such as a file or [key-value state](/laser-sdk/state). ## Good to know * Turn the feed on per binding with `notify()`. A binding without it sends no change records. * Notifications are best effort. A lost one does not lose data, because the rows are already in the view. * The feed shows that a view moved. To be sure a query includes your own write, use a read-your-writes query. * The SDK does not push. Your code calls `poll()` and decides how long to wait between polls. * The index filter runs in the client. Use the same client setup, including the default stream, that you query with. * On a deployment without the feed, opening the reader returns an unsupported error. Source: https://docs.laserdata.com/laser-sdk/change-feed --- # Graph The graph stores entities as nodes and relationships as edges, such as which host runs which service or which incident touched which component. You can walk relationships, start from the nodes closest to an embedding, and read the graph as it was at an earlier time. Build it with direct writes, or derive it from the messages on your log. ## Quick example This program records that a host runs two services, then reads the host's neighbors. It needs [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. ```ts import { Laser, graphNodeEntity } from "@laserdata/laser-sdk" await using laser = await Laser.connectWithStream("iggy:iggy@127.0.0.1:8090", "ops") const graph = laser.graph("kg") for (const service of ["service:auth", "service:metrics"]) { await graph.link("host:node-7", "runs", service) } const host = graphNodeEntity("host", "node-7") const result = await graph.neighbors(host.id, "out", "runs", 1) for (const node of result.nodes) { for (const [key, value] of node.attrs) { if (key === "value" && value.kind === "str") console.log(`${node.labels[0]}:${value.value}`) } } ``` ```rust use laser_sdk::prelude::full::*; #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_with_stream("iggy:iggy@127.0.0.1:8090", "ops").await?; for service in ["service:auth", "service:metrics"] { laser.graph("kg").link("host:node-7", "runs", service).await?; } let host = GraphNode::entity("host", "node-7").id; let result = laser .graph("kg") .neighbors(host, EdgeDir::Out, Some("runs".to_owned()), 1) .await?; for node in &result.nodes { let label = node.labels.first().map_or("entity", String::as_str); if let Some((_, Value::Str(value))) = node.attrs.iter().find(|(key, _)| key == "value") { println!("{label}:{value}"); } } Ok(()) } ``` ```python import asyncio import laser_sdk as ls async def main(): async with await ls.Laser.connect("iggy:iggy@127.0.0.1:8090", stream="ops") as laser: graph = laser.graph("kg") for service in ("service:auth", "service:metrics"): await graph.link("host:node-7", "runs", service) host_id = ls.node_id_content("host", "node-7") result = await graph.neighbors(host_id, "out", "runs", 1) for node in result.get("nodes", []): print(f"{node['labels'][0]}:{dict(node['attrs']).get('value')}") asyncio.run(main()) ``` `link` takes `"label:value"` strings and writes both nodes and the edge. A node's ID comes from its label and value, so every agent that links `host:node-7` updates the same node, and repeating a link changes nothing. The reply includes the start node. ## Change a fact `relink(from, relation, to)` replaces a relationship that has one current value, such as which host a service runs on. It closes the old edges and writes the new one. `unlink(from, relation, to)` closes one edge. Closed edges stay in the history. ```ts const closed = await laser.graph("kg").relink("service:auth", "runs_on", "host:node-9") ``` ```rust let closed = laser.graph("kg").relink("service:auth", "runs_on", "host:node-9").await?; ``` ```python closed = await laser.graph("kg").relink("service:auth", "runs_on", "host:node-9") ``` ## Walk several hops The traversal builder picks a start, adds one hop per call, and chooses what to return. Start from node IDs with `start_ids`, from nodes that match a filter with `start_match`, or from the nodes closest to an embedding with `start_nearest(embedding, k)`. ```ts const incident = graphNodeEntity("incident", "INC-101") const paths = await laser.graph("kg").startIds([incident.id]).out("affected").out("runs_on").returnPaths().fetch() ``` ```rust let incident = GraphNode::entity("incident", "INC-101").id; let paths = laser .graph("kg") .start_ids(vec![incident]) .out("affected") .out("runs_on") .return_paths() .fetch() .await?; ``` ```python incident_id = ls.node_id_content("incident", "INC-101") paths = await ( laser.graph("kg").start_ids([incident_id]).out("affected").out("runs_on").return_paths().fetch() ) ``` ## Read the graph at a past time Add `as_of(micros)` (TypeScript: `asOf`, as a `bigint`) before a read to see only the edges that were valid at that time, in epoch microseconds. An edge closed by `unlink` or `relink` still shows up in a read from before it closed. ## Build the graph from messages Register a graph projection with an entity schema, and bind it to a topic. The deployment reads each record, extracts nodes and edges with JSON pointers, and writes them, so your services need no extraction code. [Graph in depth](/laser-sdk/advanced/graph#build-the-graph-from-a-topic) shows the schema. Inside a [session](/laser-sdk/session), `linked_graph(name)` (TypeScript: `linkedGraph`) records which session and agent wrote each fact. ## Good to know * The graph needs Laser Stack or LaserData Cloud. Check `capabilities.graph`. Against Apache Iggy alone, every graph call fails with an unsupported error. * A traversal goes at most 8 hops, a reply holds at most 10,000 nodes and edges, and the default limit is 100. * Reads are eventually consistent, so a write can take a moment to show up in a traversal. * Graph names are scoped to the default stream, so `laser.graph("kg")` addresses `stream:/kg`. * In Rust, each graph call consumes its handle, so call `laser.graph(name)` for each operation. Rust also needs the `graph` feature. Source: https://docs.laserdata.com/laser-sdk/graph --- # Filters A filter belongs to a consumer group. The server runs it next to the data and sends the group only the messages that match. Each delivered message keeps its original offset, headers, and payload bytes. Use filters when one shared topic feeds many consumers that each care about a small part of it, such as alert routing, change data capture (CDC), or backfills that skip most of a topic. In the [CDC example](https://github.com/laserdata/laser-sdk/tree/main/examples/rust/src/cdc), the reader receives 4 of 240 records and 424 of 27,953 payload bytes, so 98.5% of the payload never leaves the server. Rust, Python, and TypeScript produce the same result. ## Quick example A monitoring agent wants only the readings where CPU is at 90 or above. Create the group with its filter once. Every consumer of that group then gets only the matches. ```ts import { ConsumerFilter, FilterExpr, Laser } from "@laserdata/laser-sdk" await using laser = await Laser.connectEnv() const topic = laser.stream("telemetry").topic("metrics") await topic.ensure(2) for (const [host, cpu] of [["node-1", 42], ["node-2", 97], ["node-3", 55]] as const) { await topic.publish().json({ host, cpu }).send() } // One-time setup. The group owns the filter. const hot = topic.consumerGroup("hot-hosts") await hot.create({ filter: ConsumerFilter.json(FilterExpr.pred("cpu", "gte", 90)) }) await using consumer = await hot.consumer({ startAt: { kind: "first" }, commitPolicy: { kind: "disabled" } }) for await (const message of consumer) { console.log(message.position.offset, message.json()) await consumer.commit(message) } ``` ```rust use laser_sdk::filters::{ConsumerFilter, FilterExpr}; use laser_sdk::prelude::*; use laser_sdk::query::CmpOp; use serde::{Deserialize, Serialize}; #[derive(Debug, Serialize, Deserialize)] struct Reading { host: String, cpu: u32, } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?; let topic = laser.stream("telemetry").topic("metrics"); topic.ensure(2).await?; for (host, cpu) in [("node-1", 42), ("node-2", 97), ("node-3", 55)] { let reading = Reading { host: host.into(), cpu }; topic.publish().json(&reading)?.send().await?; } // One-time setup. The group owns the filter. let hot = topic.consumer_group("hot-hosts"); hot.create() .filter(ConsumerFilter::json(FilterExpr::pred("cpu", CmpOp::Gte, 90_i64))) .build() .await?; let mut consumer = hot .consumer() .start_at(ConsumerStart::First) .commit_policy(CommitPolicy::Disabled) .build() .await?; while let Some(message) = consumer.next().await { let message = message?; let reading: Reading = message.json()?; println!("{} {reading:?}", message.position.offset); consumer.commit(&message).await?; } consumer.shutdown().await } ``` ```python import asyncio import laser_sdk as ls async def main() -> None: laser = await ls.Laser.connect_env() topic = laser.stream("telemetry").topic("metrics") await topic.ensure(2) for host, cpu in (("node-1", 42), ("node-2", 97), ("node-3", 55)): await topic.publish().json({"host": host, "cpu": cpu}).send() # One-time setup. The group owns the filter. hot = topic.consumer_group("hot-hosts") await hot.create(filter=ls.ConsumerFilter.json(ls.FilterExpr.pred("cpu", "gte", 90))) consumer = hot.consumer(polling="first", auto_commit="disabled") try: async for message in consumer: print(message.position.offset, message.json()) await consumer.commit(message) finally: await consumer.shutdown() asyncio.run(main()) ``` Only the `node-2` reading arrives. The consumer names the group, not the filter. Run `create` from a setup step. It is safe to repeat with the same filter. ## Combine conditions Build an expression from predicates, then wrap it in the codec of the payload. `all` needs every child to match, `any` needs one, and `negate` inverts a child. Field paths use dots for object keys and `[n]` for array items, such as `after.ground_stations[0]`. ```ts const critical = ConsumerFilter.json( FilterExpr.all([ FilterExpr.pred("severity", "eq", "critical"), FilterExpr.any([ FilterExpr.pred("service", "eq", "billing-agent"), FilterExpr.pred("service", "eq", "triage-agent") ]) ]) ) ``` ```rust let critical = ConsumerFilter::json(FilterExpr::all([ FilterExpr::pred("severity", CmpOp::Eq, "critical"), FilterExpr::any([ FilterExpr::pred("service", CmpOp::Eq, "billing-agent"), FilterExpr::pred("service", CmpOp::Eq, "triage-agent"), ]), ])); ``` ```python critical = ls.ConsumerFilter.json( ls.FilterExpr.all([ ls.FilterExpr.pred("severity", "eq", "critical"), ls.FilterExpr.any([ ls.FilterExpr.pred("service", "eq", "billing-agent"), ls.FilterExpr.pred("service", "eq", "triage-agent"), ]), ]) ) ``` Comparisons are `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `contains`, and `prefix`. Text matching, presence checks, and timestamp coercion are in [Filters in depth](/laser-sdk/advanced/consumer-filters#what-a-filter-can-test). JSON, CBOR, Avro, and Protobuf payloads all work. ## Filter on headers without decoding A `headers_only` filter reads typed message headers and never decodes the payload, so it works with any payload format and costs the least to run. ```ts const pager = laser.stream("ops").topic("alerts").consumerGroup("pager") await pager.create({ filter: ConsumerFilter.headersOnly(FilterExpr.header("priority", "eq", 2)) }) ``` ```rust let pager = laser.stream("ops").topic("alerts").consumer_group("pager"); pager .create() .filter(ConsumerFilter::headers_only(FilterExpr::header("priority", CmpOp::Eq, 2_i32))) .build() .await?; ``` ```python pager = laser.stream("ops").topic("alerts").consumer_group("pager") await pager.create( filter=ls.ConsumerFilter.headers_only(ls.FilterExpr.header("priority", "eq", 2)) ) ``` Producers set typed headers through `topic.producer()`. See [Filter on headers only](/laser-sdk/advanced/consumer-filters#filter-on-headers-only). ## Check a filter before you rely on it `test` judges one sample payload and explains each predicate. `preview` judges stored messages in one partition. Neither joins the group or stores progress. ```ts const tested = await hot.filter().test(JSON.stringify({ host: "node-9", cpu: 93 })) console.log(tested.explanation.verdict) const preview = await hot.filter().preview(0, { maxRecords: 10 }) console.log(preview.examined, preview.matched) ``` ```rust let sample = r#"{"host":"node-9","cpu":93}"#; let tested = hot.filter().test(sample, Vec::new()).await?; println!("{:?}", tested.explanation.verdict); let preview = hot.filter().preview(0).await?.max_records(10).send().await?; println!("{} {}", preview.examined, preview.matched); ``` ```python tested = await hot.filter().test('{"host":"node-9","cpu":93}') print(tested["explanation"]["verdict"]) preview = await hot.filter().preview(0, max_records=10) print(preview["examined"], preview["matched"]) ``` ## Change or remove a filter A group keeps the filter it was created with. Configuring a different filter on the same group fails with `conflict`, so two selections never share one set of offsets. Create a new group for a new selection. `group.filter().release()` removes the group's filter, and its consumers then receive every message. A released group, or one whose filter was deleted, can take a filter again only with the same digest it ran before. ## Good to know * Configuring a filter needs `laser-plane`, so use [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. Plain Apache Iggy has no filters. * A group with no filter receives every message. A read never falls back to unfiltered delivery when the filter cannot run. It fails instead. * A batch of 100 means the server examines up to 100 messages per partition. It can deliver fewer, or none, and an empty poll does not mean the topic is empty. * Progress covers the messages the server skipped, so a group that matches one message in a million still moves forward. * Raw Apache Iggy consumers ignore the filter and receive every message. Do not mix them with Laser SDK consumers on one group. A filter is not an access control boundary. * Filters judge each message on its own. They do not compare it with an earlier message. Source: https://docs.laserdata.com/laser-sdk/consumer-filters --- # Laser Stack [Laser Stack](https://github.com/laserdata/laser-stack) runs the LaserData Apache Iggy fork and `laser-plane` in Docker for local development, SDK examples, and CI. It waits until both services are healthy and prints a connection string. Plain Apache Iggy runs messages, agents, and session recording. Queries and views, changes, key-value state, graph, forks, filters, session listings, and managed permissions also need `laser-plane`. Laser Stack gives you both. ## Requirements * A current Docker Engine or Docker Desktop. * Docker Compose v2. The scripts stop if Docker is not installed, not running, or has no Compose v2. * A 64-bit `amd64` or `arm64` system. Docker picks the matching image, including `arm64` on Apple Silicon. ## Start the stack ```bash git clone https://github.com/laserdata/laser-stack cd laser-stack ./scripts/up ``` The first run creates `.env` from `.env.example` with mode `600`, so only its owner can read it. The script pulls the latest published images and starts both services in the background. When both report healthy, it prints: ```bash export LASER_CONNECTION_STRING='iggy:laser@127.0.0.1:8090' ``` Copy the printed value into your shell. The default credentials are `iggy:laser`. If the registry is unreachable, the script uses cached images. If there are no published or cached images, it builds them from signed binaries. ## Check the managed path ```bash ./scripts/smoke ``` The smoke test checks Iggy TCP health and plane readiness. It then installs the published TypeScript SDK in a Node.js container, connects, reads the capabilities, and runs a managed key-value write and read through Iggy and `laser-plane`. ## What runs | Service | Role | Local endpoint | | -------------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | LaserData Apache Iggy fork | Durable log, authentication, and forwarding of managed requests | TCP `127.0.0.1:8090`, HTTP `127.0.0.1:3000` | | `laser-plane` | Views, queries, key-value state, graph, changes, the filter catalog, authorization, and run read models | Internal Unix socket shared with Iggy | Applications use one SDK connection for both. Iggy data and plane data live in separate Docker volumes. ## Connect Export the value from `./scripts/up`, then connect: ```ts import { Laser } from "@laserdata/laser-sdk" await using laser = await Laser.connectEnv() ``` ```rust use laser_sdk::prelude::*; let laser = Laser::connect_env().await?; ``` ```python import laser_sdk as ls laser = await ls.Laser.connect_env() ``` The environment helpers read `LASER_CONNECTION_STRING` and the optional `LASER_STREAM`. See [Connect](/laser-sdk/connect) for every option. ## Run the SDK examples The [`laser-sdk` examples](https://github.com/laserdata/laser-sdk/tree/main/examples) run the same steps in each language. Rust needs the toolchain version pinned in `rust-toolchain.toml`. Python needs 3.10 or later and uses `uv`. TypeScript needs Node.js 22.14 or later. ```bash git clone https://github.com/laserdata/laser-sdk ``` From the directory that holds the clone, run an example. `query` uses both Iggy and `laser-plane`: ```bash cd laser-sdk/examples/typescript npm run setup npm run example:query ``` ```bash cd laser-sdk/examples/rust cargo run --example query ``` Published package: ```bash cd laser-sdk/examples/python uv venv uv pip install laser-sdk uv run python query.py ``` Local source: ```bash cd laser-sdk/examples/python uv sync --project ../../foreign/python --locked --extra testing --extra examples uv run --project ../../foreign/python python query.py ``` Run `npm run setup` once after cloning. It installs dependencies and builds the TypeScript SDK from the clone. Each `npm run example:` compiles the examples first. Run `npm run setup` again after you change the SDK source. Focused examples are `log`, `query`, `watch`, `kv`, `cdc`, `graph`, `recall`, `context`, and `agent`. Larger ones are `native-streaming`, `event-analytics`, `fleet-tape`, `firehose`, `incident-desk`, `memory`, `interop`, `orchestra`, `governance`, and `sessions`. Rust and TypeScript use hyphenated names. Python files use underscores, such as `event_analytics.py`. Each example writes to its own stream, `laser--`. A run deletes the previous run's stream first and keeps its own data for you to inspect. Set `LASER_STREAM` to use a stream of your own. The examples never delete that stream, so repeated runs share its state. Against plain Apache Iggy, a step that needs `laser-plane` prints the missing capability and exits. ## Stack commands | Command | Action | | -------------------------------- | ----------------------------------------------------------------------- | | `./scripts/up` | Start in the background and wait until healthy | | `./scripts/up --foreground` | Run attached until `Ctrl-C` | | `./scripts/up --build` | Build images from signed, pinned binaries | | `./scripts/up --random-password` | Generate a random password for a new data volume | | `./scripts/smoke` | Check health and readiness, then run a managed SDK key-value round trip | | `./scripts/logs [service]` | Follow all logs or one service (`iggy` or `plane`) | | `./scripts/down` | Stop services and keep data | | `./scripts/down --volumes` | Stop services and delete this stack's volumes | | `./scripts/reset` | Stop services and delete this stack's data after a prompt | | `./scripts/reset --yes` | Reset without a prompt | ## Configuration Edit `.env` before you start the stack. | Variable | Default | Purpose | | ------------------------------------------- | ----------------------- | ----------------------------------------------------------- | | `LASER_BIND_ADDRESS` | `127.0.0.1` | Host interface for the Iggy ports | | `LASER_IGGY_PORT` | `8090` | Iggy TCP host port | | `LASER_IGGY_HTTP_PORT` | `3000` | Iggy HTTP host port | | `LASER_IGGY_USERNAME` | `iggy` | Root username | | `LASER_IGGY_PASSWORD` | `laser` | Root password. Letters, digits, `.`, `_`, `~`, and `-` only | | `LASER_START_TIMEOUT` | `120` | Seconds `up` waits for each service to become healthy | | `LASER_IGGY_VERSION`, `LASER_PLANE_VERSION` | Tested versions | Binaries that `./scripts/up --build` downloads | | `LASER_SDK_VERSION` | Tested version | TypeScript SDK version that the smoke test installs | | `LASER_SMOKE_TIMEOUT` | `300` | Seconds before the smoke test gives up | | `LASER_SMOKE_NODE_IMAGE` | Pinned `node:22-alpine` | Node.js image for the smoke test | Iggy stores the root account in its data volume on first start. Changing the password in `.env` later does not change an existing volume. To use a random password, reset first: ```bash ./scripts/reset --yes ./scripts/up --random-password ``` `--random-password` refuses to run while the Iggy volume exists or while `LASER_IGGY_PASSWORD` is set in your shell. The ports bind to `127.0.0.1` by default, so other machines cannot reach them. The stack has no TLS certificates. Set up Iggy TLS and a trusted CA before you change `LASER_BIND_ADDRESS`. To smoke test another npm version or a local `.tgz`, set `LASER_SMOKE_SDK_SPEC` in your shell. This leaves `.env` unchanged: ```bash LASER_SMOKE_SDK_SPEC=/path/to/laser-sdk.tgz ./scripts/smoke ``` See the [Laser Stack README](https://github.com/laserdata/laser-stack#readme) for image signatures and troubleshooting. ## Choose a target | Target | Messages, agents, and sessions | Managed features | | ----------------- | ------------------------------ | ------------------------------------------------ | | Laser Stack | Yes | Yes, locally through `laser-plane` | | LaserData Cloud | Yes | Yes, managed | | Plain Apache Iggy | Yes | No. Managed calls fail with an unsupported error | Change `LASER_CONNECTION_STRING` to switch targets. The code stays the same. Read `laser.capabilities()` before you use an optional managed feature. Source: https://docs.laserdata.com/laser-sdk/laser-stack --- # Connect Create one connected client and share it. It carries authentication, transport, and connection state, and it reaches every stream on the server. ## Connection strings Pass a connection string to `connect`: ```ts import { Laser } from "@laserdata/laser-sdk" await using laser = await Laser.connect("iggy:iggy@127.0.0.1:8090") ``` ```rust use laser_sdk::prelude::*; let laser = Laser::connect("iggy:iggy@127.0.0.1:8090").await?; ``` ```python import laser_sdk as ls laser = await ls.Laser.connect("iggy:iggy@127.0.0.1:8090") ``` Use `user:pwd@host:port` or `token@host:port`. The port is optional and defaults to `8090`, the Apache Iggy TCP port: ```text # username and password iggy:iggy@127.0.0.1:8090 # a token in place of user:pwd @starter-123.us-west-1.aws.laserdata.cloud:8090 ``` All three SDKs read the string with the same rules, and every broken rule is a configuration error before any connection attempt: * The scheme is optional. Only `iggy://` and `iggy+tcp://` are accepted. * Credentials are required. An empty user, password, or token, more than one `:`, or an `@` inside the credentials is refused. The SDK takes them as written, without percent-decoding, so a `/` in a password works. * The address is `host` or `host:port`. A present port must be a number from 1 to 65535. IPv6 literals, paths, and fragments are refused. * The options are exactly `tls`, `tls_domain`, `tls_ca_file`, `reconnection_retries`, `reconnection_interval`, `reestablish_after`, `heartbeat_interval`, and `nodelay`. Names are case-sensitive and each appears at most once as `key=value`. An unknown, repeated, or malformed option, or an empty `?`, is refused. * `tls` and `nodelay` take only `true` or `false`. To keep the credentials apart from the address, pass them separately: ```ts const laser = await Laser.builder() .address("127.0.0.1", 8090) .credentials("iggy", "iggy") .connect() ``` ```rust let laser = Laser::builder() .address("127.0.0.1:8090") .credentials("iggy", "iggy") .build() .await?; ``` ```python laser = await ls.Laser.connect( address="127.0.0.1:8090", credentials=("iggy", "iggy"), ) ``` Pick one form per client. The builder fails without credentials, as the string does. The builder port is optional too and defaults to `8090`. `Laser::local()` in Rust and `Laser.local()` in TypeScript and Python connect to `iggy:iggy@127.0.0.1:8090`. [Laser Stack](/laser-sdk/laser-stack) uses `iggy:laser` by default, so use the connection string it prints: ```bash export LASER_CONNECTION_STRING='iggy:laser@127.0.0.1:8090' ``` ## Environment variables `Laser::connect_env()` in Rust, `Laser.connectEnv()` in TypeScript, and `Laser.connect_env()` in Python read the environment: ```ts await using laser = await Laser.connectEnv() ``` ```rust let laser = Laser::connect_env().await?; ``` ```python laser = await ls.Laser.connect_env() ``` | Variable | Effect | | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `LASER_CONNECTION_STRING` | The connection string, exactly what `connect()` takes. Required. The helpers return a configuration error without it | | `LASER_STREAM` | Optional default stream | | `LASER_TLS_CERT` | Path to a CA certificate file. Turns on TLS with this CA for any host and replaces the bundled LaserData CA | | `LASER_NO_TLS` | `1`, `true`, `yes`, or `on` turns off automatic TLS setup. Use it only for local development | | `LASER_CONNECT_TIMEOUT_MS` | The budget for the first connect. See [Connect timeout](#connect-timeout) | | `LASER_PUBLISH_TIMEOUT_MS`, `LASER_PUBLISH_MAX_RETRIES`, `LASER_PUBLISH_RETRY_BACKOFF_MS` | Publish limits. See [Publish timeouts and retries](#publish-timeouts-and-retries) | TypeScript's `connectEnv` also takes an environment map in place of `process.env`. The SDK does not read `LASER_SERVER`, `LASER_TOKEN`, `LASER_USERNAME`, or `LASER_PASSWORD`. Some examples use them to build `LASER_CONNECTION_STRING`. ## Token or password Use `@host:port` for a token or `user:pwd@host:port` for a username and password. Tokens suit services and CI. A username and password suit local development. Create tokens in [API keys](/security/api-keys). Both sign in an Iggy user with stream and topic permissions. A permission failure is a typed error. Rust has `is_permission_denied()`, Python sets `permission_denied` on the exception, and TypeScript has `isPermissionDenied(error)`. ## TLS For `*.laserdata.cloud` and `*.laserdata.com` hosts, the SDK turns on TLS with the bundled LaserData root CA. Other hosts keep the TLS settings of their connection string. * `LASER_TLS_CERT=` or the `tls_ca_file=` parameter selects your own CA instead of the bundled one. * `tls_ca_file=` turns TLS on by itself. `tls=true` next to it is allowed, and `tls=false` next to it is a configuration error. * An explicit `tls=false` without a CA file turns off automatic TLS for LaserData hosts and for `LASER_TLS_CERT`. * `LASER_NO_TLS=1` turns off automatic TLS setup. `0` and `false` do not. It never overrides a `tls_ca_file=` in the connection string, which still turns TLS on. ## Connect timeout The first connect has a 30 second budget. It covers the TCP connection, the TLS handshake, the login, and the managed capability probe. The Iggy client retries a failed connection by itself until the budget runs out. | Rust builder | Python `Laser.connect` | TypeScript builder | Environment variable | Default | | --------------------------- | ---------------------- | -------------------- | -------------------------- | ------- | | `connect_timeout(Duration)` | `connect_timeout_ms=` | `connectTimeout(ms)` | `LASER_CONNECT_TIMEOUT_MS` | `30000` | An explicit setting wins over the variable. The budget must be positive and at most 2147483647 milliseconds. An invalid value fails before any connection attempt. When the budget runs out, the connect fails with a timeout error that names the stage that stopped. `the Iggy server to accept the connection` means the TCP or TLS handshake did not finish. `the Iggy login reply` means the transport was up but the server did not answer the login. TypeScript drops the leading `the`, and for a client you supply it can also report `Iggy client readiness`. If only the capability probe is still open when the budget runs out, the connect succeeds with the baseline capability set. `capabilities()` probes again by itself when the set it holds has no managed plane and its last probe is at least one second old. Callers that ask at the same time share one probe. `refresh_capabilities()` always probes. ```ts const laser = await Laser.builder() .connectionString("iggy:iggy@127.0.0.1:8090") .connectTimeout(10_000) .connect() ``` ```rust use std::time::Duration; let laser = Laser::builder() .connection_string("iggy:iggy@127.0.0.1:8090") .connect_timeout(Duration::from_secs(10)) .build() .await?; ``` ```python laser = await ls.Laser.connect( "iggy:iggy@127.0.0.1:8090", connect_timeout_ms=10_000, ) ``` ## Reconnection The connect budget applies only to the first connect. After a lost connection, the client reconnects without a limit by default, once a second, so long-running consumers survive a server restart. After it reconnects, it signs in again with the connection string credentials. Change this with connection string parameters: ```text user:pwd@host:8090?reconnection_retries=&reconnection_interval= ``` Durations look like `250ms`, `1s`, or `1m`. The other options are `tls`, `tls_domain`, `tls_ca_file`, `heartbeat_interval`, `reestablish_after`, and `nodelay`. TypeScript checks `reestablish_after` but its Node transport reconnects on its own schedule. ## Publish timeouts and retries Each publish attempt has a 60 second timeout and up to three more attempts. Retry delays start at 250 milliseconds, double after each failure, and stop at 30 seconds. The timeout bounds one attempt. The transport can report a failure sooner. | Setting | Rust builder | Python `Laser.connect` | TypeScript builder | Environment variable | Default | | ----------------- | --------------------------------- | --------------------------- | ------------------------- | -------------------------------- | ------- | | Attempt timeout | `publish_timeout(Duration)` | `publish_timeout_ms=` | `publishTimeout(ms)` | `LASER_PUBLISH_TIMEOUT_MS` | `60000` | | More attempts | `publish_max_retries(n)` | `publish_max_retries=` | `publishMaxRetries(n)` | `LASER_PUBLISH_MAX_RETRIES` | `3` | | First retry delay | `publish_retry_backoff(Duration)` | `publish_retry_backoff_ms=` | `publishRetryBackoff(ms)` | `LASER_PUBLISH_RETRY_BACKOFF_MS` | `250` | Explicit settings win over the variables. Set retries to zero to turn off automatic resends. The timeout and the delay must be positive and at most 2147483647 milliseconds. Invalid settings fail before the connection opens. ```ts const laser = await Laser.builder() .connectionString("iggy:iggy@127.0.0.1:8090") .publishTimeout(90_000) .publishMaxRetries(5) .publishRetryBackoff(500) .connect() ``` ```rust use std::time::Duration; let laser = Laser::builder() .connection_string("iggy:iggy@127.0.0.1:8090") .publish_timeout(Duration::from_secs(90)) .publish_max_retries(5) .publish_retry_backoff(Duration::from_millis(500)) .build() .await?; ``` ```python laser = await ls.Laser.connect( "iggy:iggy@127.0.0.1:8090", publish_timeout_ms=90000, publish_max_retries=5, publish_retry_backoff_ms=500, ) ``` The variables apply to the builders and the `connect` helpers, and to TypeScript's `Laser.fromClient`. Rust's `Laser::from_client` uses the built-in defaults because it cannot return an error. To configure a client you supply in Rust, use `Laser::builder().client(client)`. That client must already be connected and signed in. These settings cover fluent, typed, agent, and direct producer publishes. A producer that sets its own retry count or delay keeps it, and one that leaves them unset inherits the connection values. A background producer returns once its queue accepts the records, so the publish timeout does not cover that step. See [Producing at volume](/laser-sdk/advanced/log#producing-at-volume). ### What triggers a retry A timeout or a temporary connection failure triggers a retry within the limits. An `Unauthenticated` reply on a connection that had signed in before also triggers recovery, because a reconnect or a leader change can land on a node that holds no session for the client. The SDK reconnects with the connection string credentials, then retries. Permission errors, rejected credentials, invalid requests, and malformed commit confirmations fail at once. Reconnecting before a retry uses time from that attempt. TypeScript spends at most half of the attempt on reconnecting, so time is left to send. After an attempt times out, Rust can spend one more timeout period closing the old connection and reconnecting. Rust reconnects the shared client in place, so consumers and reply readers keep their connection. TypeScript closes a failed socket and rejoins registered consumer groups on the new one. A client you supply in TypeScript has no connection recipe, so you must replace it after a failure. Retries keep message IDs, payloads, headers, and routing. Delivery stays at least once. A lost acknowledgement can deliver a record twice, so effects must be safe to repeat. ### Longer outages When retries run out, Rust returns an error, Python raises a typed exception, and TypeScript rejects the promise. Nothing panics or exits. The error reports what was confirmed and what was not: | Language | Error | Confirmed ranges | Records without a confirmation | Original error | | ---------- | --------------------------- | ---------------- | ------------------------------ | --------------------------------------------------- | | Rust | `LaserError::PublishFailed` | `committed` | `unconfirmed` | `publish_cause()` | | TypeScript | `PublishFailedError` | `committed` | `unconfirmed` | `publishCause()`, or the free `publishCause(error)` | | Python | `PublishFailedError` | `committed` | `unconfirmed` | `__cause__` | All three also carry `stream` and `topic`. The retryable and permission checks answer for the original error: Rust `is_retryable()` and `is_permission_denied()`, TypeScript `isRetryable(error)` and `isPermissionDenied(error)`, and the Python `retryable` and `permission_denied` attributes. A permanent error, such as a missing resource or a denied permission, is not retryable. Do not publish the confirmed records again. See [When a publish fails](/laser-sdk/advanced/log#when-a-publish-fails). The SDK has no durable queue for offline publishes. A long-running service keeps failed work in its own durable queue, slows intake during an outage, and retries later. Keep retry limits finite. Consumer and agent tasks can also finish with an error, so watch their result and restart them. ## Managed readiness A successful connect gives you Iggy access. Queries, key-value state, forks, graph, and other managed features also need a ready backend, and a connected server can still be replaying managed state. `capabilities()` returns the capability snapshot. Refresh it after startup or a backend restart, or wait for readiness with a deadline: ```ts const fresh = await laser.refreshCapabilities() const ready = await laser.waitUntilReady(30_000) console.log(fresh.kv.available, ready.query.available) ``` ```rust use std::time::Duration; let fresh = laser.refresh_capabilities().await; let ready = laser.wait_until_ready(Duration::from_secs(30)).await?; println!("{} {}", fresh.kv.available, ready.query.available); ``` ```python fresh = await laser.refresh_capabilities() ready = await laser.wait_until_ready(30_000) print(fresh.kv.available, ready.query.available) ``` The wait takes milliseconds in TypeScript and Python and a `Duration` in Rust. It reports unsupported when the server has no managed backend. A configured backend stays unavailable until replay finishes and it reports ready. Capabilities include operation versions, backend descriptions, readiness reasons, and the hello outcome. `is_open_only()` reports a server with only baseline capabilities, and `serves_consistency(level)` reports a supported read consistency. TypeScript has these as free functions: `isOpenOnly`, `servesConsistency`, `isReady`, `readinessReasons`, `enabledBackends`, and `unreadyBackends`. The SDK never repeats a lease acquisition after an unclear disconnect. See [Key-value state in depth](/laser-sdk/advanced/state). ## Default stream Without a default stream, address topics with the full path, `laser.stream(name).topic(name)`. If most calls use one stream, pick it when you connect: ```ts await using laser = await Laser.connectWithStream("iggy:iggy@127.0.0.1:8090", "telemetry") // shorthand for laser.stream("telemetry").topic("host-metrics") const metrics = laser.topic("host-metrics") ``` ```rust let laser = Laser::connect_with_stream("iggy:iggy@127.0.0.1:8090", "telemetry").await?; // shorthand for laser.stream("telemetry").topic("host-metrics") let metrics = laser.topic("host-metrics"); ``` ```python laser = await ls.Laser.connect("iggy:iggy@127.0.0.1:8090", stream="telemetry") # shorthand for laser.stream("telemetry").topic("host-metrics") metrics = laser.topic("host-metrics") ``` Python also has `connect_with_stream`. The builders take the default as `stream(name)` in Rust and TypeScript. Other streams stay reachable through `laser.stream(other).topic(name)`. Without a default stream, `laser.topic(name)` fails with a no-stream error. TypeScript throws `NoStreamError` from `topic(name)`. Rust returns `LaserError::NoStream` and Python raises `NoStreamError`, a subclass of `ConfigError`, when you call an operation on the topic. `with_default_stream(name)` in Rust and Python, or `withDefaultStream(name)` in TypeScript, returns a client with another default that shares the same connection. ## Stream-scoped resource names A client with a default stream scopes the managed resources it names to that stream. `laser.kv("profiles")` on a client whose default stream is `telemetry` addresses `stream:telemetry/profiles`. The same rule covers memory namespaces, lease and fence namespaces, the key registry, graph names, projection and index IDs, query indexes, fork IDs, and the change feed index filter. Each stream keeps its own managed data, and grants can name one stream's resources with a `stream:/` prefix. * A name that already starts with `stream:` is sent as is, so you can address another stream's resources. * Listings of namespaces, projections, and forks return only your stream's names, without the prefix. * Writer schemas are registered and looked up in the default stream's registry. * `resource_name(name)` (TypeScript `resourceName`) returns the name the client sends. A key-value handle reports its scoped namespace as `resource_namespace` (TypeScript `resourceNamespace`), and a fork handle reports its scoped ID as `resource_id` (TypeScript `resourceId`). * A client without a default stream sends bare names. * A record's projection selector stays the local projection ID. The backend matches it against the bindings of the record's own stream, so a record cannot select another stream's projection. * Consumer filter names are not scoped. A group's own filter is named from the group's identity, which already includes its stream. Bare naming opts out and sends every name exactly as written. Use it to read data written under bare names: ```ts const bare = laser.withResourceNaming("bare") const profiles = bare.kv("profiles") // the namespace "profiles" ``` ```rust let bare = laser.with_resource_naming(ResourceNaming::Bare); let profiles = bare.kv("profiles"); // the namespace "profiles" ``` ```python bare = laser.with_resource_naming("bare") profiles = bare.kv("profiles") # the namespace "profiles" # or for the whole connection laser = await ls.Laser.connect("iggy:iggy@127.0.0.1:8090", stream="telemetry", resource_naming="bare") ``` The builders also take the choice: `resource_naming(ResourceNaming::Bare)` in Rust and `resourceNaming("bare")` in TypeScript. `with_resource_naming` returns a client that shares the connection. A deployment that enforces stream scoping reports `stream_tenancy` in its capabilities (TypeScript `streamTenancy`). It refuses unscoped names and role grants that span streams, and it publishes each stream's change records on that stream's own change topic. ## Close `close()` ends the shared connection for every clone of the client. Consumers and reply readers on that connection lose it too. It also closes the fenced-lease connection when the client opened one. You can call it more than once. TypeScript `await using` calls `close()` when the scope of the root client ends. Clones made with `withDefaultStream`, `withResourceNaming`, and the other `with*` methods do nothing on dispose. A client you hand to TypeScript with `builder().client(c)` or `fromClient(c)` is borrowed by default, so `close()` leaves it open. Pass `{ ownership: "owned" }` to let `close()` destroy it. Python `async with` does not close the connection on exit, because a clone can still be in use. The connection closes when the last handle is dropped or when you call `close()`. `laser.stream(name).delete()` removes a stream with all of its topics and messages. It returns false when the stream did not exist. Each partition costs the server open files and memory, so delete streams you no longer need, especially on small tiers. Source: https://docs.laserdata.com/laser-sdk/connect --- # Agents This page is the full reference for agents, contracts, discovery, and workflows. For a short introduction, read [Agents](/laser-sdk/fabric). ## Agent settings Rust and TypeScript build an agent with `Agent::builder()` and `Agent.builder()`, then call `spawn(laser)`. Python calls `laser.spawn_agent(..)`. Spawning returns a handle, and `ready()` on the handle waits until the agent reads its topic. The handler is a Rust type that implements `AgentHandler`, a TypeScript object with a `handle(message, ctx)` method, or a Python callable or object with `handle(ctx, message)`. Note the argument order in Python. | Setting | Rust `Agent::builder()` | TypeScript `Agent.builder()` | Python `spawn_agent(..)` | | -------------------------------- | ----------------------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------- | | Identity | `id(..)` | `id(..)` | First argument, a name or `None` | | Input topic | `listen_on(..)` | `listenOn(..)` | Second argument | | Handler | `handler(..)` | `handler(..)` | Third argument | | Reply topic for `respond` | `respond_on(..)` | `respondOn(..)` | `respond_on=` | | Consumer group | `consumer_group(..)` | `consumerGroup(..)` | `consumer_group=` | | Capabilities | `capabilities(vec![..])` | `capabilities([..])` | `capabilities=[..]` | | Pickup status | `ack_on_pickup(true)` | `ackOnPickup()` | `ack_on_pickup=True` | | Served operations | `operations(vec![..])` | `operations([..])` | `operations=[..]` | | Session configuration | `sessions(SessionConfig)` | `sessions(config)` | `sessions=laser.sessions(..)` | | Default route for directed sends | `inbox_route(..)` | `inboxRoute(..)` | `fixed_inbox=` | | Poll interval | `poll_interval(Duration)` | `pollInterval(ms)` | `poll_interval_ms=` | | Shutdown grace | `shutdown_grace(Duration)` | `shutdownGrace(ms)` | `shutdown_grace_ms=` | | Retry | `retry(RetryPolicy)` | `retry({ maxAttempts, baseDelayMs })` | `retry_max_attempts=`, `retry_base_delay_ms=` | | Dedup window | `dedup_window(n)` | `dedupWindow(n)` | `dedup_window=` | | Custom deduplicator | `deduplicator(..)` | `deduplicator(..)` | `dedup=` | | Load dedup keys on start | `warm_dedup(true)` | `warmDedup()` | `warm_dedup=True` | | Partition lanes | `concurrency(ConcurrencyPolicy::SerialPerPartition { max_partitions })` | `concurrency({ kind: "serial-per-partition", maxPartitions })` | `max_partitions=` | | Queue bounds | `max_queued_records(n)`, `max_queued_bytes(n)` | `maxQueuedRecords(n)`, `maxQueuedBytes(n)` | `max_queued_records=`, `max_queued_bytes=` | | Understood feature bits | `understood_features(bits)` | `understoodFeatures(bits)` | `understood_features=` | | Middleware | `middleware(vec![..])` | `middleware(..)`, once per hook object | `middleware=[..]` | | Dead-letter callback | `on_dead_letter(..)` | `onDeadLetter(..)` | `dead_letter=` | | Governor | `governor((governor, mode))` | `governor([governor, mode])` | `governor=`, `governor_mode=` | | Governor retention | `governor_retention(GovernorRetention { capacity, idle_ttl })` | `governorRetention({ capacity, idleTtlMs })` | `governor_retention=(capacity, idle_ttl_ms)` | | Periodic consolidation | `consolidate_every(..)`, `consolidator(..)` | `consolidateEvery(ms)`, `consolidator(..)` | `consolidate_every_ms=`, `consolidator=` | | Signing and verification | `signing_key(..)`, `verifier(..)` | `signingKey(..)`, `verifier(..)` | `signing_key=`, `verifier=` | Defaults in all three SDKs: | Setting | Default | | ----------------- | ---------------------------------------------------------------------- | | Consumer group | The agent ID | | Poll interval | 10 ms | | Shutdown grace | 30 seconds | | Retry | 5 attempts in total, 200 ms before the first retry, doubling each time | | Dedup window | 10,000 keys in memory | | Concurrency | Serial, one message at a time across all partitions | | Queue bounds | 4,096 records and 64 MiB across partition lanes | | Pickup status | Off | | Served operations | All | More on individual settings: * `respond_on` is required for `respond` and `fan_out`. Without it, `respond` fails with a no-respond-topic error. * An agent with capabilities publishes a capability card to the registry on spawn and advertises its inbox (its `listen_on` topic) as live presence. Cards published at spawn never expire. Presence belongs to a connection, so on a server that serves presence give each advertising agent its own connection. A second one on the same connection fails to start with a presence conflict error. * `ack_on_pickup` makes the agent report a `Working` status when it picks up a command, before the handler runs. * Python accepts capability names or descriptor dictionaries. A dictionary can carry `skill_id`, `input`, `output`, `cost_class`, `latency_class`, `max_concurrency`, `health`, and `load`. The `health=` keyword sets one value (`healthy`, `degraded`, or `unavailable`) on every advertised skill. * Pass `None` as the identity and an explicit `consumer_group` to get an unscoped reliable consumer in Python. It has no agent identity, cannot advertise capabilities, and skips the session budget check. * Run more copies of the same agent to share its load. Never put two different agent IDs in one consumer group, because a record could land on the wrong member and be skipped there. * With `SerialPerPartition`, each partition gets its own ordered lane, so a slow or retrying message on one partition does not stall the others. Order within a partition stays strict. * A message that requires an AGDX feature bit the agent does not declare in `understood_features` is dead-lettered before the handler runs. * Periodic consolidation needs both an interval and a consolidator. A zero interval is a configuration error when the agent starts. Each pass is scoped to the agent ID. Python accepts a callable or an object with `consolidate(scope)`. [Memory](/laser-sdk/advanced/memory) covers consolidation. * An agent governor replaces the connection's governor for that agent. [Governance](/laser-sdk/advanced/governance#actiongovernor) covers governors. Run `bootstrap(partitions, retention)` once per stream before agents join. It creates `agent.sessions` with your retention, `agent.heartbeats` with a one-hour expiry, and `agent.streams`, `agent.memory`, `agent.dlq`, `agent.audit`, and `agent.workflow_journal`. The registry topic appears with the first card. `agent.control` is not created, because only operators may send to it. `bootstrap` needs a default stream and is safe to run again. The seven topics hold seven times `partitions` partitions, so keep the count small on small tiers. ## Inside a handler The context passed to a handler acts as the agent: | Call | What it does | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `respond(payload)` | Reply on `respond_on`. An AGDX command gets a typed response with its correlation, addressed to the requester. A plain request gets a plain reply matched by correlation | | `reply_on(topic, payload)` | Reply on another topic, chained to the handled message | | `send(topic, payload, provenance)` | Send with an explicit provenance | | `request(request_topic, reply_topic, payload, provenance, timeout)` | Send a request and wait for the correlated reply | | `fan_out(selector, payload, policy, deadline)` | Send one task to every agent with a capability and gather the replies. See [Fan out from a handler](#fan-out-from-a-handler) | | `approval_gate(reply_topic, prompt, timeout)` | Wait for a human decision. See [Interop](/laser-sdk/advanced/interop#pause-for-a-human-decision) | | `respond_input(reply_topic, response)` | Answer a human-input request | | `session()` | The handled record's session, writing as this agent | | `spawn_subconversation()` | A child conversation of the handled one | TypeScript uses camelCase. The request and approval timeouts are required in every SDK and have no default. Python passes them as `timeout_ms=` in milliseconds. `ctx.session()` is a lens on the handled record's session. It holds no lease. When a command carries the submitted marker, as `submit` and child contracts do, the runtime marks the session working before the handler runs and holds a lease until it returns. If that pickup write fails, the runtime logs a warning and still runs the handler. [Advanced sessions](/laser-sdk/advanced/sessions#hand-work-to-an-agent) covers submitted sessions. To unit test a handler without a server, build a message with `agent_message(payload, provenance)` and a context with `agent_ctx(laser, message, ..)`. Rust has them in `laser_sdk::testing`, TypeScript exports `agentMessage` and `agentCtx` (also from `@laserdata/laser-sdk/testing`), and Python has `ls.agent_message` and `ls.agent_ctx`. The `laser` only needs a live server for the context calls the handler makes. ## Delivery and retries * Delivery is at least once. The runtime commits a message only after the handler succeeds. If the handler crashes first, the agent receives the message again. * Deduplication uses each message's idempotency key, scoped to the agent that produced it. Messages without a key are not deduplicated. The default backend is an in-memory window. Supply your own deduplicator for durable deduplication, and turn on `warm_dedup` to refill the window from the partition tail on start. * External effects still need an idempotency key or a fenced write before commit. * A retryable handler error is retried until the attempts run out. A non-retryable error, such as an invalid error you raise, dead-letters the message at once. * A dead-lettered message goes to `agent.dlq` and the consumer commits past it, so one bad message does not block the partition. If publishing the dead-letter record fails, the worker stops and leaves the offset uncommitted. * A command picked up after its deadline is dead-lettered with reason `DeadlineExceeded` before the handler runs. * In Rust, a panic in the handler dead-letters the record as rejected, and the agent keeps running. * In Python, an SDK error keeps its class and retry classification. An exception you raise is classified by type: `InvalidError` and `TypeError` dead-letter at once, `PermissionError` maps to unauthorized, the built-in `TimeoutError` is retryable, and a plain `ValueError` is a retryable handler error. * Python fills an unset retry setting with its default when you set only the other one. * Publishes from agents use the connection's publish timeout and retries. See [Connect](/laser-sdk/connect#publish-timeouts-and-retries). ## Which records reach the handler Agents share `agent.sessions` by default, so the runtime classifies every record before the handler sees it. Only a command addressed to this agent or to every agent, for an operation the handler serves, reaches the handler. Replies, status records, events, and records addressed to another agent are skipped and committed, and `consumed` reports them as skipped. The author never decides, so an agent can send work to itself. On a server that resolves group policies and serves filtered reads, a client built from a connection string binds the agent's group to the addressee filter `agdx.to In [, "*"]`, so the server delivers only the agent's own and broadcast records. Otherwise the agent classifies every record itself, with the same result. [Advanced sessions](/laser-sdk/advanced/sessions#agents-and-consumer-groups) has the details. Session rules the runtime applies: * Every agent follows `agent.control` across all its partitions, outside its consumer group, so every instance sees each pause and cancel request. A handler reads them through `ctx.session().pending_control()` and decides when to stop. While a session is paused, the runtime holds its new work and handles it after the resume. * On a deployment that indexes sessions, work for a session over its budget fails that session once with reason `budget` and never reaches the handler. * Dead letters keep the record's conversation. An agent whose session configuration turns on `fail_on_dead_letter` fails the session of a dead-lettered record. ## Middleware and dead letters Middleware wraps every handler dispatch. `before_handle` runs once before the retry loop and can reject the message, which dead-letters it as rejected without running the handler. `after_handle` runs after every attempt with its result and a one-based attempt number. A dead-letter callback runs for every dead letter and reports whether the dead-letter record itself was published. ```ts import { Agent, AgentId, AgentTopic } from "@laserdata/laser-sdk" await using triage = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .retry({ maxAttempts: 3, baseDelayMs: 100 }) .middleware({ afterHandle: async (_message, result, attempt) => { console.log(`attempt ${attempt} succeeded: ${result.kind === "ok"}`) } }) .onDeadLetter({ onDeadLetter: async (_message, capsule, publishError) => { console.log( `dead letter after ${capsule.attempts} attempts, published: ${publishError === undefined}` ) } }) .handler({ handle: (_message, ctx) => ctx.respond(new TextEncoder().encode("triaged")) }) .build() .spawn(laser) ``` ```rust use laser_sdk::prelude::full::*; use laser_sdk::wire::agent::AgentDeadLetter; use std::sync::Arc; use std::time::Duration; struct Triage; impl AgentHandler for Triage { async fn handle(&self, _message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { ctx.respond("triaged").await } } struct Metrics; #[async_trait::async_trait] impl AgentMiddleware for Metrics { async fn after_handle( &self, _message: &AgentMessage, result: &Result<(), LaserError>, attempt: u32, ) { println!("attempt {attempt} succeeded: {}", result.is_ok()); } } struct Alert; #[async_trait::async_trait] impl DeadLetterSink for Alert { async fn on_dead_letter( &self, _message: Option<&AgentMessage>, capsule: &AgentDeadLetter, publish_result: &Result<(), LaserError>, ) { println!( "dead letter after {} attempts, published: {}", capsule.attempts, publish_result.is_ok() ); } } let triage = Agent::builder() .id("triage".parse::()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .retry(RetryPolicy::backoff(3, Duration::from_millis(100))) .middleware(vec![Arc::new(Metrics)]) .on_dead_letter(Arc::new(Alert)) .handler(Triage) .build() .spawn(laser.clone()); ``` ```python import laser_sdk as ls async def handle(ctx, message): await ctx.respond(b"triaged") class Metrics: async def after_handle(self, message, result, attempt): print(f"attempt {attempt} succeeded: {result['ok']}") async def alert(message, capsule, publish_error): print(f"dead letter after {capsule['attempts']} attempts, published: {publish_error is None}") triage = laser.spawn_agent( "triage", ls.AgentTopic.Sessions, handle, respond_on=ls.AgentTopic.Sessions, retry_max_attempts=3, retry_base_delay_ms=100, middleware=[Metrics()], dead_letter=alert, ) ``` The Rust `AgentMiddleware` and `DeadLetterSink` traits use `#[async_trait]`, so add the `async-trait` crate. The Python middleware result is a dictionary with `ok` and `error`, where `error` is the exception or `None`. The Python dead-letter callback receives the message (`None` when the record does not decode), the full capsule as a dictionary, and a typed SDK exception when publishing the dead letter failed. Python hooks can return directly or through an awaitable. An asynchronous hook runs on the event loop that spawned the agent, and a synchronous one runs on an SDK worker thread, so it must not block. To reprocess a dead letter after a fix, `redrive_dead_letter(capsule)` (TypeScript: `redriveDeadLetter`) reads the original record and publishes it again unchanged. ## Check consumption `consumed(target, at)` reports whether a consumer has committed past a log position, such as a dead-letter capsule's `source` or an envelope's `cause_at`. The target is a consumer group or a named consumer. | Result | Meaning | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `Consumed` | Committed past the position. Carries the committed offset and the partition head | | `NotYetConsumed` | Still behind, by `behind_by` records | | `Skipped` | The agent's group committed past the record without handling it. Carries the offsets and `dispatch`, the reason, such as `foreign` or `reply` | A consumer with no stored offset reads as not consumed. The server gives the same answer for a read it cannot authorize, so hold topic read access before you act on the result. ```ts const status = await laser.consumed({ kind: "group", name: "triage" }, capsule.source) if (status.kind === "notYetConsumed") { console.log(`behind by ${status.behindBy}`) } ``` ```rust use laser_sdk::agent::{ConsumerRef, ConsumptionStatus}; let status = laser .consumed(ConsumerRef::Group("triage".parse()?), capsule.source) .await?; if let ConsumptionStatus::NotYetConsumed { behind_by } = status { println!("behind by {behind_by}"); } ``` ```python status = await laser.consumed( ls.ConsumerRef.Group("triage"), ls.LogPosition.from_bytes(capsule["source"]), ) if isinstance(status, ls.ConsumptionStatus.NotYetConsumed): print(f"behind by {status.behind_by}") ``` A position is a `LogPosition` of stream ID, topic ID, partition ID, and offset. Rust builds one with `LogPosition::new` and Python with `ls.LogPosition(stream_id, topic_id, partition_id, offset)`. Both read the packed 20-byte form of a dead-letter `source` with `from_bytes`. TypeScript has `newLogPosition(streamId, topicId, partitionId, offset)` and `logPositionFromBytes`. ## Contracts `laser.contract(router)` sends one task to one agent and waits for its terminal reply. In Rust and TypeScript it returns a builder that ends with `send()`. Python calls `laser.contract(skill, payload, source=.., ..)` directly, with `skill=None` and `agent=` to name an agent. `laser.agent(id).contract(..)` sends as that agent. | Option | Rust and TypeScript | Python | Default | | ------------------ | ------------------------------------------ | ---------------------------- | -------------------- | | Sender | `from(agent)` | `source=` | Required | | Task body | `payload(..)` | Second argument | Empty | | Inbox route | `inbox_route(..)` | `fixed_inbox=` | The advertised inbox | | Reply topic | `reply_on(topic)` | `reply_on=` | `agent.sessions` | | Consumption expiry | `expire_if_not_consumed(d)` | `expire_if_not_consumed_ms=` | None | | Reply deadline | `deadline(d)` | `deadline_ms=` | 30 seconds | | Conversation | `conversation(id)` | `conversation=` | A new one per send | | Fence token | `fence(token)` | `fence=` | None | | Child session | `parent(parent, root)` | `parent=`, `root=` | None | | Route policy | In the router | `policy=` | Any | | Required principal | `Router::to_principal`, `routeToPrincipal` | `principal=` | None | A contract ends in one of four outcomes: * `Completed`: the agent replied within the deadline. * `Failed`: the agent replied with a terminal error. * `NotConsumed`: no pickup report and no reply arrived before the consumption expiry. This is reliable only when the agent runs with `ack_on_pickup`. Without pickup reports, a slow handler also reads as not consumed. * `TimedOut`: no terminal reply arrived within the deadline. Python returns a `Contract`: `Contract.Completed(reply)` and `Contract.Failed(reply)` carry the reply message at index `0`, and `Contract.NotConsumed()` and `Contract.TimedOut()` carry nothing. TypeScript returns `{ kind: "completed" | "failed" | "notConsumed" | "timedOut" }`, with `reply` on the first two. More contract rules: * The consumption expiry rides the command as its deadline, so an agent that picks it up later dead-letters it instead of handling it. * A fence token buys two guarantees. A consumer drops a command whose fence is below the highest it has seen for the conversation, which only works when every attempt uses the same pinned `conversation`. For an at-most-once external effect across agents or replicas, the handler must commit through a fenced compare-and-swap with the token. * A child contract writes the child's submitted start on `agent.sessions` before the command and ends the child by the outcome: completed on a reply, failed otherwise. [Advanced sessions](/laser-sdk/advanced/sessions#child-sessions) covers child sessions. * A reply completes a contract only when it carries the request's correlation, belongs to the request's session, and is addressed to the requester when the request named one. `respond` addresses the requester for you. Under a verifier, the reply must be signed by the routed agent, or by the required principal. * A broadcast or all-capable route is refused, because a contract goes to exactly one agent. ### Inbox routes and presence By default a contract sends work to the inbox the agent advertises in its live presence. Presence needs a server that serves it and, in Rust, the `query` feature. Against a server without presence, or for an agent with no capabilities, use a fixed inbox: `InboxRoute::Fixed(topic)` in Rust, `{ kind: "fixed", topic }` in TypeScript, or `fixed_inbox=` in Python. A route that resolves no inbox fails instead of falling back to a shared topic. A handler's directed sends, fan-out included, use the agent's `inbox_route`. A route can also require the agent's live connection to authenticate as a given principal. Use `Router::to_principal(agent, principal)` or `CapabilitySelector::new(skill, policy).principal(p)` in Rust, `routeToPrincipal(agent, principal)` or `capabilitySelector(skill, policy, principal)` in TypeScript, and `principal=` in Python on contracts, scatter, fan-out, and workflow steps. ## Route policies A capability route picks one agent among those that advertise the skill. The policy decides which one. | Policy | Rust | TypeScript | Python | | -------------------- | ----------------------------- | ---------------------------- | ------------------------------------- | | Any candidate | `RoutePolicy::Any` | `{ kind: "any" }` | `"any"` or `None` | | Lowest cost class | `RoutePolicy::Cheapest` | `{ kind: "cheapest" }` | `"cheapest"` | | Lowest latency class | `RoutePolicy::Fastest` | `{ kind: "fastest" }` | `"fastest"` | | Lowest load | `RoutePolicy::LeastLoaded` | `{ kind: "leastLoaded" }` | `"least_loaded"` | | Prefer one agent | `RoutePolicy::Sticky(agent)` | `{ kind: "sticky", agent }` | `"sticky:"` | | Your own ranking | `RoutePolicy::Custom(scorer)` | `{ kind: "custom", scorer }` | A callable or an object with `select` | Python passes the policy as `policy=` on contracts, scatter, and workflow steps, and as `route_policy=` on handler fan-out. The built-in policies compare the advisory classes on the capability descriptor, where lower is better. An agent that does not advertise the ranked field sorts last. `Sticky` falls back to any candidate when the preferred agent is absent. Candidates are ordered by agent ID in byte order, so `Any` and ranking ties pick the same agent in every SDK. A route that cannot address an agent fails with one of three errors. `NoCapableAgent` means no live agent advertises the skill. `NoInbox` means the chosen agent advertises no inbox. `RoutePrincipalMismatch` means the agent is not bound to the required principal. TypeScript and Python name them `NoCapableAgentError`, `NoInboxError`, and `RoutePrincipalMismatchError`. Python groups them under `RoutingError`. TypeScript has no shared base class, so catch the three classes or test with `isNoCapableAgent(error)`. A custom scorer receives the skill ID and the same candidate view as the built-in policies. Each candidate carries the agent, its registered card, and the descriptor it advertises for the skill, which can be missing. Return the index of the chosen candidate. Return nothing, or an index out of range, to refuse every candidate, which fails the route with no capable agent. Python accepts a synchronous callable or an object with a `select(skill_id, candidates)` method, and rejects an asynchronous scorer. An exception from a Python scorer is raised in place of the routing error. ```ts import { type RouteScorer, routeToCapable } from "@laserdata/laser-sdk" const preferPrimary: RouteScorer = { select: (_skillId, candidates) => { const index = candidates.findIndex((candidate) => candidate.agent.asStr().endsWith("-primary") ) return index === -1 ? undefined : index } } const router = routeToCapable("triage-ticket", { kind: "custom", scorer: preferPrimary }) ``` ```rust use std::sync::Arc; struct PreferPrimary; impl RouteScorer for PreferPrimary { fn select(&self, _skill_id: &str, candidates: &[RouteCandidate<'_>]) -> Option { candidates .iter() .position(|candidate| candidate.agent.as_str().ends_with("-primary")) } } let router = Router::to_capable("triage-ticket", RoutePolicy::Custom(Arc::new(PreferPrimary))); ``` ```python def prefer_primary(skill_id, candidates): for index, candidate in enumerate(candidates): if candidate.agent.endswith("-primary"): return index return None outcome = await laser.contract( "triage-ticket", b"disk full on node-7", source="intake", policy=prefer_primary, fixed_inbox=ls.AgentTopic.Sessions, ) ``` ## Discover agents Agent objects come from the connection: `laser.agent(id)` returns an `AgentScope`, `agent_registry()` an `AgentRegistry`, and `workflow(name)` a `Workflow`. In Rust and TypeScript, `contract(router)` returns a `ContractBuilder` and a workflow `step` returns a `StepHandle`. Python returns results directly and changes a workflow in place. `laser.agent(id)` acts as that agent with `send`, `ask`, `contract`, `publish_card`, and `advertise` (TypeScript: `publishCard`). `advertise` publishes a card and live presence the way a spawning agent does, and fails when the connection already advertises another agent. The registry surface is `agent_registry()`, `publish_card`, `advertise_presence`, `clear_presence`, `quarantine`, `unquarantine`, `quarantine_signed`, `unquarantine_signed`, and `client_metadata` (TypeScript uses camelCase). A quarantined agent is excluded from routing until it is lifted. With a verifier on the registry, unsigned quarantine facts are dropped, so use the signed variants, which need the `sign` feature in Rust. Python passes cards and presence as dictionaries and returns registry entries as `RegisteredCard` objects. Python's `client_metadata(..)` returns a `ClientMetadataPage` (`clients`, `next_cursor`) and adds `client_metadata_all`, while Rust and TypeScript return a request builder. In Rust, presence and client metadata need the `query` feature. `resolve(skill, now)` returns the cards that advertise a skill, are fresh, do not mark the skill unavailable, and are not quarantined. `agents()` and `resolve` list cards by agent ID in byte order, so a route that picks any agent, and a ranking tie, choose the same agent in every SDK. The card predicates apply the same checks to cards you read yourself. A card is fresh when it has no lifetime or its lifetime has not run out. `serves` checks that the card advertises the skill. `available_for` also checks that the skill is not marked unavailable. ```ts import { SystemClock, cardAvailableFor, cardIsFresh } from "@laserdata/laser-sdk" const registry = await laser.agentRegistry() const now = new SystemClock().nowMicros() await registry.refresh(now) const ready = registry .agents() .filter((card) => cardIsFresh(card, now) && cardAvailableFor(card, "triage-ticket")) .map((card) => card.agent.asStr()) ``` ```rust use laser_sdk::agent::{Clock, SystemClock}; let mut registry = laser.agent_registry()?; let now = SystemClock.now_micros(); registry.refresh(now).await?; let ready: Vec = registry .agents() .filter(|card| card.is_fresh(now) && card.available_for("triage-ticket")) .map(|card| card.agent.to_string()) .collect(); ``` ```python registry = laser.agent_registry() now = ls.SystemClock().now_micros() await registry.refresh(now) ready = [ card.agent for card in registry.agents() if card.is_fresh(now) and card.available_for("triage-ticket") ] ``` Rust and Python call the predicates on `RegisteredCard` as `is_fresh`, `serves`, and `available_for`. TypeScript exports `cardIsFresh`, `cardServes`, and `cardAvailableFor`. The freshness check takes the time in epoch microseconds. Python's `refresh()` and `resolve()` default the time to now. ## Scatter to every capable agent `scatter` sends one task to every agent that advertises a capability, at once, and returns the reply bodies of those that completed. Unavailable and quarantined agents are left out, and no capable agent at all fails with no capable agent. Rust calls `laser.scatter(source, &selector, payload, &route, deadline)`, TypeScript `laser.scatter(source, selector, payload, inboxRoute, deadlineMs)`, and Python `laser.scatter(skill, payload, source=.., deadline_ms=..)`. The deadline is required in all three. `scatter_report` (TypeScript: `scatterReport`) returns a `ScatterReport` with each agent's own result in `outcomes`. `completed()` lists the agents that replied, with their replies. `failures()` lists only branches that errored before reaching a terminal outcome. An agent that answered `Failed`, `TimedOut`, or `NotConsumed` is in `outcomes` only. ## Fan out from a handler Inside a handler, `fan_out(selector, payload, policy, deadline)` (TypeScript: `fanOut`) sends a task to every agent with a capability, each on its own child conversation, and gathers the replies on the agent's `respond_on` topic. It returns a `Gather` with `ok` (agent and reply pairs) and `failures` (agent and error pairs). A target with no inbox is a failure entry, never rerouted. | Gather policy | Rust | TypeScript | Python | | -------------------------------- | -------------------------- | ------------------------------- | ----------------------------------- | | Wait for every branch | `GatherPolicy::RequireAll` | `{ kind: "requireAll" }` | `policy="require_all"`, the default | | Stop after n successes | `GatherPolicy::Quorum(n)` | `{ kind: "quorum", needed: n }` | `policy="quorum", quorum=n` | | Take what landed by the deadline | `GatherPolicy::BestEffort` | `{ kind: "bestEffort" }` | `policy="best_effort"` | Python's `fan_out(skill, payload, ..)` takes a required `deadline_ms=`, plus `principal=` and `route_policy=`, the parts that Rust and TypeScript carry in the capability selector. Branches follow the agent's inbox route in every SDK, and no call overrides it. ## Workflows `laser.workflow(name)` runs dependency-ordered steps. The name is the orchestrator identity the run publishes as, so it must be a valid agent ID, which is checked when the run starts. Each step targets one agent, one capable agent, or every capable agent, and builds its payload from the outputs of earlier steps. ```ts import { AgentTopic, WorkflowBudget, routeToCapable } from "@laserdata/laser-sdk" const text = new TextEncoder() const outcome = await laser .workflow("incident-response") .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .budget(WorkflowBudget.unlimited().invocations(8).wallClock(60_000)) .step("isolate", routeToCapable("isolate-host", { kind: "any" }), () => text.encode("isolate node-7") ) .compensateWith(() => text.encode("release node-7")) .step("patch", routeToCapable("patch-host", { kind: "any" }), ({ outputs }) => { const isolated = new TextDecoder().decode(outputs.get("isolate") ?? new Uint8Array()) return text.encode(`patch after: ${isolated}`) }) .after("isolate") .verifyWith((output) => output.byteLength > 0) .run() ``` ```rust let outcome = laser .workflow("incident-response") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .budget(WorkflowBudget::unlimited().invocations(8).wall_clock(Duration::from_secs(60))) .step( "isolate", Router::to_capable("isolate-host", RoutePolicy::Any), |_ctx: &StepContext<'_>| b"isolate node-7".to_vec(), ) .compensate_with(|_ctx: &StepContext<'_>| b"release node-7".to_vec()) .step( "patch", Router::to_capable("patch-host", RoutePolicy::Any), |ctx: &StepContext<'_>| { let isolated = ctx.outputs.get("isolate").cloned().unwrap_or_default(); [b"patch after: ".to_vec(), isolated].concat() }, ) .after("isolate") .verify_with(|output: &[u8]| !output.is_empty()) .run() .await?; ``` ```python workflow = laser.workflow("incident-response", fixed_inbox=ls.AgentTopic.Sessions) workflow.budget(ls.WorkflowBudget.unlimited().invocations(8).wall_clock(60_000)) workflow.step( "isolate", to_capable="isolate-host", build=lambda outputs: b"isolate node-7", compensate=lambda outputs: b"release node-7", ) workflow.step( "patch", to_capable="patch-host", after=["isolate"], build=lambda outputs: b"patch after: " + outputs["isolate"], verify=lambda output: len(output) > 0, ) outcome = await workflow.run() ``` All three return a `WorkflowOutcome` with the outputs by step label and the run ID (`outputs` and `run_id`, TypeScript: `runId`). ### Step options | Option | Rust | TypeScript | Python | | ----------------------------- | --------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------ | | Depend on a step | `after(label)` | `after(label)` | `after=[..]` | | Verify the output | `verify_with(..)` | `verifyWith(..)` | `verify=` | | Compensate on a later failure | `compensate_with(..)` | `compensateWith(..)` | `compensate=` | | Hold a fenced lease | `exclusive()`, `exclusive_in(namespace)` | `exclusive()`, `exclusiveIn(namespace)` | `exclusive=True`, `fence_namespace=` | | On timeout | `on_timeout(OnTimeout::Fail)` or `OnTimeout::Reassign` | `onTimeout("fail")` or `"reassign"` | `on_timeout="fail"` or `"reassign"` | | Target | `Router::to`, `Router::to_capable`, `Router::all_capable` | `routeTo`, `routeToCapable`, `{ kind: "allCapable", selector }` | `to=`, `to_capable=`, `all_capable=` | ### How a run behaves * A run is a root [session](/laser-sdk/session) whose ID is the run ID. Each step and compensation is a child session. The run ends as completed, failed, or canceled. * Steps run one at a time in dependency order. Independent steps do not run in parallel. An unknown dependency or a cycle fails the run. * Each dispatch waits up to 30 seconds, cut to the wall-clock budget that remains. No SDK has a per-step deadline setting. * An all-capable step scatters the task and joins the completed replies with newlines into one output. * A step that fails its verifier, or ends as failed, timed out, or not consumed, fails the run with a handler configuration error after the compensations. * Compensations run in reverse order when a later step fails. They are best effort: each goes to the step's router with a 30 second deadline and no fence, so a capability route may reach a different agent, and errors are ignored. An all-capable step's compensation never runs. * Between steps the engine checks `agent.control` for a cancel request. On a cancel it runs the compensations and fails with a cancelled error. Operators cancel a run with `laser.sessions().control(stream, run_id).as_operator(op).cancel()`. Python's `CancelledError` also subclasses `asyncio.CancelledError`. TypeScript also accepts `run({ signal })`. * `run_id(id)` (TypeScript: `runId`) resumes an earlier run from its journal on `agent.workflow_journal`. The engine skips the steps recorded as complete and dispatches the rest. Rust and TypeScript take a `ConversationId`, Python a string. * Builders, verifiers, and compensations can be asynchronous in all three SDKs. Builders and compensations return bytes and verifiers return a boolean. Python accepts a callable or an object with a `build` or `verify` method, and a compensation object also needs `build`. An SDK error raised in a callback keeps its classification, and cancelling the run cancels the active asynchronous callback. ### Budgets A `WorkflowBudget` caps tokens, wall-clock time, and invocations across the run. Build it with `WorkflowBudget.unlimited()` and chain `wall_clock` and `invocations` (TypeScript: `wallClock`). `WorkflowBudget.tokens(n)` starts a budget from a token cap. Rust takes the wall-clock time as a `Duration`, TypeScript and Python in milliseconds. Each chained call returns a new budget, and `workflow.budget(budget)` replaces the whole budget. * Invocations are counted before each dispatch, and every limit is checked again after each step. * The token cap counts only the usage that replies carry, so it is advisory. An all-capable step reports no tokens. * The token cap is also written as the run session's budget and checked at every step boundary, on plain Apache Iggy too. * A run with less than 100 ms of wall-clock budget left fails as budget exceeded instead of timing out. * A breach runs the compensations and fails with a budget exceeded error (Rust `LaserError::BudgetExceeded`, TypeScript and Python `BudgetExceededError`), and ends the run session failed with reason `budget`. ### Exclusive steps An exclusive step holds a fenced lease, an ownership grant with an increasing token, while it runs. The lease key is the run ID, in the namespace `agdx.workflow.fence` unless you name another. The lease lasts 60 seconds, is renewed at half that, and is released on every exit. The handler's fenced write must use the same namespace and the run ID as its fence key, so the effect and the workflow check one fence sequence. [Advanced key-value state](/laser-sdk/advanced/state) explains fenced writes. `Reassign` on timeout releases the lease, takes it again with a higher fence, and hands the task to a fresh holder, at most 2 times, so 3 attempts in total. It needs an exclusive step and is refused on an all-capable step. Exclusive steps need the `kv_fenced_leases` capability, which Laser Stack and LaserData Cloud provide. Without it, the step fails with an unsupported error when it is dispatched, after earlier steps ran, so their compensations run. ## Stop an agent The handle owns the worker. Keep it until you stop or join the agent. * `shutdown()` stops new polls, drains the message in flight, and returns the consumer result. * `join()` waits for the worker to exit on its own and returns its result. Periodic consolidation stays active until the worker exits. * `abort()` stops at once. ```ts import { Agent, AgentId, AgentTopic } from "@laserdata/laser-sdk" const triage = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .handler({ handle: async () => {} }) .shutdownGrace(10_000) .build() .spawn(laser) await triage.ready() await triage.shutdown() ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; struct Triage; impl AgentHandler for Triage { async fn handle(&self, _message: &AgentMessage, _ctx: &AgentCtx<'_>) -> Result<(), LaserError> { Ok(()) } } let mut triage = Agent::builder() .id("triage".parse::()?) .listen_on(AgentTopic::Sessions) .handler(Triage) .shutdown_grace(Duration::from_secs(10)) .build() .spawn(laser.clone()); triage.ready().await?; triage.shutdown().await?; ``` ```python import laser_sdk as ls async def handle(ctx, message): pass triage = laser.spawn_agent( "triage", ls.AgentTopic.Sessions, handle, shutdown_grace_ms=10_000, ) await triage.ready() await triage.shutdown() ``` The shutdown grace sets how long pending SDK work may run after shutdown starts. When it expires, `shutdown()` fails with a timeout and the SDK cancels the work still in flight. The TypeScript `ReliableConsumer` takes the same limit as `shutdownGraceMs`. Dropping the Rust handle signals a graceful shutdown and stops consolidation. Python does the same when the handle is collected. A dropped handle reports no result, so call `shutdown()` to see a drain failure. TypeScript has no destructor and stops through `shutdown()`, `abort()`, or `await using`. Python also supports `async with laser.spawn_agent(..) as agent`, which waits for readiness on entry and shuts the agent down on exit. Handlers, middleware, deduplicators, dead-letter callbacks, and consolidators are your code. The SDK requests cancellation but cannot stop a callback that ignores it. In Python, let `asyncio.CancelledError` propagate out of every callback. ## Layouts and many sessions in one stream The stream is the isolation boundary. One stream can hold many independent root sessions, each with its own child sessions, and every agent topic name is the same in every stream. Give each application and environment its own stream. On the default shared layout, `agent.sessions` carries all work, keyed by session, so different sessions can share a partition. Order holds within a partition, and causal links connect results across sources. Parent and root IDs do not merge state, add up budgets, or cancel descendants. A layout is declared with the session configuration, `laser.sessions_with(SessionConfig::new().layout(..))` in Rust, `laser.sessions(new SessionConfig().layout(..))` in TypeScript, or `laser.sessions(layout=..)` in Python. The declaration holds for later sends on that stream through the same connection, so declare it in every process that writes to the stream. The per-agent partition layout moves commands to the addressee's declared partition and replies to the requester's. The single partition layout makes `bootstrap` create one partition per topic. The per-agent topic layout moves work addressed to a declared agent off the lane onto that agent's own topic. Agents keep `listen_on` set to `agent.sessions` and pass the same session configuration, and a declared agent then reads its own topic. Callers keep sending to `agent.sessions`, and the SDK routes at send time. [Declare a topic per agent](/laser-sdk/advanced/sessions#declare-a-topic-per-agent) shows it end to end, and [Advanced sessions](/laser-sdk/advanced/sessions#layouts) compares the layouts. ## Key operations | Call | What it does | | ----------------------------------------- | ------------------------------------------------------------------------- | | `bootstrap(partitions, retention)` | Create the agent topics on a stream once | | `Agent::builder()...spawn(laser)` | Define and start an agent (Python: `spawn_agent`) | | `ready()` | Wait until the agent reads its topic | | `contract(router)` | Send one task to one agent and wait for the outcome | | `laser.agent(id)` | Act as one agent: send, ask, contract, publish a card, advertise presence | | `agent_registry()` | Read cards, presence, and quarantine facts | | `quarantine(..)`, `quarantine_signed(..)` | Exclude an agent from routing | | `scatter(..)`, `scatter_report(..)` | Send one task to every capable agent | | `ctx.fan_out(..)` | Gather replies from every capable agent inside a handler | | `consumed(target, at)` | Check whether a consumer committed past a position | | `sessions().submit(agent, input)` | Hand a new session to an agent | | `workflow(name)` | Run ordered steps with budgets, verifiers, and compensation | | `shutdown()`, `join()`, `abort()` | Stop an agent | ## Where it runs Agents, contracts, scatter, fan-out, and workflows work on plain Apache Iggy, Laser Stack, and LaserData Cloud. Live presence needs a server that serves it. Exclusive steps need `kv_fenced_leases`. Budget enforcement in agents needs the managed session index. In Rust, the agent runtime needs the `agent` feature. Add `query` for live presence and client metadata, `sign` for signing keys, verifiers, and signed quarantine, and `kv` for exclusive steps. `managed` includes `query` and `kv` but not `agent` or `sign`. ## Related Source: https://docs.laserdata.com/laser-sdk/advanced/agents --- # Sessions in depth This page holds the full detail behind the [Sessions](/laser-sdk/session) guide. Read the guide first. ## Identity A session is one unit of agent work with a recorded lifecycle. Its ID is the conversation ID, so every conversation read in the SDK also finds the session. A session is addressed by its stream and its ID. The stream is the isolation boundary: a session and all of its child sessions live in one stream, and no link, derived ID, or read crosses streams. The same ID written into two streams is two unrelated sessions. The factory starts a session in one of three ways: * `create(label)` derives the ID from the stream name, the namespace, and the label. The same three always reach the same session. Two applications on one stream share a session only when they pick the same namespace and label. * `start()` gives the session a fresh ID. * `.with_id(id)` (TypeScript: `withId`) on either builder starts the session under an ID you choose. `.id()` on a builder returns the ID it will start. For an unlabeled builder without an explicit ID, each call returns a new fresh ID. `derive_session_id(stream, namespace, label)` (TypeScript: `deriveSessionId`) computes a labeled ID without a builder. Reopening a derived ID does not reset a session that already ended. Use a new ID for a new lifecycle. A derived ID holds hash bits where a fresh ID holds time, so order sessions by start time, never by ID. Many agents can write in one session. Each handle writes as one agent. `as_agent(id)` (TypeScript: `asAgent`) returns a copy of a handle that writes as another agent. ## Bootstrap a stream `sessions.bootstrap(partitions, retention)` creates the agent topics in the stream. `agent.sessions` holds every session's records, so it needs an explicit retention. `TopicRetention::expire_after(age)` keeps records for that long with the server's default size bound. `TopicRetention::new(expiry, max_size)` sets both. A policy that never expires and has no size bound is refused with an invalid error. | Language | Retention | | ---------- | ---------------------------------------------------------------------------------------------------------------- | | Rust | `TopicRetention::expire_after(Duration)`, `TopicRetention::new(IggyExpiry, MaxTopicSize)` | | TypeScript | `TopicRetention.expireAfter(ageMs)`, `TopicRetention.new(expiryMs?, maxSizeBytes?)` | | Python | `TopicRetention.expire_after(age_ms)`, `TopicRetention(expiry_ms=None, max_size=None)` in milliseconds and bytes | Bootstrap uses a stream that already exists as it is, so an account with permission on the stream's topics can bootstrap a stream that was created for it. With the `SinglePartition` layout, bootstrap creates every agent topic with one partition, whatever count you pass. When the server announces the `sessions` capability, bootstrap also registers the stream as a session source, then waits until session reads see the registration, up to the publish timeout. Registration needs `session:admin` on `stream:`. A refused registration is logged and reported as `registered: false`, never as an error, so provisioning can register the stream for an account without that right. Turn registration off with `register_source(false)` in the session configuration. `laser.bootstrap(partitions, retention)` creates the agent topics without the registration step. It knows no session configuration, so it never creates declared per-agent topics and never forces one partition. ## Start a session Both `create(label)` and `start()` return a builder. Chain its settings, then call `begin()`. | Setting | Rust | TypeScript | Python | | ---------------------------- | ------------------------- | --------------------------------- | ------------------------------------------- | | Owning agent, required | `.agent(id)` | `.agent(id)` | `.agent(id)` | | Namespace for a labeled ID | `.namespace(ns)` | `.namespace(ns)` | `.namespace(ns)` | | Parent and root, for a child | `.parent(parent, root)` | `.parent(parent, root)` | `.parent(parent, root)` | | Explicit ID | `.with_id(id)` | `.withId(id)` | `.with_id(id)` | | Idle timeout | `.idle_timeout(Duration)` | `.idleTimeout(ms)` | `.idle_timeout(timeout_ms)` | | Budget | `.budget(Budget { .. })` | `.budget({ tokens, costMicros })` | `.budget(ls.Budget(tokens=, cost_micros=))` | | Searchable tag, repeatable | `.tag(tag)` | `.tag(tag)` | `.tag(tag)` | `begin()` writes the start record on the session's partition of `agent.sessions` and returns the session with a lease. Rust returns `(Session, SessionLease)`, TypeScript returns `{ session, lease }`, and Python returns a `(session, lease)` tuple. A builder without an agent fails with an invalid error. `open(id)` returns a session for an existing ID as a lens. It does no I/O, takes no lease, and writes nothing until `as_agent(id)` names an author. | Field | Limit | | --------- | ----------------------------------------------- | | Label | 256 bytes of valid UTF-8, no control characters | | Namespace | 128 bytes, the key-value namespace rule | | Tags | 16 tags of at most 64 bytes each | ## End a session `end()`, `fail(error)`, and `cancel()` write the terminal record. Every copy of a session handle shares one terminal latch. The first call fixes the outcome and the record ID. Calling the same verb again resends that same record, which is safe after a failed publish. A different terminal verb fails with an invalid error. `end()` first writes a state snapshot when the session state changed since the last one. `fail(error)` takes an `AgentErrorBody` with a code, an optional message, a retryable flag, and an optional detail map. Rust takes the struct, TypeScript an object, and Python a dict such as `{"code": 7, "message": "timed out", "retryable": False}`, where the code is the wire number (7 is `Internal`). A Python `async with` block that raises records code `Internal` with the exception text. Each SDK has a guard that ends the session by the outcome of your work and releases the lease: * In Rust, `session.run(lease, |session| async move { .. })` ends the session as completed when the work returns `Ok`, and as failed on an error or a panic. A panic is recorded with `panic: true` in the error detail and then raised again. * In TypeScript, `session.run(lease, async (session) => { .. })` catches thrown errors and rejected promises, records the failure, and rethrows the original error. A thrown value that is not a `LaserError` is recorded with the panic flag. A promise you do not await stays yours, and the SDK adds no process-wide rejection listener. * In Python, `async with builder as session:` begins the session, ends it when the block finishes, and fails it with the exception type and traceback when the block raises. `await session.run(lease, work)` does the same for a function that takes the session and may be async. Both raise the original exception. When the terminal write also fails, the error from your work is the one you see. No guard can capture a dropped future, process death, or a Rust build with `panic = "abort"`. Such a session shows as idle once its heartbeat stops. ## Leases and liveness A lease keeps the session listed in the client's heartbeat. Starting a session or picking up submitted work takes a lease. Opening a session as a lens or handling a record with `ctx.session()` does not. Drop or release the lease when the process stops working on the session. A terminal verb does not release leases held elsewhere in the process. In TypeScript, `SessionLease` also works with `using`. While it holds at least one lease, the client publishes one heartbeat per stream on `agent.heartbeats`, listing the sessions it holds leases on. It beats at the heartbeat interval or at one fifth of the shortest idle timeout among its leases, whichever is shorter, and never more often than every 10 ms. One heartbeat record lists at most 2,048 sessions, and a larger set splits across records. Heartbeats stop when the last lease is released. A lease is scoped to the stream name and the stream's creation time, so a lease from a deleted stream never reports liveness for a new stream with the same name. | Setting | Default | Rust | TypeScript | Python | | ------------------ | ---------- | ------------------------ | ----------------- | ------------------ | | Idle timeout | 5 minutes | `idle_timeout(Duration)` | `idleTimeout(ms)` | `idle_timeout_ms=` | | Heartbeat interval | 60 seconds | `heartbeat(Duration)` | `heartbeat(ms)` | `heartbeat_ms=` | Idle is never written to the log. A reader derives it from the later of the last event and the last heartbeat, plus the session's idle timeout. A session whose process died shows as idle, never as failed, because nothing proves it failed. The next record makes it active again. Idle never replaces `submitted`, `paused`, or a terminal status. ## Session statuses | Status | Set by | | ----------- | ------------------------------------------------------------------------------------------ | | `submitted` | A submission handed the session to an agent | | `active` | The owning agent started it, or an agent picked up submitted work or resumed after a pause | | `paused` | An agent acknowledged a pause request | | `completed` | `end()` | | `failed` | `fail(error)`, a failing guard, a budget breach, or `fail_on_dead_letter` | | `canceled` | `cancel()`, a cancel request handled by the agent, or `force_cancel` | The spelling is `canceled` everywhere. A status the client does not know decodes as `unrecognized`. Records written after the terminal record still count, and the managed index flags them `after_end`. A session with no start record is implicit: active, shown idle after its timeout, and never completed. ## Topics | Topic | Holds | | ------------------------ | -------------------------------------------------------------------------------------------------------------- | | `agent.sessions` | The session lane: commands, replies, user turns, model and tool records, lifecycle, state, and context records | | `agent.streams` | Token chunk streams, collapsed into their reply on a timeline | | `agent.heartbeats` | Client heartbeats, with a one-hour expiry. Never on a timeline | | `agent.control` | Operator control requests, keyed by session | | `agent.memory` | Memory records | | `agent.dlq` | Dead-letter records | | `agent.audit` | Policy evidence | | `agent.workflow_journal` | Workflow step outcomes | | `agent.registry` | Agent cards, created by the first card | Bootstrap never creates `agent.control`. Create it during provisioning and give send permission on it only to operator accounts, because that permission is what makes a control request authoritative. Every record on `agent.sessions` names its addressee in the `agdx.to` header: an agent ID, or `*` for every agent. Records describe themselves, so the record kind and operation live in the record, not in the topic name. ## Layouts A stream picks one layout for its agents' work: | Layout | Where work rides | Isolation between agents | | ------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------- | | Shared, the default | `agent.sessions`, keyed by session | None. A capable server delivers each agent only its own and broadcast records | | Per-agent topic | Each declared agent's own topic, keyed by session | Enforced by per-topic permissions | | Per-agent partition | `agent.sessions`, with a declared partition per agent | None | | Single partition | Every agent topic has one partition | None | Routing follows these rules: * Lifecycle and state always ride the session's partition of `agent.sessions`, in every layout. That partition is the session lane. * The session's partition comes from the message key, which is the session ID. * In the per-agent partition layout, a command lands on its addressee's partition and a reply on its requester's partition. Records for an agent the map does not name stay on the session's partition. Partition IDs start at zero. Declare the map, because hashing agent names can collide. * In the per-agent topic layout, work addressed to a declared agent moves from `agent.sessions` to that agent's declared topic, keyed by session. See [Declare a topic per agent](#declare-a-topic-per-agent). * Order is exact within one partition. Across partitions, readers order records by broker append time and then by stream, topic, partition, and offset. The producer's clock is never used for order. A declared layout holds for every later send on that stream through the same connection, so agents and session handles route the same way. In every layout the SDK keeps agents from misreading each other's records. Only separate topics with separate permissions keep agents from reading each other's records. ### Declare a topic per agent The per-agent topic layout maps agent IDs to topic names. Each declared agent then reads its own topic, and per-topic permissions can keep agents from reading each other's work. Everyone still sends to `agent.sessions`. The SDK moves the record when it is sent: * An AGDX command, response, error, or chunk whose target is a declared agent goes to that agent's topic. * A plain record addressed to a declared agent goes there too. That covers plain requests and plain `respond` replies. * Lifecycle, state, status, events, broadcast records (`*` or no target), and records for an agent the map does not name stay on `agent.sessions`. * On a declared topic the partition is keyed by session, so one session's records keep their order there. `bootstrap(partitions, retention)` creates each declared topic with the same partition count and retention as `agent.sessions`. A declared topic named `agent.sessions` or `agent.control`, or an invalid topic name, is an invalid error. An agent that listens on `agent.sessions` with this session configuration reads its declared topic plus `agent.control`. A declared caller waits for replies on its own topic, which covers `request`, `contract`, `request_input`, and MCP tool calls. Session source registration covers every topic of the stream, so session reads still see the moved records. ```ts import { Agent, AgentId, AgentTopic, Laser, SessionConfig, TopicRetention, agentMessageBody, routeTo } from "@laserdata/laser-sdk" await using root = await Laser.connectEnv() const laser = root.withDefaultStream("support") const config = new SessionConfig().layout({ kind: "perAgentTopic", topics: new Map([ ["planner", "planner.inbox"], ["worker", "worker.inbox"] ]) }) await laser.sessions(config).bootstrap(1, TopicRetention.expireAfter(86_400_000)) // The worker listens on agent.sessions and reads worker.inbox. await using worker = Agent.builder() .id(AgentId.new("worker")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .sessions(config) .handler({ handle: (message, ctx) => ctx.respond(agentMessageBody(message)) }) .build() .spawn(laser) await worker.ready() // The command lands on worker.inbox and the reply on planner.inbox. const outcome = await laser .contract(routeTo(AgentId.new("worker"))) .from(AgentId.new("planner")) .payload(new TextEncoder().encode("plan the node-7 drain")) .inboxRoute({ kind: "fixed", topic: AgentTopic.Sessions }) .send() console.log(outcome.kind) ``` ```rust use laser_sdk::prelude::full::*; use std::time::Duration; struct Worker; impl AgentHandler for Worker { async fn handle(&self, message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { ctx.respond(message.body().to_vec()).await } } #[tokio::main] async fn main() -> Result<(), LaserError> { let laser = Laser::connect_env().await?.with_default_stream("support"); let config = SessionConfig::new().layout(SessionLayout::per_agent_topic([ (AgentId::new("planner")?, "planner.inbox"), (AgentId::new("worker")?, "worker.inbox"), ])); laser .sessions_with(config.clone()) .bootstrap(1, TopicRetention::expire_after(Duration::from_secs(86_400))) .await?; // The worker listens on agent.sessions and reads worker.inbox. let mut worker = Agent::builder() .id("worker".parse::()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .sessions(config) .handler(Worker) .build() .spawn(laser.clone()); worker.ready().await?; // The command lands on worker.inbox and the reply on planner.inbox. let outcome = laser .contract(Router::to("worker".parse::()?)) .from("planner".parse::()?) .payload("plan the node-7 drain") .inbox_route(InboxRoute::Fixed(AgentTopic::Sessions)) .send() .await?; println!("{}", matches!(outcome, Contract::Completed(_))); worker.shutdown().await } ``` ```python import asyncio import laser_sdk as ls async def work(ctx, message): await ctx.respond(bytes(message.body())) async def main(): async with await ls.Laser.connect_env() as root: laser = root.with_default_stream("support") layout = ls.SessionLayout.PerAgentTopic({"planner": "planner.inbox", "worker": "worker.inbox"}) sessions = laser.sessions(layout=layout) await sessions.bootstrap(1, ls.TopicRetention.expire_after(86_400_000)) # The worker listens on agent.sessions and reads worker.inbox. async with laser.spawn_agent( "worker", ls.AgentTopic.Sessions, work, respond_on=ls.AgentTopic.Sessions, sessions=sessions, ): # The command lands on worker.inbox and the reply on planner.inbox. outcome = await laser.contract( None, b"plan the node-7 drain", agent="worker", source="planner", fixed_inbox=ls.AgentTopic.Sessions, ) print(isinstance(outcome, ls.Contract.Completed)) asyncio.run(main()) ``` Declare the same layout in every process that writes to the stream, because each connection routes by its own declaration. Give each declared topic the grants of its agent only. An agent that the map does not name keeps reading `agent.sessions`. ### Choose a layout Agent topic names are the same in every stream, and nothing session related crosses a stream. Give each application, and each environment of an application, its own stream instead of sharing one. * Keep the shared layout when a stream has a few roles. * Move to per-agent topics when the roles are many or the lane is busy. On a shared lane the server examines the whole lane once for every filtered role, so each delivered record costs as many examined records as there are roles. Per-agent topics remove that cost, and per-topic permissions keep the roles apart. * Use the per-agent partition layout when each declared agent's work should land on one known partition. * Use a single partition when everything must keep one order, for example in tests or a small app. Partitions are how one role scales. A consumer group gives each partition to one member at a time, so instances of a role beyond the partition count sit idle. Bootstrap `agent.sessions` with at least as many partitions as the busiest role has instances. ## Agents and consumer groups Each agent ID reads through its own consumer group, named after the agent ID unless the agent builder names another group. Every running instance of one agent joins that group, so the instances share its work and each record reaches one of them. Two different agent IDs never share a group. On `agent.sessions` and `agent.control`, the runtime binds the agent's group to the addressee filter `agdx.to In [, "*"]` before its first read. It does so when the server resolves group policies and serves filtered reads and the filter catalog, and the client was built from a connection string. The server then sends the agent only records addressed to it or to every agent. A group already bound to another filter is refused. In any other case, such as on plain Apache Iggy or with a client you built yourself, the group stays unbound. The agent reads every record, classifies it, and commits what is not its own work without calling the handler. Delivery is the same either way. The filter only saves reading, and it is not an access boundary. Set the session configuration an agent runs with through its builder: `Agent::builder().sessions(config)` in Rust, `Agent.builder().sessions(config)` in TypeScript, and `spawn_agent(..., sessions=laser.sessions(...))` in Python. Inside a handler, `ctx.session()` follows that configuration. ## Record model and tool calls The SDK never calls a model. Your code calls its provider, and the session records the call. * `assemble(policy)` reads the session lane, applies a [context policy](/laser-sdk/advanced/context#policies), and returns the selected records with a manifest. It writes nothing. `text()` on the result joins the records' payloads, one per line. * `model(request, assembled)` records the model request and, when you pass the assembled context, its manifest. It returns a call. Its `complete(response)` records the answer with its usage, the answering model, the finish reason, and the duration, and its `fail(error)` records a failed call. When the response carries no duration, the SDK measures it from the request. * `tool(name, args)` records a tool call. Its `complete(result)` and `fail(error)` record the outcome with the measured duration. * `record_model_call(request, response, assembled)` (TypeScript: `recordModelCall`) records a call that already happened. It records the response's duration, or zero when the response carries none. One call keeps one correlation across its request, manifest, and result. `correlation()` on a call returns it (Python: the `correlation` property). `ModelRequest` takes a model and a prompt body, with an optional provider and an optional operation such as `text_completion`, `chat` by default. A manifest lists at most 1,024 fragments. Tool arguments and JSON model request bodies pass a redactor before they are published. The default replaces the values of `authorization`, `api_key`, `token`, `password`, `secret`, and `cookie` with `[redacted]` at any depth, matching keys without regard to case. `redact(fn)` returns a handle with your own redactor. In Python, a redactor that raises publishes `[redacted]`. `default_redact` (TypeScript: `defaultRedact`) is exported so your redactor can extend it. Redaction is a convenience, not a guarantee: a secret under another key is published as it is. ## Session state `state()` returns one JSON document per session, built from patches on the session lane. * `set(key, value)` adds or replaces one top-level key. * `patch(ops)` appends a [JSON Patch](https://www.rfc-editor.org/rfc/rfc6902) delta. * `replace(document)` replaces the whole document as one patch that removes the keys it drops and sets every key it holds. * `snapshot()` writes the whole document. The SDK also writes a snapshot after every 64 deltas and before `end()`. * `get()` folds the retained lane from the newest snapshot. It reports `complete: false` when the records the document starts from are no longer retained. A handle that has not written yet starts from the folded lane, so a lens such as `ctx.session()` writes against the current revision. A reader applies each patch as a whole: if one operation fails, the whole patch is rejected and the document stays unchanged. A repeated delta applies once. A snapshot replaces the document only when it was written against the current revision, so a delayed snapshot from one agent cannot overwrite a later change from another. A successful append does not prove that a patch applied. Read the state at or after your record to learn the outcome. | Item | Limit | | ---------------------------- | ------------------------------------------ | | Operations per delta | 256 | | Patch body or state document | 8 MiB of JSON | | JSON integers | From -9007199254740991 to 9007199254740991 | Session state is separate from [Key-value state](/laser-sdk/state). ## Context, memory, and linked writes `context()` returns the last 50 records of the session lane, trimmed to about 4,000 estimated tokens. Change the defaults with `context_turns` and `context_tokens` in the session configuration. Python's `context(last_n=, token_budget=)` also takes them per call. `context_with(policy)` (TypeScript: `contextWith`) takes your own policy. Every SDK estimates tokens the same way: the byte count divided by four, rounded up. `record_retrieval(query, items)` records which memory items entered the context, with their scores. `record_compaction(compaction)` records that a summary replaced a range of records. `memory()` returns memory for this session in the `agent.session` namespace. Change it with `memory_namespace` in the session configuration. Rust and Python `memory_in(namespace)`, TypeScript `memory(namespace)`, and Python `memory(handle)` pick another namespace or handle. `linked_memory()` also stamps each remembered item with the record that motivated it and the agent that wrote it. See [Memory](/laser-sdk/memory). `kv(namespace)` and `linked_graph(name)` return key-value and graph handles that link every write to this session. `graph(name)` returns the shared graph without a link. `reference()` returns the session's stream and ID, for a write that lands outside the session's stream. `acting_on(source)` (TypeScript: `actingOn`) returns a handle that stamps a source record on graph writes and on remembered items. Inside a handler, `ctx.session()` already acts on the handled record. In Rust, `kv` needs the `kv` feature and `graph` and `linked_graph` need the `graph` feature. `scope()` (TypeScript: the `scope` property) returns the underlying [context scope](/laser-sdk/context) for a topic outside the session lane. `append(envelope)` writes one typed envelope on the session lane. The envelope must name this session as its conversation. ## Hand work to an agent `submit(agent, input)` starts a session on behalf of another agent. Name yourself with `.from(id)` (Python: `from_`), which is required. You can add `.label(..)`, `.namespace(..)`, `.budget(..)`, `.tag(..)`, and `.operation(..)`, which is `invoke_agent` by default. `send()` writes a submitted session and a command addressed to the agent, and returns the session ID and the command's correlation. The agent's reliable consumer marks the session active when it picks the command up, and holds a lease while the handler runs. Inside the handler, `ctx.session()` returns that session, writing as the handling agent and acting on the handled record. ## Budgets A budget states the tokens and cost a session may use. Set it with `.budget(..)` on a session builder or a submission, and it rides the session's start record. Cost is in integer micro-units of the deployment's currency. `over_budget()` (TypeScript: `overBudget`) tells you whether the summed input and output tokens of the session's records passed the token ceiling, or their summed cost passed the cost ceiling. A deployment that indexes sessions answers from its index. On plain Apache Iggy, and for a session the index does not know yet, the SDK folds the retained lane by the same rule. A session without a budget is never over it. On a deployment that indexes sessions, an agent with an ID checks each session before its handler runs, once per session per poll batch. When the session is over its budget, the agent fails it once with reason `budget` and an error naming the ceiling, then commits the work without calling the handler. Later work for that session is committed the same way. On plain Apache Iggy the runtime does not enforce budgets, because folding the lane for every record would cost too much. A workflow checks its run session at every step boundary, stops with a budget exceeded error, and ends the run session failed with reason `budget`. Budget reads are eventually consistent, so a budget is a cooperative limit, not a hard spending cap. ## Dead letters A dead-lettered record never fails its session by itself. It counts as an error, shows as `dead_letter` on the timeline, and keeps the session's ID in the dead-letter record. To fail the session instead, turn on `fail_on_dead_letter` in the session configuration and pass it to the agent: `SessionConfig::new().fail_on_dead_letter(true)` in Rust, `new SessionConfig().failOnDeadLetter(true)` in TypeScript, and `laser.sessions(fail_on_dead_letter=True)` in Python. ## Operator control `control(stream, id).as_operator(op)` returns a control handle that writes requests on `agent.control`. Only accounts with send permission on that topic can write them. | Verb | Request | | ---------------- | --------------------------------------------------------------------------------- | | `pause()` | Ask the session's agents to stop taking new actions | | `resume()` | Lift a pause | | `cancel()` | Ask the session's agents to end it as canceled at their next boundary | | `force_cancel()` | End the session as canceled without its agents, for a session whose agent is gone | The same requests found on any other topic are never applied, and a timeline shows them as `unauthorized_control`. Control is cooperative. A running agent follows `agent.control` and records every pause and cancel request addressed to it or to every agent, including requests sent before it started. The follower reads every partition of `agent.control` directly, outside any consumer group, so every instance of an agent sees every request. Inside a handler, `session.pending_control()` (TypeScript: `pendingControl`) returns the recorded requests as `pause_requested` and `cancel_requested`. The runtime never interrupts a handler, so the handler decides when to stop. `cancel_requested()` reads the session's control records directly, so it also answers on plain Apache Iggy. Workflows check it between steps. `signed_by(key)` (TypeScript: `signedBy`) on the control handle signs each control record, so a verifying reader can prove which operator sent it. `signed_by(key)` on a session handle signs its terminal record. The managed session index verifies signed records against the stream's key registry and reports the signer as `verified_actor` on the event. Without a signature, the operator ID in a control record is a claim. In Rust, signing needs the `sign` feature. ## Pause and resume Pause means no new actions in the session until it resumes. An action already in flight finishes and is recorded. Pause is cooperative, so a client that ignores it can still append. 1. `pause()` writes a request that names its participants, the agents whose acknowledgments complete the pause. Set them with `participants(..)`, or let the SDK read them from the session lane: the agents its work was sent to and the agents that picked it up. 2. Each named participant acknowledges with a `session.paused` record that names the exact request. An agent outside the set acknowledges when it first receives work for the paused session. 3. An agent holds work that arrives while the session is paused. It writes a `session.parked` record on the lane before it commits the source, and it never treats held work as handled. 4. `resume()` lifts the pause. Each agent acknowledges with `session.resumed`, handles every held record at least once before new work, and writes `session.unparked` after each one. A crash between the effect and that record can repeat the effect, so effects still need an idempotency key or a fenced write. 5. After a restart, a rebalance, or a reconnect, an agent rebuilds its held work from the lane. A bounded read that cannot prove the list complete reports it as incomplete instead of dropping work. A cancel while paused ends the session as canceled, and held records are not handled. `parked()` lists the held records that no agent reported handled, with a `complete` flag. The flag is false when the read did not reach the oldest retained record of the session's partition, or when a held record can no longer be read back. The managed index counts held records as `held` on the session summary. No deployment advertises the pause runtime as a capability yet. ```ts const control = laser .sessions() .control("support", session.conversation) .asOperator(AgentId.new("ops")) .participants([AgentId.new("worker")]) await control.pause() // ... review ... await control.resume() const held = await laser.sessions().open(session.conversation).parked() console.log(held.records.length, held.complete) ``` ```rust let control = laser .sessions() .control("support", session.conversation()) .as_operator("ops".parse::()?) .participants(["worker".parse::()?]); control.pause().await?; // ... review ... control.resume().await?; let held = laser.sessions().open(session.conversation()).parked().await?; println!("{} {}", held.records.len(), held.complete); ``` ```python control = ( laser.sessions() .control("support", session.conversation) .as_operator("ops") .participants(["worker"]) ) await control.pause() # ... review ... await control.resume() held = await laser.sessions().open(session.conversation).parked() print(len(held.records), held.complete) ``` ## Child sessions Work with its own lifecycle is a child session: a new session with a parent and a root, in the same stream. A root session has neither. Start one with `.parent(parent, root)` on a session builder, where `root` is the parent itself when the parent has no parent. A child's history stays in the child. Nothing copies it into the parent, so collect child results in your application and record them on the parent. The SDK also starts child sessions for you: * A [workflow](/laser-sdk/fabric) run is a root session, and each step and compensation is a child session whose ID derives from the run ID and the step label. * A contract runs as a child with `.parent(parent, root)` on the contract builder. * The [A2A and MCP bridges](/laser-sdk/advanced/interop) run calls as children with `submit_in` and `call_tool_in`. ## Read a session back Each record read back from a session carries its display type, derived from the record: | Display type | Record | | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `session.submitted`, `session.started`, `session.resumed`, `session.paused` | Lifecycle records and pause acknowledgments | | `session.parked`, `session.unparked` | Work held while paused, and its completion after the resume | | `session.completed`, `session.failed`, `session.canceled` | Terminal records | | `session.control`, `unauthorized_control` | A control request on `agent.control`, and the same request found on any other topic | | `model.request`, `model.response`, `model.stream` | Model calls and their token chunks | | `tool.call`, `tool.result` | Tool calls and their outcomes | | `user.message`, `agent.handoff`, `agent.message` | User turns, work handed to another agent, and other agent records | | `state.updated` | State deltas and snapshots | | `context.assembled`, `context.compacted`, `context.retrieved` | Context records | | `policy.decision`, `task.status`, `workflow.step` | Policy evidence, other status records, and workflow journal records | | `memory.created`, `memory.forgotten`, `memory.feedback` | Memory records | | `error`, `dead_letter`, `undecodable`, `invalid` | Failures, and records whose header and body disagree | A Rust `SessionTurn` holds `display` and `message`, and `text()` returns the body as UTF-8. Python has the same `display`, `message`, and `text()`. TypeScript has `display` and `message`, and `sessionTurnText(turn)` returns the text. `checkpoint()` records where the session lane ends now. It is a client bookmark that serializes, never a record on the log. `turns_at(checkpoint)` reads the records before it, and `turns_since(checkpoint)` the records after it. `state_at(checkpoint, init, fold)` and `replay(checkpoint, init, fold)` fold those records into your own state. ```ts const state = session.state() await state.set("tasks", ["triage"]) await state.patch([{ op: "add", path: "/tasks/-", value: "diagnose" }]) const view = await state.get() console.log(view.revision, view.document, view.complete) const checkpoint = await session.checkpoint() const saved = JSON.stringify(checkpoint) const toolCalls = (count: number, turn: SessionTurn) => turn.display === "tool.call" ? count + 1 : count const later = await session.replay(Checkpoint.fromJSON(JSON.parse(saved)), 0, toolCalls) ``` ```rust let state = session.state(); state.set("tasks", json!(["triage"])).await?; let patch = serde_json::from_value(json!([ { "op": "add", "path": "/tasks/-", "value": "diagnose" } ])) .map_err(|error| LaserError::Codec(error.to_string()))?; state.patch(patch).await?; let view = state.get().await?; println!("{} {} {}", view.revision, view.document, view.complete); let checkpoint = session.checkpoint().await?; let saved = serde_json::to_string(&checkpoint) .map_err(|error| LaserError::Codec(error.to_string()))?; let restored: Checkpoint = serde_json::from_str(&saved).map_err(|error| LaserError::Codec(error.to_string()))?; let tool_calls = |count: usize, turn: &SessionTurn| { if turn.display == DisplayType::ToolCall { count + 1 } else { count } }; let later = session.replay(restored, 0, tool_calls).await?; ``` ```python state = session.state() await state.set("tasks", ["triage"]) await state.patch([{"op": "add", "path": "/tasks/-", "value": "diagnose"}]) view = await state.get() print(view["revision"], view["document"], view["complete"]) checkpoint = await session.checkpoint() saved = checkpoint.to_json() def tool_calls(count, turn): return count + 1 if turn.display == "tool.call" else count later = await session.replay(ls.Checkpoint.from_json(saved), 0, tool_calls) ``` In Rust, `DisplayType` is part of `laser_sdk::prelude::full`. ## Managed session reads A deployment can keep a session index for a registered stream. The server sets the `sessions` capability only when it serves the index, and a client starts with it off. The session factory reads the index of its stream: * `get(id)` returns one session's summary: status, label, namespace, owning agent, parent and root, start and end times, first and last event and heartbeat times, counts of events, model calls, tool calls, and errors, input and output tokens and cost, the budget, the SDK, the derived `idle`, `over_budget`, `pause_requested`, and `cancel_requested` flags, and `held`, the count of records held while paused and not yet handled. `session.status()` reads the same summary for one handle. * `list()` pages the stream's sessions, newest first. Filter by `status`, `agent`, `text` (a label or ID substring), `root` (the tree under one session), or `label_prefix` (TypeScript: `labelPrefix`). Continue with `cursor`, set the page size with `limit`, and ask for a count of every match with `total`. A zero limit leaves the page size to the server. * `events(id)` pages one session's timeline in broker time order, without payloads. Each event names its position, display type, agent, addressee, correlation, cause, tool, usage, and a value-free summary. A signed record names its verified signer as `verified_actor`. `fixed_frontier` pins the first page's frontier for a consistent historical walk. * `state(id, history_limit)` returns the folded state document with its change history. Each history row records the revision, the operation ID, the outcome (`applied`, `rejected`, `stale`, or `duplicate`), the source position, and the reason. A zero limit leaves the history size to the server. * `links(id, surface)` lists the memory items, keys, graph nodes and edges, projections, and child sessions the session wrote, recalled, or touched. Pass a surface to narrow the list. * `sources(id)` lists the source partitions that hold the session's records, with how far the index has folded each one. * `changes(after, limit)` reads the stream's change feed after a sequence number. `watch(poll_every)` follows it from now and reports the changed session IDs of each poll, or a resync when it fell behind the retained feed. List the sessions again after a resync. Index summaries never hold values. A tool call keeps its tool name, argument key names, size, and content hash. A model record keeps model, provider, usage, and finish reason, never the prompt or completion text. Raw values stay on the log under its native read permission. Every reply that comes from the index carries the fold frontier of each source, so you can tell a settled view from one that is still catching up. An events page also lists gaps, each a missing range with its reason: `expired_before_fold`, `pruned`, `truncated`, or `rebuilding`. The `liveness_unknown` flag says the server's heartbeat view has not caught up, so `idle` is not meaningful yet. Other flags mark a partial view: `label_truncated`, `overflow`, `events_truncated`, `lane_conflict`, and `rebuilding`. A read fails with a session error: `LaserError::Session` in Rust and `SessionError` in Python and TypeScript. Its kinds are `unsupported`, `not_found`, `not_registered`, `invalid`, `unauthorized`, `stale`, `backend`, and `unavailable`. Without the capability, the reads fail with an unsupported error before sending. Access needs `session:read` on `stream:` and read permission on the whole stream, so a reader limited to some topics of the stream gets `unauthorized`. `laser.read_at(source)` (TypeScript: `readAt`) fetches the one record a source reference names, such as a timeline row's position. It checks that the topic was not recreated and that the record sits at that offset, and returns nothing when the record is gone. It uses a standard poll, so it works on plain Apache Iggy too. ```ts const sessions = laser.sessions() const page = await sessions.list().status("active").limit(20).fetch() for (const info of page.items) { console.log(info.id.toString(), info.status, info.idle, info.overBudget) } const events = await sessions.events(session.conversation).limit(100).fetch() for (const event of events.items) { const record = await laser.readAt(event.at) } const watch = await sessions.watch(1_000) const change = await watch.next() ``` ```rust let sessions = laser.sessions(); let page = sessions .list() .status(SessionStatus::Active) .limit(20) .fetch() .await?; for info in &page.items { println!("{} {:?} {} {}", info.id, info.status, info.idle, info.over_budget); } let events = sessions.events(session.conversation()).limit(100).fetch().await?; for event in &events.items { let record = laser.read_at(&event.at).await?; } let mut watch = sessions.watch(Duration::from_secs(1)).await?; let change = watch.next().await?; ``` ```python sessions = laser.sessions() page = await sessions.list(status="active", limit=20) for info in page["items"]: print(info["id"], info["status"], info["idle"], info["over_budget"]) events = await sessions.events(session.conversation, limit=100) for event in events["items"]: record = await laser.read_at(event["at"]) watch = await sessions.watch(1_000) change = await watch.next() ``` Python returns the replies as dicts and takes the list and event filters as keywords. In Rust, `SessionStatus` is part of `laser_sdk::prelude::full`. The [AGDX page](/laser-sdk/advanced/agdx) describes the read surface on the wire and over HTTP. ## Read limits `context()`, `context_with`, `assemble`, `state().get()`, `cancel_requested()`, and `parked()` examine at most `CONTEXT_READ_WINDOW` (10,000) raw records of each partition, the newest ones, then keep this session's records. Many sessions share a partition, so older records of a quiet session on a busy partition can fall outside that window. `turns_at`, `turns_since`, `state_at`, and `replay` read the whole range their checkpoint selects. For a long session, save your state with a checkpoint at regular points, then `replay` reads only the records after the last one. ## Recreated streams and topics A session factory remembers the stream's creation time, the creation time of `agent.sessions`, and its partition count when it first touches the lane. If the stream or `agent.sessions` is deleted and created again, or the partition count changes, every later write through that factory and its handles fails with a stale session error before anything is published. Build a new factory with `laser.sessions()` to write to the new topic. On a managed deployment, a new factory also checks that the lane matches the registered session source, and a stale error means the registration must be recovered first. ## Submit and control example ```ts await using worker = Agent.builder() .id(AgentId.new("worker")) .listenOn(AgentTopic.Sessions) .handler({ handle: async (_message, ctx) => { const session = ctx.session() await session.state().set("seen", true) await session.end() } }) .build() .spawn(laser) await worker.ready() const submitted = await laser .sessions() .submit(AgentId.new("worker"), new TextEncoder().encode('{"ticket": 42}')) .from(AgentId.new("intake")) .label("ticket-42") .budget({ tokens: 4_000n }) .send() // An operator with send permission on agent.control await laser .sessions() .control("support", submitted.session) .asOperator(AgentId.new("ops")) .cancel() ``` ```rust struct Worker; impl AgentHandler for Worker { async fn handle(&self, _message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { let session = ctx.session(); session.state().set("seen", json!(true)).await?; session.end().await } } let mut worker = Agent::builder() .id("worker".parse()?) .listen_on(AgentTopic::Sessions) .handler(Worker) .build() .spawn(laser.clone()); worker.ready().await?; let submitted = laser .sessions() .submit("worker".parse::()?, br#"{"ticket": 42}"#.to_vec()) .from("intake".parse::()?) .label("ticket-42") .budget(Budget { tokens: Some(4_000), cost_micros: None }) .send() .await?; // An operator with send permission on agent.control laser .sessions() .control("support", submitted.session) .as_operator("ops".parse::()?) .cancel() .await?; ``` ```python async def handle(ctx, message): session = ctx.session() await session.state().set("seen", True) await session.end() worker = laser.spawn_agent("worker", ls.AgentTopic.Sessions, handle) await worker.ready() submitted = await ( laser.sessions() .submit("worker", b'{"ticket": 42}') .from_("intake") .label("ticket-42") .budget(ls.Budget(tokens=4_000)) .send() ) # An operator with send permission on agent.control await laser.sessions().control("support", submitted.session).as_operator("ops").cancel() ``` The submitted session reads `session.submitted` and `agent.handoff`, then `session.resumed` when the worker picks the command up, then `state.updated` for the delta and for the snapshot `end()` writes, and `session.completed`. In Rust, `Budget` is part of `laser_sdk::prelude::full`. ## Budget example ```ts const session = laser.sessions().open(submitted.session) if (await session.overBudget()) { console.log("the session passed its budget") } ``` ```rust let session = laser.sessions().open(submitted.session); if session.over_budget().await? { println!("the session passed its budget"); } ``` ```python session = laser.sessions().open(submitted.session) if await session.over_budget(): print("the session passed its budget") ``` ## Custom layout example ```ts const sessions = laser.sessions( new SessionConfig() .stream("support") .layout({ kind: "perAgentPartition", partitions: new Map([ ["planner", 0], ["worker", 1] ]) }) .idleTimeout(600_000) .memoryNamespace("support.sessions") ) await sessions.bootstrap(2, TopicRetention.expireAfter(86_400_000)) ``` ```rust let sessions = laser.sessions_with( SessionConfig::new() .stream("support") .layout(SessionLayout::per_agent_partition([ (AgentId::new("planner")?, 0), (AgentId::new("worker")?, 1), ])) .idle_timeout(Duration::from_secs(600)) .memory_namespace("support.sessions"), ); sessions .bootstrap(2, TopicRetention::expire_after(Duration::from_secs(86_400))) .await?; ``` ```python sessions = laser.sessions( stream="support", layout=ls.SessionLayout.PerAgentPartition({"planner": 0, "worker": 1}), idle_timeout_ms=600_000, memory_namespace="support.sessions", ) await sessions.bootstrap(2, ls.TopicRetention.expire_after(86_400_000)) ``` The session factory takes its configuration differently in each language: Rust `laser.sessions_with(SessionConfig)`, TypeScript `laser.sessions(new SessionConfig())`, and Python keywords on `laser.sessions(...)`. `laser.sessions()` alone uses the defaults on the connection's default stream. | Setting | Default | Rust and TypeScript | Python keyword | | --------------------------- | ------------------------------- | ------------------------------------------ | --------------------- | | Stream | The connection's default stream | `stream` | `stream` | | Layout | Shared | `layout` | `layout` | | Idle timeout | 5 minutes | `idle_timeout` / `idleTimeout` | `idle_timeout_ms` | | Heartbeat | 60 seconds | `heartbeat` | `heartbeat_ms` | | Register the session source | On | `register_source` / `registerSource` | `register_source` | | Fail on dead letter | Off | `fail_on_dead_letter` / `failOnDeadLetter` | `fail_on_dead_letter` | | Memory namespace | `agent.session` | `memory_namespace` / `memoryNamespace` | `memory_namespace` | | Context turns | 50 | `context_turns` / `contextTurns` | `context_turns` | | Context tokens | 4,000 | `context_tokens` / `contextTokens` | `context_tokens` | ## Where each feature runs Starting and ending sessions, model and tool records, state, context, checkpoints, replay, `over_budget()`, `read_at`, submission, control, and pause work on plain Apache Iggy, on Laser Stack, and on LaserData Cloud. These need the managed session index on Laser Stack or LaserData Cloud: * `get`, `list`, `events`, `state`, `links`, `sources`, `changes`, `watch`, and `status()`. * Source registration during bootstrap. * Runtime budget enforcement before a handler runs. * Verified signers on timeline events. `kv(namespace)` and `linked_graph(name)` need the managed key-value and graph views. Default memory recall reads the managed key-value view, and folded recall works on plain Apache Iggy. In Rust, sessions are part of the `agent` feature. Python and TypeScript include them. ## Key operations | Verb | What it does | | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `sessions()` / `sessions_with(config)` | The session factory | | `bootstrap(partitions, retention)` | Create the agent topics, and register the stream when the server serves session reads | | `create(label)` / `start()` | A builder for a labeled or fresh session | | `.agent`, `.namespace`, `.parent`, `.with_id`, `.idle_timeout`, `.budget`, `.tag` | Builder settings | | `begin()` | Write the start record and return the session and its lease | | `open(id)` | A lens over an existing session | | `as_agent(id)` | A handle that writes as another agent | | `end()` / `fail(error)` / `cancel()` | Write the terminal record once, shared by every handle copy | | `run(lease, work)` | End the session by the outcome of `work`. Python also has `async with` | | `assemble(policy)` | The selected context and its manifest, nothing written | | `model(request, assembled)` / `tool(name, args)` | Record a call, then `complete` or `fail` it | | `record_model_call`, `record_retrieval`, `record_compaction` | Record facts after they happened | | `redact(fn)` | A handle with your own redactor | | `state()` | `set`, `patch`, `replace`, `snapshot`, and `get` on the session document | | `context()` / `context_with(policy)` | The lane's recent records with display types | | `memory()` / `linked_memory()` | Session memory, plain or stamped with lineage | | `kv(namespace)` / `linked_graph(name)` / `graph(name)` | Key-value and graph handles | | `reference()` / `acting_on(source)` | The session's stream and ID, and a handle that stamps a source record | | `append(envelope)` | Write one typed envelope on the lane | | `submit(agent, input)` | Hand a new session to an agent | | `control(stream, id)` | Operator `pause`, `resume`, `cancel`, and `force_cancel` | | `cancel_requested()` / `pending_control()` | Whether an operator asked to cancel, and the requests this agent recorded | | `participants(..)` / `pause()` / `resume()` | Pause a session for named agents and lift it | | `parked()` | Held work that no agent reported handled | | `over_budget()` | Whether the session passed its budget | | `signed_by(key)` | Sign the terminal record, or on the control handle every control record | | `checkpoint()`, `turns_at`, `turns_since`, `state_at`, `replay` | Bookmark and replay the lane | | `get`, `list`, `events`, `state`, `links`, `sources`, `changes`, `watch` | Managed session index reads on the factory | | `status()` | The managed summary of one session | | `read_at(source)` | Fetch the one record a source reference names | TypeScript uses camelCase names, for example `contextWith`, `recordModelCall`, `linkedGraph`, `cancelRequested`, `asOperator`, and `readAt`. Source: https://docs.laserdata.com/laser-sdk/advanced/sessions --- # Messages in depth The full reference for topics and messages. For a short introduction, start with [Messages](/laser-sdk/log). ## Topics, partitions, and routing A stream groups topics. A topic is a named log with no schema required by the SDK. Partitions store its messages in order. One connection reaches every stream. Address a topic with `laser.stream("telemetry").topic("metrics")`, or set a default stream and use `laser.topic("metrics")`. See [Connect](/laser-sdk/connect#default-stream). Order holds within a partition. There is no total order across a topic's partitions. A publish picks a partition in one of three ways: * Balanced, the default, spreads messages for throughput. Only messages in the same partition share an order. * Key routing hashes a non-empty key to a partition. Messages with the same key keep their relative order. * Partition routing names a partition number directly. More partitions allow more parallel consumers. Fewer partitions keep more messages in one ordered sequence. Use a host, agent, or session key when those messages need a shared order. Payloads are bytes. The raw publish adds no encoding. JSON, CBOR, MessagePack, BSON, and Avro helpers encode values into those bytes. TypeScript snippets on this page turn text into bytes with `new TextEncoder().encode(text)`. ## Typed handles and readers `topic.json(..)` returns a typed handle. Its `publish` encodes a value and its `records(name)` reader decodes each message back. `topic.cbor(..)` does the same in CBOR. Rust takes a type parameter, `topic.json::()`. Python takes a class, `topic.json(Reading)`. TypeScript takes a codec, `topic.json(new Json(decode))`, whose decode function checks each value because types do not exist at runtime. `Cbor`, `Msgpack`, and `Bson` take the same decode function. A Python typed `publish(body)` returns the publish builder, so finish with `.send()`. A Rust typed `publish(&value)` also returns a builder. A TypeScript typed `publish(value)` sends at once and takes `{ key }`, `{ partition }`, or `{ headers }` as a second argument. A typed reader is a cursor that you own. It is not consumer group delivery, and the server does not store its offsets. It starts at offset 0 unless you restore saved offsets: | | Rust | Python | TypeScript | | --------------------------------- | ------------------------------------------------------- | ----------------------------------------------------- | -------------------------------------------------- | | Read one record | `reader.next().await` | `await reader.next()` | `await reader.next()` | | At the current tail | `None` | `None` | `undefined` | | Read a batch | `reader.poll().await` | `await reader.poll()` | `await reader.poll()` | | Save offsets | `reader.offsets()`, a list with one entry per partition | `reader.offsets`, a list with one entry per partition | `reader.offsets`, a map from partition to `bigint` | | Resume | `records(name)?.from_offsets(saved)` | `records(name, from_offsets=saved)` | `(await records(name)).fromOffsets(saved)` | | Records per request and partition | 1000 | 1000 | 1000 | | Record position | `MessageId` | `MessageId` with `partition_id` and `offset` | `{ partitionId, offset }` | Change the request size with `batch(n)` in Rust and TypeScript and `records(name, batch=n)` in Python. One poll drains each partition to its tail, up to 10,000 records per partition, and returns the records in log timestamp order. The offsets are empty until the first poll or until you restore saved ones. Restoring replaces the offsets, it does not merge them. A record that does not decode fails with a decode error that names its log position, and the next read moves past it. Rust `next()` returns the error, TypeScript `next()` returns a result with `kind: "error"`, and Python `next()` raises `TypedDecodeError`. `poll()` returns it inline next to the good records in all three. A Python `async for` loop ends on the error, so loop over `next()` to skip it. Rust and TypeScript `stream()` and Python `async for` end once the reader is caught up. `topic.replay()` is the raw cursor with the same offset rules, across every partition. ## Producing at volume `publish()` builds one message, sends it, and waits for the result. It uses the connection's publish timeout and retries. See [Connect](/laser-sdk/connect#publish-timeouts-and-retries). For more throughput, pick a write path: * `publish_batch()` collects messages on the client and sends them in one request. You decide when the batch goes out. * `topic.producer()` creates a long-lived producer that sends each batch directly and retries failures. Use it as the default production path. * A producer in background mode queues sends and writes them from a worker. It gives the most throughput. See [Background producers](#background-producers). * `topic.batching()` returns a client-side accumulator that flushes on a record count, a byte size, or a timer. See [Batching producers](#batching-producers). ### Direct producer The Rust producer builder takes these options. Python takes the same values as keyword arguments, and TypeScript takes them as options. | Option | Default | What it does | | ----------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `batch_length(n)` | 1000 | Most messages in one server request. A larger `send_batch` is split into requests of this size, sent one after another | | `linger(d)` | 0 | Minimum gap between sends. A send waits out what is left of it since the previous one. Direct mode only | | `retries(n, interval)` | The connection's publish retries, 3 retries with a first delay of 250 ms | Resend attempts after a failure | | `retry_backoff(d)` | The connection's first retry delay | Changes the delay and keeps the retry count | | `routing(r)` | `Balanced` | Default routing for every send: `Routing::Balanced`, `Routing::key(k)`, or `Routing::Partition(n)` | | `create_stream(bool)` / `create_topic(bool)` | `true` | Create a missing stream or topic on init. Turn off to fail fast on a wrong name | | `partitions(n)` | 1 | Partition count when the producer creates the topic | | `expire_after(d)` / `never_expire()` | Server default | Message retention when the producer creates the topic. Setting both forms, or a zero value, fails with an invalid error before any network call | | `max_topic_bytes(n)` / `unlimited_topic_size()` | Server default | Size limit when the producer creates the topic. Setting both forms, or a zero value, fails with an invalid error before any network call | | `background(config)` | Direct mode | Switch to Apache Iggy's buffered mode. `batch_length` and `linger` are then ignored | Python's `topic.producer(..)` takes `batch_length`, `linger_ms`, `retries`, `retry_interval_ms`, `key` or `partition` for the default routing, `create_stream`, `create_topic`, `partitions`, `expire_after_ms` or `never_expire`, `max_topic_bytes` or `unlimited_topic_size`, and `background`. `linger_ms` and `expire_after_ms` take a float, so a fraction of a millisecond is kept. `max_topic_bytes` is a positive byte count. `send` and `send_batch` initialize the producer on first use. Call `await producer.init()` when startup must fail before the service accepts work. `send_batch` takes raw payloads or `(payload, headers)` pairs. TypeScript's `topic.producer(options)` takes `routing`, `retries`, `retryBackoffMs`, `createStream`, `createTopic`, `partitions`, `expireAfterMs` or `neverExpire`, `batchLength`, `lingerMs`, `maxTopicBytes` or `unlimitedTopicSize`, and `background`. `expireAfterMs` is a number of milliseconds and `maxTopicBytes` is a `bigint`. The producer needs no init call. It creates a missing stream and topic before the first send unless you turn that off. A transport that cannot set topic expiry or size limits refuses those options. This sample follows the [`native-streaming`](https://github.com/laserdata/laser-sdk/tree/main/examples/rust/src/native-streaming) example: ```ts import { HeaderValue } from "@laserdata/laser-sdk" const text = new TextEncoder() await using producer = topic.producer({ batchLength: 1_000, lingerMs: 5, retries: 3, retryBackoffMs: 1_000 }) await producer.send(text.encode("reading-0"), { key: text.encode("node-7"), headers: { type: HeaderValue.uint16(7) } }) const payloads = [text.encode("reading-1"), text.encode("reading-2")] await producer.sendBatch(payloads) ``` ```rust use laser_sdk::prelude::*; use laser_sdk::stream::{HeaderKey, HeaderValue}; use std::time::Duration; let producer = topic .producer() .batch_length(1_000) .linger(Duration::from_millis(5)) .retries(Some(3), Some(Duration::from_secs(1))) .routing(Routing::Balanced) .build() .await?; producer .send_keyed( ProducerMessage::new(b"reading-0".as_slice()) .header(HeaderKey::try_from("type")?, HeaderValue::from(7_u16)), b"node-7".to_vec(), ) .await?; let batch = vec![ ProducerMessage::new(b"reading-1".as_slice()), ProducerMessage::new(b"reading-2".as_slice()), ]; producer.send_batch(batch).await?; ``` ```python producer = topic.producer( batch_length=1000, linger_ms=5, retries=3, retry_interval_ms=1000, ) await producer.init() await producer.send( b"reading-0", headers={"type": ("uint16", 7)}, key=b"node-7", ) values = [b"reading-1", b"reading-2"] await producer.send_batch(values) ``` A send can override the producer's routing. Rust uses `send_keyed(msg, key)` or `send_to_partition(msg, n)`. Python's `send` takes `key=` or `partition=`. TypeScript's `send` takes `{ key }` or `{ partition }`. Do not pass a key and a partition together. Batches take the same override through Rust `send_batch_with_routing`, Python `send_batch(key=, partition=)`, and TypeScript `sendBatch(messages, { key })`. All records in one batch share the key or partition. Producers use the connection's publish retry count and delay unless you set them. In direct mode, retry delays start at the configured delay, 250 ms by default, double after each failure, and stop at 30 seconds. Only temporary errors are retried. In background mode the first resend goes at once, and later resends wait the configured delay each time without doubling. Background mode resends after any error except a lost confirmation, which means the batch may have been written. Each producer setup attempt also uses the publish timeout and retry budget. A producer that loses a race to create the same stream or topic treats it as success and reads the winner's resource. These are the producer-level overrides: | Goal | Rust | Python | TypeScript | | ----------------------------- | ------------------------ | --------------------------------------------------------- | ------------------------------------------ | | Inherit the connection values | Leave `retries` unset | `retries=None` and `retry_interval_ms=None`, the defaults | Leave `retries` and `retryBackoffMs` unset | | Disable resends | `retries(Some(0), None)` | `retries=0` | `retries: 0` | | Change only the delay | `retry_backoff(d)` | `retry_interval_ms=` with `retries=None` | `retryBackoffMs` without `retries` | In Rust direct mode, `retries(None, ..)` also means no retries. Creation settings apply only to topics that do not exist yet. A producer never changes an existing topic's expiry or size limit. A topic created by `topic.ensure(..)` never expires, and `ensure` leaves an existing topic as it is. Two producers that create the same topic at once both succeed in all three SDKs, because the one that loses the race accepts the existing topic. ### Background producers Background mode queues each send and writes it from a worker. A send returns once the queue accepts the records, so it carries no commit confirmation, and the publish timeout does not cover that step. Drain the queue before you exit, or the queued records are lost. All three SDKs take the same settings, with Apache Iggy's background defaults: one ordered shard, a flush at 1000 sends, 1 MiB, or the linger time, a 32 MiB buffer budget, and the `block` failure mode. Ordered sharding, the default, keeps every send of the producer on one shard in send order. Balanced sharding spreads sends over every shard and gives up that order. | Setting | Rust `BackgroundConfig::builder()` | Python `BackgroundConfig(..)` | TypeScript `background: { .. }` | | ---------------- | ------------------------------------------------- | --------------------------------------- | ------------------------------- | | Shards | `num_shards(n)` | `num_shards=` | `shards` | | Sharding | `sharding(Box::new(BalancedSharding::default()))` | `sharding="balanced"` | `sharding: "balanced"` | | Flush at sends | `batch_length(n)` | `batch_length=` | `batchLength` | | Flush at bytes | `batch_size(n)` | `batch_size=` | `batchBytes` | | Linger | `linger_time(d)` | `linger_ms=` | `lingerMs` | | Buffer budget | `max_buffer_size(n)` | `max_buffer_size=` | `maxBufferBytes` | | Writes in flight | `max_in_flight(n)` | `max_in_flight=` | `maxInFlight` | | Full buffer | `failure_mode(..)` | `failure_mode=` and `block_timeout_ms=` | `failureMode` | | Failed write | `error_callback(..)` | `error_callback=` | `onError` | Python selects the mode with `background=True` for the defaults or `background=BackgroundConfig(..)` to change them. Rust's builder takes Apache Iggy types: `linger_time` takes an `IggyDuration` (convert a `Duration` with `.into()`), `max_buffer_size` takes an `IggyByteSize`, `failure_mode` takes a `BackpressureMode` from `laser_sdk::iggy::clients::producer_config`, and `error_callback` takes an `ErrorCallback` from `laser_sdk::iggy::clients::producer_error_callback`. Python's `linger_ms` takes a float, so a fraction of a millisecond is kept. Close a background producer with `shutdown()`. It waits for sends in flight and writes every queued record before it closes. After shutdown, a send raises `InvalidError` in Python and TypeScript. A second `shutdown()` returns once the first one finishes. In Rust, `shutdown()` takes the producer by value and needs the last live handle. By default, Rust and Python log a failed background write and drop its records. Set the error callback to handle it instead. Rust's callback receives Apache Iggy's `ErrorCtx` with the cause, the stream and topic, the confirmed ranges, and the messages. A Python `error_callback` receives a `PublishFailedError` with `stream`, `topic`, `committed`, `unconfirmed`, and the cause as `__cause__`. It can be a plain or async callable. TypeScript hands each failure to `onError` and awaits a returned promise. Without `onError`, or when it throws, TypeScript keeps the failure and `shutdown()` rejects with one `PublishFailedError` that lists the records of every failed background batch. ```ts const text = new TextEncoder() const producer = topic.producer({ background: { shards: 2, lingerMs: 5, onError: (error) => console.error(error.publishCause()) } }) await producer.send(text.encode("reading-0")) await producer.send(text.encode("reading-1")) await producer.shutdown() ``` ```rust use laser_sdk::stream::BackgroundConfig; use std::time::Duration; let producer = topic .producer() .background( BackgroundConfig::builder() .num_shards(2) .linger_time(Duration::from_millis(5).into()) .build(), ) .build() .await?; producer.send(b"reading-0".as_slice()).await?; producer.send(b"reading-1".as_slice()).await?; producer.shutdown().await?; ``` ```python import laser_sdk as ls def report(error: ls.PublishFailedError) -> None: print(f"{len(error.unconfirmed)} record(s) unconfirmed: {error.__cause__}") producer = topic.producer( background=ls.BackgroundConfig(num_shards=2, linger_ms=5, error_callback=report) ) await producer.init() await producer.send(b"reading-0") await producer.send(b"reading-1") await producer.shutdown() ``` ### Batching producers `topic.batching()` returns a producer that queues records on the client and sends each flush as one append. A flush runs when the queue reaches `max_records`, when the queued payload reaches `max_bytes`, or when `linger` expires, whichever comes first. A linger below 1 ms is raised to 1 ms. With a `partition_key`, every record of the handle goes to that key. Without one, each flushed batch is balanced across partitions. Rust needs the `agent` feature, because the linger timer runs on tokio. | Setting | Default | Rust | Python | TypeScript | | ------------- | ------- | -------------------- | ---------------- | ------------------- | | Record limit | 512 | `max_records(n)` | `max_records=` | `maxRecords(n)` | | Byte limit | 1 MiB | `max_bytes(n)` | `max_bytes=` | `maxBytes(n)` | | Linger | 5 ms | `linger(Duration)` | `linger_ms=` | `linger(ms)` | | Partition key | None | `partition_key(key)` | `partition_key=` | `partitionKey(key)` | `send()` queues one payload with headers. Rust always takes a headers map, which can be empty. Python and TypeScript make headers optional. It flushes inline when a size limit trips, so backpressure lands on the sender. `flush()` sends what is queued, and `close()` flushes and stops the timer. Flushes run one at a time, so batches reach the log in queue order. After `close()`, a send raises `InvalidError` in Python and TypeScript. Rust's `close()` takes the producer by value. In TypeScript, `await using` closes the producer at the end of its scope. A failed timer flush never stops the timer. Its failure is kept until the next `send()`, `flush()`, or `close()` reports it. A `send()` that finds a kept failure does not queue its own record. It returns the kept failure with that record added to the unconfirmed records. `flush()` and `close()` report the kept failure after they drain the queue. When several flushes fail before you ask, you get one publish failure that lists the records of every failed batch. ```ts await using batching = topic .batching() .maxRecords(512) .maxBytes(1_048_576) .linger(5) .partitionKey("node-7") .build() const text = new TextEncoder() await batching.send(text.encode("reading-0")) await batching.send(text.encode("reading-1")) await batching.flush() ``` ```rust use laser_sdk::stream::Headers; use std::time::Duration; let batching = topic .batching()? .max_records(512) .max_bytes(1_048_576) .linger(Duration::from_millis(5)) .partition_key("node-7") .build(); batching.send(b"reading-0".to_vec(), Headers::new()).await?; batching.send(b"reading-1".to_vec(), Headers::new()).await?; batching.flush().await?; batching.close().await?; ``` ```python batching = topic.batching( max_records=512, max_bytes=1_048_576, linger_ms=5, partition_key="node-7", ) await batching.send(b"reading-0") await batching.send(b"reading-1") await batching.flush() await batching.close() ``` ### When a publish fails A publish that gives up reports what it left behind. `committed` lists the ranges the server confirmed, each with the stream and topic IDs, the partition, and the base offset. `unconfirmed` holds the records without a confirmation. Do not publish the confirmed ranges again. An unconfirmed record can already exist on the server if its acknowledgement was lost. Unconfirmed records keep the message IDs that the attempts used. A direct producer splits a large batch into requests of `batch_length` records. A failure reports the confirmed requests and every record from the failed request on. Inspect the report before you retry. | | Rust | Python | TypeScript | | ------------------- | -------------------------------------- | -------------------------------------- | --------------------------------------------------- | | Failure | `LaserError::PublishFailed` | `PublishFailedError` | `PublishFailedError` | | Confirmed ranges | `committed` | `committed` | `committed` | | Unconfirmed records | `unconfirmed`, a list of `IggyMessage` | `unconfirmed`, a list of `IggyMessage` | `unconfirmed` | | Target | `stream` and `topic` | `stream` and `topic` | `stream` and `topic` | | Original error | `publish_cause()` | `__cause__` | `publishCause()`, or the free `publishCause(error)` | | Retryable | `is_retryable()` | `retryable` | `isRetryable(error)` | The retryable check answers for the original error. To resend the unconfirmed records with their original message IDs, pass them back to `topic.batch(..)`. A server that deduplicates by message ID then drops any record that already landed. The records carry no routing, so the resend does not keep the original key or partition. ```ts import { PublishFailedError } from "@laserdata/laser-sdk" try { await producer.sendBatch(payloads) } catch (error) { if (!(error instanceof PublishFailedError)) throw error console.error( `${error.committed.length} range(s) confirmed, ${error.unconfirmed.length} record(s) unconfirmed`, error.publishCause() ) await topic.batch(error.unconfirmed) } ``` ```rust use laser_sdk::prelude::LaserError; if let Err(error) = producer.send_batch(batch).await { eprintln!("cause: {}", error.publish_cause()); if let LaserError::PublishFailed(failure) = error { eprintln!( "{} range(s) confirmed, {} record(s) unconfirmed", failure.committed.len(), failure.unconfirmed.len() ); topic.batch(failure.unconfirmed, None).await?; } } ``` ```python import laser_sdk as ls try: await producer.send_batch(values) except ls.PublishFailedError as error: print( f"{len(error.committed)} range(s) confirmed, " f"{len(error.unconfirmed)} record(s) unconfirmed", error.__cause__, ) await topic.batch(error.unconfirmed) ``` ## Consumers, offsets, and commit policies `topic.consumer(name, partition)` builds a named reader for one partition. `topic.consumer_group(group)` returns a group handle, and `group.consumer()` builds a reader that joins the group. The server gives each partition to one member and rebalances as members join and leave. In Python, `partition` is a keyword that defaults to 0. In TypeScript, `topic.consumer(name, partitionId, options)` builds the named reader and `await topic.consumerGroup(group).consumer(options)` builds the group reader. Every consumed message carries its log position in `position`, a partition and an offset. Read the offset as `message.position.offset`. The message also keeps the exact Iggy headers, the message ID, the checksum, and the timestamps. A malformed header block in TypeScript marks the record instead of failing the whole poll. Delivery is at least once. A crash between delivery and commit delivers the message again, so handlers must be safe to repeat. ### Two group paths A group consumer reads one of two ways. On a server that resolves consumer group policies, such as Laser Stack and LaserData Cloud, it reads through the group and acknowledges with fenced group commits. This page calls that the group-policy path. On plain Apache Iggy it is the native Iggy group consumer, called the native path here. A group-policy consumer always joins its group, and all three SDKs refuse `auto_join_group` set to false on that path (TypeScript `autoJoinGroup`). The group-policy path is also how [Filters](/laser-sdk/consumer-filters) run. `create_group` and `auto_join_group` default to true (Python `create_group=` and `auto_join_group=`, TypeScript `createGroup` and `autoJoinGroup`). The server stores offsets by consumer name or group. Reconnect with the same name to resume from the stored offset. ### Where a consumer starts `start_at(..)` picks the first position when no stored offset exists. The default is `Next`. * Rust takes `ConsumerStart::First`, `Last`, `Next`, `Offset(n)`, or `TimestampMicros(t)`. * TypeScript takes `startAt` with `{ kind: "first" | "last" | "next" }`, or `{ kind: "offset" | "timestampMicros", value }` where `value` is a `bigint`. * Python takes `polling="first" | "last" | "next"`, or `offset=` or `timestamp_micros=`. `allow_replay()` in Rust, `allow_replay=True` in Python, and `allowReplay: true` in TypeScript permit reads at or below a stored offset. Use it to replay a group from `First`. Without it, a native consumer skips records it already consumed. A group-policy consumer honors its start position as given. `Next` does not skip the first record of a fresh consumer. With no stored offset, it starts at the first retained record. With a stored offset of 0, it starts at offset 1. Storing offset 0 acknowledges that record. It is not an empty starting state. ### Commit and offset control `commit(message)` acknowledges a processed record. Native consumers also expose `store_offset(offset, partition)` and `delete_offset(partition)` for direct offset control (TypeScript `storeOffset(offset, partitionId?)` and `deleteOffset(partitionId?)`, Python `store_offset(offset, partition=)` and `delete_offset(partition=)`). Group-policy consumers refuse those two calls, because their acknowledgments must pass the group and policy checks. `last_consumed_offset(p)` and `last_stored_offset(p)` read local progress (TypeScript `lastConsumedOffset(p)` and `lastStoredOffset(p)`, awaited in Python). They are diagnostics, not a resume point. On the native path in Rust and Python, the Iggy cache can hold an initial zero before any commit. On the group-policy path, `last_stored_offset` reports the last acknowledged offset. Resume with the `Next` start instead of computing a position from them. `stored_offset(p)` (TypeScript `storedOffset(p)`) reads the stored offset and the partition's current offset from the server, or returns nothing when the server stores none yet. TypeScript refuses it on an unnamed partition consumer. A purge restarts every partition at offset 0. An open native consumer can keep its old position and skip the new records, so shut it down and build a new one with the same name or group. Group-policy consumers detect the changed history, reset partition progress, and report `source_changed` before the next read resumes. A group handle built from a numeric ID refuses a recreated stream or topic and needs a new reader. ### Commit policies The commit policy decides when the consumer stores offsets on the server: | Policy | Stores the offset | | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `Polling` (default) | Native consumers: the end of the polled batch, before delivery. Group-policy consumers: the handled prefix at the next poll or shutdown | | `All` | After consuming everything a poll returned | | `Each` | After every yielded record | | `Every(n)` | After every `n` yielded records | | `Interval(d)` | On a timer | | `IntervalOrPolling(d)` / `IntervalOrAll(d)` / `IntervalOrEach(d)` / `IntervalOrEvery(d, n)` | The interval or the event, whichever comes first | | `Disabled` | Never. You call `commit(message)` yourself | On the native path the default policy can skip unprocessed records after a crash, because it commits before delivery. Group-policy consumers acknowledge the delivered prefix when consumption continues or shuts down, so a crash delivers the current batch again. Use `Disabled` and commit after successful processing when only finished work may advance. Native consumers send an offset store, and group-policy consumers send a fenced acknowledgment. ```ts await using consumer = await topic.consumerGroup("workers").consumer({ batchLength: 100, commitPolicy: { kind: "disabled" }, startAt: { kind: "first" }, allowReplay: true, pollIntervalMs: 5 }) for await (const message of consumer) { handle(message) await consumer.commit(message) } ``` ```rust let mut consumer = topic .consumer_group("workers") .consumer() .batch_length(100) .poll_interval(Duration::from_millis(5)) .start_at(ConsumerStart::First) .allow_replay() .commit_policy(CommitPolicy::Disabled) .build() .await?; while let Some(message) = consumer.next().await { let message = message?; handle(&message)?; consumer.commit(&message).await?; } consumer.shutdown().await?; ``` ```python consumer = topic.consumer_group("workers").consumer( batch_length=100, poll_interval_ms=5, polling="first", auto_commit="disabled", allow_replay=True, ) await consumer.init() async for message in consumer: handle(message) await consumer.commit(message) await consumer.shutdown() ``` To commit automatically instead, drop the manual commits and set a policy: `commit_policy(CommitPolicy::IntervalOrEach(Duration::from_secs(1)))` in Rust, `auto_commit="each", commit_interval_ms=1000` in Python, or `commitPolicy: { kind: "intervalOrEach", intervalMs: 1_000 }` in TypeScript. In TypeScript, `consumer.stream({ signal })` takes an `AbortSignal`. Aborting it stops the loop with a `CancelledError`. ### Waiting with a timeout `next_within(timeout)` returns the next record, or fails when the time runs out. Rust takes a `Duration`, and Python and TypeScript take milliseconds. A timeout is a typed error: `LaserError::Timeout` in Rust and `TimeoutError` in Python and TypeScript. After shutdown, the call fails with an invalid-state error. TypeScript keeps a poll that outlives the timeout. The next call resumes that poll and can receive its late record. A timeout never starts overlapping polls and never acknowledges a record that the caller did not receive. ```ts import { TimeoutError } from "@laserdata/laser-sdk" try { const message = await consumer.nextWithin(5_000) handle(message) } catch (error) { if (!(error instanceof TimeoutError)) throw error // nothing arrived in five seconds } ``` ```rust match consumer.next_within(Duration::from_secs(5)).await { Ok(message) => handle(&message)?, Err(LaserError::Timeout(_)) => { // nothing arrived in five seconds } Err(error) => return Err(error), } ``` ```python import laser_sdk as ls try: message = await consumer.next_within(5_000) handle(message) except ls.TimeoutError: pass # nothing arrived in five seconds ``` ### Differences by language All three SDKs share these consumer defaults: `batch_length` 1000, the `Polling` policy, the `Next` start, a 1 second wait before a failed poll is retried, and group creation and join on build. All three also set the failed-poll wait and the init retries: Rust `polling_retry_interval` and `init_retries`, Python `polling_retry_interval_ms`, `init_retries`, and `init_retry_interval_ms`, TypeScript `pollingRetryIntervalMs` and `initRetries: { retries, intervalMs }`. These parts differ: * Rust sets one of the ten `CommitPolicy` variants with `commit_policy(..)`. Without `poll_interval`, a native consumer polls again at once after an empty poll. A group-policy consumer waits 250 ms. * Python picks a policy with `auto_commit`: `disabled`, `interval`, `polling` (the default), `all`, `each`, or `every`. `commit_interval_ms` defaults to 0, which keeps the plain policy. A positive value selects the interval variant, such as `IntervalOrEach`. `interval` needs a positive `commit_interval_ms`, and `every` needs `commit_every`. `poll_interval_ms` follows the Rust default. The consumer initializes on first read. Call `await consumer.init()` when startup must fail before the service accepts work. * TypeScript sets `commitPolicy` with the same ten variants as `{ kind, intervalMs, count }` objects, such as `{ kind: "disabled" }` or `{ kind: "every", count: 10 }`. `pollIntervalMs` defaults to 0 on native consumers, which poll again without a delay. `shutdown()` stops polling and leaves the group. Automatic policies store the handled prefix first. Native polling commits before delivery, so its shutdown does not prove that every record was processed. If a native consumer delivered offset 0 on a partition, automatic shutdown stores offset 0 explicitly unless an earlier store already covers it. With `Disabled`, progress follows your commits. Unnamed TypeScript consumers have isolated identities and default to no automatic commit. Use a stable name and an explicit commit policy to resume durable progress. ## Receive only matching records [Filters](/laser-sdk/consumer-filters) let one shared topic deliver a different subset to each consumer group. The server selects records before they cross the network. Configure the policy through `topic.consumer_group(name).filter()`. Normal group consumers and `group.reader()` apply it, and the group's progress still covers every record the server scanned. Raw Iggy consumers ignore the filter and receive every record, so do not mix them with Laser consumers on one group. ## Apache Iggy access Rust's `topic.iggy_producer()`, `topic.iggy_consumer(..)`, and `topic.iggy_consumer_group(group)` return native Iggy builders on the same connection, and `laser.client()` returns the client. Use them for settings that Laser SDK does not expose. TypeScript exposes the underlying client as `laser.client`. Python has no raw Iggy accessor. ## Durability Publish completion follows the topic's durability policy. A local queue that accepted a record does not prove the server stored it, and neither does a background send that returned. Message durability and consumer offset durability are independent. See [Topic durability](/deployments/server#topic-durability) and [Acknowledgements and offset commits](/deployments/server#acknowledgements-and-offset-commits). ## Producer statistics Producers in all three SDKs can report statistics over a separate observer connection. Set the variables before you create producers. A telemetry error never fails a publish. | Setting | Environment variable | Default | Range | | ------------------------------- | ---------------------------------------- | ------- | ---------------------------------------------- | | Report interval | `LASER_PRODUCER_TELEMETRY_INTERVAL_MS` | `10000` | `0` disables, otherwise `1000` through `60000` | | Reported handles per connection | `LASER_PRODUCER_TELEMETRY_MAX_PRODUCERS` | `32` | `0` through `32` | | Counter | Meaning | | --------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Submitted records and bytes | Each application send once, before retries. Payload bytes only, no headers or framing | | Confirmed records and bytes | Fully successful synchronous send calls. Unavailable for background sends | | Failed calls | Send calls that returned an error | | Last success | The last successful application call | | Retries | Unavailable when the native client does not expose its attempts | | Latency p50, p99, p99.9 | Publish-call time including retry delays, as power-of-two bucket upper bounds. p99 needs 100 calls, p99.9 needs 1000 | * A failed batch can have committed a prefix that the confirmed counters leave out. The counters are SDK reports. Do not use them as delivery totals, exactly-once evidence, usage metering, or authorization input. * A value outside its range is clamped to the range. A report expires after three intervals. Handles past the limit publish normally and are left out. A producer ID stays stable for the life of its handle, across reconnects. * A Laser built from an injected client that cannot open a second authenticated connection does not report. Reports also need a server that answers the AGDX hello, so plain Apache Iggy gets none. * The Producers tab of a topic in Stream UI shows the connected node only. The observer node can differ from the publishing node. Unknown values show as unavailable, never as zero. ## Key operations | Operation | What it does | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `stream(name).topic(name)` | Address a topic on any stream | | `topic.ensure(partitions)` | Create the topic if it does not exist, with a fixed partition count and no message expiry | | `publish().payload(bytes)` | Append one raw message | | `.json(v)` / `.msgpack(v)` | Encode the body before publish | | `.avro(schema, schema_id, v)` | Encode a body as an Avro datum under a registered writer schema. Rust needs the `schema-codecs` feature | | `.claim_check(store, threshold)` | Store a large body in a blob store and publish a reference (TypeScript `claimCheck`). Rust needs the `agent` feature | | `.partition_key(key)` | Route by key for per-key order | | `.index(key, value)` | Stamp an indexed text header, so a [view](/laser-sdk/views) can query it | | `.header(key, value)` | Attach a text header that stays on the record and is not queryable. Use `topic.producer()` for typed headers | | `publish_batch()` | Collect several messages into one request | | `topic.json(..)` / `topic.cbor(..)` | A typed handle. `publish` encodes, `publish_batch` encodes many, `records(name)` opens a typed cursor | | `topic.schema(id)` | A typed handle bound to a registered schema. Async in all three SDKs. Needs Laser Stack or LaserData Cloud, and Rust needs the `schema-codecs` feature | | `topic.producer()` | A long-lived producer with batching, linger, and retries. See [Producing at volume](#producing-at-volume) | | `topic.batching()` | A client-side accumulator with `flush()` and `close()`. See [Batching producers](#batching-producers) | | `topic.consumer(..)` / `topic.consumer_group(name).consumer()` | A live consumer, partitioned across group members | | `consumer.next()` / `next_within(timeout)` | Wait for the next record, optionally with a typed timeout | | `consumer.commit(message)` | Commit after the record is handled | | `store_offset(..)` / `delete_offset(..)` | Native consumers only | | `topic.replay()` | A raw cursor from offset 0, or from saved offsets, across every partition | An indexed header wins over a value that the projection schema extracts for the same field. A record with no indexed fields from headers or the schema produces no view row. [Queries and views](/laser-sdk/views) explains extraction and projection registration. A typed publish through `topic.schema(id)` encodes Avro and JSON Schema bodies. For Protobuf, Rust and Python publish the encoded bytes with the schema ID, and TypeScript encodes the body. `BatchPublishRequest.add_record` in Python takes per-record metadata. Batch defaults fill what the record leaves unset, and batch index entries and headers merge with the record's own, which win. Its `logical_schema_fingerprint=` parameter takes 32 bytes and needs Arrow content. TypeScript exposes the same metadata through `Record.logicalSchemaFingerprint(bytes)`. Use the Arrow IPC helpers when the SDK must check metadata and payload length. Source: https://docs.laserdata.com/laser-sdk/advanced/log --- # Filters in depth The full reference for consumer group filters. For a short introduction, start with [Filters](/laser-sdk/consumer-filters). ## How a filter works A consumer filter selects records on the streaming server, before they cross the network. A reader asks for a page of one partition. The server scans it and returns only the records the filter selects. Each record keeps its original offset, ID, timestamp, headers, and payload bytes. Everything else stays on the server, and the log stays the single source of truth. A filter belongs to a consumer group: `Laser`, then stream, topic, consumer group, filter. You create the group once and give it a filter. Every Laser SDK consumer of that group then runs it, and a consumer names only the group. A group with no filter receives every record. One topic can serve many selective groups without an intermediate topic per subset. Raw Iggy polling ignores the filter. Rust's `topic.iggy_consumer_group(..)`, the Iggy CLI, and any native Iggy client poll the same group with standard semantics and receive every record. The filter is a delivery policy for Laser SDK consumers, not an access control boundary, and a native source-read grant still permits native consumption. Do not mix the two on one group. A native poll stores the group's offset past records that the filtered consumers never saw, and they resume after it. Savings depend on the share of payload bytes that match. Suppose ten consumers each need 5% of a topic's payload bytes: | Payload from one 1 GiB source range | Read everything, filter in each application | Filter on the server | | ----------------------------------- | ------------------------------------------- | -------------------- | | One consumer | 1 GiB | 51.2 MiB | | Ten consumers combined | 10 GiB | 512 MiB | | Payload transfer saved | 0% | 95% | This illustrates payload transfer only, without protocol overhead. The server still examines records for each group. Measured results: * In the [CDC example](https://github.com/laserdata/laser-sdk/tree/main/examples/rust/src/cdc), the reader receives 4 of 240 records and 424 of 27,953 payload bytes, 98.5% less payload transfer. Rust, Python, and TypeScript produce the same result. * The [Frostline example](https://github.com/laserdata/laser-example-frostline) publishes ten million change records, 6.30 GB of payload, and reads them with four filtered groups. In its recorded run, made before SDK 0.5.1, the groups receive 777.64 MB, while reading the full feed four times moves 25.21 GB, 96.9% of payload avoided. Counting both TCP directions with `strace` gives 1.11 GB against 31.40 GB, 96.5% less application traffic, without TCP/IP headers and retransmissions. The repository holds the benchmark, latency tables, server CPU, and memory results. Header predicates can skip payload decoding. Payload predicates add server work while they cut downstream bytes and application work. ## What a filter can test A filter is declarative. It never runs your code. | Test | Operators | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Payload field comparison | `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `contains`, `prefix` | | Text matching on a field or a header | `equals`, `prefix`, `suffix`, `contains`, `glob`, `regex`, each optionally case-insensitive | | Typed header comparison | The same comparisons, by exact header key, without decoding the payload | | Field presence | `present`, `absent` | | Composition | `all`, `any`, `not` (built with `negate`) | | Explicit coercion | `pred_as` (TypeScript `predAs`) compares a timestamp (RFC 3339, epoch seconds, milliseconds, or microseconds) as an instant, or a decimal string as a number | A payload field is named by a path. Dots separate object keys, and `[n]` selects an array item, as in `after.ground_stations[0]`. Escape a literal `.`, `[`, `]`, or `\` inside a key with a backslash. A digit-only key such as `codes.200` stays a key. A path holds up to 16 segments and 256 bytes. The payload codec is part of the filter. JSON, CBOR, Avro, and Protobuf support field filtering. `headers_only` never decodes the payload, so it works with any format. Avro and Protobuf need registered writer schemas. Truth has three values. Most comparisons with a missing field or a type mismatch return unknown, and `not` keeps unknown. An unknown result selects nothing, unless the mismatch policy asks to see type mismatches. Null comparisons are the exception: `eq null` matches a missing or explicit-null field, and `ne null` excludes both. Use `present` and `absent` when the difference matters. ### Faults A payload that cannot be decoded is a fault, which is separate from truth. The filter's fault policy decides what happens: | Policy | What happens to the record | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `stop` (default) | The page stops before it. The reader raises the fault and reads that partition again after its idle interval, so the block clears once the record ages out of the partition. To move past it sooner, create a reader with an explicit start position or use another policy | | `pass` | Delivered and marked as not evaluated | | `drop` | Skipped | Set it with `with_fault_policy` (TypeScript `ConsumerFilter.withFaultPolicy(filter, policy)`). The Python and TypeScript constructors also take it as their last argument. ### Foreign and mismatched records Two record policies decide what happens to records a filter cannot judge on its own terms. Both default to `reject`. | Policy | Covers | `reject` (default) | `pass` | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ------------------------------------- | | `foreign_policy` | A record whose `agdx.ct` header names another codec than the filter's, or whose writer schema is missing, not listed by the filter, or of another schema family | Skipped. Never decoded and never a fault | Delivered and marked as not evaluated | | `mismatch_policy` | A record where a predicate's value exists but has a type the predicate cannot compare, for example `xyz == "Abc"` against `"xyz": 1`, an object, or an array | Skipped | Delivered and marked as not evaluated | A missing or null field is not a mismatch. It is plain unknown and the mismatch policy never delivers it. A record without an `agdx.ct` header is decoded with the filter's codec. Content type `any` and codes a server does not know also decode as usual. A payload that is broken in the filter's own codec is still a fault and follows the fault policy. A group reader marks a record delivered under `pass` with `evaluated` set to false, so a consumer can quarantine it, log it, or apply its own rules. Every record of an unbound group also has `evaluated` set to false, because nothing was evaluated. Records from the normal consumer carry no such flag, so use the group reader when you need to tell them apart. ```ts import { ConsumerFilter, FilterExpr } from "@laserdata/laser-sdk" const strict = ConsumerFilter.json(FilterExpr.pred("xyz", "eq", "Abc")) const seeEdgeCases = ConsumerFilter.withMismatchPolicy(strict, "pass") ``` ```rust use laser_sdk::filters::{ConsumerFilter, FilterExpr, RecordPolicy}; use laser_sdk::query::CmpOp; let see_edge_cases = ConsumerFilter::json(FilterExpr::pred("xyz", CmpOp::Eq, "Abc")) .with_mismatch_policy(RecordPolicy::Pass); ``` ```python import laser_sdk as ls strict = ls.ConsumerFilter.json(ls.FilterExpr.pred("xyz", "eq", "Abc")) see_edge_cases = strict.with_mismatch_policy("pass") ``` ### Digest Every filter has a digest, a SHA-256 hash over a fixed domain tag and the filter's JSON encoding. Two filters with the same digest behave the same way. Equivalent expressions written differently get different digests. A group remembers the digest it was configured with. Read it with `digest()` in Rust, the `digest` property in Python, or `ConsumerFilter.digest(filter)` in TypeScript. ## Define a filter Build the expression, then wrap it in the codec it reads. This filter follows a satellite fleet from the CDC example. It selects a producer-reported mode change to safe, a decommission, or a telemetry report of safe mode. ```ts import { ConsumerFilter, FilterExpr } from "@laserdata/laser-sdk" const satellites = () => FilterExpr.pred("table", "eq", "satellites") const transitionsOnly = FilterExpr.all([ satellites(), FilterExpr.pred("op", "eq", "u"), FilterExpr.pred("changed", "contains", "mode"), FilterExpr.pred("after.mode", "eq", "safe") ]) const safeMode = ConsumerFilter.json( FilterExpr.any([ transitionsOnly, FilterExpr.all([satellites(), FilterExpr.pred("op", "eq", "d")]), FilterExpr.all([ FilterExpr.pred("event", "eq", "satellite.telemetry_changed"), FilterExpr.pred("fields.mode", "eq", "safe") ]) ]) ) ``` ```rust use laser_sdk::filters::{ConsumerFilter, FilterExpr}; use laser_sdk::query::CmpOp; let satellites = || FilterExpr::pred("table", CmpOp::Eq, "satellites"); let transitions_only = FilterExpr::all([ satellites(), FilterExpr::pred("op", CmpOp::Eq, "u"), FilterExpr::pred("changed", CmpOp::Contains, "mode"), FilterExpr::pred("after.mode", CmpOp::Eq, "safe"), ]); let safe_mode = ConsumerFilter::json(FilterExpr::any([ transitions_only.clone(), FilterExpr::all([satellites(), FilterExpr::pred("op", CmpOp::Eq, "d")]), FilterExpr::all([ FilterExpr::pred("event", CmpOp::Eq, "satellite.telemetry_changed"), FilterExpr::pred("fields.mode", CmpOp::Eq, "safe"), ]), ])); ``` ```python import laser_sdk as ls def satellites() -> ls.FilterExpr: return ls.FilterExpr.pred("table", "eq", "satellites") transitions_only = ls.FilterExpr.all([ satellites(), ls.FilterExpr.pred("op", "eq", "u"), ls.FilterExpr.pred("changed", "contains", "mode"), ls.FilterExpr.pred("after.mode", "eq", "safe"), ]) safe_mode = ls.ConsumerFilter.json( ls.FilterExpr.any([ transitions_only, ls.FilterExpr.all([satellites(), ls.FilterExpr.pred("op", "eq", "d")]), ls.FilterExpr.all([ ls.FilterExpr.pred("event", "eq", "satellite.telemetry_changed"), ls.FilterExpr.pred("fields.mode", "eq", "safe"), ]), ]) ) ``` Group reads, tests, previews, and filter administration are part of the default Rust `streaming` feature. The `filters` Cargo feature, also included by `managed`, adds only the local evaluator and the reader's local guard: ```toml laser-sdk = { version = "0.7", features = ["filters"] } ``` Create Laser from a connection string, so the SDK can open authenticated connections to the serving nodes. A raw client you supply without connection settings supports local diagnostics only, without acknowledgments. ## Configure once, consume by group ID Create the group with its filter once. Consumers then name only the group, by name or by native ID. The server runs the group's filter, or none, on every read. Several instances of the same group share partition assignments and progress. Different groups keep separate progress and can each receive their own copy of the same matching records. ```ts const topic = laser.stream("orbit").topic("fleet_changes") const desk = topic.consumerGroup("anomaly-desk") // One-time setup. Omit filter to create a group that reads everything. const info = await desk.create({ filter: safeMode }) const groupId = info.id // Consumer instances only need the topic and group ID. const consumer = await topic.consumerGroupId(groupId).consumer({ commitPolicy: { kind: "disabled" }, batchLength: 100 }) try { for await (const record of consumer) { // FleetChange is the record union of the complete example. const change = record.json() console.log(record.position.partitionId, record.position.offset, change) await consumer.commit(record) } } finally { await consumer.shutdown() } ``` ```rust use laser_sdk::prelude::CommitPolicy; let topic = laser.stream("orbit").topic("fleet_changes"); let desk = topic.consumer_group("anomaly-desk"); // One-time setup. Omit .filter(..) to create a group that reads everything. let info = desk.create().filter(safe_mode.clone()).build().await?; let group_id = info.id; // Consumer instances only need the topic and group ID. let mut consumer = topic .consumer_group_id(u64::from(group_id)) .consumer() .batch_length(100) .commit_policy(CommitPolicy::Disabled) .build() .await?; while let Some(record) = consumer.next().await { let record = record?; // FleetChange is the serde model of the complete example. let change: FleetChange = record.json()?; println!("{} {} {change:?}", record.position.partition_id, record.position.offset); consumer.commit(&record).await?; } consumer.shutdown().await?; ``` ```python topic = laser.stream("orbit").topic("fleet_changes") desk = topic.consumer_group("anomaly-desk") # One-time setup. Omit filter to create a group that reads everything. info = await desk.create(filter=safe_mode) group_id = info.id # Consumer instances only need the topic and group ID. consumer = topic.consumer_group_id(group_id).consumer( auto_commit="disabled", batch_length=100 ) try: async for record in consumer: print(record.position.partition_id, record.position.offset, record.json()) await consumer.commit(record) finally: await consumer.shutdown() ``` `create` is idempotent. An existing group is kept, and the same definition keeps its binding. A different definition on a group that already runs one fails with `conflict`. Create another group instead, so two selections never share one set of offsets. You can also create a plain group first and call `group.filter().configure(filter)` later. The definition is saved as the group's own filter in the catalog and the group is bound to it in one step. The first definition becomes revision 1. A later matching definition reuses its revision. A released group, or a group whose own filter was deleted, can be configured again only with the digest it ran before. Only a group that never had a binding takes any filter. Native group creation and catalog configuration are two operations. If configuration fails after the group exists, Rust returns `LaserError::ConsumerGroupSetup`, TypeScript throws `ConsumerGroupSetupError`, and Python raises `ConsumerGroupSetupError`, a `FilterError`. Each carries the group's ID, name, and identity (`group_id`, `group_name`, and `identity` in Python) and the reason the catalog gave, such as `conflict`. The group stays as it was. In Rust the reason sits on the error's `source`, in TypeScript read it with `filterReason(error.cause)`, and in Python it is `reason`. Pass an `operation_id` (TypeScript `operationId`) you recorded first to resume the same configuration after a crash and read its first outcome. ### What a group consumer needs On a server that serves filters, a group consumer needs the `group_policy_reads` capability. There it runs whatever the group holds when it reads, a filter or none. On original Apache Iggy, the same call builds the native group consumer. A server that serves filters but not group reads refuses the build with `unsupported` and asks for an upgrade. A capability probe that never answered fails the build with a retryable error. The SDK never falls back to a native read that ignores the group's policy. Python builds the consumer without awaiting it, so call `await consumer.init()` when startup must fail on any of these errors. ## What a batch of 100 means A normal consumer with batch length 100 examines at most 100 source records per partition request and delivers zero through 100 records. If seven match, you receive seven. If none match, the SDK keeps the scan position and continues through the backlog. An empty poll does not mean the topic is empty. On the normal consumer, `batch_length` is both the page size and the scan budget, so one poll never costs the server more than one batch of source records. A selective filter over a long backlog therefore takes many polls to reach the first match. On a group-aware server, the batch length runs from 1 to 1000. When you want the server to scan further per request, use the group reader. ### Group reader The group reader separates the two limits. `count` bounds delivered matches and `max_examined` bounds scanned source records, within the server's own limits. `count` defaults to 100 and runs from 1 to 1000. Without `max_examined`, the server's own scan budget applies. The reader returns pages, acknowledges explicitly, and reads every record when the group is unbound. Use `next_record` for a record loop or `next_page` for batch handling (TypeScript `nextRecord` and `nextPage`). Both wait until a record arrives, so bound the wait yourself. TypeScript takes `timeoutMs`, and in Rust and Python wrap the call in a timeout. ```ts const reader = await desk .reader() .start({ kind: "first" }) .count(100) .maxExamined(1000) .build() try { const page = await reader.nextPage({ timeoutMs: 15_000 }) for (const record of page.records) { console.log(record.partitionId, record.offset, record.json()) } await reader.ackPage(page) } finally { await reader.close() } ``` ```rust use laser_sdk::filters::FilteredStart; let mut reader = desk .reader()? .start(FilteredStart::First) .count(100) .max_examined(1000) .build() .await?; let page = reader.next_page().await?; for record in &page.records { let change: FleetChange = record.json()?; println!("{} {} {change:?}", record.partition_id, record.offset); } reader.ack_page(&page).await?; reader.close().await?; ``` ```python reader = await desk.reader(start="first", count=100, max_examined=1000) try: page = await reader.next_page() for record in page.records: print(record.partition_id, record.offset, record.message.json()) await reader.ack_page(page) finally: await reader.close() ``` A page is one bounded filtered poll result, with matching records and scan progress. It uses standard Iggy transport and leaves stored records unchanged. For batch handling, process every returned record before `ack_page` (TypeScript `ackPage`). For record handling, call `ack(record)` after processing. The SDK advances stored progress only through the completed prefix. Acknowledge the record or page object that the reader returned. In TypeScript, a copy of its payload in another task does not carry the acknowledgment. `try_next_page()` returns at once when nothing is new. ## Safe progress and policy changes A fresh group using `Next` starts at the first retained record, including offset 0. After offset 0 is acknowledged, `Next` resumes at offset 1. An unset offset and a stored offset of zero are different states, so do not initialize a new group by storing zero. `Next` is the default start. Pick another with `start_at(ConsumerStart::First)` in Rust, `polling="first"` in Python, and `startAt: { kind: "first" }` in TypeScript. Acknowledgments cover finished work, including records the server skipped. A page that examines records reports a safe offset, the last record it finished classifying. When you acknowledge the page, the server stores that offset, skipped records included. A group that matches one record in a million still moves forward. A page that examines offsets 0 through 99 and returns matches at 10 and 20 can store progress through 99 once both are done. It cannot cross an earlier delivery that is not yet handled. On the normal consumer, the commit policy decides when the handled prefix is stored. `Polling` (the default) stores it at the next poll and at shutdown, `Interval` on a timer, `Each`, `Every(n)`, and `All` after deliveries, and `Disabled` only when you call `commit(record)`. Python names them with `auto_commit` (`"polling"`, `"interval"`, `"each"`, `"every"`, `"all"`, or `"disabled"`) plus `commit_interval_ms` and `commit_every`. TypeScript takes `commitPolicy`, such as `{ kind: "disabled" }`. Nothing is stored before a record reaches your code, so a crash delivers the batch you were handling again instead of skipping it. Use `Disabled` plus `commit` when processing must finish before progress is stored. The server fences every acknowledgment by source incarnation, group identity, execution mode, and policy generation. An old unfiltered page cannot commit after a filter is configured. An old filtered page cannot commit after the group is released. A purge or a recreated topic invalidates old delivery handles. A revoked partition can still drain accepted pages while its commit fence permits it. A permission change takes effect inside a page. The server checks source read permission before every read round. A revocation between rounds fails the request and returns no partial page. The group's policy resolves once when the request starts, so a catalog change applies from the next request. An acknowledgment checks offset store permission when it runs. The SDK rejects a page or record from another reader and checks response identity and progress before it shows them to you. It reads ahead and lets you finish records in any order. It stores only the offset up to which every returned record is done. Pages with no matches store their scanned range on their own. A reader keeps at most 1024 unacknowledged pages with records per partition and stops reading that partition until you acknowledge, so read-ahead memory stays bounded. Change the bound with `max_unacked_pages` in Rust and Python or `maxUnackedPages` in TypeScript. Zero is refused: TypeScript throws when you set it, Rust fails at `build()`, and Python fails when you await `reader(..)`. Empty pages do not count. Reads go to the partition primary by default. That is the only mode that can acknowledge. The `local` read mode serves diagnostics from whatever replica the connection reaches. Set it with `read_mode(ReadMode::Local)` in Rust, `read_mode="local"` in Python, and `readMode("local")` in TypeScript. A local page offers no acknowledgment offset, because a lagging replica can still hold records from before a purge. A group reader joins over its own connection, so two readers of one Laser are two members, and a reader dropped without `close` leaves the group with its connection. A partition the group hands a reader later resumes after the group's stored offset, whatever start the reader was built with, so the previous owner's unacknowledged backlog is read, not skipped. A partition that fails waits one idle interval while the other partitions keep reading. A read never falls back to unfiltered delivery when a filter should apply. A failed catalog lookup, a node that cannot prove its catalog is current, and a paused revision all fail the read. With the plane disabled, a group read is unfiltered only when the server holds no catalog history and the read carries no required catalog position. Otherwise it fails with `catalog_unavailable`. ### When the policy changes A group keeps running when its policy changes. Its consumers and readers stay joined and keep their place. When the server reports that the policy moved, the partition restarts from its stored offset and the next read runs whatever the group holds now. | Change | What running consumers do | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | Configure a filter on a group that never had one | The next read runs the filter | | Release or delete the filter | The next read returns every record | | Pause the active revision | New reads fail with `revision_disabled`. Records already delivered can still be acknowledged. Enable the revision again to resume | | Draft a revision | Nothing changes. The group keeps running the revision it is bound to | Records delivered under the old policy but not yet stored arrive again from the stored offset, so nothing is skipped. Only an explicit `commit` or `ack` of a record read under the old policy fails, with `conflict`, because the server never stored that offset. ### Checkpoints inside a page Persist your application checkpoint before you acknowledge its offset. `ack_through(record)` (TypeScript `ackThrough(record)`) marks every earlier returned record on that partition as handled and stores the completed prefix. Process that full prefix first. Later records in the same page stay pending. Use it when a window receipt or a transaction ends inside a page. Use `ack_page` when the whole page is done. ## Cluster freshness Configuration outcomes carry a durable operation receipt and a control log position. The group handle remembers its last configuration, also across a cloned handle, and every read built from it carries that position. The serving plane confirms that operation before it answers, so an application never sees its own configuration as missing through another node. A read from a group that nobody configured recently can hit a cached unbound answer. `IGGY_PLANE_FILTERS_UNBOUND_CONFIRM_MS` bounds how long a server shard reuses that answer before it asks the plane for a lookup confirmed against the control log head. The default is 1000 ms, the maximum is 60,000 ms, and zero confirms every read. A filter configured through another node takes effect within that window at the latest. It is not an instant cross-node guarantee. Acknowledgments confirm the current policy again. A node that cannot prove its catalog is current returns the retryable `catalog_unavailable`. Cluster membership, primary routing, and replication keep their native Iggy behavior. ## What a change record proves Filters judge each record on its own. They do not compare it with an earlier version of the entity. In the example, the producer supplies `changed`, a list of columns that changed. The transition branch needs both `changed` containing `mode` and `after.mode` equal to `safe`. A battery update from a satellite already in safe mode does not match that branch. A filter on values only selects it. Choose that kind of filter when you need current state rather than evidence of a transition. The telemetry branch selects a report of safe mode. Repeated or forced telemetry reports can match even when the mode did not change. The decommission branch selects deletions whatever the mode. Keep these producer semantics in mind when you adapt the example to your feed. ## Test and preview Check the group's filter before you depend on it. Neither call joins the group or stores an offset. * `test` judges one sample payload you supply and explains every predicate. * `preview` judges stored records in one partition and lists each verdict. Its first argument is the partition ID. It takes `from_offset`, `max_examined`, `max_records`, and `explain` (TypeScript `fromOffset`, `maxExamined`, `maxRecords`, `explain`). `max_records` defaults to 20. Pass `explain` to list rejected records too. ```ts const tested = await desk.filter().test(JSON.stringify(sample)) console.log(tested.explanation.verdict) const preview = await desk.filter().preview(0, { maxRecords: 10 }) console.log(preview.examined, preview.matched, preview.stop) ``` ```rust let payload = serde_json::to_string(&sample).expect("the sample serializes"); let tested = desk.filter().test(payload, Vec::new()).await?; println!("{:?}", tested.explanation.verdict); let preview = desk.filter().preview(0).await?.max_records(10).send().await?; println!("{} {} {:?}", preview.examined, preview.matched, preview.stop); ``` ```python tested = await desk.filter().test(json.dumps(sample)) print(tested["explanation"]["verdict"]) preview = await desk.filter().preview(0, max_records=10) print(preview["examined"], preview["matched"], preview["stop"]) ``` `stop` says why the scan ended: `filled`, `budget`, `end_of_visible`, `fault`, or `oversized_record`. Python returns catalog and preview replies as dicts. `test` also takes typed headers, as `FilterHeader` values in Rust and TypeScript and a `headers=` dict in Python, so a headers-only filter can be checked the same way. ### Local evaluator The local evaluator gives the same verdict without a server. Compile the filter once with `CompiledFilter.compile(filter)`, then call `evaluate` and `explain` on each record. Avro and Protobuf need the schemas at compile time: a `schemas` argument in Python and TypeScript, and `compile_with_schemas` in Rust. Python takes `evaluate(payload, headers)`. TypeScript takes a `{ payload, headers }` record. Rust uses `CompiledFilter` from `laser_sdk::filters`, which needs the `filters` feature, and its `evaluate` and `explain` also take `&DecodeLimits`. `evaluate_with_fault` (TypeScript `evaluateWithFault`) also returns the fault reason when the evaluator cannot judge a record. ### Local guard The local guard checks the server's work. A reader built with the guard evaluates every returned record again with the shared evaluator and fails the page on any disagreement, before your code sees it. Rust calls `local_guard(true)`, Python passes `local_guard=True`, and TypeScript calls `localGuard(true)`. The server sends its decoder bounds so the guard can reproduce a size or depth fault. A record marked as not evaluated without the matching `pass` policy is rejected. Older servers without these bounds cannot verify pass-through decode faults locally. ## Filter on headers only A `headers_only` filter reads typed user headers and never touches the payload. Use it when producers stamp a routing header and the payload is Protobuf, Avro, or any other format. The pager group below receives only critical alert frames. ```ts import { ConsumerFilter, FilterExpr, HeaderValue } from "@laserdata/laser-sdk" const alerts = laser.stream("orbit").topic("fleet_alerts") await alerts.producer().send(frame, { headers: { priority: HeaderValue.uint8(2) } }) const pager = alerts.consumerGroup("pager") await pager.create({ filter: ConsumerFilter.headersOnly(FilterExpr.header("priority", "eq", 2)) }) const reader = await pager.reader().start({ kind: "first" }).build() const record = await reader.nextRecord({ timeoutMs: 15_000 }) console.log(record.offset, record.message.payload.byteLength) await reader.ack(record) await reader.close() ``` ```rust use laser_sdk::filters::{ConsumerFilter, FilterExpr, FilteredStart}; use laser_sdk::query::CmpOp; use laser_sdk::stream::{HeaderKey, HeaderValue, ProducerMessage}; let alerts = laser.stream("orbit").topic("fleet_alerts"); let producer = alerts.producer().build().await?; let message = ProducerMessage::new(frame) .header(HeaderKey::try_from("priority")?, HeaderValue::from(2_u8)); producer.send_message(message).await?; let pager = alerts.consumer_group("pager"); pager .create() .filter(ConsumerFilter::headers_only(FilterExpr::header("priority", CmpOp::Eq, 2_i32))) .build() .await?; let mut reader = pager.reader()?.start(FilteredStart::First).build().await?; let record = reader.next_record().await?; println!("{} {}", record.offset, record.message.payload.len()); reader.ack(&record).await?; reader.close().await?; ``` ```python alerts = laser.stream("orbit").topic("fleet_alerts") producer = alerts.producer() await producer.send(frame, headers={"priority": ("uint8", 2)}) pager = alerts.consumer_group("pager") await pager.create( filter=ls.ConsumerFilter.headers_only(ls.FilterExpr.header("priority", "eq", 2)) ) reader = await pager.reader(start="first") record = await reader.next_record() print(record.offset, len(record.message.payload)) await reader.ack(record) await reader.close() ``` `frame` is your encoded payload. Header keys are text names matched exactly, including dots. Values can be booleans, signed or unsigned integers up to 64 bits, floating-point numbers, or strings. Integers compare by value across widths, so a `uint8` header matches the integer `2`. Numeric `2` and text `"2"` are different values. Pick an exact-width Iggy header when space matters: `uint8` uses one value byte. Use the streaming producer for typed headers. The `publish().header(..)` shortcut takes strings. Non-text keys and 128-bit numeric comparisons are outside the filter contract. ### Agent groups Agents use this kind of filter without any setup. Each agent ID has its own consumer group. On `agent.sessions` and `agent.control`, the agent runtime binds that group to the headers-only filter `agdx.to In [, "*"]`. It does so when the server serves filtered reads, group policies, and the catalog, and the client was built from a connection string. On a server without filters, the group stays unbound and the agent sorts records on the client. A server that serves filters but not group reads makes the agent fail to start with `unsupported`, a capability probe that never answered fails it with a retryable timeout, and a group already bound to another filter is refused. See [Agents in depth](/laser-sdk/advanced/agents). ## One log, many kinds of events Put every producer's events in one ordered log and let each group select its part on the server. One partition keeps a single global order. More partitions add parallel group workers. Events can differ in type, payload shape, envelope, and codec. The pattern that holds up: 1. Stamp a routing header on every record, such as `event.type = "metrics.cpu.v1.reported"`, and the content type through the codec helpers, which set `agdx.ct`. 2. Select the part with a header text match, such as `prefix` `metrics.`, `contains` `.v1.`, or a `glob` or `regex`. A `headers_only` filter never decodes a payload, so every format mixes safely, and it is the cheapest filter to run. 3. Add payload conditions only where needed, inside `all([header match, payload predicates])`. Header children run first and `all` stops at the first child that does not match, so records of other kinds are rejected before any decode. 4. Keep `foreign_policy` at `reject`. A record in another codec is then skipped instead of stalling the partition, even when its routing header is missing. ```ts const metricsV1 = ConsumerFilter.headersOnly( FilterExpr.headerText("event.type", "glob", "metrics.*.v1.*") ) const hotHosts = ConsumerFilter.json( FilterExpr.all([ FilterExpr.headerText("event.type", "prefix", "metrics.cpu."), FilterExpr.pred("cpu", "gte", 90) ]) ) ``` ```rust use laser_sdk::filters::TextMatch; let metrics_v1 = ConsumerFilter::headers_only(FilterExpr::header_text( "event.type", TextMatch::Glob, "metrics.*.v1.*", )); let hot_hosts = ConsumerFilter::json(FilterExpr::all([ FilterExpr::header_text("event.type", TextMatch::Prefix, "metrics.cpu."), FilterExpr::pred("cpu", CmpOp::Gte, 90_i64), ])); ``` ```python metrics_v1 = ls.ConsumerFilter.headers_only( ls.FilterExpr.header_text("event.type", "glob", "metrics.*.v1.*") ) hot_hosts = ls.ConsumerFilter.json(ls.FilterExpr.all([ ls.FilterExpr.header_text("event.type", "prefix", "metrics.cpu."), ls.FilterExpr.pred("cpu", "gte", 90), ])) ``` ### Text matching `FilterExpr.text` matches a payload field and `header_text` (TypeScript `headerText`) matches a header. Both take one of these kinds: | Kind | Matches when | Relative cost | | ---------- | -------------------------------------------------------------------------- | ------------------------------- | | `equals` | The whole value equals the pattern | Lowest | | `prefix` | The value starts with the pattern | Lowest | | `suffix` | The value ends with the pattern | Lowest | | `contains` | The pattern occurs anywhere | Low, one scan of the value | | `glob` | The whole value matches. `*` is any run, `?` is one character, `\` escapes | Low to moderate with many stars | | `regex` | The value contains a match | Moderate, linear in the value | Any kind can be case-insensitive. Rust and Python call `.case_insensitive()` on the expression, such as `FilterExpr.text("host", "prefix", "node-").case_insensitive()` in Python. TypeScript passes `true` as the last argument of `text` or `headerText`, or wraps the expression in `filterExprCaseInsensitive(expr)`. Every kind except `regex` lowercases the pattern and the value first, which copies the value. Case-insensitive `regex` uses the engine's Unicode folding. The value must be text. A number, object, or array is a type mismatch and follows the mismatch policy. On a JSON payload, parsing costs more than the text test, so the text kinds cost about the same as a plain equality test. A header-only test does not parse the payload and costs much less. Measure your own records with the SDK's `filter_evaluation` and `filter_paths` benchmarks. Regular expressions run in linear time on the server, so a pattern cannot backtrack without bound. Lookaround, backreferences, and inline flags such as `(?i)` are refused. Use the case-insensitive option instead of `(?i)`. The server's Rust engine decides pattern syntax. Character classes such as `\d` and `\w` are Unicode-aware. Rust and Python run that engine in their local evaluator and guard. TypeScript runs a port of the same syntax in linear time, so its evaluator and local guard also handle regex. It refuses the few Unicode properties that Node cannot express, such as `Age` and the segmentation break properties, and a pattern close to the size limit can pass locally and still be refused by the server. ## Revisions, pause and resume, A/B A group's filter has immutable revisions. `revise` drafts a new revision of the group's own definition. Readers keep running the active one. A draft never replaces what a bound group runs. To run a stricter variant, create a second group with that definition. Each group has its own offsets, so one variant never skips the other's records. `set_revision_enabled(revision, false)` pauses the active revision. New reads then fail with `revision_disabled`. Records already delivered can still be acknowledged, and `test` and `preview` still work. Set it to `true` to resume. The flag changes neither the definition nor its digest. `release()` removes the group's policy and returns the released binding. Its consumers then receive every record, and its filter stays saved in the catalog. `delete()` removes the group's own definition and every revision, and releases its binding in the same catalog operation. It returns whether the group had one. After either call the group keeps its digest restriction, so configure it again with the same digest or create another group for a different policy. A named filter dropped from the console or over HTTP also releases its groups and leaves a tombstone that reserves the name. Past the tombstone limit the oldest tombstones are deleted. An ID is never reused. ```ts import { filterReason } from "@laserdata/laser-sdk" const binding = await desk.filter().get() if (binding === undefined) throw new Error("the desk is unbound") // Draft on the desk's own filter. The desk keeps running its revision. const draft = await desk.filter().revise(binding.revision, ConsumerFilter.json(transitionsOnly)) const revisions = await desk.filter().revisions(0, 10) // Run the variant in its own group, with its own offsets. const variant = topic.consumerGroup("anomaly-desk-transitions") const created = await variant.create({ filter: ConsumerFilter.json(transitionsOnly) }) // Pause and resume the variant. if (created.filter === undefined) throw new Error("the variant was created unbound") await variant.filter().setRevisionEnabled(created.filter.revision, false) await variant.filter().setRevisionEnabled(created.filter.revision, true) // Another definition on a running group is refused. try { await desk.filter().configure(ConsumerFilter.json(transitionsOnly)) } catch (error) { if (filterReason(error) !== "conflict") throw error } // Release: the desk receives every record again. const released = await desk.filter().release() // Delete the desk's own filter with every revision. False when it had none. const deleted = await desk.filter().delete() ``` ```rust use laser_sdk::filters::FilterErrorReason; let binding = desk.filter().get().await?.expect("the desk is bound"); // Draft on the desk's own filter. The desk keeps running its revision. let draft = desk .filter() .revise(binding.revision, ConsumerFilter::json(transitions_only.clone())) .await?; let revisions = desk.filter().revisions(0, 10).await?; // Run the variant in its own group, with its own offsets. let variant = topic.consumer_group("anomaly-desk-transitions"); let created = variant .create() .filter(ConsumerFilter::json(transitions_only.clone())) .build() .await?; let variant_revision = created.filter.expect("bound").revision; // Pause and resume the variant. variant.filter().set_revision_enabled(variant_revision, false).await?; variant.filter().set_revision_enabled(variant_revision, true).await?; // Another definition on a running group is refused. match desk.filter().configure(ConsumerFilter::json(transitions_only)).await { Err(error) if error.filter_reason() == Some(FilterErrorReason::Conflict) => {} other => panic!("expected conflict, got {other:?}"), } // Release: the desk receives every record again. let released = desk.filter().release().await?; // Delete the desk's own filter with every revision. False when it had none. let deleted = desk.filter().delete().await?; ``` ```python binding = await desk.filter().get() if binding is None: raise RuntimeError("the desk is unbound") # Draft on the desk's own filter. The desk keeps running its revision. draft = await desk.filter().revise(binding["revision"], ls.ConsumerFilter.json(transitions_only)) revisions = await desk.filter().revisions(page=0, page_size=10) # Run the variant in its own group, with its own offsets. variant = topic.consumer_group("anomaly-desk-transitions") created = await variant.create(filter=ls.ConsumerFilter.json(transitions_only)) # Pause and resume the variant. await variant.filter().set_revision_enabled(created.filter["revision"], False) await variant.filter().set_revision_enabled(created.filter["revision"], True) # Another definition on a running group is refused. try: await desk.filter().configure(ls.ConsumerFilter.json(transitions_only)) except ls.FilterError as error: assert error.reason == "conflict" # Release: the desk receives every record again. released = await desk.filter().release() # Delete the desk's own filter with every revision. False when it had none. deleted = await desk.filter().delete() ``` `revisions` takes a page and a page size. Rust needs both. TypeScript makes both optional. Python takes them as keywords, `page=0` and `page_size=50` by default. Every catalog change carries an operation ID. A retry by the same caller with the same ID and change returns the first retained outcome. Reusing an ID with another change or caller is refused. `configure_as(operation_id, ..)` (TypeScript `configureAs`) sends a configuration under an ID you recorded first. `configure_with` (TypeScript `configureWith`) takes a definition or one of the group's own revisions. Python has the same three verbs: `configure(filter)`, `configure_with(filter=, filter_id=, revision=)`, and `configure_as(operation_id, filter=, filter_id=, revision=)`. Results stay in durable plane storage, even after control topic messages expire, so retry protection survives a recreated control topic. Recent results stay in memory up to the cache limit, and a miss reads durable storage. Each change is authorized again when the catalog applies it, against the catalog as it stands at that point of its log, so a scoped deny holds even on a node that had not seen the group yet. A group ID belongs to one topic incarnation. Deleting and recreating a group, a topic, or a stream makes a new identity that inherits no policy. A reader of the old ID does not join the replacement. ## Errors Filter failures carry a typed reason. Rust exposes it through `error.filter_reason()`, Python through `FilterError.reason`, and TypeScript through the `filterReason(error)` helper or `FilterExecutionError.detail.reason`. | Reason | What it means | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `conflict` | A precondition failed. The group runs another definition, the expected revision moved, or an acknowledgment names a policy the group no longer holds. A reader that meets it on a read restarts that partition from its stored offset and keeps reading | | `source_changed` | The source history or the group's identity changed. The reader restarts that partition from its stored offset once and raises this error so you know | | `revision_disabled` | The active revision is paused. New server reads stop until it is enabled. Records already delivered can still be acknowledged | | `catalog_unavailable` | The plane cannot prove its catalog is current, or the needed configuration has not reached this node yet. Retryable | | `membership_stale` | The consumer no longer owns the partition. The SDK refreshes its assignment | | `not_primary` | The node is no longer the partition primary. The SDK resolves the route again | | `not_found` | The group does not exist, or an operation that needs a policy met an unbound group | | `forbidden` | The caller lacks the `filter` grant on the group path or read permission on the source | | `unsupported` | The server does not serve this operation, such as the catalog without `laser-plane`, or a filter whose evaluator version or codec it does not match | | `capacity_exhausted` | A configured catalog limit is reached, such as definitions, revisions, group identities, or stored revision bytes | | `version_skew` | The client and the server speak different filter operation versions | | `invalid_request` | The filter or request is malformed or out of range | | `too_large` | The request is too large | | `unauthenticated` | The connection is not signed in | | `unavailable` | A transient failure. The same request can succeed later | | `backend` | An unexpected failure that a retry cannot fix | | `unknown` | A reason this SDK version does not know. The error's result code still classifies it | Faults, oversized records, and group setup failures are separate error types. A `stop` fault or a matching record larger than the page's reply cap blocks its partition. The reader delivers every earlier match first, then raises the error with the partition and offset. Rust returns `LaserError::FilterFault` or `LaserError::FilterOversizedRecord`, and `filter_reason()` returns `None` for them and for `ConsumerGroupSetup`. TypeScript throws `FilterFaultError`, with the fault in `reason`, or `FilterOversizedRecordError`, each with `partitionId` and `offset`. `filterReason()` returns `undefined` for them and for `ConsumerGroupSetupError`. Python raises `FilterFaultError` or `FilterOversizedRecordError`, both `FilterError` subclasses with `reason` set to `fault` or `oversized_record`, plus `fault_reason`, `partition_id`, and `offset`. A Python `ConsumerGroupSetupError` has `reason` `setup_failed` when the catalog gave no other reason. ## Key operations | Operation | What it does | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `topic.consumer_group(name)` / `consumer_group_id(id)` | The group handle, by name or native ID. Free to construct | | `group.create().filter(filter).build()` | Create the group, optionally with its filter. Idempotent | | `group.info()` | ID, name, exact identity, and the active binding | | `group.consumer()` | The normal consumer. Runs the group's policy. `batch_length` bounds page and scan | | `group.reader()` | The match-oriented reader. `count`, `max_examined`, `max_reply_bytes`, `start`, `partition`, `read_mode`, `local_guard`, `idle_interval` (Python: `idle_interval_ms`), `max_unacked_pages` | | `next_page()` / `next_record()` | Read the next page or record. `try_next_page()` returns at once when nothing is new | | `ack(record)` / `ack_through(record)` / `ack_page(page)` | Mark records done, which stores progress | | `group.filter().configure(filter)` | Give an existing group its policy. `configure_with` takes a definition or one of its own revisions, `configure_as` adds an operation ID | | `group.filter().get()` | The active binding, or none | | `group.filter().revisions(page, size)` / `revise(expected, filter)` | List revisions, draft a new one | | `group.filter().set_revision_enabled(revision, enabled)` | Pause or resume without changing content | | `group.filter().release()` | Remove the policy. Consumers then receive every record | | `group.filter().delete()` | Delete the group's own filter with every revision, releasing the group first. Returns whether it had one | | `group.filter().test(payload, headers)` / `preview(partition)` | Judge a sample or stored records without storing state | TypeScript spells these `consumerGroup`, `consumerGroupId`, `nextPage`, `ackThrough`, `configureWith`, `setRevisionEnabled`, and so on. Python passes options as keyword arguments and returns catalog replies as dicts. The Python reader takes `partitions` where Rust and TypeScript take `partition`. Rust takes `idle_interval` as a `Duration`, and Python (`idle_interval_ms`) and TypeScript take milliseconds. ## Permissions Grants for a group's filter use the group path `stream/topic/group`. `filter:read` inspects and lists, `filter:write` drafts a revision, `filter:admin` configures, pauses, resumes, and releases, and `filter:delete` deletes the group's own filter. Native source read permission still controls consumption, and a consumer does not need to inspect the definition to run its group's policy. In the LaserData Cloud Console, open a topic and its consumer group. The group's filter shows above its members, with its revision, digest, and policy generation. You can configure an unbound group there with the visual, schema-aware editor, pause or resume the active revision, release the policy, and preview it. Policy information that cannot be read shows as unavailable, never as "no filter". ## Limits | Limit | Value | | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Encoded filter size | 8 KiB | | Field path | 256 bytes, and 16 path segments | | Header key in a filter | 256 bytes. Iggy itself stores header keys of at most 255 bytes, and a test sample header key is limited to 255 | | Test sample payload | 1 MiB | | Test sample headers | 64, each value up to 1 KiB | | Expression nodes | 128 | | Nesting depth | 8 | | Items in one `in` list | 64 | | Text pattern | 1 KiB | | Compiled glob or regex programs per filter | 4 | | Budget per compiled program | 256 KiB | | Records in one page | 1000 | | Source records one request can ask a page to examine | 100,000, lowered to the server's own budget (10,000 by default) | | Unacknowledged pages with records per partition | 1024 by default, configurable per reader | | Bytes in one page | 8 MiB, and 1 MiB by default for a reader | | Records one preview examines | 1,000 by default, configurable up to 10,000 | | Records one preview returns | 100 | | Live saved definitions | 1,024 | | Revisions per definition | 256 | | Retained group identities | 4,096 | | Stored revision bytes, all filters together | 64 MiB | | Dropped filters kept as tombstones, oldest deleted past the limit | 4,096 | | Results cached in memory by default | 256 | | Concurrent changes per plane by default | 32 | | Concurrent changes per caller by default | 8 | | Changes per caller per second by default | 32 | ### Plane settings The definition, tombstone, revision, and group-identity values above are defaults. `LD_PLANE_FILTER_MAX_DEFINITIONS`, `LD_PLANE_FILTER_MAX_TOMBSTONES`, `LD_PLANE_FILTER_MAX_REVISIONS`, `LD_PLANE_FILTER_MAX_GROUP_IDENTITIES`, and `LD_PLANE_FILTER_MAX_CATALOG_BYTES` accept `0` for no limit. `LD_PLANE_FILTER_OUTCOME_CACHE_ENTRIES=0` turns off the cache and keeps durable retries. `LD_PLANE_FILTER_MAX_CONCURRENT_MUTATIONS` and `LD_PLANE_FILTER_MAX_CONCURRENT_MUTATIONS_PER_ACTOR` set positive limits on changes in progress at once. `LD_PLANE_FILTER_MAX_MUTATIONS_PER_ACTOR_PER_SECOND` limits each caller's change rate, with a minimum of 1. Past any of them, the plane returns a retryable `unavailable`. The system control topic defaults to no expiry. Set `LD_PLANE_CONTROL_EXPIRY_SECS` to `604800` for seven days, `2592000` for thirty days, or `0` for no expiry. Startup applies the configured value to existing topics. With finite retention, recovery uses the durable plane snapshot and the remaining log. Expiring topic messages does not delete group filters, bindings, or operation results. `LD_PLANE_DLQ_EXPIRY_SECS` and `LD_PLANE_CHANGES_EXPIRY_SECS` keep the plane's dead-letter and change topics for one day by default. ### Server settings `IGGY_PLANE_FILTERS_ENABLED` turns the filter evaluator on or off for a deployment. It is on by default. Group reads of unbound groups work without it. A bound group is refused until it is on. | Setting | Default | What it bounds | | ---------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------- | | `IGGY_PLANE_FILTERS_MAX_CONCURRENT` | 16 per shard | Concurrent scans. A scan past it gets a retryable refusal | | `IGGY_PLANE_FILTERS_MAX_EXAMINED_RECORDS` | 10,000 | Source records one page examines | | `IGGY_PLANE_FILTERS_MAX_EXAMINED_BYTES` | 64 MiB | Bytes one page scan examines | | `IGGY_PLANE_FILTERS_PREVIEW_MAX_EXAMINED` | 1,000, at most 10,000 | Records one preview examines | | `IGGY_PLANE_FILTERS_MAX_ROUND_RECORDS` | 1,000, from 1 to 1,000 | Records in one internal owner read | | `IGGY_PLANE_FILTERS_MAX_PAGE_ROUNDS` | 256 | Owner reads per page | | `IGGY_PLANE_FILTERS_MAX_ROUNDS` | Separate value | Owner reads per preview | | `IGGY_PLANE_FILTERS_UNBOUND_CONFIRM_MS` | 1,000, at most 60,000, 0 confirms every read | How long a cached unbound answer is reused | | `IGGY_PLANE_FILTERS_POLICY_CONFIRM_TIMEOUT_MS` | 1,000, from 1 to 60,000 | Each policy version confirmation | | `IGGY_PLANE_FILTERS_POLICY_CACHE_ENTRIES` | 1,024, at most 65,536, 0 resolves every request | Cached policies per shard | | `IGGY_PLANE_FILTERS_POLICY_CACHE_TTL_MS` | 60 seconds, at most 10 minutes, 0 resolves every request | Lifetime of a cached policy | Fewer owner reads per page cost less server CPU. On a one-million-record trial, a round cap of 1,000 used about 19% less server CPU than 128 with the same results. The first owner read of a page probes at most 128 records. Each later read shrinks to what the remaining byte budget covers at the average record size seen so far, so a sudden jump in record size can pass the byte budget by one owner read. The scan yields every 1 MiB of records and sizes the next read by how selective the filter was so far, so a sparse filter can scan further within the page budget. A request's `max_examined` caps the budget below the server's. Consumer group reads also respect the remaining wanted match count to keep the native polling fence. A page also stops at its round and byte budgets, so a 1,000-record request can return fewer matches. Previews size their owner reads the same way and also check bytes before each record. These bounds limit work per page. They do not promise throughput or latency. Each server shard caches resolved policies and watches its own plane for catalog changes. Before it reuses an entry, it confirms the current catalog version through the sidecar, so a delayed watch cannot keep an old binding active after a finished local change. Changes made on another node reach the local plane through control log replay. A plane restart invalidates earlier version tokens. The server also checks current reader grants before reuse. Cache hits skip repeated definition transfer and decoding, while a small version confirmation still runs for each hit. Watch failures and confirmation failures turn off reuse. ## Payload formats All formats use the same predicates, fault policies, previews, and acknowledgment rules. The server returns the original payload bytes, so consumers keep their existing decoders. | Format | Filter constructor | Value rules | | ------------ | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JSON | `ConsumerFilter.json(expression)` | Exact 64-bit integers, finite numbers, arrays, objects, strings, booleans, and null | | CBOR | `ConsumerFilter.cbor(expression)` | Definite-length values, text map keys, byte strings as integer arrays. No tags, duplicate keys, or non-finite numbers | | Avro | `ConsumerFilter.avro(expression, schema_refs)` | Registered writer schemas. Logical types use their physical values, such as integer timestamps or decimal bytes. Enums are their symbol strings | | Protobuf | `ConsumerFilter.protobuf(expression, schema_refs)` | Registered descriptor sets and message names. Field names follow the schema, enums are numbers, and bytes are integer arrays. A field with presence tracking (proto2, `optional`, messages) is absent when unset. A proto3 scalar or repeated field without it always exists, at its default when unset, so `counter == 0` matches | | Headers only | `ConsumerFilter.headers_only(expression)` | Typed user headers, with no payload decoding | The table uses Python names. Rust uses `ConsumerFilter::avro(expression, schema_refs)` and TypeScript uses `ConsumerFilter.avro(expression, schemaRefs)`. The [CDC examples](https://github.com/laserdata/laser-sdk/tree/main/examples) show typed producers and group readers for every format in all three languages. Register Avro and Protobuf schemas once, then reference their IDs. Put the allowed writer IDs in the filter's `schema_refs`. Stamp each published record with its writer ID through `schema_id` in Rust and Python or `schemaId` in TypeScript, which sets the `agdx.sid` header. A filter can allow several immutable writer schemas of the same codec. The schema IDs are part of its digest. A schema ID that is missing, not listed, or registered for another schema family follows the filter's `foreign_policy` when a predicate needs the payload: the record is skipped by default, or delivered as not evaluated under `pass`. The server resolves schemas through the plane schema catalog before it scans records. Without a schema stream, `schema_refs` resolve in the deployment-wide registry. `with_schema_stream(stream)` in Rust and Python, and `ConsumerFilter.withSchemaStream(filter, stream)` in TypeScript, name the stream whose registry holds them, and change the digest. Rust, Python, and TypeScript local guards use the same references. CBOR needs no registered schema. Protobuf groups are not supported. Empty repeated and map fields are present with their default empty values. Use explicit timestamp coercions for Avro integer timestamps. Byte and nesting limits bound decoding. The server announces its evaluator version and supported codecs, and the SDKs refuse unsupported capabilities explicitly. ## Availability and cost Group reads, tests, and previews run on the LaserData Iggy fork, in [Laser Stack](/laser-sdk/laser-stack) and LaserData Cloud. Configuring a group's filter needs `laser-plane`. `create` with a filter is refused before the group is created when catalog support is unavailable. Original Apache Iggy keeps working for native streaming. Check the filter capabilities before you use them. `native` covers filtered reads, tests, and previews. `catalog` covers configuration. `group_policy_reads` covers group-aware consumers. ```ts const { filters } = await laser.capabilities() console.log(filters.native, filters.catalog, filters.groupPolicyReads) ``` ```rust let filters = laser.capabilities().await.filters; println!("{} {} {}", filters.native, filters.catalog, filters.group_policy_reads); ``` ```python filters = (await laser.capabilities()).filters print(filters.native, filters.catalog, filters.group_policy_reads) ``` Measure bandwidth savings and server cost together. The server still scans source records per group. Header-only policies avoid decoder work. Payload policies add decoding and evaluation while they cut bytes sent to consumers. Use the [Frostline benchmark documentation](https://github.com/laserdata/laser-example-frostline/blob/main/docs/benchmarks.md) to measure CPU, memory, sample counts, and p50, p99, and p99.9 latency on your workload. The figures on this page describe recorded workloads. They are not a latency guarantee or a replacement for benchmarks on your deployment. Source: https://docs.laserdata.com/laser-sdk/advanced/consumer-filters --- # Memory in depth This page is the full reference for [Memory](/laser-sdk/memory). Start there for the short version. ## How memory is stored `laser.memory(namespace)` returns a `MemoryHandle`. Every `remember`, `improve`, and `forget` appends a record to the memory topic. `forget` writes a tombstone and `improve` writes a feedback weight, so nothing is edited in place and the topic keeps the full history. A managed deployment folds the memory topic into a versioned key-value read view. Default recall and `fetch` read that view, so they need [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. Folded recall rebuilds memory from the topic inside your process instead, so it works with Apache Iggy alone. When a `forget` or `improve` call names a conversation in its scope, it acts only on an item remembered in that conversation. A call without a conversation acts on the item in any conversation. A [scoped memory](/laser-sdk/context) always passes its own conversation, so it cannot forget or reweight another conversation's item. ## Namespaces A client with a default stream scopes the memory namespace to the stream its memory records ride, so `laser.memory("notes")` reads and writes `stream:/notes`. A name that already starts with `stream:` is sent as written. A client without a default stream sends the bare name. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names) for the opt out. Every record carries its namespace in a header. Folded recall reads only records with the handle's namespace, and the managed view keys rows by namespace, so two namespaces on one topic never mix. ## Scope Each item has a scope. A scope names the stream, the agent, and the conversation the item belongs to, plus an optional user and application. | Field | Remember (Rust, TypeScript) | Python keyword | `MemoryScope` field | | ------------ | --------------------------- | --------------- | ---------------------------- | | Conversation | `.scope(conversation)` | `conversation=` | `conversation` | | Agent | `.agent(id)` | `agent=` | `agent` | | Stream | `.stream(name)` | `stream=` | `stream` | | User | `.user(user)` | `user=` | `user` | | Application | `.application(app)` | `application=` | `app` in Rust and TypeScript | Recall narrows by every field you set. A field you leave out widens the read, so a recall without a conversation reads across conversations. TypeScript calls `recall()` with no argument, Rust calls `recall(None)`, and Python omits `conversation=`. A stream filter on a log handle returns nothing when it names a stream other than the one the records ride. `.durable()` (Python: `durable=True`) sets the scope lifetime to durable. The default is session. Custom backends receive the lifetime. Built-in backends do not store it, and recall never filters on it. ## Memory topic By default memory records ride `agent.memory` on the connection's default stream. `bootstrap(partitions, retention)` creates that topic with the other agent topics. `memory_on_topic(topic, stream)` (TypeScript: `memoryOnTopic`) uses a topic that already exists. The stream is optional and defaults to the connection's default stream: Rust takes an `Option<&str>`, TypeScript an optional second argument, and Python `stream=`. `memory_topic(topic)` (TypeScript: `memoryTopic`) creates the topic first and sets its partition count and message expiry. Both use the topic name as the namespace. | Setting | TypeScript | Rust | Python | | ----------------------- | -------------------- | -------------------- | ------------------------- | | Stream | `.stream(name)` | `.stream(name)` | `stream=` | | Partitions, default 1 | `.partitions(count)` | `.partitions(count)` | `partitions=` | | Expiry, default 30 days | `.ttl(ttlMs)` | `.ttl(Duration)` | `ttl_ms=` in milliseconds | | Never expire | `.noExpiry()` | `.no_expiry()` | `ttl_ms=0` | TypeScript and Rust finish the builder with `.build()`. Python awaits `laser.memory_topic(topic, ..)` directly. Records of one conversation share a partition key, so each conversation keeps its write order on a multi-partition topic. Topic expiry and view retention are separate. Once a record expires from the topic, folded recall can no longer rebuild it, while the managed view keeps its own retention. ## Recall `recall(conversation)` starts a recall builder in Rust and TypeScript, and `recall(None)` in Rust or `recall()` in TypeScript recalls every conversation. Finish it with `.fetch()`. Python passes everything as keywords to `recall(..)`. `.limit(n)` (Python: `limit=`) caps the result. The default is 50. ### Managed and folded recall Default recall reads the managed key-value view and returns the newest items first. Newest means the latest broker append time, with the source position breaking ties, so items from different partitions come back in arrival order. The view folds the topic in the background, so a fresh write can take a moment to appear. `.folded()` (Python: `folded=True`) rebuilds memory from the topic in your process. Keep one handle for folded reads. A handle remembers what it has read and reads only new records on the next call. A fresh handle reads the whole topic on its first recall. Feedback from `improve` reorders folded recall and vector recall for every strategy except recent. Managed recall always returns newest first. ### Strategies | Strategy | Rust and TypeScript | Python | Ranks by | | -------- | ------------------- | ----------------------------------- | ------------------------------------------------------ | | Recent | `.recent()` | `strategy="recent"` | Newest first. Query text and feedback do not change it | | Semantic | `.semantic(text)` | `semantic=text` | Embedding similarity to the text | | Keyword | `.keyword(text)` | `strategy="keyword", semantic=text` | Exact terms, which suits names and identifiers | | Hybrid | `.hybrid(text)` | `strategy="hybrid", semantic=text` | The sum of semantic, keyword, and feedback scores | Python also accepts `auto`, `graph`, and `temporal`. Rust and TypeScript set any value with `.strategy(..)`. `auto` is the default. Only the vector backend ranks by text. Log memory uses the text as a filter: semantic, keyword, and hybrid recall keep items that share at least one word with the text, then apply the limit. Recent recall ignores the text. Each item's `signals` records which strategy surfaced it, its rank within that signal (0 is best), and that signal's score. Feedback appears as an `auto` signal in every backend. `fuse_reciprocal_rank(signals, limit)` (TypeScript: `fuseReciprocalRank`) merges several ranked lists into one. Each item scores the sum of `1 / (60 + rank)` over the lists that contain it, so agreement between lists outranks one list's top hit. ### Vector memory The vector backend embeds each item when you remember it and ranks recall by cosine similarity. It keeps items in process memory and drops them when the handle goes away. | Task | TypeScript | Rust | Python | | ---------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------- | | Open a governed vector handle | `laser.memoryWith(ns, MemoryBackend.Vector, embedder)` | `laser.memory_with(ns, MemoryBackend::Vector).embedder(e)` | `laser.memory_with(ns, "vector", embedder=e)` | | Same backend, built directly | `VectorMemory.governed(laser, embedder)` | `VectorMemory::governed(laser, embedder)` | `VectorMemory.governed(laser, embedder)` | | Standalone, no connection, no governance | `MemoryHandle.vector(embedder)` or `new VectorMemory(embedder)` | `MemoryHandle::vector(embedder)` or `VectorMemory::new(embedder)` | `MemoryHandle.vector(embedder)` or `VectorMemory(embedder)` | The embedder comes from your application. It is a Rust `Embedder` implementation, a TypeScript object with `embed(text)`, or a Python callable or object with `embed(text)` that returns a list of floats, directly or through an awaitable. A synchronous Python embedder runs on an SDK worker thread, so it must not block. Python and TypeScript take the embedder when the handle opens, so they check it there: the vector backend without an embedder, or another backend with one, is an invalid error. Rust attaches the embedder after opening with `.embedder(..)`, so a Rust vector handle without one fails with a configuration error at the first remember or semantic recall. `.embedder(..)` on a vector handle builds a fresh, empty index over the new embedder, so attach it before you write. On a log or custom handle `.embedder(..)` changes nothing. Keyword recall on the vector backend scores the share of query words found in the item and never calls the embedder. Without query text, vector recall returns the newest items, ranked by feedback when any exists. ### Rerankers A reranker is a second pass over the recalled candidates. Attach one with `.reranker(..)` on any handle, or wrap a backend in `RerankedMemory` (Rust: `RerankedMemory::new(inner, reranker)`, Python: `RerankedMemory(inner, reranker)`, TypeScript: `new RerankedMemory(inner, reranker)`). It runs only when the query carries text, and it also runs on folded recall. Writes pass through unchanged. Rust takes a `Reranker` implementation. TypeScript takes an object with `rerank(query, items)`. Python takes a callable `(query, items) -> items` or an object with `rerank(query, items)`, synchronous or async. ```ts import { ConversationId, MemoryBackend } from "@laserdata/laser-sdk" const conversation = ConversationId.new() const memory = laser.memoryWith("runbooks", MemoryBackend.Vector, embedder).reranker(reranker) const encoder = new TextEncoder() await memory.remember(encoder.encode("node-7 sits in the eu-west pool")).scope(conversation).send() await memory.remember(encoder.encode("node-9 rotates keys monthly")).scope(conversation).send() const hits = await memory.recall(conversation).semantic("which pool is node-7 in").limit(3).fetch() ``` ```rust use laser_sdk::prelude::full::*; let conversation = ConversationId::new(); let memory = laser .memory_with("runbooks", MemoryBackend::Vector) .embedder(embedder) .reranker(reranker); memory .remember("node-7 sits in the eu-west pool".as_bytes()) .scope(conversation) .send() .await?; memory .remember("node-9 rotates keys monthly".as_bytes()) .scope(conversation) .send() .await?; let hits = memory .recall(conversation) .semantic("which pool is node-7 in") .limit(3) .fetch() .await?; ``` ```python import laser_sdk as ls conversation = ls.new_conversation_id() memory = laser.memory_with("runbooks", "vector", embedder=embed).reranker(rerank) await memory.remember("node-7 sits in the eu-west pool", conversation=conversation) await memory.remember("node-9 rotates keys monthly", conversation=conversation) hits = await memory.recall( conversation=conversation, semantic="which pool is node-7 in", limit=3, ) ``` `embedder` and `reranker` are your own implementations. The runnable [memory example](https://github.com/laserdata/laser-sdk/tree/main/examples/rust/src/memory) ships a small bag-of-words embedder in each language. ## Writing items ### Kinds `remember(..)` stores a `Fact` unless you set a kind with `.kind(..)` (Python: `kind="summary"` and the other lowercase names). | Kind | Class | Holds | | ----------- | ---------- | ------------------------------------ | | `Fact` | Semantic | A standalone fact. The default | | `Message` | Episodic | A conversation turn or event | | `Summary` | Semantic | A summary distilled from other items | | `Entity` | Semantic | An extracted entity | | `Feedback` | Semantic | A ranking signal | | `Procedure` | Procedural | A reusable workflow | `MemoryKind::class()` in Rust, `memory_kind_class(kind)` in Python, and `memoryClass(kind)` in TypeScript report the class. `MemoryKind::code()`, `memory_kind_code(kind)`, and `memoryKindCode(kind)` return the stable one-byte code. ### Content IDs `remember` returns a random ID unless you add `.dedup()` (Python: `dedup=True`). With dedup the ID comes from the owner scope (stream, agent, user, and application), the kind, and the body. The conversation is not part of the ID, so remembering the same fact twice stores it once, and two users never share an ID. `MemoryId::content(scope, kind, body)` in Rust, `memory_id_content(kind, body, stream=, agent=, user=, application=)` in Python, and `MemoryId.content(scope, kind, body)` in TypeScript compute the same ID without writing anything. Every SDK produces the same ID for the same input. [Graph](/laser-sdk/advanced/graph) uses the same approach for node IDs. ### Typed append Typed append writes an ID and a kind that you supply. Built-in backends keep both. Use it when the ID comes from somewhere else, such as a content ID. Rust calls the `Memory::append` trait method, Python uses `append(id, payload, kind=, ..)`, and TypeScript uses `append(scope, id, kind, payload)`. ```ts import { AgentId, MemoryId, MemoryKind } from "@laserdata/laser-sdk" const scope = { agent: AgentId.new("planner"), conversation } const body = new TextEncoder().encode("node-7 rotates keys monthly") const id = MemoryId.content(scope, MemoryKind.Entity, body) await memory.append(scope, id, MemoryKind.Entity, body) ``` ```rust let scope = MemoryScope::builder() .agent("planner".parse()?) .conversation(conversation) .build(); let body = b"node-7 rotates keys monthly".to_vec(); let id = MemoryId::content(&scope, MemoryKind::Entity, &body); Memory::append(&memory, &scope, id, MemoryKind::Entity, body).await?; ``` ```python body = b"node-7 rotates keys monthly" memory_id = ls.memory_id_content("entity", body, agent="planner") await memory.append( memory_id, body, kind="entity", agent="planner", conversation=conversation, ) ``` ## Lineage An item can record where it came from. `origin` is the log record that motivated it, such as the message a fact was extracted from. `producer` names the agent, extraction policy, or consolidator that wrote it, with a name and a version. Both are optional, never filter recall, and never change the item's content ID. Set them on one write with `.origin(source)` and `.producer(info)` on the remember builder (Python: `origin=` and `producer=`). `with_lineage(origin, producer)` on a [scoped memory](/laser-sdk/context) (TypeScript: `withLineage`) stamps every item it remembers. Inside a [session](/laser-sdk/session), `linked_memory()` (TypeScript: `linkedMemory`) stamps the session's agent as the producer and the record the session acts on as the origin. Call `acting_on(source)` (TypeScript: `actingOn`) on the session first to name that record. ```ts const facts = laser .context(conversation) .memory("facts") .withLineage(undefined, { name: "fact-extractor", version: "2" }) await facts.remember(new TextEncoder().encode("the reporter prefers email")).send() await session.linkedMemory().remember(new TextEncoder().encode("ticket 42 is a login bug")).send() ``` ```rust use laser_sdk::wire::graph::ProducerInfo; let facts = laser.context(conversation).memory("facts").with_lineage( None, Some(ProducerInfo { name: "fact-extractor".into(), version: "2".into(), }), ); facts.remember("the reporter prefers email".as_bytes()).send().await?; session .linked_memory() .remember("ticket 42 is a login bug".as_bytes()) .send() .await?; ``` ```python facts = ( laser.context(conversation) .memory("facts") .with_lineage(producer={"name": "fact-extractor", "version": "2"}) ) await facts.remember("the reporter prefers email") await session.linked_memory().remember("ticket 42 is a login bug") ``` A recalled item also carries `source`, the memory record it was folded from (stream, topic, partition, and offset), so a reader can open the original record while the log still holds it. A message source can carry the topic creation time, so a reader can tell a recreated topic from the original. In Python, `source` is a tuple `(stream, topic, partition, offset, generation, conversation)`. A managed deployment also links an item to the session that wrote it, and `record_retrieval(query, items)` (TypeScript: `recordRetrieval`) on a session records which items entered its context. ## Item fields | Field | Holds | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id` | The item's ID, a ULID unless it is a content ID | | `payload` | The body. Read it as text with `text()` in Rust and Python or `memoryItemText(item)` in TypeScript, or as JSON with `json()` or `memoryItemJson(item)` | | `kind` | The item kind | | `provenance` | The conversation and agent it was stored under | | `score` | The ranking score, empty for unranked recall | | `signals` | The per-strategy rank and score behind a ranked result | | `source` | The memory record it was folded from | | `origin`, `producer` | Lineage, when the item was written with it. Managed recall does not return them | ## Named items For facts you address directly, use named items instead of a recall search. All three SDKs call `set(key, body)`, `fetch(key)`, `update(key, patch)`, and `remove(key)` on the memory handle. Each write is a record on the memory topic, stored under `/`. `update` applies a JSON merge patch (RFC 7386): fields in the patch overwrite and `null` removes. It reads the current value by folding the topic, so it works with Iggy alone. `fetch` reads the managed key-value view. `fetch_folded(key)` (TypeScript: `fetchFolded`) folds the topic in your process instead. `remove` of an absent key succeeds. Vector and custom handles return an unsupported error for named items. The `LogMemory` class carries the same operations as `set_named`, `fetch_named`, `fetch_named_folded`, `update_named`, and `forget_named` (TypeScript: camelCase). ## Context blocks `context(..)` recalls a conversation's items and formats them as one prompt block. Rust takes `context(conversation, Some(token_budget))`, Python takes `context(conversation, token_budget=)`, and TypeScript takes `context(scope, { tokenBudget })`. The recall builder renders the same block with `.block(Some(budget))` in Rust, `.block(budget)` in TypeScript, and `recall(.., token_budget=n, block=True)` in Python. The token estimate is the byte count divided by four, rounded up, in every SDK. When the budget runs out, the block drops the remaining items and appends a marker such as `[... 3 more recalled item(s) omitted ...]`. The first item is always kept, even when it alone exceeds the budget. `context` and a plain `block` use default recall, so a log handle needs the managed view. With Iggy alone, use `.folded().block(Some(budget))` in Rust, `.folded().block(budget)` in TypeScript, or `recall(.., folded=True, token_budget=budget, block=True)` in Python. To format items you already hold, call `to_context_block(items, token_budget)` (TypeScript: `toContextBlock(items, tokenBudget)`). In Python, `token_budget=` without `block=True` is passed to the backend as a hint and does not trim the returned list. ## Consolidation `consolidate(scope, max_items)` runs one pass over a scope. The default pass keeps the newest `max_items` items in arrival order and forgets the rest. It examines at most 10,000 items per pass, and a limit of zero prunes that whole window. Python takes the scope as keywords: `consolidate(max_items, conversation=, agent=, ..)`. The pass returns a `ConsolidationReport` with four counts: `summarized`, `reweighted`, `pruned`, and `derived`. The default pass fills only `pruned`, and it counts an item after the backend accepts the forget. `reweighted` and `derived` are for custom consolidators. A summarizer adds a step before pruning. It groups the `Message` items by conversation and writes one `Summary` per conversation with a durable lifetime. With `prune_summarized`, the pass then forgets the messages it folded and counts them in `pruned`. Your application supplies the summarizer, usually a model call. The SDK ships no model client. | Control | TypeScript | Rust | Python | | ---------------------- | --------------------------------------- | ------------------------------------------------------------------- | ---------------------------- | | Run a pass | `consolidate(scope, maxItems, options)` | `consolidate(&scope, max_items)` | `consolidate(max_items, ..)` | | Add a summarizer | `{ summarizer }` | `consolidate_with(&scope, max_items, summarizer, prune_summarized)` | `summarizer=` | | Forget folded messages | `{ pruneSummarized: true }` | `prune_summarized` set to `true` in `consolidate_with` | `prune_summarized=True` | The TypeScript summarizer is an object with `summarize(bodies)`. Rust implements the `Summarizer` trait from `laser_sdk::memory`. Python passes a callable from a list of bytes to a body, or an object with `summarize(bodies)`, synchronous or async. Scoped handles take the same options, and Rust spells it `consolidate_with(max_items, summarizer, prune_summarized)` on a scoped handle. `DefaultConsolidator::new(&memory, max).with_summarizer(..).prune_summarized()` builds the same pass as a Rust `Consolidator` for an agent timer. Consolidation reads through default recall, so a log handle needs the managed view. A vector handle consolidates in process. A custom backend without a typed `append` receives the summary through its `remember`. ```ts const report = await memory.consolidate({ conversation }, 100, { summarizer: { summarize: async (bodies) => new TextEncoder().encode(`${bodies.length} turns about node-7`) }, pruneSummarized: true }) console.log(report.summarized, report.pruned) ``` ```rust use laser_sdk::memory::Summarizer; use laser_sdk::prelude::full::*; struct Digest; impl Summarizer for Digest { async fn summarize(&self, bodies: Vec>) -> Result, LaserError> { Ok(format!("{} turns about node-7", bodies.len()).into_bytes()) } } let scope = MemoryScope::builder().conversation(conversation).build(); let consolidator = DefaultConsolidator::new(&memory, 100) .with_summarizer(Digest) .prune_summarized(); let report = consolidator.consolidate(&scope).await?; println!("{} summarized, {} pruned", report.summarized, report.pruned); ``` ```python report = await memory.consolidate( 100, conversation=conversation, summarizer=lambda bodies: f"{len(bodies)} turns about node-7".encode(), prune_summarized=True, ) print(report.summarized, report.pruned) ``` ## Agent turns and periodic consolidation `MemoryHandler` wraps an agent handler and remembers each message the handler finishes. After the inner handler succeeds, it remembers the payload under the message's conversation and agent. Remembering stays off until you choose a kind with `auto_remember(kind)` (TypeScript: `autoRemember(kind)`). A handler that fails remembers nothing and follows the normal retry path. A failed memory write is dropped, so it never repeats the handler's effects. An agent can also consolidate on a timer, off the handler loop. Set both an interval and a consolidator, because one alone does nothing. The first pass runs when the agent starts and one more runs each interval. The consolidator owns its memory handle. Its scope names the agent ID, or is empty when a Python or TypeScript agent has no ID. A failed pass does not stop later ones. Shutdown stops the timer. TypeScript and Python reject an interval that is not positive. In Rust, a zero interval makes the agent fail with a configuration error, which `ready()` returns. [Agents](/laser-sdk/fabric) covers the agent lifecycle. | Control | TypeScript | Rust | Python | | ------------ | -------------------------------------------------------- | ---------------------------------- | ---------------------------------------------------- | | Interval | `consolidateEvery(intervalMs)` | `consolidate_every(Duration)` | `consolidate_every_ms=` | | Consolidator | `consolidator(obj)`, an object with `consolidate(scope)` | `consolidator(SharedConsolidator)` | `consolidator=`, an object with `consolidate(scope)` | ```ts import { Agent, AgentId, AgentTopic, MemoryHandler, MemoryKind } from "@laserdata/laser-sdk" const memory = laser.memory("triage") const handler = new MemoryHandler(inner, memory).autoRemember(MemoryKind.Message) const agent = Agent.builder() .id(AgentId.new("triage")) .listenOn(AgentTopic.Sessions) .handler(handler) .consolidateEvery(60_000) .consolidator({ consolidate: (scope) => memory.consolidate(scope, 500) }) .build() .spawn(laser) ``` ```rust use laser_sdk::memory::SharedConsolidator; use laser_sdk::prelude::full::*; use std::time::Duration; let handler = MemoryHandler::new(inner, laser.memory("triage")) .auto_remember(MemoryKind::Message); let consolidator = SharedConsolidator::new(DefaultConsolidator::new(laser.memory("triage"), 500)); let agent = Agent::builder() .id("triage".parse()?) .listen_on(AgentTopic::Sessions) .handler(handler) .consolidate_every(Duration::from_secs(60)) .consolidator(consolidator) .build() .spawn(laser.clone()); ``` ```python memory = laser.memory("triage") handler = ls.MemoryHandler(inner, memory).auto_remember("message") class Compact: async def consolidate(self, scope): await memory.consolidate(500) agent = laser.spawn_agent( "triage", ls.AgentTopic.Sessions, handler, consolidate_every_ms=60_000, consolidator=Compact(), ) await agent.ready() ``` `inner` is your own handler. ## Custom backends `memory_custom(backend)` (TypeScript: `memoryCustom`) plugs in your own store. The backend owns its scoping and ID minting. It implements `remember`, `recall`, `improve`, and `forget`, and it can add a typed `append`. Without `append`, the SDK falls back to `remember` and your backend picks the ID and kind. The handle's `backend` reports `log`, `vector`, or `custom`. In Rust, implement the `Memory` trait and pass an `Arc` to `memory_custom`. In TypeScript, pass an object that implements the `Memory` interface. In Python, pass an object with `remember(scope, payload)`, `recall(scope, query)`, `improve(scope, feedback)`, `forget(scope, id)`, and optionally `append(scope, id, kind, payload)`. Each method returns directly or through an awaitable, and scopes, queries, and feedback arrive as dicts. `recall` returns `MemoryItem` objects or dicts with a `payload` and optional `id`, `kind`, `score`, and `conversation`. Synchronous Python methods run on an SDK worker thread, so they must not block. Python also builds a handle without a connection with `MemoryHandle.custom(backend)`. ## Governance and access Memory writes from a handle built on a `Laser` pass through the connection's action governor. That covers log and vector handles. The policy sees the proposed item body. A forget or feedback write presents its encoded record. A standalone vector handle has no connection, so no policy applies. See [Governance](/laser-sdk/advanced/governance). The managed read view is shared across principals. Reading needs `kv:read` on the memory namespace, and writing needs permission to publish to the memory topic. A conversation filter narrows a read but does not grant or limit access. Sharing a namespace between sessions does not merge their histories. ## Requirements Run `bootstrap(partitions, retention)` once per stream before you write to the default topic. In Rust, memory needs the `agent` feature, and default recall and `fetch` also need `kv`, which `managed` includes. Without `kv`, those calls return an unsupported error. | Works with Apache Iggy alone | Needs Laser Stack or LaserData Cloud | | --------------------------------------- | ------------------------------------- | | Remember, improve, forget, typed append | Default recall | | Folded recall and `fetch_folded` | `fetch` | | Named `set`, `update`, `remove` | `context` and `block` without folding | | Vector and custom handles | Consolidation on a log handle | ## Language differences * TypeScript uses camelCase names and `Uint8Array` payloads. * Python passes scope and options as keywords (`agent=`, `conversation=`, `user=`, `application=`, `kind=`, `durable=`, `dedup=`, `origin=`, `producer=`) instead of builder calls, and accepts `str` or `bytes` payloads. * Python `improve(target, weight, ..)` and `forget(id, ..)` take the scope as keywords. Rust and TypeScript take a scope and a `Feedback` or ID. ## Key operations | Call | What it does | | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `laser.memory(namespace)` | Open log-backed memory in a namespace | | `laser.memory_with(namespace, backend)` | Choose the log or vector backend. TypeScript adds the embedder as a third argument and Python as `embedder=` | | `remember(payload)` | Store an item, with `.scope`, `.agent`, `.stream`, `.user`, `.application`, `.kind`, `.dedup`, `.durable`, `.origin`, `.producer` | | `append(..)` | Write an item under an ID and kind you supply | | `recall(..)` + `.recent()`, `.semantic(t)`, `.keyword(t)`, `.hybrid(t)` | Choose a recall strategy | | `.folded()` | Read the memory topic in process, with Iggy alone | | `.limit(n)` | Cap the result, 50 by default | | `.embedder(..)`, `.reranker(..)` | Attach an embedding function or a reranking pass | | `improve(..)` | Add a feedback weight to an item | | `forget(..)` | Write a tombstone for an item | | `memory_on_topic(topic)`, `memory_topic(topic)` | Use your own memory topic | | `memory_custom(backend)` | Use your own backend | | `LogMemory`, `VectorMemory`, `RerankedMemory` | Build a backend directly | | `set`, `fetch`, `fetch_folded`, `update`, `remove` | Read and write named items | | `context(..)`, `block(budget)`, `to_context_block(..)` | Render items as one prompt block | | `consolidate(..)` | Prune a scope, optionally with a summarizer | | `MemoryHandler` + `auto_remember(kind)` | Remember each successful agent turn | | `consolidate_every` + `consolidator` | Consolidate on a timer from the agent builder | | `fuse_reciprocal_rank(lists, limit)` | Merge ranked lists | Source: https://docs.laserdata.com/laser-sdk/advanced/memory --- # Context in depth This page holds the full detail behind the [Context](/laser-sdk/context) guide. Read the guide first. ## Open a context scope `laser.context(conversation_id)` returns a context scope for one conversation. The call does no I/O. Every operation on the scope uses that conversation ID. A [session](/laser-sdk/session) is a conversation with a recorded lifecycle, and its ID is a conversation ID, so a scope reads a session's records too. `session.scope()` (TypeScript: the `scope` property) returns the scope of a session. `append(topic, payload)` writes one message to a topic. The message carries the conversation ID in its provenance headers, and the conversation ID is also its partition key, so one conversation's messages on one topic share a partition. `fetch(topics, n)` reads the last `n` messages of this conversation across the selected topics. Each result is a context message: | Field | Rust | TypeScript | Python | | ----------------------------------- | ----------------------- | --------------------- | ----------------------- | | Log position | `id` | `id` | `id` | | Provenance | `provenance` | `provenance` | `provenance` | | Raw body | `payload` | `payload` | `payload` | | Decoded AGDX envelope, when present | `envelope` | `envelope` | `envelope` (a dict) | | Topic name | `topic` | `topic` | `topic` | | Broker append time in microseconds | `timestamp_micros` | `timestampMicros` | `timestamp_micros` | | Numeric stream and topic IDs | `stream_id`, `topic_id` | `streamId`, `topicId` | `stream_id`, `topic_id` | The numeric IDs let you build a source reference without another lookup. A read collects the conversation's messages from each topic and sorts them by the timestamp Iggy assigns. Each topic has its own offsets, so the timestamp is the only clock shared across topics. Messages with the same timestamp are ordered by the position of their topic in the list you passed, then by partition, then by offset. A message whose provenance does not decode is skipped. ## Policies `fetch` limits a read by count. For other limits, pass an assembly policy to `fetch_with(topics, policy)` (TypeScript: `fetchWith`). | Policy | Keeps | | -------------------- | -------------------------------------------------------------------------- | | `LastN(n)` | The newest `n` messages | | `TokenBudget(n)` | The newest messages that fit `n` estimated tokens, and always at least one | | `RoleFilter(agents)` | Messages whose provenance names one of the given agents | | `Chain([...])` | The result of applying each policy in turn to the output of the one before | The default token estimate is the payload byte count divided by four, rounded up. Every SDK and the managed session index use this same estimate. To use your own tokenizer, pass an estimator: Rust `TokenBudget::with_estimator(max, fn)`, TypeScript `new TokenBudget(max, fn)`, and Python `TokenBudget(max, estimator=fn)`. `fetch(topics, n)` and `block(topics, n)` keep the last `n` messages and take an optional token budget applied after `n`, the same as `Chain([LastN(n), TokenBudget(t)])`. Rust passes the budget as an `Option` third argument and TypeScript as an optional third argument. Python takes `topics`, `n`, and `token_budget` as keywords, `n` defaults to 50, and without `topics` it reads `agent.sessions`. Each built-in policy reports a `name`, a `version`, and a `selection(history)` with the kept and dropped messages and the reason. The names are `last_n(n)`, `token_budget(n)`, `role_filter`, and `chain(..)` with the inner names. A [session](/laser-sdk/advanced/sessions#record-model-and-tool-calls) records the name and version in the manifest of an assembled context. ### Write your own policy A policy takes the ordered history and returns the messages to keep. Each kept message still names its source topic. | Language | Shape | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Rust | Implement `ContextPolicy` with `select(&self, history: &[ContextMessage]) -> Vec`. `name` defaults to `custom` and `version` to `1` | | TypeScript | An object with `select(history)`, and optionally `name()`, `version()`, and `selection(history)` | | Python | A synchronous callable that takes the messages, or an object with a `select(messages)` method. It can set `name` and `version` attributes | This policy drops large messages before the count limit applies. ```ts const small: ContextPolicy = { select: (history) => history.filter((message) => message.payload.byteLength < 2_000) } const turns = await ctx.fetchWith([AgentTopic.Sessions], new Chain([small, new LastN(20)])) ``` ```rust struct Small; impl ContextPolicy for Small { fn select(&self, history: &[ContextMessage]) -> Vec { history .iter() .filter(|message| message.payload.len() < 2_000) .cloned() .collect() } } let turns = scope .fetch_with( vec![AgentTopic::Sessions], Box::new(Chain(vec![Box::new(Small), Box::new(LastN(20))])), ) .await?; ``` ```python def small(messages): return [message for message in messages if len(message.payload) < 2_000] turns = await ctx.fetch_with([ls.AgentTopic.Sessions], ls.Chain([small, ls.LastN(20)])) ``` ## Assemble without a scope `ContextAssembler` reads a conversation directly and exposes every read control. Rust and TypeScript use a builder and finish with `.build().assemble(laser)`. Python's `laser.assemble_context(conversation_id, ..)` takes the same controls as keywords. | Control | Rust builder | TypeScript builder | Python keyword | | -------------------------------- | --------------------------------- | ----------------------------- | ------------------------------ | | Conversation, required | `.conversation_id(id)` | `.conversationId(id)` | First argument | | Topics, default `agent.sessions` | `.topics(vec)` | `.topics([..])` | `topics=` | | Policy, default `LastN(50)` | `.policy(Box)` | `.policy(policy)` | `policy=` | | Include child conversations | `.across_subconversations(true)` | `.acrossSubconversations()` | `across_subconversations=True` | | Start offsets per partition | `.from_offsets(map)` | `.fromOffsets(map)` | `from_offsets=` | | Resume after a checkpoint | `.from_checkpoint(checkpoint)` | `.fromCheckpoint(checkpoint)` | `from_checkpoint=` | | Stop at a checkpoint | `.to_checkpoint(checkpoint)` | `.toCheckpoint(checkpoint)` | `to_checkpoint=` | Child conversations are those whose parent or root conversation is this one. Start offsets are per partition, and one map applies to every topic. A checkpoint is per topic and per partition, and it overrides the start offsets. When resuming, a partition missing from the checkpoint starts at zero. When stopping, a missing partition reads as empty. Python also takes the shorthand `last_n=`, `roles=`, and `token_budget=`. `policy=` replaces them, and combining `policy=` with any of them raises an invalid error. With `roles=` and no `last_n=`, Python applies no count limit. Otherwise `last_n` defaults to 50. ## Read window A context read does not walk an unlimited history. Each partition read examines at most `CONTEXT_READ_WINDOW` (10,000) raw records before it keeps this conversation's messages. * An open read takes the newest 10,000 records. * A read that stops at a checkpoint takes the 10,000 that end at the checkpoint. * A read that resumes from offsets or a checkpoint starts at the later of that position and the start of the newest 10,000. Many conversations share a partition, so older messages of a quiet conversation on a busy partition can fall outside the read. All three SDKs use the same window. State folds are different: every replay bound except the last-N bound reads its whole range, see [Fold state from the log](#fold-state-from-the-log). ## The scope reaches every primitive One scope also gives you: * `block(topics, n, token_budget)`, the last `n` messages as one newline-joined, prompt-ready string, with the same optional token budget as `fetch`. Python takes `block(topics=, n=, token_budget=)`. * `memory(..)`, this conversation's memory. See [Scoped memory](#scoped-memory). * `graph(name)`, the shared knowledge graph. It is not narrowed to the conversation, because a graph holds relationships shared across conversations. In Rust, it needs the `graph` feature. * `state(topics, bound, init, fold)`, state rebuilt from the log. See [Fold state from the log](#fold-state-from-the-log). * `checkpoint(topics)`, where the topics end now. Python's `topics` defaults to `agent.sessions`. `context::checkpoint(&laser, &topics)` in Rust, `context_checkpoint(laser, topics)` in Python, and `contextCheckpoint(laser, topics)` in TypeScript record the same without a scope. ## Scoped memory `scope.memory(..)` returns a scoped memory, a memory handle with the conversation already applied, so `remember` and `recall` take no conversation argument. Rust takes a namespace, and `memory_with(namespace, backend)` picks the backend. Python takes a namespace or a memory handle, and `memory_with(namespace, backend, embedder=)` picks the backend. TypeScript takes a namespace or a memory handle, such as `laser.memory("notes")` or a vector handle, and `memoryWith(namespace, backend, embedder?)` picks the backend. All three SDKs support `remember`, `recall`, `search`, `block(token_budget)`, `consolidate(max_items)`, `forget(id)`, and `improve(..)`. | Verb | Rust | TypeScript | Python | | -------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------- | | Remember | `remember(bytes).send()` | `remember(bytes).send()` | `remember(payload)` returns the ID | | Recall | `recall()` builder, then `fetch()` | `recall()` builder, then `fetch()` | `recall(limit=, folded=, ..)` | | Keyword search | `search(query)` builder, then `.limit(n)`, `.folded()`, and `.await` | `search(query, { limit, folded })` | `search(query, limit=, folded=)` | | Feedback | `improve(Feedback::new(id, weight))` | `improve({ target, weight })` | `improve(target, weight, note=)` | | Consolidate | `consolidate(max_items)`, or `consolidate_with(max_items, summarizer, prune_summarized)` | `consolidate(maxItems, { summarizer, pruneSummarized })` | `consolidate(max_items, summarizer=, prune_summarized=)` | `search` is a keyword recall. It needs no embedder, returns at most 50 items unless you set a limit, and reads the managed view unless you fold. The `handle` accessor returns the underlying memory for reads across conversations, and unscoped `laser.memory(..)` reads across conversations too. Default recall reads the managed key-value view on Laser Stack or LaserData Cloud. Folded recall rebuilds memory from the topic and works on plain Apache Iggy. [Memory](/laser-sdk/memory) covers recall strategies, backends, and consolidation. This example remembers a note in the conversation, recalls it with folded recall, gives it feedback, and forgets it. ```ts const notes = laser.context(conversation).memory("notes") const id = await notes.remember(new TextEncoder().encode("drain node-7 before the upgrade")).send() const hits = await notes.recall().limit(5).folded().fetch() await notes.improve({ target: id, weight: 1 }) await notes.forget(id) ``` ```rust let notes = laser.context(conversation).memory("notes"); let id = notes .remember("drain node-7 before the upgrade".as_bytes()) .send() .await?; let hits = notes.recall().limit(5).folded().fetch().await?; notes.improve(Feedback::new(id, 1.0)).await?; notes.forget(id).await?; ``` ```python notes = laser.context(conversation).memory("notes") note_id = await notes.remember("drain node-7 before the upgrade") hits = await notes.recall(limit=5, folded=True) await notes.improve(note_id, 1.0) await notes.forget(note_id) ``` ## Fold state from the log `scope.state(topics, bound, init, fold)` folds the conversation's messages into your own state under a replay bound. | Bound | Rust `ReplayBound` | TypeScript | Python keyword | Reads | | ------------------ | -------------------- | ----------------------------------------- | ------------------ | ------------------------------------------- | | Last N messages | `Last(n)` | `{ kind: "last", count }` | `last_n=` | The newest 10,000 records of each partition | | From offsets | `FromOffsets(map)` | `{ kind: "from-offsets", offsets }` | `from_offsets=` | From the offsets to the tail | | After a checkpoint | `FromCheckpoint(cp)` | `{ kind: "from-checkpoint", checkpoint }` | `from_checkpoint=` | From the checkpoint to the tail | | Up to a checkpoint | `At(cp)` | `{ kind: "at", checkpoint }` | `at=` | From the start to the checkpoint | | Whole partition | `Full` | `{ kind: "full" }` | `full=True` | Everything | Python takes exactly one bound keyword. For state folds, the offsets are grouped by topic name, unlike the single map that `ContextAssembler` takes, so one topic's offsets never skip records in another topic. `ConversationState.load(laser, conversation, topics, bound, init, fold)` runs the same fold without a scope. Rust calls `ConversationState::load`, TypeScript `ConversationState.load`, and Python `ConversationState.load(laser, conversation, topics, init, fold, *, ..)` with the bound as a keyword. `scope.checkpoint(topics)` records the next offset of each partition of each topic. A checkpoint covers whole partitions, so other conversations on those partitions move its offsets too. It is a client bookmark, never a record on the log. Persist it with `serde` in Rust, `JSON.stringify` and `Checkpoint.fromJSON` in TypeScript, and `to_json` and `Checkpoint.from_json` in Python. This example records a checkpoint, appends one more message, and counts the conversation's messages after the checkpoint. ```ts const topics = [AgentTopic.Sessions] const checkpoint = await ctx.checkpoint(topics) await ctx.append(AgentTopic.Sessions, new TextEncoder().encode("drain node-9")) const since = await ctx.state(topics, { kind: "from-checkpoint", checkpoint }, 0, (count) => count + 1) ``` ```rust let topics = vec![AgentTopic::Sessions]; let checkpoint = scope.checkpoint(&topics).await?; scope.append(AgentTopic::Sessions, "drain node-9".as_bytes()).await?; let since = scope .state(topics, ReplayBound::FromCheckpoint(checkpoint), 0usize, |count, _message| count + 1) .await?; ``` ```python topics = [ls.AgentTopic.Sessions] checkpoint = await ctx.checkpoint(topics) await ctx.append(ls.AgentTopic.Sessions, b"drain node-9") since = await ctx.state(topics, 0, lambda count, message: count + 1, from_checkpoint=checkpoint) ``` ## Snapshots A snapshot saves folded state at a checkpoint, so a later fold replays only the records after it. `state_with(store, topics, init, fold)` (TypeScript: `stateWith`) starts from the store's latest snapshot and folds only the records after it. A conversation with no snapshot folds fully from `init`. `ConversationState.load_with(laser, store, conversation, topics, init, fold)` (TypeScript: `loadWith`) does the same without a scope. Rust and Python decode the saved state as JSON. TypeScript decodes JSON by default and takes an optional `decodeState` function as the last argument. To save one, build it with `snapshot_from_checkpoint(laser, conversation, fold, checkpoint, state)` (TypeScript: `snapshotFromCheckpoint`) and pass it to the store's `save`. The state is the folded value encoded as JSON bytes. A snapshot stores its stream name, stream ID, and stream creation time, the conversation, and the fold name. It stores inclusive offsets for each topic ID, topic creation time, and partition. Before a fold uses a snapshot, it checks those creation times. When the stream or a topic was recreated, the fold refuses the snapshot with an invalid error. Two built-in stores keep snapshots under `agent.snapshots` by default. `TopicSnapshotStore` keeps them on a topic and works on plain Apache Iggy. `KvSnapshotStore` keeps them in the managed key-value view and needs Laser Stack or LaserData Cloud. | Store | TypeScript | Rust | Python | | ---------------------------- | --------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------ | | Topic, default name | `new TopicSnapshotStore(laser, fold)` | `TopicSnapshotStore::new(laser, fold)` | `TopicSnapshotStore(laser, fold)` | | Topic, your name | `new TopicSnapshotStore(laser, fold, topic)` | `TopicSnapshotStore::on_topic(laser, topic, fold)` | `TopicSnapshotStore.on_topic(laser, topic, fold)` | | Key-value, default namespace | `new KvSnapshotStore(laser, fold)` | `KvSnapshotStore::new(laser, fold)` | `KvSnapshotStore(laser, fold)` | | Key-value, your namespace | `new KvSnapshotStore(laser, fold, namespace)` | `KvSnapshotStore::in_namespace(laser, namespace, fold)` | `KvSnapshotStore.in_namespace(laser, namespace, fold)` | A custom store implements `latest(conversation)` and `save(snapshot)`. Python accepts the backend directly or wraps it with `SnapshotStore(backend)`, and its methods can return directly or through an awaitable. Rust `snapshot::encode` and `decode`, Python `encode_snapshot` and `decode_snapshot`, and TypeScript `encodeSnapshot` and `decodeSnapshot` share the stored CBOR bytes. `resume_offsets` (TypeScript: `resumeOffsets`) converts inclusive snapshot offsets to next-read offsets and stops at the unsigned 64-bit maximum. `checkpoint_from_snapshot` (TypeScript: `checkpointFromSnapshot`) turns a snapshot into a checkpoint after the same creation time checks. ## Where it runs Context scopes, policies, assembly, state folds, checkpoints, and topic snapshots use ordinary Iggy topics and need no managed backend. Run `bootstrap(partitions, retention)` once per stream to create the agent topics, and give `agent.sessions` an explicit retention. In Rust, context needs the `agent` feature. Default recall through `scope.memory(..)` and `KvSnapshotStore` need the managed key-value view on Laser Stack or LaserData Cloud. ## Key operations | Verb | What it does | | --------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `context(conversation_id)` | Scope everything below to one conversation | | `append(topic, payload)` | Write one message under this conversation | | `fetch(topics, n)` | Read the last `n` messages. Python: `fetch(topics=, n=, token_budget=)` | | `fetch_with(topics, policy)` | Read under a policy | | `block(topics, n, token_budget)` | The last `n` messages as one prompt-ready string | | `LastN`, `TokenBudget`, `RoleFilter`, `Chain` | The built-in policies | | `ContextAssembler` | Read a conversation with offsets, checkpoints, and child conversations. Python: `assemble_context` | | `memory(..)` / `graph(name)` | This conversation's memory, and the shared graph | | `state(topics, bound, init, fold)` | Fold the conversation's log into your own state | | `state_with(store, ..)` | The same fold seeded from a snapshot | | `ConversationState.load` / `load_with` | The same folds without a scope | | `checkpoint(topics)` | Save the next offset of each partition | | `TopicSnapshotStore` / `KvSnapshotStore` | The built-in snapshot stores | TypeScript uses camelCase names, for example `fetchWith`, `stateWith`, and `loadWith`. Source: https://docs.laserdata.com/laser-sdk/advanced/context --- # Key-value state in depth This page is the full reference for [Key-value state](/laser-sdk/state). Start there for the short version. ## How the store works `laser.kv(namespace)` opens a store within one namespace. Keys are unique within a namespace, and scans and bulk deletes stay inside it. Every managed write goes through a mutation topic on the log first. A stored entry records the log record that wrote it in `source`, so you can find the source message while the log still holds it. The store needs the managed services of [Laser Stack](/laser-sdk/laser-stack) or LaserData Cloud. Key-value operations, compare-and-swap, fenced leases, and forks have separate capabilities. Check them before you depend on them. Without a capability, the matching call returns an unsupported error before anything is sent. | Capability | Rust and Python | TypeScript | Covers | | ---------------- | ------------------------------- | ------------------------------ | --------------------------------------------------------------------- | | Key-value | `capabilities.kv.available` | `capabilities.kv.available` | Every call on `kv(..)` | | Compare-and-swap | `capabilities.kv.cas` | `capabilities.kv.cas` | `.commit()` | | Fenced leases | `capabilities.kv.fenced_leases` | `capabilities.kv.fencedLeases` | `lease`, `renew_lease`, `release`, `get_entry_at_least`, `cas_fenced` | | Forks | `capabilities.forks` | `capabilities.forks` | `fork(..)` and `forks()` | ## Names and limits A client with a default stream scopes the namespace to that stream, so `laser.kv("config")` addresses `stream:/config`. Lease and fence namespaces and fork IDs follow the same rule. `kv_namespaces()` and `forks()` return your stream's names without the prefix. `namespace` on a handle returns the name you gave it, and `resource_namespace` (TypeScript: `resourceNamespace`) returns the name it sends. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names) for the opt out. | Item | Limit | | --------------- | ------------------------------------------------------------- | | Key | Non-empty, at most 512 bytes | | Namespace | Non-empty, at most 128 bytes, no ASCII control characters | | Value | At most 8 MiB | | Scan page | 100 entries by default, at most 1,000 | | Lease TTL | From 1 second to 5 minutes | | Lease holder ID | Non-empty, at most 128 bytes | | Fork ID | At most 128 bytes of ASCII letters, digits, `-`, `_`, and `.` | Keys are bytes. Rust takes anything byte-like, Python takes `str` or `bytes`, and TypeScript takes `Uint8Array`. A key or value over its limit fails with an invalid error before the request is sent. ## Write and read values `set(key)` starts a write. Encode with `.json(value)` or `.msgpack(value)`, pass raw bytes with `.bytes(payload)`, or use any codec with `encode_with` (TypeScript: `encodeWith`). Add an optional expiry with `.ttl(..)`, which counts from now, or with `.expires_at(epoch_micros)` for an absolute time (TypeScript: `expiresAt`). Then call `.send()`. Read bytes with `get(key)`, decode JSON with `get_typed(key)`, or decode with any codec through `get_as` (TypeScript: `getAs`). TypeScript's `getTyped(key, decode)` also takes a decode function that checks the value. `get_entry(key)` (TypeScript: `getEntry`) returns the value with its key, version, expiry, and source. A missing or expired key reads as empty. | TTL argument | TypeScript | Rust | Python | | --------------------------------- | ------------------------------------------ | --------------------------- | ------------------------------ | | Value TTL in `.ttl(..)` | `ttlMs`, milliseconds | `Duration` | `ttl_ms`, milliseconds | | `expire(key, ttl)` | Optional `ttlMs` | `Option` | Optional `ttl_ms` | | Lease TTL | `ttlMs` | `Duration` | `ttl_ms` | | Granted TTL on the returned lease | `grantedTtlMs`, milliseconds as a `number` | `granted_ttl`, a `Duration` | `granted_ttl_ms`, milliseconds | MessagePack encodes structured values as named maps. TypeScript emits the shortest integer encoding and rejects a `bigint` outside the signed and unsigned 64-bit ranges. Small decoded integers are JavaScript numbers, and values encoded as 64-bit integers decode as `bigint`. ## Compare-and-swap Compare-and-swap rejects an update based on an outdated version: 1. Read the value and version with `get_entry(key)`. 2. Set `expect_version(version)` (TypeScript: `expectVersion`) on the next write and finish with `.commit()`. 3. If the version changed, the write fails with a version conflict that carries the current version. Read again before you retry. `expect_absent()` (TypeScript: `expectAbsent`) permits a write only when the key is absent. Use guarded writes for counters and any value with more than one writer. `.commit()` applies a guarded write and returns the new version. It needs `expect_version` or `expect_absent`, or it fails with an invalid error. `.send()` writes unconditionally. If you set a precondition and then call `.send()`, all three SDKs fail with an invalid error that tells you to call `.commit()`, so a precondition is never dropped. Check a version conflict with `error.is_version_conflict()` in Rust, `error.version_conflict` in Python, or `isVersionConflict(error)` in TypeScript, which checks for a `KvExecutionError` whose `detail.kind` is `"versionConflict"`. ## More operations All three SDKs provide these, with snake\_case in Rust and Python and camelCase in TypeScript: * Call `delete(key)` to remove a key. It reports whether it removed a live entry. `exists(key)` returns the version, expiry, and size without the value, or nothing when the key is absent. * Call `expire(key, ttl)` to change the expiry without rewriting the value or changing its version. The TTL counts from now. Leave it out (Rust: pass `None`) to clear the expiry. `expire_at(key, epoch_micros)` (TypeScript: `expireAt`) sets an absolute expiry the same way. * Call `patch(key, patch)` to apply a codec-specific merge patch. It returns the new version. For JSON values, pass JSON merge-patch bytes. * Call `copy_to(key, to_key)` or `move_to(key, to_key)` to copy or rename in one transaction. Both return the new version of the destination. The destination is overwritten and the value keeps its remaining expiry. An absent or expired source returns a not-found error. For another namespace, chain `into_namespace(ns)` (TypeScript: `intoNamespace`) and finish with `.send()`. Python also accepts `to_namespace=` and can await the request directly. * Call `get_many(keys)` to read several keys in one round trip. It returns one optional value per key, in key order. * Call `delete_many()` and select entries with `.prefix(p)`, `.range(start, end)`, or `.key_contains(s)`. `.send()` returns the number removed. Without a bound, it clears the namespace. * Call `scan()` with the same bounds plus `.limit(n)` and `.cursor(c)`. Finish with `.fetch()` for one page and its cursor, or `.entries()` to walk every page. `range` includes the start and excludes the end. `key_contains` skips keys that are not UTF-8. * Call `laser.kv_namespaces()` (TypeScript: `kvNamespaces`) to list the namespaces that hold entries for the caller. ```ts const key = new TextEncoder().encode("service:auth") await kv.expire(key, 3_600_000) // one hour from now await kv.expireAt(key, BigInt(Date.now()) * 1000n + 86_400_000_000n) // an absolute time await kv.expire(key) // clear the expiry const removed = await kv.deleteMany().prefix(new TextEncoder().encode("session:")).send() ``` ```rust use std::time::{Duration, SystemTime, UNIX_EPOCH}; kv.expire("service:auth", Some(Duration::from_secs(3_600))).await?; // one hour from now let tomorrow = SystemTime::now().duration_since(UNIX_EPOCH)? + Duration::from_secs(86_400); kv.expire_at("service:auth", Some(tomorrow.as_micros() as u64)).await?; // an absolute time kv.expire("service:auth", None).await?; // clear the expiry let removed = kv.delete_many().prefix("session:").send().await?; ``` ```python import time await store.expire("service:auth", 3_600_000) # one hour from now tomorrow = time.time_ns() // 1_000 + 86_400_000_000 await store.expire_at("service:auth", tomorrow) # an absolute time await store.expire("service:auth") # clear the expiry removed = await store.delete_many().prefix("session:").send() ``` ## Locks and fenced writes A lease grants temporary ownership of a key. A fencing token identifies the current grant, so a stale holder cannot keep writing. A lease belongs to a holder, a coordination namespace, and a lease key. Acquire it with `lease(key, holder, ttl)`. The call returns the token, the granted TTL, and the commit position of the grant. TypeScript spells the other calls `renewLease`, `getEntryAtLeast`, and `casFenced`. The store can grant less TTL than you asked for, never more. A live lease always conflicts with a new acquisition, so extend it with `renew_lease`, not by acquiring again. Before planning a protected update, call `get_entry_at_least(key, lease.position)`. It waits until the read view includes the lease grant. If the view does not catch up in time, the read fails with a stale error, which you can retry. Then write with `cas_fenced(key, fence_namespace, fence_key, token)`, which takes the coordination namespace and lease key and needs a precondition. The write succeeds only while the lease is held and the token is current. ```ts const kv = laser.kv("config") const key = new TextEncoder().encode("service:auth") const lock = new TextEncoder().encode("lease:service:auth") const holder = "worker-a" const lease = await kv.lease(lock, holder, 30_000) try { const entry = await kv.getEntryAtLeast(key, lease.position) if (entry === undefined) throw new Error("config entry missing") await kv .casFenced(key, "config", lock, lease.token) .json({ log_level: "debug" }) .expectVersion(entry.version) .commit() await kv.renewLease(lock, holder, lease.token, 30_000) } finally { await kv.release(lock, holder, lease.token) } ``` ```rust let kv = laser.kv("config"); let lease = kv .lease("lease:service:auth", "worker-a", Duration::from_secs(30)) .await?; let entry = kv .get_entry_at_least("service:auth", lease.position) .await? .ok_or_else(|| LaserError::Invalid("config entry missing".to_owned()))?; kv.cas_fenced("service:auth", "config", "lease:service:auth", lease.token) .json(&serde_json::json!({"log_level": "debug"}))? .expect_version(entry.version) .commit() .await?; kv.renew_lease("lease:service:auth", "worker-a", lease.token, Duration::from_secs(30)) .await?; kv.release("lease:service:auth", "worker-a", lease.token).await?; ``` ```python store = laser.kv("config") lease = await store.lease("lease:service:auth", "worker-a", 30_000) try: entry = await store.get_entry_at_least("service:auth", lease.position) if entry is None: raise RuntimeError("config entry missing") await ( store.cas_fenced("service:auth", "config", "lease:service:auth", lease.token) .json({"log_level": "debug"}) .expect_version(entry.version) .commit() ) await store.renew_lease("lease:service:auth", "worker-a", lease.token, 30_000) finally: await store.release("lease:service:auth", "worker-a", lease.token) ``` The example needs an existing entry and the fenced-lease capability. Renewal keeps the same token. Release revokes it at once and returns true when a held lease was released, or false when none was held. A stale holder's next write fails with lease-lost, even before another holder acquires the lease. Check it with `error.is_lease_lost()` in Rust, `error.lease_lost` in Python, or `isLeaseLost(error)` in TypeScript. Renew before expiry and stop protected effects if renewal fails. Give each independently running worker its own holder ID, and use a stable worker ID rather than a connection value. If the outcome of an acquisition is unknown, for example because the reply was lost, the SDK waits for the requested TTL and then returns an ambiguous-mutation error. The wait lets any grant the server made expire, so you can acquire again afterwards. `lease` runs over a dedicated coordination connection, one acquisition at a time per client. That gate stays closed through the wait, even if the caller dropped the call. The dedicated connection needs a `Laser` that connected from a connection string. A `Laser` wrapped around your own Iggy client returns a configuration error, so use `FencedLeaseClient` with your own transport there. Lease TTL, read consistency, and value TTL are separate settings. ### Prepared coordination requests `FencedLeaseClient` exposes the two-phase coordination API. Use it when you need to keep the prepared request and read its recovery action. `connect_dedicated(connection_string)` (TypeScript: `connectDedicated`) builds a client over its own connection, opened on first use. To use another transport, pass it to the constructor. A custom transport supplies `send` and `reset`, and can add `ready` and `close`. `reset` must stop all in-flight work before it returns. First call `prepare_acquire`, `prepare_renew`, `prepare_release`, or `prepare_cas_fenced`. Each validates its request once and returns a `PreparedMutation` bound to that client and operation. Then pass it to `acquire`, `renew`, `release`, or `cas_fenced`. Rust takes the `KvLease`, `KvLeaseRenew`, `KvRelease`, and `KvCasFenced` structs. Python takes dictionaries with the same field names, `bytes` keys, and `"v": 1`. TypeScript takes objects with camelCase fields, `Uint8Array` keys, and `bigint` numbers. Repeating a prepared object sends the same operation ID and the same request bytes. A prepared object from another client, or one passed to the wrong operation, is refused with an invalid error. ```ts import { FencedLeaseClient, isAmbiguousMutation } from "@laserdata/laser-sdk" const client = FencedLeaseClient.connectDedicated(connectionString).withAttemptTimeout(5_000) const key = new TextEncoder().encode("lease:service:auth") try { const lease = await client.acquire( client.prepareAcquire({ namespace: "config", key, holderId: "worker-a", leaseTtlMicros: 30_000_000n }) ) const renewal = client.prepareRenew({ namespace: "config", key, holderId: "worker-a", leaseToken: lease.token, leaseTtlMicros: 30_000_000n }) try { await client.renew(renewal) } catch (error) { if (!isAmbiguousMutation(error)) throw error // renewal.ambiguousRecovery is { kind: "repeatPrepared" } await client.renew(renewal) } } finally { await client.close() } ``` ```rust use laser_sdk::kv::{FencedLeaseClient, KV_LEASE_OP_VERSION, KvLease, KvLeaseRenew}; use std::time::Duration; let client = FencedLeaseClient::connect_dedicated(connection_string) .with_attempt_timeout(Duration::from_secs(5)); let acquire = client.prepare_acquire(&KvLease { v: KV_LEASE_OP_VERSION, namespace: "config".to_owned(), key: b"lease:service:auth".to_vec(), lease_ttl_micros: 30_000_000, holder_id: "worker-a".to_owned(), subject_user_id: None, })?; let lease = client.acquire(&acquire).await?; let renewal = client.prepare_renew(&KvLeaseRenew { v: KV_LEASE_OP_VERSION, namespace: "config".to_owned(), key: b"lease:service:auth".to_vec(), holder_id: "worker-a".to_owned(), subject_user_id: None, lease_token: lease.token, lease_ttl_micros: 30_000_000, })?; if let Err(error) = client.renew(&renewal).await { if !error.is_ambiguous_mutation() { return Err(error); } // renewal.ambiguous_recovery() is AmbiguousMutationRecovery::RepeatPrepared client.renew(&renewal).await?; } client.close().await; ``` ```python import laser_sdk as ls client = ls.FencedLeaseClient.connect_dedicated(connection_string).with_attempt_timeout(5_000) try: lease = await client.acquire( client.prepare_acquire({ "v": 1, "namespace": "config", "key": b"lease:service:auth", "lease_ttl_micros": 30_000_000, "holder_id": "worker-a", }) ) renewal = client.prepare_renew({ "v": 1, "namespace": "config", "key": b"lease:service:auth", "holder_id": "worker-a", "lease_token": lease.token, "lease_ttl_micros": 30_000_000, }) try: await client.renew(renewal) except ls.LaserError as error: if not error.ambiguous_mutation: raise # renewal.ambiguous_recovery == ls.AmbiguousMutationRecovery.repeat_prepared() await client.renew(renewal) finally: await client.close() ``` An attempt times out after 10 seconds by default. Override it with `with_attempt_timeout(Duration)` in Rust, `with_attempt_timeout(timeout_ms)` in Python, or `withAttemptTimeout(timeoutMs)` in TypeScript, both in milliseconds. A zero timeout is refused before anything is sent. A timeout or an uncertain transport failure retires the connection and returns an ambiguous-mutation error. Read `ambiguous_recovery()` in Rust, the `ambiguous_recovery` property in Python, or `ambiguousRecovery` in TypeScript to learn what to do next. Python returns an `AmbiguousMutationRecovery` whose `kind` names the action and which compares equal to the matching factory. | Operation | Rust | Python | TypeScript | Recovery | | ----------------------- | ------------------------------ | ----------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------- | | Acquire | `WaitForLeaseExpiry(Duration)` | `AmbiguousMutationRecovery.wait_for_lease_expiry(ttl_ms)` | `{ kind: "waitForLeaseExpiry", ttlMs }` | Wait through the requested lifetime before acquiring under a new identity | | Renew, release | `RepeatPrepared` | `AmbiguousMutationRecovery.repeat_prepared()` | `{ kind: "repeatPrepared" }` | Repeat the same prepared object | | Fenced compare-and-swap | `ReconcileTargetPrecondition` | `AmbiguousMutationRecovery.reconcile_target_precondition()` | `{ kind: "reconcileTargetPrecondition" }` | Read the target and reconcile it against the write's precondition | The low-level client reports the action and leaves the acquisition wait to you. The ordinary `lease` call performs the wait. `get(request)` is the barriered read on the same dedicated connection. A timed-out `get` returns a plain timeout, not an ambiguous-mutation error. Call `close()` when finished. Python supports `async with` and TypeScript supports `await using`. Closing the client or its `DedicatedKvTransport` is final, and later calls fail before anything is sent. `reset()` on a transport retires its connection and allows later reuse. Readiness, authentication, unsupported-capability, and request validation failures happen before the request is sent, so they are definite failures. ## Local state stores A state store is the small `get`, `set`, and `delete` interface that agents use for checkpoints, cursor offsets, and duplicate-suppression keys. Keys are strings and values are bytes. `InMemoryStore` keeps entries in process and loses them on restart. `FileStore(root)` writes one file per key under a directory, with the key hex-encoded in the file name, and survives restarts. Both work with Iggy alone and have no size limits. Rust defines the `StateStore` trait in `laser_sdk::state_store` behind the `agent` feature, and Rust's `Kv` implements it, so `laser.kv(namespace)` is the managed drop-in. A `Kv` used as a `StateStore` writes without expiry and keeps the key and value limits. TypeScript exports a `StateStore` interface, and `new KvStore(kv)` is the managed store behind it. Python exposes a `StateStore` base class that `InMemoryStore`, `FileStore`, and `KvStore(kv)` extend. In every SDK the managed store's `set` never expires. ```ts import { FileStore, type StateStore } from "@laserdata/laser-sdk" const store: StateStore = new FileStore("/var/lib/agent-state") await store.set("cursor:metrics", new TextEncoder().encode("42")) const saved = await store.get("cursor:metrics") await store.delete("cursor:metrics") ``` ```rust use laser_sdk::state_store::{FileStore, StateStore}; let store = FileStore::new("/var/lib/agent-state"); store.set("cursor:metrics", b"42".to_vec()).await?; let saved = store.get("cursor:metrics").await?; store.delete("cursor:metrics").await?; ``` ```python store = ls.FileStore("/var/lib/agent-state") await store.set("cursor:metrics", b"42") saved = await store.get("cursor:metrics") await store.delete("cursor:metrics") ``` ## Forks A fork is a copy-on-write branch of the materialized [views](/laser-sdk/advanced/views). It stores its rows apart from the main rows, so you can try a change before you apply it. Forks branch view tables, not the key-value store, so `kv.set(..)` is never redirected into a fork. Choose the branch behavior when you create it: * Continuous, the default, keeps seeing later appends to the main data while it keeps its own writes. * Severed, selected with `.severed()`, captures the main data at its current offsets and hides later appends. `.tables([...])` narrows the snapshot to some tables. An empty list captures every table. The fork lifecycle: * Create the branch with `fork(id).create()`. It returns its metadata: ID, kind, parent, row count, status, owner, and creation time. Rust and TypeScript finish it with `.send()` and select the default explicitly with `.continuous()`. `.parent(id)` records lineage for audit only, and the data still branches from the main data. Python passes `severed`, `continuous`, `parent`, and `tables` as keywords to `create(..)`, and setting both `severed` and `continuous` raises an invalid error. * Write a row with `put_row(table, partition, offset)` (TypeScript: `putRow`, with the offset as a `bigint`). It writes at an exact source position. Add `.field(..)`, `.payload(..)`, `.embedding(..)`, `.metadata(..)`, or `.projection(id, version)`, or call `.tombstone()` to hide the main row at that position. `.embedding(..)` takes numbers, and a value that is not finite is an invalid error. TypeScript raises it from `.embedding(..)`, and Rust and Python raise it from `.send()`. * Read the branch with `query(index).fork(id)`. Ordinary reads and queries of the main data do not see its rows until promotion. * Call `promote()` to apply the fork's writes to the main data. It closes the fork and returns the number of rows applied. * Call `squash()` to discard the fork. It returns true when an open fork was removed. * Call `laser.forks()` to list the open forks of the authenticated user. `fork.id` returns the fork ID (Rust: `fork.id()`), and `resource_id` (TypeScript: `resourceId`) returns the scoped ID to pass to other requests. Before you run this example, register and bind a view whose index is named `service_config` through [Queries and views](/laser-sdk/views). ```ts const fork = laser.fork("experiment-1") await fork.squash() await fork.create().severed().tables(["service_config"]).send() await fork.putRow("service_config", 0, 0n).field("log_level", "trace").send() const applied = await fork.promote() ``` ```rust let fork = laser.fork("experiment-1"); fork.squash().await?; fork.create().severed().tables(["service_config"]).send().await?; fork.put_row("service_config", 0, 0) .field("log_level", "trace") .send() .await?; let applied = fork.promote().await?; ``` ```python fork = laser.fork("experiment-1") await fork.squash() await fork.create(severed=True, tables=["service_config"]) await fork.put_row("service_config", 0, 0).field("log_level", "trace").send() applied = await fork.promote() ``` ## Session links A key-value write can name the [session](/laser-sdk/session) it belongs to. `in_session(reference)` (TypeScript: `inSession`) on a handle stamps every `set`, compare-and-swap, `delete`, and `patch` with the session's stream and ID. Inside a session, `session.kv(namespace)` returns a handle that is already linked. The link names the stream, never a numeric ID. The server checks the link before it stores it. The caller needs its usual key-value grant, `session:write` on the session's stream, and send permission on that stream's `agent.sessions`. A managed deployment then lists the key on the session, and the session's state view shows the write as a `kv.set` entry. The write has no position on the session timeline, because it travels through the mutation log rather than the session's stream. ```ts const flags = session.kv("ticket-flags") await flags.set(new TextEncoder().encode("ticket:42")).json({ escalated: true }).send() const linked = laser.kv("ticket-flags").inSession(session.reference()) ``` ```rust let flags = session.kv("ticket-flags")?; flags.set("ticket:42").json(&serde_json::json!({ "escalated": true }))?.send().await?; let linked = laser.kv("ticket-flags").in_session(session.reference()?); ``` ```python flags = session.kv("ticket-flags") await flags.set("ticket:42").json({"escalated": True}).send() linked = laser.kv("ticket-flags").in_session(session.reference()) ``` ## Session documents and shared key-value resources Session state, the JSON document a session keeps on its lane, is a different thing from the key-value store. It is built from the session's `state_delta` and `state_snapshot` records. See [Sessions in depth](/laser-sdk/advanced/sessions). A key-value namespace lives on the managed mutation log and can be shared by independent sessions. It has its own versions and conflict rules. A session link records which work wrote or used a key. It does not copy the value into the session document. ## Errors | Failure | When | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid | An empty or oversized key, an oversized value or namespace, a lease TTL outside its range, `.commit()` without a precondition, or `.send()` with one | | Unsupported | The deployment does not serve the key-value store, compare-and-swap, fenced leases, or forks | | Version conflict | A compare-and-swap precondition did not match. It carries the current version | | Lease lost | A renewal, release, or fenced write used an expired, released, or replaced lease | | Stale | A barriered read did not catch up in time. Retry it | | Not found | A copy or move named an absent or expired source | | Ambiguous mutation | A coordination request may or may not have been applied. Follow its recovery action | Rust returns these as `LaserError` with helpers such as `is_version_conflict()`, `is_lease_lost()`, `is_stale()`, `is_not_found()`, and `is_ambiguous_mutation()`. Python raises `KvError` for store failures, plus `InvalidError`, `UnsupportedError`, and `AmbiguousMutationError`, and each carries flags such as `version_conflict` and `lease_lost`. TypeScript throws `KvExecutionError`, whose `detail.kind` names the failure, plus `InvalidError`, `UnsupportedError`, and `AmbiguousMutationError`. ## Key operations | Call | What it does | | ---------------------------------------------------------------------- | ---------------------------------------------------------- | | `kv(namespace)` | Open a namespace | | `set(key).json(v).ttl(d).send()` | Write a value with an optional expiry | | `set(..).expect_version(v).commit()` | Guarded write that returns the new version | | `expect_version(v)`, `expect_absent()` | Compare-and-swap preconditions for `commit()` | | `get(key)`, `get_typed(key)`, `get_as(key)` | Read a value, raw or decoded | | `get_entry(key)` | Read a value with its version | | `delete(key)`, `exists(key)` | Remove a key or read its metadata | | `expire(key, ..)`, `expire_at(key, ..)` | Change expiry in place | | `patch(key, patch)` | Merge-patch a structured value | | `copy_to(..)`, `move_to(..)` | Copy or rename a value in one transaction | | `get_many(..)`, `delete_many()`, `scan()` | Batch reads, bulk deletes, paged reads | | `kv_namespaces()` | List the namespaces that hold entries | | `lease(..)`, `renew_lease(..)`, `release(..)` | Hold a revocable lease | | `get_entry_at_least(key, position)`, `cas_fenced(..)` | Barriered read and fenced compare-and-swap | | `FencedLeaseClient` + `prepare_*` | Keep one operation identity through recovery | | `StateStore`, `InMemoryStore`, `FileStore` | Local store for checkpoints and duplicate-suppression keys | | `fork(id).create()`, `put_row(..)`, `promote()`, `squash()`, `forks()` | Branch, write, apply, discard, and list forks | | `in_session(reference)`, `session.kv(namespace)` | Link writes to a session | Source: https://docs.laserdata.com/laser-sdk/advanced/state --- # Queries and views in depth This is the full reference for [Queries and views](/laser-sdk/views). Start there for the short version. A view is a queryable table that the managed plane keeps up to date from messages in the log. A projection says which fields to extract. A binding says which topic feeds the projection and which index the rows land in. Queries read the index. Views need Laser Stack or LaserData Cloud. On standalone Apache Iggy the query capability is off and every call on this page returns an unsupported error. ## How a view is built 1. Register a projection. It names the fields to extract from each payload. 2. Apply a binding. It connects a `(stream, topic)` source to the projection and names the operational index. 3. Publish to the topic as usual. The managed plane extracts the fields from each new record and writes a row to the index. 4. Query the index. Registration and binding both publish control commands. The managed plane applies them asynchronously, so the calls return before the index exists. Register the projection before you apply a binding that names it, or the binding is rejected. A query fails until the index exists, then returns an empty page. A client with a default stream scopes projection IDs, index names, and the index a query names to that stream, as `stream:/`. Listings return your stream's projections without the prefix. Schema registration and lookup use the stream's own registry. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names). In Rust, views need the `projections` and `query` features. The `managed` feature includes both. ## Register a projection All three SDKs build a projection with the same builder: `Projection::builder(id)` in Rust, `new ProjectionBuilder(id)` in TypeScript, and `ls.ProjectionBuilder(id)` in Python. TypeScript uses camelCase method names. `build()` returns the projection, which is a dict in Python. Pass it to `projections().register(projection)`. * `field(name)` indexes a top-level JSON field under the same name. * `fields([...])` indexes several top-level fields. * `field_at(name, pointer)` indexes the value at an RFC 6901 JSON pointer under `name`. `field_at("cpu_pct", "/cpu/value")` indexes `cpu.value` as `cpu_pct`. * `field_typed(name, type)` and `field_at_typed(name, pointer, type)` add a storage type hint for typed-column backends. Rust uses `FieldType::Int`, `Float`, `Bool`, or `Text`. Python and TypeScript use `"int"`, `"float"`, `"bool"`, or `"text"`. The embedded engine ignores the hint. * `vector_field(pointer)` extracts an embedding vector for nearest-neighbor search. Without it, the vector is read from `/embedding`. * `content_type(..)` selects the decoder, such as JSON or Avro. The builder default is `Any`, a best-effort decode. * `inline_payload()` keeps a copy of the original payload with each row, so typed reads decode the row without reading the log. This is the builder default. `index_only()` keeps only the declared fields. * `version(n)` sets the schema version. The default is 1. The recommended ID form is `name.vN`, for example `readings_v1.v1`. * `name(..)` sets the display name. It is separate from the ID, so you can rename a projection without changing the ID that producers send. * `extraction(schema)` replaces the whole extraction plan. `IndexSchemaBuilder` builds one with `field`, `field_at`, `vector_field`, and `inline_payload`. A hand-written object or dict with the same fields also works. It states the payload choice with `inline_payload` (`inlinePayload`) on the extraction and `inline_payload_default` (`inlinePayloadDefault`) on the projection. A dict that omits them stores no payload, unlike the builder. A producer can also ask for one record's payload to be kept with its row with `inline_payload()` on the publish request (TypeScript: `inlinePayload()`). Limits: * A record can carry at most 32 `.index(..)` values. The SDK refuses more when you send. * The managed plane decodes and copies payloads up to 8 MiB. A larger record keeps only the values its `.index(..)` headers set. The projection's fields are not extracted from it, and the row does not copy the payload. * A record with no indexed fields produces no row. The log keeps the original bytes according to the topic's retention. Rows have their own retention, set on the binding, so a row can outlive the message that produced it. Only declared fields are queryable. Other fields stay in the inline payload, when it is kept, or in the log. `projections().register(..)` rejects a graph projection with an invalid error. Register those with `register_graph` as shown in [Graph](/laser-sdk/advanced/graph). ### Read and change the registry * `projections().get(id)` returns one projection with its bindings, or nothing when the ID is unknown. * `projections().list()` browses the registry. Rust and TypeScript narrow it with `for_topic`, `for_topics`, `name_contains`, `id_prefix`, and `search` (camelCase in TypeScript) and finish with `fetch()`. Python passes `topic=`, `topics=`, `name_contains=`, `id_prefix=`, and `search=` to `list(..)`. No filter lists every projection. * `projections().drop(id)` stops materialization for that projection. Rows already written stay. Writes are applied asynchronously. Poll `get(id)` to see when a registration has been applied. ## Bind a topic A binding selects one `(stream, topic)` source and the projections allowed to read it. The builder is `ProjectionBinding::builder()` in Rust, `new ProjectionBindingBuilder()` in TypeScript, and `ls.ProjectionBindingBuilder()` in Python. Pass the result to `bindings().apply(binding)`. * `source(stream, topic)` selects the topic. A binding has exactly one source. `selector(source)` takes a prebuilt source instead. * `allow(projection_id)` adds a permitted projection. Call it once per projection. * `default_projection(id)` handles records that name no projection. Without a default, those records are skipped. * `index(name)` names the operational index. Without it, the index takes the source topic name. * `backend(binding)` pins the index to an exact backend resource ID and generation. Without it, the index uses the replica-local backend. * `retention(policy)` sets how long rows live, independent of topic retention. Without it, the binding uses the deployment default, which mirrors the log unless an operator changed it. * `notify()` turns on [change records](/laser-sdk/advanced/change-feed) for this binding. Notifications are off by default. A producer tags a record for one projection with `projection_ref(id)` on the publish request (TypeScript: `projectionRef`). It sets the `agdx.ref` header. A record tagged for a projection outside the allowed set is skipped or dead-lettered, according to the deployment's policy. The header keeps your local projection ID, and the managed plane resolves it against the bindings of the record's own stream, so a record never selects another stream's projection. The build step differs by language when no source is set: | Call | Rust | TypeScript | Python | | ---------------------------- | --------------------------- | --------------------- | --------------------- | | `build()` | Panics | Throws `InvalidError` | Raises `InvalidError` | | `try_build()` / `tryBuild()` | Returns `Err(InvalidError)` | Returns `undefined` | Raises `InvalidError` | A hand-written binding uses the fields `source`, `allowed_projections`, `default_projection`, `backend`, `index`, `notify`, and `retention` (camelCase in TypeScript). ### Retention policies | Policy | Rust | Python | TypeScript | | -------------------------------------- | -------------------------------------------- | ------------------------------------------- | -------------------------------------- | | Mirror the log, the default | `RetentionPolicy::MirrorLog` | `{"kind": "mirror_log"}` | `{ kind: "mirrorLog" }` | | Keep forever | `RetentionPolicy::Keep` | `{"kind": "keep"}` | `{ kind: "keep" }` | | Keep until the source topic is deleted | `RetentionPolicy::KeepUntilSourceDeleted` | `{"kind": "keep_until_source_deleted"}` | `{ kind: "keepUntilSourceDeleted" }` | | Time to live after materialization | `RetentionPolicy::TimeToLive { ttl_micros }` | `{"kind": "time_to_live", "ttl_micros": n}` | `{ kind: "timeToLive", ttlMicros: n }` | | Newest rows only | `RetentionPolicy::MaxRows { rows }` | `{"kind": "max_rows", "rows": n}` | `{ kind: "maxRows", rows: n }` | In TypeScript, `n` is a `bigint`. Mirror the log removes rows when Iggy removes the messages that produced them. Mirror the log and keep until the source topic is deleted both drop the rows when the source topic is deleted. Keep, time to live, and newest rows only keep their rows after the topic is gone. A policy kind that a newer server sends and your SDK does not know decodes as `Unknown`. Do not apply a binding you read back with an unknown policy, because its original kind is lost. The index and the topic have separate names. The `readings` topic can feed the `readings_v1` index, so you can version the view without renaming the topic that producers use. `bindings().remove(source, projection_ref)` stops materialization for that source. Rows already written stay. Pass a projection ID to stop routing into that one projection only. Rust takes `Option` for the second argument. TypeScript and Python make it optional. ### Register and bind, in code ```ts import { ContentType, ProjectionBindingBuilder, ProjectionBuilder } from "@laserdata/laser-sdk" const projection = new ProjectionBuilder("readings_v1.v1") .name("readings_v1") .version(1) .contentType(ContentType.Json) .indexOnly() .fields(["host", "cpu", "status"]) .build() await laser.projections().register(projection) const binding = new ProjectionBindingBuilder() .source("telemetry", "readings") .allow("readings_v1.v1") .defaultProjection("readings_v1.v1") .index("readings_v1") .retention({ kind: "timeToLive", ttlMicros: 7n * 24n * 3_600_000_000n }) .notify() .build() await laser.bindings().apply(binding) ``` ```rust use laser_sdk::prelude::full::*; let projection = Projection::builder("readings_v1.v1") .name("readings_v1") .version(1) .content_type(ContentType::Json) .index_only() .fields(["host", "cpu", "status"]) .build(); laser.projections().register(projection).await?; let binding = ProjectionBinding::builder() .source("telemetry", "readings") .allow("readings_v1.v1") .default_projection("readings_v1.v1") .index("readings_v1") .retention(RetentionPolicy::TimeToLive { ttl_micros: 7 * 24 * 3_600_000_000, }) .notify() .build(); laser.bindings().apply(binding).await?; ``` ```python import laser_sdk as ls projection = ( ls.ProjectionBuilder("readings_v1.v1") .name("readings_v1") .version(1) .content_type("json") .index_only() .fields(["host", "cpu", "status"]) .build() ) await laser.projections().register(projection) binding = ( ls.ProjectionBindingBuilder() .source("telemetry", "readings") .allow("readings_v1.v1") .default_projection("readings_v1.v1") .index("readings_v1") .retention({"kind": "time_to_live", "ttl_micros": 7 * 24 * 3_600_000_000}) .notify() .build() ) await laser.bindings().apply(binding) ``` After the binding is applied, every new message on `telemetry/readings` updates the view. ## Two ways to index a field With a registered projection, the producer sends a normal payload and the managed plane extracts the fields through the projection's pointers. The producer needs no projection details. The producer can also set an indexed value directly. `.index(key, value)` on a publish request adds an `agdx.idx.` header with a string value. Use it for raw payloads, for writer schemas the plane cannot decode, or for a constant on every record in a batch. A header wins over an extracted value with the same field name. A record can carry at most 32 of these headers. ## Writer schemas `laser.schemas()` is the writer schema registry for Avro, Protobuf, and JSON Schema payloads. Registration is synchronous: the managed plane checks that the definition compiles, allocates the next free ID, and returns it. IDs are permanent and concurrent callers never get the same one. A definition that does not compile returns an unsupported error and allocates nothing. * Rust and TypeScript call `register(source)`, optionally chain `.name(..)` and `.version(..)`, and finish with `.send()`. Python passes `name=` and `version=` to `register(..)`. Name and version are stored labels only. * `get(id)` returns the schema at that ID, active or dropped, or nothing when the ID is free. * `list()` returns every known schema, active and dropped. * `drop(id)` marks the schema dropped. Records stamped with that ID keep decoding, and the ID can never be reused for a different definition. Dropping an unknown ID does nothing. Producers stamp the ID with `.schema_id(id)` on the publish request (TypeScript: `schemaId`). ## Wait for the view Materialization is asynchronous. A message you just published can be missing from the next query. * Wait for a query to succeed after you apply a binding. The examples do this before they publish, so new records reach a running projector. * Use the [change feed](/laser-sdk/advanced/change-feed) to learn when an index advanced, then query. * Use a read-your-writes query when one query must see your own earlier writes. See [Consistency](#consistency). ## Query an index `laser.query(index)` opens a query against an operational index. Chain conditions, then run it with a terminal such as `fetch()`. TypeScript uses camelCase names, such as `whereEq`, `filterGte`, and `countDistinct`. ### Match and filter * `where_eq(field, value)` matches an indexed field exactly. It is the cheap path that index keys answer directly. * `filter_eq`, `filter_ne`, `filter_gt`, `filter_gte`, `filter_lt`, `filter_lte`, `filter_in`, `filter_contains`, and `filter_prefix` compare any indexed field. Chained filters combine with AND. * `filter(expr)` adds a composed condition. Build it with `Filter::pred`, `Filter::all`, `Filter::any`, and `Filter::negate` in Rust, `ls.Filter.pred`, `all`, `any`, and `negate` in Python, and `filterPred`, `filterAll`, `filterAny`, and `filterNegate` in TypeScript. * `message_type(value)` matches the reserved `message_type` field. * `time_range(start, end)` matches the reserved `ts` field, in epoch microseconds. The range includes both ends, and `start` must be before `end`. Value types differ. Rust takes anything that converts into a `TypedValue`, such as `&str`, `i32`, `i64`, or `f64`. Python takes plain Python values. TypeScript `whereEq` takes a string or a `TypedValue`, and the `filter*` comparisons take a `TypedValue` object such as `{ kind: "long", value: 80n }`. `filterContains` and `filterPrefix` take a string. ### Sort and page * `order_asc(field)` and `order_desc(field)` sort. Several calls sort by several fields in order. * `limit(n)` sets the page size. The default is 50 and the maximum is 1,000. Zero or a value above 1,000 fails with an invalid error before anything is sent. * `offset(n)` skips rows. `cursor(c)` resumes from a cursor that a previous page returned. Setting one clears the other. * `with_total()` asks for an exact total count. It costs a full scan on a wide filter. Without it, the total is absent and `has_more` is still exact. Each result has a `page` with `offset`, `limit`, `total`, `has_more`, and `next_cursor` (`hasMore` and `nextCursor` in TypeScript). `next_cursor` is present exactly when `has_more` is true. Aggregate and vector queries always return one page. ### Aggregate * `count`, `sum`, `avg`, `min`, `max`, and `count_distinct` work on every backend. `stddev` and `percentile(field, fraction)` need a columnar backend. * Each aggregate writes to a result field named after itself, such as `count` or `avg`. TypeScript aggregate methods take an optional alias as their last argument. * `agg_as` adds an aggregate with a chosen alias, so you can return several of the same kind. Rust: `agg_as(func, field, fraction, alias)`. Python: `agg_as(func, alias, field=, fraction=)`. TypeScript: `aggAs(func, alias, { field, fraction })`. * `group_by([...])` groups the aggregate. `window(field, every_micros)` buckets it into tumbling windows over a timestamp field, and each result row carries the bucket start in `window_start`. * `having(expr)` filters aggregate output. Its fields name an aggregate alias or a group key, not raw row fields. It needs an aggregate. * An aggregate query cannot also select fields or ask for payloads. ```ts import { queryResultValueI64, queryResultValueText } from "@laserdata/laser-sdk" const byStatus = await laser.query("readings_v1").count().groupBy(["status"]).fetch() for (const row of byStatus.rows) { const status = queryResultValueText(byStatus, row, "status") const count = queryResultValueI64(byStatus, row, "count") console.log(`${status ?? "?"}: ${String(count ?? 0n)}`) } ``` ```rust let by_status = laser .query("readings_v1") .count() .group_by(["status"]) .fetch() .await?; for row in &by_status.rows { let status = by_status.value_text(row, "status").unwrap_or_default(); let count = by_status.value_i64(row, "count").unwrap_or(0); println!("{status}: {count}"); } ``` ```python by_status = await laser.query("readings_v1").count().group_by(["status"]).fetch() for row in by_status.rows: status = by_status.value_text(row, "status") count = by_status.value_i64(row, "count") or 0 print(f"{status}: {count}") ``` ### Search * `nearest(embedding, top_k)` runs a nearest-neighbor search on the `embedding` field. Rust and TypeScript use `nearest_in(field, embedding, top_k)` (`nearestIn`) for another field. Python passes `field=` to `nearest`. * `text(query)` runs a lexical search over every text-hinted indexed field. `text_in(field, query)` narrows it to one field. Relevance lands in each row's `score`. Lexical search needs the `keyword` capability, and the SDK refuses it before sending when the deployment does not advertise it. ### Select fields and payloads * `select_fields([...])` returns only some fields. * `distinct()` returns distinct rows over the selected fields. It needs `select_fields`. * `with_payload()` returns the stored payload bytes on each row. ### Raw SQL `raw_sql` runs one read-only SQL statement on the index's backend. It works on SQL backends only, is not portable, and cannot be combined with any structured condition, sort, aggregate, selection, or payload request. | Language | Call | Dialect | | ---------- | ------------------------------------------------------------------------------------------- | --------------------------- | | Rust | `raw_sql(dialect, sql)`, or `raw_sql_with(dialect, sql, params)` with positional parameters | Required, a `SqlDialect` | | TypeScript | `rawSql(sql, params?, dialect?)` | Defaults to `"data_fusion"` | | Python | `raw_sql(sql, params=None, dialect="data_fusion")` | Defaults to `"data_fusion"` | The dialects are DataFusion, Postgres, MySQL, and SQLite. The statement reads the queried index by the same name the query uses. A stream-scoped name such as `stream:orders/orders.v1` holds `:` and `/`, so quote it: `FROM "stream:orders/orders.v1"`, or with backticks on MySQL. ### Consistency Queries are eventually consistent by default. `read_your_writes()` waits until the projector reaches the end of the source log as it was when the query started. If the projector does not catch up in time, the query fails with a stale error instead of returning older data. `consistency(level)` takes eventual, read-your-writes, or strong. Strong adds agreement across replicas. Read-your-writes and strong need backend support. The SDK checks `capabilities.query.consistency` and returns an unsupported error before sending when the level is not served. Check for a stale error with `error.is_stale()` in Rust, `isStale(error)` in TypeScript, and the `stale` attribute in Python. ### Scope a query * `conversation(id)` returns only the rows one conversation produced. The deployment indexes every message's conversation header, so this works on any projection without producer changes. Rust takes a `ConversationId`. TypeScript and Python take a string. * `fork(id)` reads a fork's copy-on-write view instead of the trunk. See [Key-value state](/laser-sdk/advanced/state). ### Deadlines Each query has one deadline for all its pages, 30 seconds after the request was created by default. The server never extends it. Set a relative deadline with `deadline(duration)` in Rust and `deadline(timeout_ms)` in milliseconds in Python and TypeScript. `deadline_micros(..)` (TypeScript: `deadlineMicros`) sets an absolute time in epoch microseconds in all three SDKs. ## Read results A result has `fields`, `rows`, `page`, and `context`. Each row's values follow the order of `fields`. Read values by field name with the accessors: | Language | Accessors | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Rust and Python | `result.value(row, name)`, `value_text`, `value_u64`, `value_i64`, and `field_index(name)` | | TypeScript | `queryResultValue(result, row, name)`, `queryResultValueText`, `queryResultValueU64`, `queryResultValueI64`, and `queryResultFieldIndex(result, name)` | Use typed accessors for logic and the text accessor for display. `value_u64` and `value_i64` return nothing when the value is missing, out of range, or not an integer. TypeScript `typedValueDiagnosticText(value)` and Python `ls.typed_value_diagnostic_text(value)` render any typed value as text. See [Managed data](/laser-sdk/advanced/managed-data#typed-query-results) for the type rules. ### Typed reads Typed reads decode the stored payload of each row, so they need a view that keeps payloads. A row without a payload fails. * `fetch_typed` returns one page of decoded values. Rust decodes JSON into your type with `fetch_typed::()` and takes another decoder with `fetch_typed_with::()`. Python decodes JSON into Python values, and `fetch_typed_with(codec)` takes any object with `decode(data)`. TypeScript takes a codec, as in `fetchTyped(new Json())`. * `fetch_one` returns at most one decoded value. Rust and Python also have `fetch_one_with`. TypeScript `fetchOne(codec)` takes the codec directly. * `fetch_all()` reads every matching row into memory. `fetch_all_typed()` (TypeScript: `fetchAllTyped(codec)`) decodes them. Use these only when the result fits in memory. ## Walk many rows `rows()` walks matching rows across pages. It needs an explicit ceiling, `max_rows(n)` (TypeScript: `maxRows(n)`), and fails with an invalid error without one. It stops at the ceiling or the last page, whichever comes first. Rust returns a walker that you advance with `next().await`. TypeScript returns an async iterable. Python returns an asynchronous iterator. `rows_typed()` does the same with decoded payloads (TypeScript: `rowsTyped(codec)`). ```ts let degraded = 0 const walk = laser.query("readings_v1").whereEq("status", "degraded").maxRows(5_000).rows() for await (const _row of walk) degraded += 1 ``` ```rust let mut rows = laser .query("readings_v1") .where_eq("status", "degraded") .max_rows(5_000) .rows()?; let mut degraded = 0; while rows.next().await?.is_some() { degraded += 1; } ``` ```python rows = laser.query("readings_v1").where_eq("status", "degraded").max_rows(5_000).rows() degraded = 0 async for _row in rows: degraded += 1 ``` A zero ceiling returns no rows and sends no request in all three SDKs. TypeScript rejects a negative or unsafe integer ceiling. The walk fetches pages of `limit` rows, 50 by default. ## Execution identity, status, and cancel Every query carries an execution ID. Rust reads it with `execution_id()`, TypeScript with `executionId()`, and Python from the `execution_id` property. `status()` reads the execution state and `cancel()` asks the server to stop it. Cursor paging, status, and cancellation each have their own capability flag: `cursor_paging`, `execution_status`, and `cancellation` on `capabilities.query` (camelCase in TypeScript). The SDK returns an unsupported error before sending when the deployment does not advertise the one you call. The lower-level calls on the client are `execute_query(query)`, `query_page(execution_id, cursor, deadline_micros)`, `query_status(execution_id)`, and `cancel_query(execution_id)` (TypeScript: `executeQuery`, `queryPage`, `queryStatus`, `cancelQuery`). Each checks that the reply carries the execution ID you asked for, and a reply for another execution fails with a protocol error. When the server advertises a different query protocol version, the SDK fails with a typed version error before sending. ### Build a query value To build a whole query instead of chaining on `laser.query(..)`, use `Query::builder()` in Rust, `new QueryBuilder()` in TypeScript, or `ls.QueryBuilder()` in Python, then pass the result to `execute_query` (TypeScript: `executeQuery`). The execution ID, target, and absolute deadline in epoch microseconds are required, and `build()` fails with an invalid error when one is missing. The builder has one setter per query field. In all three SDKs, `into_query()` (TypeScript: `intoQuery()`) turns a chained request into the same value, and `query_target(target)` (TypeScript: `queryTarget`) starts a chained request from an explicit target. ## Operational views and destinations A binding keeps an operational index on the deployment's backend. A materialization destination is a separate resource with its own identity, source scope, generation, checkpoints, and lifecycle. It is not another target of the binding. A query names its target. `laser.query(index)` reads an operational index. `laser.query_lakehouse(destination, generation)` (TypeScript: `queryLakehouse`) reads one destination generation. On a lakehouse target, `at_snapshot(id)` and `at_timestamp_micros(ts)` (TypeScript: `atSnapshot`, `atTimestampMicros`) read an earlier table state. Both values must be positive. On an operational target they fail with an invalid error: Rust returns it, TypeScript throws it, and Python raises it. A lakehouse query cannot read a fork. What a backend can serve depends on what it advertises. See [Managed data](/laser-sdk/advanced/managed-data). Columnar data enters the log as Arrow IPC. Publish one complete stream per message with `arrow_ipc(bytes, metadata)` on a publish request (TypeScript: `arrowIpc`). The SDK checks the metadata and that the payload length matches it before sending. The managed plane enforces the rest of the acceptance policy. ## Errors | Situation | What you get | | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | The deployment does not serve queries | Unsupported error, before sending | | An unadvertised consistency level, lexical search, cursor paging, status, or cancel | Unsupported error naming the feature, before sending | | `limit` of zero or above 1,000, an empty time range, both offset and cursor, or `having` without an aggregate | Invalid error, before sending | | `rows()` or `rows_typed()` without `max_rows` | Invalid error | | Read-your-writes that cannot catch up in time | Stale query error | | A typed read of a row without a payload | Config error | | The index does not exist yet | An error until the managed plane creates it | | `register` with a graph projection | Invalid error | ## Key operations | Verb | What it does | | ---------------------------------------------------------------- | ----------------------------------------------------------------------- | | `Projection::builder(id)` | Start a projection (TypeScript and Python: `ProjectionBuilder(id)`) | | `.field` / `.fields` / `.field_at` | Index top-level fields or a JSON pointer | | `.field_typed` / `.field_at_typed` | Index a field with a storage type hint | | `.vector_field(pointer)` | Extract an embedding vector | | `.inline_payload()` / `.index_only()` | Keep the payload with each row, or only the indexed fields | | `projections().register` / `get` / `list` / `drop` | Manage the projection registry | | `ProjectionBinding::builder()` | Start a binding (TypeScript and Python: `ProjectionBindingBuilder()`) | | `.source` / `.allow` / `.default_projection` | Which topic, which projections, and the projection for untagged records | | `.index` / `.backend` / `.retention` / `.notify` | Index name, backend, row lifetime, and change records | | `bindings().apply` / `remove` | Apply or remove a binding | | `schemas().register` / `get` / `list` / `drop` | Manage writer schemas | | `publish().index(key, value)` | Set an indexed value at publish time | | `publish().projection_ref(id)` | Tag a record for one allowed projection | | `query(index)` / `query_lakehouse` / `query_target` | Open a query | | `where_eq` and the `filter_*` family | Match and filter | | `order_asc` / `order_desc` / `limit` / `offset` / `cursor` | Sort and page | | `count`, `sum`, `avg`, `group_by`, `window`, `having` | Aggregate | | `nearest` / `text` | Vector and lexical search | | `read_your_writes` / `consistency` | Choose read consistency | | `fetch` / `fetch_typed` / `fetch_one` / `fetch_all` | Run the query | | `max_rows(n).rows()` / `rows_typed()` | Walk rows across pages under a ceiling | | `status` / `cancel` | Read or stop the execution | | `execute_query` / `query_page` / `query_status` / `cancel_query` | Lower-level query lifecycle | | `raw_sql` | One read-only SQL statement on a SQL backend | Source: https://docs.laserdata.com/laser-sdk/advanced/views --- # Changes in depth This is the full reference for [Changes](/laser-sdk/change-feed). Start there for the short version. The change feed tells you when an operational view advanced. Each change record says which index moved, over which source offsets, and how many rows landed. It does not carry the rows. Read the feed, then query the index only when it moved. The feed is an ordinary topic read by offset over the connection you already have, so it adds no new connection or push channel. ## Turn it on Call `notify()` on the [projection binding](/laser-sdk/advanced/views#bind-a-topic). After each committed projector batch for that binding, the managed plane publishes one change record. A binding without `notify()` serves queries only. Watch the same index name that you query. The feed needs the `watch` capability, which Laser Stack and LaserData Cloud advertise. Reading the rows afterward also needs the `query` capability. In Rust, the reader needs the `watch` feature, which the `managed` feature includes. ## Where records go Change records go to the `changes` topic on the ops stream, `_agdx`, the stream the managed services use for their own records. On a deployment that reports `stream_tenancy`, each stream has its own change topic, `stream:/_agdx/changes`. A client with a default stream reads its stream's topic, and each record names its `stream`. A client with a default stream also scopes the index name it filters on, the same way a query scopes it. ## Open a reader | Language | Open a reader for one index | Read every index | Returns | | ---------- | ------------------------------------------- | ------------------------------- | ------------------------------------------------ | | Rust | `laser.watch().index(name).records()?` | `laser.watch().records()?` | `Result`, synchronously | | TypeScript | `await laser.watch().index(name).records()` | `await laser.watch().records()` | `Promise` | | Python | `laser.watch(index=name)` | `laser.watch()` | The `WatchReader` directly, not awaited | When the deployment does not publish the feed, opening the reader fails with an unsupported error: `records()` in Rust and TypeScript, and `watch()` in Python. The SDK never waits on a topic that nothing writes to. `.index(name)` keeps one index. The filter runs in the client, so the reader still reads every record on the topic. The reader skips any record that does not decode as a change record. ## Read records Your application owns the loop. The SDK tracks the position between polls and has no push callbacks. 1. Call `poll()`. It returns every matching record that arrived since the last poll, or an empty list when the reader is caught up. 2. Handle the records. Query the index if it moved. 3. If nothing arrived, wait briefly and poll again. To read one record at a time, Rust's `stream()` returns a `futures::Stream`, TypeScript's `stream()` returns an async generator, and the Python reader is itself an async iterator. All three end when the reader is caught up. In TypeScript and Python a later loop on the same reader resumes where the last one stopped. Rust's `stream()` takes ownership of the reader, so save its offsets first if you need them. ```ts const feed = await laser.watch().index("readings_v1").records() for await (const change of feed.stream()) { console.log(`${change.index} advanced ${change.rows} row(s) on partition ${change.partitionId}`) } ``` ```rust use futures::StreamExt; let feed = laser.watch().index("readings_v1").records()?; let mut changes = std::pin::pin!(feed.stream()); while let Some(change) = changes.next().await { let change = change?; println!( "{} advanced {} row(s) on partition {}", change.index, change.rows, change.partition_id ); } ``` ```python feed = laser.watch(index="readings_v1") async for change in feed: print(f"{change.index} advanced {change.rows} row(s) on partition {change.partition_id}") ``` ## Change record fields | Field | TypeScript | Meaning | | -------------- | ------------- | ------------------------------------------------------------------------- | | `index` | `index` | The index that advanced | | `partition_id` | `partitionId` | The source partition | | `from_offset` | `fromOffset` | First source offset in the batch, inclusive | | `to_offset` | `toOffset` | Last source offset in the batch, inclusive | | `rows` | `rows` | Rows the batch wrote | | `stream` | `stream` | The source stream, set only when the deployment keeps one feed per stream | | `v` | `v` | The change record version, currently 1 | In TypeScript, `fromOffset` and `toOffset` are `bigint` values and `stream` is optional. In Rust, `stream` is an `Option`. In Python, it is `None` when absent. ## Resume after a restart Save the reader's offsets and restore them in a new reader. Where you store them is up to you. | Language | Read the position | Restore it | | ---------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | Rust | `feed.offsets()`, a slice with one `u64` per partition | `.from_offsets(saved)` on a new reader, with a `Vec` | | TypeScript | The `feed.offsets` property, a `ReadonlyMap` from partition to offset | `.fromOffsets(saved)` on a new reader | | Python | The `feed.offsets` property, a list of integers | `.from_offsets(saved)`, or `laser.watch(index=name, from_offsets=saved)` | Each offset is the next one to read on that partition. A new reader without restored offsets starts at the beginning of the feed. In Rust, a saved list shorter than the partition count reads the missing partitions from the start. ```ts const saved = feed.offsets // After a restart const resumed = (await laser.watch().index("readings_v1").records()).fromOffsets(saved) ``` ```rust let saved = feed.offsets().to_vec(); // After a restart let mut resumed = laser .watch() .index("readings_v1") .records()? .from_offsets(saved); ``` ```python saved = feed.offsets # After a restart resumed = laser.watch(index="readings_v1", from_offsets=saved) ``` ## Delivery guarantees * Notifications are best effort after commit. A lost notification does not lose projected data, because the rows are already in the view. * A reader that falls behind the feed's retention window misses the records that expired. Query the view directly to catch up. * The feed reports progress. It does not prove that a specific write is in the view. To make sure a query includes your own write, use a read-your-writes query. See [Consistency](/laser-sdk/advanced/views#consistency). * The feed reports operational view changes only. It does not report key-value or session changes. Sessions have their own change feed on the session reads. Destination checkpoints have their own [lifecycle](/laser-sdk/advanced/managed-data#destinations-and-query-routes). ## Key operations | Verb | What it does | | --------------------------------- | ---------------------------------------------------------------------------------- | | `notify()` on a binding | Turn on change records for that index | | `watch().index(name).records()` | Open a reader for one index (Rust, TypeScript) | | `watch(index=name)` | Open a reader for one index (Python) | | `watch().records()` / `watch()` | Read every index on one feed | | `poll()` | Read the records that arrived since the last poll | | `stream()` | Read records one at a time until caught up (Rust, TypeScript. Python: `async for`) | | `offsets` / `from_offsets(saved)` | Save and restore the position across restarts (TypeScript: `fromOffsets`) | Source: https://docs.laserdata.com/laser-sdk/advanced/change-feed --- # Graph in depth This page is the full reference for [Graph](/laser-sdk/graph). Start there for the short version. ## How the graph works A graph stores entities as nodes and relationships as edges. `laser.graph(name)` returns a handle that writes, reads, and traverses one named graph. A node ID comes from the node's label and value. `GraphNode::entity("service", "auth")` (TypeScript: `graphNodeEntity`, Python: `graph_node_entity`) always produces the same ID in every SDK, so two services that mention `service:auth` update one node without agreeing on IDs first. An edge ID comes from its two endpoints and its type, so the same relationship recorded twice is one edge. Writes are therefore safe to repeat. A client with a default stream scopes the graph name to that stream, so `laser.graph("kg")` addresses `stream:/kg`. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names). A graph name is at most 128 bytes with no ASCII control characters. The graph is a managed feature. It works on [Laser Stack](/laser-sdk/laser-stack) and LaserData Cloud when `capabilities.graph` is true. Against Apache Iggy alone, every graph call returns an unsupported error (`LaserError::Unsupported` in Rust, `UnsupportedError` in Python and TypeScript). Graph writes enter a durable mutation log before the deployment applies them, and traversals read with eventual consistency, so a write can take a moment to appear. ## Write facts `link(from, relation, to)` takes two `"label:value"` strings. It derives both node IDs, writes both nodes, and writes the edge. A string without a colon gets the label `entity`. Repeating the same triple produces the same nodes and edge. `upsert(nodes, edges)` writes nodes and edges you build yourself, for richer payloads. One upsert carries at most 10,000 nodes and edges, and a node carries at most 16 labels. | Build | TypeScript | Rust | Python | | ---------------------------- | ----------------------------------- | ------------------------------------- | -------------------------------------- | | Node for a label and value | `graphNodeEntity(label, value)` | `GraphNode::entity(label, value)` | `graph_node_entity(label, value)` | | Node ID alone | `graphNodeEntity(label, value).id` | `GraphNode::entity(label, value).id` | `node_id_content(label, value)` | | Edge between two nodes | `graphEdgeRelate(from, type, to)` | `GraphEdge::relate(&from, type, &to)` | `graph_edge_relate(from_, type, to)` | | Valid-time window on an edge | `graphEdgeValid(edge, from, to)` | `edge.valid(from, to)` | `graph_edge_valid(edge, from_, to)` | | Source on an edge | `graphEdgeWithSource(edge, source)` | `edge.with_source(source)` | `graph_edge_with_source(edge, source)` | A node holds an ID, labels, attributes, an optional embedding, an optional source, and an optional producer. An entity node carries its label in `labels` and its value in the `value` attribute. An edge holds an ID, its endpoints, a type, a weight, attributes, an optional valid-time window (`valid_from` and `valid_to` in epoch microseconds), a source, and a producer. Attributes are `[name, value]` pairs in every SDK. Rust and TypeScript return typed structs and objects. Python returns dicts. ## Change a fact These operations keep relationship history: * Call `relink(from, relation, to)` to replace a single-valued relationship. It closes every live edge with the same source node and relation that points at another target, writes the new edge, and returns how many edges it closed. * Call `unlink(from, relation, to)` to close one edge by setting `valid_to` to now. The nodes stay. An `as_of` read at an earlier time still sees the relationship. ## Read a neighborhood `neighbors(node_id, direction, edge_type, depth)` returns the nodes reachable from one node, with the edges between them. The direction is outgoing, incoming, or both (`"out"`, `"in"`, or `"both"` in Python and TypeScript, `EdgeDir` in Rust). An edge type limits the walk to that relationship. Without one, it follows every type. Depth is the number of hops. The reply includes the start node. Python's `neighbors(node, dir="out", edge_type=None, depth=1)` defaults the last three. `limit`, `as_of`, and `conversation` on the handle also apply to `neighbors`. ## Traverse The traversal builder gives more control: * Start with `start_ids([...])`, `start_match(filter)`, or `start_nearest(embedding, k)`. These pick explicit nodes, nodes that match a query `Filter`, or the `k` nodes closest to an embedding. * Add hops with `out(edge_type)`, `incoming(edge_type)`, and `both(edge_type)`. Each call adds one hop. * Return nodes by default, or choose `return_edges()`, `return_triplets()`, or `return_paths()`. Triplets are `source, type, destination`. Paths are node and edge ID sequences. * Cap the result with `limit(n)`, 100 by default. `as_of(micros)` reads edges valid at a time. `conversation(id)` limits the walk to facts recorded by one conversation. Without it, the walk spans the whole graph. Finish with `fetch()`. TypeScript uses camelCase (`startIds`, `returnPaths`, `asOf`). A traversal goes at most 8 hops deep, and a reply holds at most 10,000 nodes and edges. The start nodes are part of the returned nodes. ```ts import { filterPred } from "@laserdata/laser-sdk" const dependencies = await laser .graph("ops") .startMatch(filterPred("label", "eq", { kind: "string", value: "Service" })) .out("depends_on") .limit(100) .fetch() const paths = await laser.graph("ops").startNearest(embedding, 5).out("depends_on").returnPaths().fetch() ``` ```rust let dependencies = laser .graph("ops") .start_match(Filter::pred("label", CmpOp::Eq, "Service")) .out("depends_on") .limit(100) .fetch() .await?; let paths = laser .graph("ops") .start_nearest(embedding, 5) .out("depends_on") .return_paths() .fetch() .await?; ``` ```python dependencies = await ( laser.graph("ops") .start_match(ls.Filter.pred("label", "eq", "Service")) .out("depends_on") .limit(100) .fetch() ) paths = await ( laser.graph("ops") .start_nearest(embedding, 5) .out("depends_on") .return_paths() .fetch() ) ``` Rust returns a `GraphResult` with `nodes`, `edges`, and `paths`. TypeScript returns a `GraphResult` object with the same fields. Python returns a dict with `nodes`, `edges`, and `paths`, each present when the traversal filled it. ## Read the graph at a past time `as_of(micros)` (TypeScript: `asOf`, as a `bigint`) reads the graph as it was at a time in epoch microseconds. Only edges whose valid-time window contains that time are followed. The window includes `valid_from` and excludes `valid_to`, and a missing bound is open, so an edge without a window is always valid. A closed edge stays visible to an earlier `as_of` read. This example writes a mitigation edge that became true at a known time, then reads before and after it. ```ts import { graphEdgeRelate, graphEdgeValid, graphNodeEntity } from "@laserdata/laser-sdk" const since = 1_900_000_000_000_000n const gateway = graphNodeEntity("Service", "gateway") const replica = graphNodeEntity("Component", "read-replica") const edge = graphEdgeValid(graphEdgeRelate(gateway, "mitigated_by", replica), since) await laser.graph("ops").upsert([gateway, replica], [edge]) const before = await laser.graph("ops").startIds([gateway.id]).out("mitigated_by").asOf(since - 1n).fetch() const after = await laser.graph("ops").startIds([gateway.id]).out("mitigated_by").asOf(since + 1n).fetch() ``` ```rust let since: u64 = 1_900_000_000_000_000; let gateway = GraphNode::entity("Service", "gateway"); let replica = GraphNode::entity("Component", "read-replica"); let edge = GraphEdge::relate(&gateway, "mitigated_by", &replica).valid(Some(since), None); laser .graph("ops") .upsert(vec![gateway.clone(), replica], vec![edge]) .await?; let before = laser .graph("ops") .start_ids(vec![gateway.id]) .out("mitigated_by") .as_of(since - 1) .fetch() .await?; let after = laser .graph("ops") .start_ids(vec![gateway.id]) .out("mitigated_by") .as_of(since + 1) .fetch() .await?; ``` ```python since = 1_900_000_000_000_000 gateway = ls.graph_node_entity("Service", "gateway") replica = ls.graph_node_entity("Component", "read-replica") edge = ls.graph_edge_valid(ls.graph_edge_relate(gateway, "mitigated_by", replica), since, None) await laser.graph("ops").upsert([gateway, replica], [edge]) before = await ( laser.graph("ops").start_ids([gateway["id"]]).out("mitigated_by").as_of(since - 1).fetch() ) after = await ( laser.graph("ops").start_ids([gateway["id"]]).out("mitigated_by").as_of(since + 1).fetch() ) ``` ## Build the graph from a topic You can write the graph directly with `link` and `upsert`, or derive it from messages. To derive it, register a graph [projection](/laser-sdk/advanced/views) with `projections().register_graph(projection)` (TypeScript: `projections().registerGraph`) and an entity schema. Each node rule names a label and a JSON pointer (an RFC 6901 path into the record) to the node value, plus an optional pointer to an embedding. Each edge rule names an edge type, pointers to its two endpoints, and optional pointers to a valid-time window. When the projection is bound to a source topic, the deployment applies the rules to each record and writes the nodes and edges, with no extraction code in your services. ```ts import { ContentType, parseProjectionId } from "@laserdata/laser-sdk" await laser.projections().registerGraph({ id: parseProjectionId("ops.v1"), name: "ops", version: 1, kind: { kind: "graph" }, contentType: ContentType.Json, extraction: { fields: [], inlinePayload: false }, entitySchema: { nodes: [ { label: "Service", valuePointer: "/service" }, { label: "Component", valuePointer: "/component" } ], edges: [{ edgeType: "depends_on", fromPointer: "/service", toPointer: "/component" }] }, inlinePayloadDefault: false }) ``` ```rust laser .projections() .register_graph( Projection::builder("ops.v1") .name("ops") .content_type(ContentType::Json) .index_only() .graph(EntitySchema { nodes: vec![ NodeExtract { label: "Service".to_owned(), value_pointer: "/service".to_owned(), embedding_pointer: None, }, NodeExtract { label: "Component".to_owned(), value_pointer: "/component".to_owned(), embedding_pointer: None, }, ], edges: vec![EdgeExtract { edge_type: "depends_on".to_owned(), from_pointer: "/service".to_owned(), to_pointer: "/component".to_owned(), valid_from_pointer: None, valid_to_pointer: None, }], }) .build(), ) .await?; ``` ```python await laser.projections().register_graph( { "id": "ops.v1", "name": "ops", "version": 1, "content_type": "json", "extraction": {"fields": [], "inline_payload": False}, "entity_schema": { "nodes": [ {"label": "Service", "value_pointer": "/service"}, {"label": "Component", "value_pointer": "/component"}, ], "edges": [ {"edge_type": "depends_on", "from_pointer": "/service", "to_pointer": "/component"}, ], }, } ) ``` [Queries and views](/laser-sdk/views) covers binding a projection to a topic. ## Session links A graph write can name the [session](/laser-sdk/session) that produced it. On a graph handle, `in_session(reference)` links every upsert to the session, `produced_by(producer)` stamps a producer name and version, and `sourced_from(source)` stamps the record the facts came from (TypeScript: `inSession`, `producedBy`, `sourcedFrom`). The producer and source apply only to nodes and edges that do not set their own, and they never change a node or edge ID. A node keeps the source it was first written with, and an edge keeps the latest one. Inside a session, `session.linked_graph(name)` (TypeScript: `linkedGraph`) returns a handle with all three set: the session, the session's agent as the producer, and the record the session acts on as the source. Call `acting_on(source)` (TypeScript: `actingOn`) on the session first to name that record. `session.graph(name)` returns the shared graph without a link. The server checks a session link like a [key-value link](/laser-sdk/advanced/state#session-links). The caller needs its graph grant, `session:write` on the stream, and send permission on the stream's `agent.sessions`. A managed deployment then lists each node and edge the session touched. One graph can serve many sessions in the same stream, and each session keeps its own history. ```ts const graph = session.linkedGraph("services") await graph.link("service:auth", "depends_on", "service:db") const extracted = laser .graph("services") .inSession(session.reference()) .producedBy({ name: "dependency-extractor", version: "3" }) ``` ```rust use laser_sdk::wire::graph::ProducerInfo; let graph = session.linked_graph("services")?; graph.link("service:auth", "depends_on", "service:db").await?; let extracted = laser .graph("services") .in_session(session.reference()?) .produced_by(ProducerInfo { name: "dependency-extractor".into(), version: "3".into(), }); ``` ```python graph = session.linked_graph("services") await graph.link("service:auth", "depends_on", "service:db") extracted = ( laser.graph("services") .in_session(session.reference()) .produced_by({"name": "dependency-extractor", "version": "3"}) ) ``` A source names a log record (numeric stream, topic, partition, and offset, plus the topic creation time and conversation when known), a key-value entry, or a memory item. TypeScript writes it as `{ kind: "message", .. }`, `{ kind: "kv", namespace, key }`, or `{ kind: "memory", id }`. Python writes it as a dict keyed by the variant, such as `{"Kv": {"namespace": "topology", "key": "gateway"}}`. ## Errors | Failure | When | | ------------ | ------------------------------------------------------------------------------------ | | Unsupported | The deployment does not serve the graph | | Invalid | An empty or oversized graph name, or an upsert over its caps. Checked before sending | | Unauthorized | The caller lacks the graph grant | | Not found | The graph does not exist | | Too large | A traversal exceeds a reply or depth cap | | Unavailable | The deployment is temporarily unavailable | Python raises `GraphError` with the cause in `detail`. TypeScript throws `GraphExecutionError` with the cause in `detail`. ## Key operations | Call | What it does | | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `graph(name).link(from, relation, to)` | Write both entity nodes and the edge in one call | | `relink(from, relation, to)` | Replace a single-valued relationship and return how many edges closed | | `unlink(from, relation, to)` | Close an edge, keeping it visible to earlier `as_of` reads | | `upsert(nodes, edges)` | Write your own nodes and edges | | `GraphNode::entity(label, value)` | Build a content-addressed node | | `GraphEdge::relate(from, type, to)` | Build an edge for `upsert` | | `neighbors(node_id, direction, edge_type, depth)` | Read the neighborhood of one node | | `start_ids`, `start_match`, `start_nearest` | Pick where a traversal starts | | `out(t)`, `incoming(t)`, `both(t)` | Add one hop | | `return_edges()`, `return_triplets()`, `return_paths()` | Choose the result shape | | `limit(n)` | Cap the result, 100 by default | | `as_of(micros)` | Read the graph at a past time | | `conversation(id)` | Limit the walk to what one conversation recorded | | `projections().register_graph(projection)` | Derive the graph from a topic | | `in_session(reference)`, `produced_by(..)`, `sourced_from(..)`, `session.linked_graph(name)` | Link writes to a session and record lineage | ## Requirements Use Laser Stack or LaserData Cloud. In Rust, the graph needs the `graph` feature. The graph example checks `capabilities.graph` and exits normally when the deployment does not serve it. Source: https://docs.laserdata.com/laser-sdk/advanced/graph --- # Managed data This page covers the managed plane that serves [Queries and views](/laser-sdk/views), [Key-value state](/laser-sdk/state), and the rest of the managed features. Read it when you run a managed deployment in production. Streaming and managed calls share one authenticated Iggy connection. Iggy forwards managed requests, such as queries, key-value calls, and destination changes, to the managed plane. Projection and binding changes are published to the control log instead. Filtered group readers and the dedicated fenced-lease client open their own connections. [Laser Stack](/laser-sdk/laser-stack) runs both services locally, and LaserData Cloud runs them for you. ## Check backend readiness A connection to Iggy does not mean the managed backend is ready. A running backend can still be replaying or restoring data, and its managed operations stay unavailable until it reports ready. After startup, failover, or a backend restart, refresh the capabilities or wait for readiness with a deadline: ```ts import { unreadyBackends } from "@laserdata/laser-sdk" const caps = await laser.waitUntilReady(30_000) console.log(caps.query.available, caps.kv.fencedLeases, unreadyBackends(caps).length) ``` ```rust use std::time::Duration; let caps = laser.wait_until_ready(Duration::from_secs(30)).await?; println!( "{} {} {}", caps.query.available, caps.kv.fenced_leases, caps.unready_backends().count() ); ``` ```python caps = await laser.wait_until_ready(30_000) print(caps.query.available, caps.kv.fenced_leases, len(caps.unready_backends())) ``` `refresh_capabilities()` (TypeScript: `refreshCapabilities()`) reads a fresh snapshot without waiting. `capabilities()` returns the held snapshot, and probes again by itself when that snapshot has no managed plane and its last probe is at least one second old. The wait returns an unsupported error when the server has no managed backend descriptors. See [Connect](/laser-sdk/connect#managed-readiness) for the connection side. Check the feature you need on its own. Query support does not imply fenced leases, destinations, or every consistency level. The capability groups are objects in all three SDKs, with camelCase fields in TypeScript: | Group | Fields | | ------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `caps.query` | `available`, `consistency` (the strongest level served), `keyword`, `cursor_paging`, `cancellation`, `execution_status` | | `caps.kv` | `available`, `cas`, `cas_fenced`, `fenced_leases` | | `caps.destinations` | `available`, `consistency` (`linearizable` or `potentially_stale`) | | `caps.filters` | `native`, `catalog`, `group_policy_reads`, `evaluation` | The flags `managed`, `graph`, `forks`, `a2a_gateway`, `sessions`, `watch`, `authz`, and `stream_tenancy` sit directly on the capabilities, beside `versions`, `backends`, and `hello`. Every feature is off until the server reports it. ### Backend descriptors Each backend descriptor names a resource ID, a mode (operational or lakehouse), a label, the `implementation` kind and version, the observed backend generation, the `runtime_configuration_revision`, the desired and observed state, the `readiness` state with reason codes, and the materialization, query, schema, maintenance, and limit support. A descriptor reports what the deployment actually serves. A destination type that exists in the protocol is not a promise that every backend implements it. | Need | Rust and Python | TypeScript | | ----------------------------------- | ------------------------------------- | ------------------------------------ | | Backends that should be running | `caps.enabled_backends()` | `enabledBackends(caps)` | | Enabled backends that are not ready | `caps.unready_backends()` | `unreadyBackends(caps)` | | Why one backend is not ready | `caps.readiness_reasons(resource_id)` | `readinessReasons(caps, resourceId)` | | One backend by ID | `caps.backend(resource_id)` | `backend(caps, resourceId)` | Rust returns iterators from the first two. Python returns lists. ## Operational indexes A projection names the fields to extract. Its binding selects a source and an `index`. An optional `backend` pins a resource ID and generation. Without it, the binding uses replica-local operational storage. `notify` adds the binding to the [change feed](/laser-sdk/advanced/change-feed). An optional `retention` sets how long rows live, independent of the source topic. Without it, the deployment default applies, which mirrors the log unless an operator changed it: rows go when Iggy removes the messages that produced them. The serialized form of a binding, as Python and the protocol use it: ```json { "source": { "stream": "telemetry", "topic": "readings" }, "allowed_projections": ["readings_v1.v1"], "default_projection": "readings_v1.v1", "index": "readings_v1", "notify": true } ``` TypeScript uses the same fields in camelCase. All three SDKs also have a binding builder. See [Queries and views in depth](/laser-sdk/advanced/views#bind-a-topic). ## Destinations and query routes A materialization destination writes a projection to a lakehouse table, such as Parquet or Iceberg v2. Each destination generation is immutable. It fixes the source scope, the partition recreation policy, the projection version, the schema fingerprint, the backend generation, the physical table, the start and new-partition behavior, the error policy, and the desired state. Progress and ownership live in checkpoint status records: prepared and completed attempts, repairs, retention gaps, blocking errors, and the current lease. A query names its target explicitly. `laser.query(index)` reads an operational index. `laser.query_lakehouse(destination, generation)` (TypeScript: `queryLakehouse`) reads one destination generation. A query route gives a name to one of those targets. Generations and source incarnations tell a recreated table or topic apart from an older one with the same name. `laser.destinations()` exposes the destination and checkpoint operations in all three SDKs, with camelCase names in TypeScript. Every mutation takes the expected global state revision first, plus the revisions that the operation checks. | Task | Calls | | ----------------- | ---------------------------------------------------------------------------------------------------------------------- | | Declare and steer | `register`, `set_desired_state`, `bind_table`, `add_partition`, `observe_partition_lifecycle` | | Own the work | `acquire_lease`, `renew_lease`, `take_over_lease` | | Record progress | `prepare`, `complete` | | Handle problems | `record_block`, `clear_block`, `record_retention_gap`, `accept_retention_gap`, `record_repair`, `supersede_generation` | | Name targets | `register_query_route`, `remove_query_route` | | Read | `get(id, consistency)`, `list(filter, after, limit, consistency)`, `query_routes(..)` | | Send any mutation | `mutate(revision, mutation)` | `accept_retention_gap`, `record_repair`, and `supersede_generation` need a signed supervisor assertion. Rust and TypeScript send any mutation with one through `mutate_with_supervisor_assertion`. Python passes `supervisor_assertion=` to `mutate`. Reads take a consistency of `linearizable` or `potentially_stale`. When a mutation fails on a revision conflict, read the current state before you try another change. Destination lease acquisition uses an expected lease sequence and epoch, which are different rules from ordinary key-value leases. Destination lease durations are microseconds in all three SDKs (`lease_duration_micros`, TypeScript `leaseDurationMicros` as a `bigint`), while key-value lease TTLs are milliseconds in Python and TypeScript. Source retention can remove records before a destination reads them. A recorded retention gap blocks progress until a repair or an explicit acceptance. It never counts as successful ingestion. ## Typed query results A result contains `fields`, `rows`, `page`, and `context`. The values in each row follow the order of `fields`. Each value is a tagged typed value that keeps booleans, integers, longs, floats and doubles, decimals, dates in days, times and timestamps in microseconds, UUIDs, fixed and variable binary values, strings, lists, maps, structs, and nulls apart. A vector or lexical query also returns a `score` on each row. | Language | Read a field | | ---------- | ------------------------------------------------------------------------------------------------------------------------------ | | TypeScript | `queryResultValue(result, row, "cpu")`, `queryResultValueU64(..)`, `queryResultValueI64(..)`, or `queryResultValueText(..)` | | Rust | `result.value(row, "cpu")`, `result.value_u64(row, "cpu")`, `result.value_i64(row, "cpu")`, or `result.value_text(row, "cpu")` | | Python | `result.value(row, "cpu")`, `result.value_u64(row, "cpu")`, `result.value_i64(row, "cpu")`, or `result.value_text(row, "cpu")` | `field_index(name)` (TypeScript: `queryResultFieldIndex(result, name)`) returns a field's position. `typedValueDiagnosticText(value)` in TypeScript and `ls.typed_value_diagnostic_text(value)` in Python render any typed value as text. Use typed accessors for logic and text accessors for display. The client rejects a result whose rows have the wrong number of values, or a value that does not match its field type. A result, its later pages, its status, and its cancellation stay bound to the execution ID of the original query. `context` records the engine, the target, backend identity and generations, the requested and delivered consistency, and resource use. Lakehouse results add the destination generation, table, and snapshot. ## Retry and recovery Every managed mutation carries a stable operation identity, so the plane can return the saved outcome when the same mutation arrives twice. Reads can reconnect and retry. Mutations retry only with their original identity, and a deterministic rejection, such as invalid input, is never retried. A lost reply does not prove that the mutation failed. Lease acquisition is the exception that needs care. The SDK never repeats a lease acquisition after an uncertain disconnect. Ordinary key-value lease calls wait out the requested lease lifetime and then report the uncertainty. Until that lifetime has passed, do not assume that no lease exists. The fenced-lease client, `FencedLeaseClient` in all three SDKs, prepares each acquisition, renewal, release, or fenced write once. The prepared request keeps its operation ID and exact bytes. Repeat it only when its recovery action allows that: renew and release can repeat, an uncertain acquisition waits out the lease lifetime, and a fenced write needs to be checked against its target. Each attempt times out after 10 seconds by default. A renewal keeps the fencing token, and a release revokes it at once. Before planning a protected write, wait for a read that includes the lease's commit position. See [Key-value state in depth](/laser-sdk/advanced/state). Source: https://docs.laserdata.com/laser-sdk/advanced/managed-data --- # Governance This page is the reference for controlling who can act and which effects can proceed. For running agents, start with [Agents](/laser-sdk/fabric). Two permission layers guard different things. Native Apache Iggy permissions decide whether a credential can see streams, create topics, send records, and poll records. Managed roles decide whether the same server-stamped user can call managed features such as queries, key-value state, graph, forks, and session reads. On top of both, an `ActionGovernor` in your process can stop or change a single effect before it happens, and intent records let several agents approve an action before anyone applies it. Access is denied by default, and a matching deny wins. ## Managed roles A grant reads `effect feature:action [on resource-pattern]`. A role is a named set of grants, and a binding gives roles to an Iggy user ID. ```ts import type { Role } from "@laserdata/laser-sdk" const role: Role = { name: "support-reader", grants: [{ effect: "allow", feature: "kv", action: "read", resource: { kind: "prefix", value: "stream:support/" } }] } await laser.defineRole(role) await laser.bindRoles(7, [role.name]) ``` ```rust use laser_sdk::prelude::PrincipalId; use laser_sdk::rbac::{Action, Effect, Feature, Grant, ResourcePattern, Role}; let role = Role { name: "support-reader".to_owned(), grants: vec![Grant { effect: Effect::Allow, feature: Feature::Kv, action: Action::Read, resource: ResourcePattern::prefix("stream:support/"), }], }; laser.define_role(role).await?; laser.bind_roles(PrincipalId::new(7), vec!["support-reader".to_owned()]).await?; ``` ```python import laser_sdk as ls grants = [ ls.Grant( "kv", "read", effect="allow", resource=ls.ResourcePattern.prefix("stream:support/"), ) ] await laser.define_role("support-reader", grants) await laser.bind_roles(7, ["support-reader"]) ``` The resource is `stream:support/` because managed names are scoped to the connection's default stream by default. `laser.kv("tickets")` on a connection whose default stream is `support` addresses `stream:support/tickets`. A role written for bare names such as `support/` only matches clients that use bare resource naming. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names). ### Role operations | Operation | What it does | Needs | | ------------------------------------------ | ---------------------------------------------- | ------------- | | `whoami()` | Return the caller's roles and effective grants | Any caller | | `list_roles`, `get_role`, `get_bindings` | Read roles and one user's bound role names | `authz:read` | | `define_role`, `delete_role`, `bind_roles` | Change roles and bindings | `authz:admin` | | `authz_history` | Read the change history of roles and bindings | `authz:read` | TypeScript uses camelCase for every name. The `authz_history` subject is all changes, one role, or one user's binding. Python passes it as `"all"`, `{"role": name}`, or `{"binding": {"user_id": id}}` with `after_revision=` and `limit=` keywords, and the limit defaults to 100. Rust takes `(subject, after_revision, limit)`. TypeScript takes `(subject, limit, afterRevision?)`. `list_roles` takes a name prefix and a `search` substring in every SDK: Rust `list_roles(name_prefix, search)`, TypeScript `listRoles({ namePrefix, search })`, and Python `list_roles(name_prefix=None, *, search=None)`. A binding update can be guarded by revision. Rust uses `bind_roles_expect_revision`, TypeScript passes `expectRevision` as the third argument of `bindRoles`, and Python passes `expect_revision=` to `bind_roles`. The update applies only if the binding's current revision equals the one you pass. A mismatch fails with a conflict that carries the current revision: `AuthzError::Conflict { current_revision }` in Rust, an `AuthzExecutionError` whose `detail` has `currentRevision` in TypeScript, and an `AuthzError` with `detail` in Python. `get_bindings` returns role names only, so take the revision from the conflict or from `authz_history`. New Iggy users receive no managed role. The default administrator has a reserved `admin` role for first setup. You can also bind roles in the Console. ### Grant rules * The features are `kv`, `kv_lease`, `kv_fence`, `memory`, `projection`, `fork`, `graph`, `query`, `destination`, `checkpoint`, `filter`, `session`, and `authz`. The retired `agent` and `workflow` features match no operation. A role that granted `agent:*` should grant `session:*` instead. * The actions are `read`, `write`, `delete`, and `admin`. An unknown feature or action word decodes as `unrecognized` and matches nothing. * A resource pattern is `all`, `literal`, or `prefix`. A prefix is a plain string prefix, so `stream:acme` also matches `stream:acme-staging`. Only `all` matches a request with no keyed resource, such as a list call. * A deny wins over an allow for the same feature, action, and resource match. This rule cannot be turned off. * Lease control is its own feature. A `kv:write` grant never lets a holder acquire, renew, or release a lease. That needs `kv_lease:admin` on the coordination namespace. A fenced write needs `kv:write` on the target and `kv_fence:read` on the coordination namespace. * Role names are 1 to 64 bytes of ASCII letters, digits, `-`, `_`, and `.`. An invalid name fails in the client before any request. * Managed grants are separate from Iggy permissions. The server enforces them on managed commands. * Session reads use the `session` feature on the resource `stream:`. `session:read` lists and reads sessions, and the caller also needs read permission on the whole stream. `session:admin` registers a stream as a session source. `session:write` covers key-value and graph writes linked to a session, and also needs native send permission on that stream's `agent.sessions`. * A deployment in stream tenancy mode accepts only grants that stay inside one stream. It refuses a whole-feature grant or a prefix that spans streams with a tenancy violation error, and it rejects an unscoped resource name. Without the `authz` capability on the server, role calls fail in the client with an unsupported error before anything is sent. Server-side failures are `Unsupported`, `Unauthorized`, `UnknownRole`, `InvalidName`, `Conflict`, `Version`, and `TenancyViolation`. In TypeScript a server `unsupported` becomes `UnsupportedError` and the rest become `AuthzExecutionError`. ### Act for a user When an agent acts for a user, both grant sets must allow the action. A user session cannot extend the agent's grants, and the user's restrictions still apply through the agent. ```ts import { delegatedAllow } from "@laserdata/laser-sdk" const allowed = delegatedAllow( agentGrants, userGrants, "kv", "write", "stream:support/tickets/acme" ) ``` ```rust use laser_sdk::rbac::{Action, Feature, delegated_allow}; let allowed = delegated_allow( &agent_grants, &user_grants, Feature::Kv, Action::Write, Some("stream:support/tickets/acme"), ); ``` ```python allowed = ls.delegated_allow( agent_grants, user_grants, "kv", "write", "stream:support/tickets/acme", ) ``` `grants_allow` (TypeScript: `grantsAllow`) applies the same deny-wins rule to one grant set. A missing resource means the operation has no keyed resource. In Rust both helpers also live in `laser_sdk::wire::authz`, which needs no `rbac` feature. `verify_delegation` checks a signed delegation on an envelope and returns the signer and the delegated user, which you then pass to `delegated_allow`. [Interop](/laser-sdk/advanced/interop#standalone-helpers) lists it. To check the audience and scope of an A2A or MCP request token before it reaches a bridge, use `authorize_edge`. [Interop](/laser-sdk/advanced/interop#authorize-edge-requests) shows it in all three languages. ## ActionGovernor `ActionGovernor` is a policy hook in your process that runs before an effect. Roles decide what a principal may do at all. The governor decides whether one specific effect may run now, for example by size, rate, or payload content. It can restrict what the server allows. It cannot widen it. `laser.with_governor(governor, mode)` returns a governed clone of the connection (TypeScript: `withGovernor`). Agents spawned from the governed connection inherit it. An agent can take its own governor through the agent builder or `spawn_agent(governor=..)`, which replaces the connection's governor for that agent. The governor sees agent sends, typed and raw topic publishes, requests, AGDX verbs, and memory writes. ```ts import { ActionDecision, type ActionGovernor, GovernorMode } from "@laserdata/laser-sdk" const sizeLimit: ActionGovernor = { decide: async (action) => action.payload.length > 65_536 ? ActionDecision.block("payload over 64 KiB") : ActionDecision.allow() } const governed = laser.withGovernor(sizeLimit, GovernorMode.Enforce) ``` ```rust use laser_sdk::prelude::full::*; use std::sync::Arc; struct SizeLimit; #[async_trait::async_trait] impl ActionGovernor for SizeLimit { async fn decide(&self, action: &GovernedAction<'_>) -> Result { Ok(if action.payload.len() > 65_536 { ActionDecision::block("payload over 64 KiB") } else { ActionDecision::allow() }) } } let governed = laser.with_governor(Arc::new(SizeLimit), GovernorMode::Enforce); ``` ```python class SizeLimit: async def decide(self, action): if len(action.payload) > 65_536: return ls.ActionDecision.block("payload over 64 KiB") return ls.ActionDecision.allow() governed = laser.with_governor(SizeLimit(), "enforce") ``` The Rust trait uses `#[async_trait]`, so add the `async-trait` crate to your dependencies. Python accepts any object with a `decide` method or a plain callable, synchronous or asynchronous. The governed action carries its kind, stream, topic, source, target, conversation, correlation, operation, tool, payload, whether it is signed, the delegated user (`on_behalf_of`), the advisory `purpose` and `data_classification` claims, and session counters in `action.counters` (`sends`, `requests`, and `bytes_sent`, TypeScript: `bytesSent`). `sends` and `requests` count every decided effect, blocked ones included. `bytes_sent` counts only effects that ran, with the replacement body when one was modified. ### Verdicts | Verdict | Effect in enforce mode | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `allow` | Runs the effect. Records no evidence | | `observe` | Runs the effect and records evidence | | `block` | Rejects the effect with `PolicyBlocked` (TypeScript and Python: `PolicyBlockedError`) | | `step_up(scope)` | Rejects the effect with `StepUpRequired { scope }` (TypeScript and Python: `StepUpRequiredError` with `scope`) | | `modify(body)` | Replaces the body, then runs the effect. The replacement applies before a large body is moved out by claim check and before signing | | `defer` | Rejects the effect with `PolicyDeferred` (TypeScript and Python: `PolicyDeferredError`), which is retryable | The factories are `ActionDecision.allow()`, `observe()`, `block(reason)`, `step_up(scope)` (TypeScript: `stepUp`), `modify(body)`, and `defer(reason)`. `with_reason` and `with_risk_score` (TypeScript: `withReason`, `withRiskScore`) add details that the evidence records. A blocked or deferred effect fails with a fixed message. The governor's reason appears only in the evidence. A decision carries its verdict as a `Verdict`, read as a string with `as_str()` (TypeScript: `verdictAsStr(verdict)`). `with_policy(PolicyRef { pack_id, pack_version, rule_ids })` pins the policy pack that decided (TypeScript: `withPolicy({ packId, packVersion, ruleIds })`, Python: `with_policy(ls.PolicyRef(pack_id, pack_version, rule_ids))`). ### Modes The mode is `enforce` or `observe`. Python defaults to `enforce`, and Rust and TypeScript require you to pass it. Enforce applies the verdict. Observe runs every effect with its original body and records what enforce would have done, with outcome `effected`. Introduce a policy in observe mode, read the evidence, then switch to enforce. An error from the governor itself fails the call in both modes, so a broken governor never lets an effect through. ### Evidence Every decision that is not `allow` is recorded as policy evidence: an AGDX event with operation `policy_decision` on the `agent.audit` topic of the governed action's stream. Its `decision` names the verdict and its `outcome` names what happened to the effect: `effected`, `blocked`, `step_up`, or `deferred`. The evidence source is the acting agent, or `governor` when there is none. * `agent.audit` is not created by the governor. Run `bootstrap` first, and give the caller send permission on it. * In enforce mode, a failed evidence write stops an otherwise allowed effect. A denial stands whether or not its evidence was stored. * In observe mode, a failed evidence write never stops the effect. Rust and Python log a warning, and TypeScript emits a `LaserWarning` with the same text. * Records chain by digest for each governed connection and conversation. Each `with_governor` call, and each agent with its own governor, starts its own chain. * The governor keeps one chain head per conversation in memory, at most 4,096 heads by default, and evicts a head after one idle hour. A head in use is never evicted. An eviction or a restart starts a new chain for that conversation. Rust and Python decode an evidence body with `PolicyEvidence::decode` and `PolicyEvidence.decode`, and encode it with `encode()`. TypeScript uses `encodePolicyEvidence` and `decodePolicyEvidence`. `verify_evidence_chain(evidence)` (TypeScript: `verifyEvidenceChain`) takes records in log order and reports whether each reproduces its own digest and names the previous one. The first record must have no previous digest. Run it over one chain, because two governed connections writing the same conversation interleave two chains on `agent.audit`. | Where | Rust | TypeScript | Python | | ------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Connection | `with_governor(governor, mode)` | `withGovernor(governor, mode)` | `with_governor(governor, mode)` | | Connection with retention | `with_governor_retention(governor, mode, GovernorRetention { capacity, idle_ttl })` | `withGovernor(governor, mode, { capacity, idleTtlMs })` | `with_governor_retention(governor, mode, ls.GovernorRetention(capacity=, idle_ttl_ms=))` | | Connect options | `governor(..)`, `governor_with_retention(..)` on the builder | `governor(policy, mode, retention?)` on the builder | `connect(..., governor=, governor_mode=, governor_retention=)` | | Agent | `governor((governor, mode))`, `governor_retention(..)` | `governor([governor, mode])`, `governorRetention({ capacity, idleTtlMs })` | `governor=`, `governor_mode=`, `governor_retention=(capacity, idle_ttl_ms)` | All three raise a capacity of 0 to 1. TypeScript throws `InvalidError` for a capacity or an `idleTtlMs` that is negative or not a safe integer. ### Combine and swap governors `QuorumGovernor` runs several governors at once and combines their verdicts. `allow`, `observe`, and `modify` count as yes. `block`, `step_up`, and `defer` do not. The policy is `All`, `Any`, or `AtLeast(n)` (TypeScript: `{ kind: "all" }`, `{ kind: "any" }`, or `{ kind: "at-least", required: n }`, Python: `QuorumPolicy.all()`, `.any()`, or `.at_least(n)`). The combined decision follows these rules, in order: * An empty voter set, or a threshold of zero or above the voter count, blocks. * A repeated voter name blocks. * A mandatory voter that raises an error blocks. An optional voter that raises an error counts as an abstention. * A mandatory voter must say yes under every policy. The first mandatory voter that does not is returned as it is. A mandatory `defer` wins even when an optional voter said `block`. * Conflicting body replacements block. * When the quorum passes, the strongest yes wins: `modify`, then `observe`, then `allow`. When it fails, the most actionable denial wins: `block`, then `step_up`, then `defer`. The decision reason lists what each voter said. The `mandatory` argument of `voter` is required in every SDK, and `voter` returns the governor, so enrollments chain. `SwappableGovernor` replaces the active policy at runtime without a reconnect. `swap(next)` returns the previous policy and `current()` returns the active one. A decision already in flight finishes under the policy it read, and a swap never reinterprets evidence already on the log. ```ts import { QuorumGovernor, SwappableGovernor } from "@laserdata/laser-sdk" const audit: ActionGovernor = { decide: async () => ActionDecision.observe() } const quorum = new QuorumGovernor({ kind: "all" }) .voter("size", sizeLimit, true) .voter("audit", audit, false) const swappable = new SwappableGovernor(quorum) const governed = laser.withGovernor(swappable, GovernorMode.Enforce) const previous = swappable.swap(sizeLimit) const active = swappable.current() ``` ```rust use laser_sdk::govern::{QuorumGovernor, QuorumPolicy, SwappableGovernor}; struct Audit; #[async_trait::async_trait] impl ActionGovernor for Audit { async fn decide(&self, _action: &GovernedAction<'_>) -> Result { Ok(ActionDecision::observe()) } } let quorum = QuorumGovernor::new(QuorumPolicy::All) .voter("size", Arc::new(SizeLimit), true) .voter("audit", Arc::new(Audit), false); let swappable = Arc::new(SwappableGovernor::new(Arc::new(quorum))); let governed = laser.with_governor(swappable.clone(), GovernorMode::Enforce); let previous = swappable.swap(Arc::new(SizeLimit)); let active = swappable.current(); ``` ```python class Audit: async def decide(self, action): return ls.ActionDecision.observe() quorum = ls.QuorumGovernor(ls.QuorumPolicy.all()) quorum.voter("size", SizeLimit(), True) quorum.voter("audit", Audit(), False) swappable = ls.SwappableGovernor(quorum) governed = laser.with_governor(swappable, "enforce") previous = swappable.swap(SizeLimit()) active = swappable.current() ``` ## Session budgets and control A session budget states the tokens and cost one [session](/laser-sdk/session) may use. Set it on a session builder or on a submission. Cost is an integer in micro-units of the deployment's currency, so sums are exact. A budget does not decide who can submit work or what the work can reach. Native permissions and managed grants do that. ```ts const submitted = await laser .sessions() .submit(AgentId.new("auditor"), new TextEncoder().encode("audit incident 7")) .from(AgentId.new("governance")) .budget({ tokens: 4_000n, costMicros: 2_000_000n }) .send() const over = await laser.sessions().open(submitted.session).overBudget() ``` ```rust use laser_sdk::wire::agent::Budget; let submitted = laser .sessions() .submit("auditor".parse::()?, b"audit incident 7".to_vec()) .from("governance".parse::()?) .budget(Budget { tokens: Some(4_000), cost_micros: Some(2_000_000), }) .send() .await?; let over = laser.sessions().open(submitted.session).over_budget().await?; ``` ```python submitted = await ( laser.sessions() .submit("auditor", b"audit incident 7") .from_("governance") .budget(ls.Budget(tokens=4_000, cost_micros=2_000_000)) .send() ) over = await laser.sessions().open(submitted.session).over_budget() ``` `over_budget()` on a session handle answers from the managed session index when the deployment has one, and otherwise folds the session's records on `agent.sessions`, so it also works on plain Apache Iggy. `sessions().get(id)` returns the same flag as `over_budget` but needs the managed index. The platform never writes your topics, so the flag never ends a session by itself. The SDK acts on it in two places: * On a deployment that indexes sessions, an agent with an ID checks each session before its handler runs. It fails an over-budget session once with reason `budget`, then commits its work without calling the handler. If reading the index fails, the record is handled. Plain Apache Iggy does not get this check. * A workflow writes the token cap of its `WorkflowBudget` as the budget of its run session and checks it at every step boundary, on any server. A breach runs the compensations, fails with a budget exceeded error, and ends the run session failed with reason `budget`. See [Advanced agents](/laser-sdk/advanced/agents#workflows). Operators stop sessions through `agent.control`. Native send permission on that topic is the authority boundary, so give it to operator accounts only. A cancel, pause, or resume found on any other topic is never applied. `signed_by(key)` (TypeScript: `signedBy`) on the control handle signs each control record, and on a session handle signs the session's terminal record. The managed session index verifies signed records against the stream's key registry and names the signer as `verified_actor`. Without a signature, operator and author names stay claims. In Rust, `signed_by` needs the `sign` feature. [Advanced sessions](/laser-sdk/advanced/sessions#operator-control) covers pause, resume, cancel, and force cancel. ## Durable approval records Approvals use typed `Intent`, `Vote`, and `Decision` records. The application publishes them and folds the recorded votes into a decision. Apply the effect only when the decision authorizes the intent. ```ts import { AgentId, ConversationId, Intent, Vote, VoteChoice, decide } from "@laserdata/laser-sdk" const now = BigInt(Date.now()) * 1_000n const safety = AgentId.new("safety") const intent = new Intent({ conversation: ConversationId.new(), proposer: AgentId.new("planner"), body: new TextEncoder().encode("rotate the storage credentials"), eligibleVoters: [safety], policy: { kind: "all" }, policyVersion: 7n, deadlineMicros: now + 30_000_000n }) const vote = Vote.cast(intent, safety, VoteChoice.Allow) const decision = decide(intent, [vote], BigInt(Date.now()) * 1_000n) if (decision?.authorizes(intent)) { // apply the fenced effect, then persist the decision } ``` ```rust use laser_sdk::agent::{Clock, SystemClock}; use laser_sdk::intent::{Intent, IntentPolicy, Vote, VoteChoice, decide}; use laser_sdk::prelude::{ConversationId, LaserError}; fn approve() -> Result<(), LaserError> { let now = SystemClock.now_micros(); let intent = Intent::builder() .conversation(ConversationId::new()) .proposer("planner".parse()?) .body(b"rotate the storage credentials".to_vec()) .eligible_voters(vec!["safety".parse()?]) .policy(IntentPolicy::All) .policy_version(7) .deadline_micros(now + 30_000_000) .build()?; let vote = Vote::cast(&intent, "safety".parse()?, VoteChoice::Allow)?; if let Some(decision) = decide(&intent, &[vote], SystemClock.now_micros())? { if decision.authorizes(&intent)? { // apply the fenced effect, then persist the decision } } Ok(()) } ``` ```python import time now = time.time_ns() // 1_000 intent = ls.Intent( conversation=ls.new_conversation_id(), proposer="planner", body=b"rotate the storage credentials", eligible_voters=["safety"], policy=ls.IntentPolicy.all(), policy_version=7, deadline_micros=now + 30_000_000, ) vote = ls.Vote.cast(intent, "safety", "allow") decision = ls.decide(intent, [vote], time.time_ns() // 1_000) if decision and decision.authorizes(intent): # apply the fenced effect, then persist the decision pass ``` The Rust intent calls return `IntentError`, which converts into `LaserError::Invalid`, so `?` works in a function that returns `LaserError`. In TypeScript `IntentError` is a subclass of `InvalidError`. In Python it is `IntentError(InvalidError)` with `kind` constants. An intent has an eligible voter list, an optional mandatory voter list, a policy (`All`, `Any`, or `AtLeast(n)`), a policy version, and a deadline in epoch microseconds. A vote is `allow`, `block`, or `abstain`, and names the digest and policy version of the intent it answers. Building an intent mints its ID, computes its digest, and stamps the proposal time now. `Vote.cast` stamps now too. No SDK takes a caller-supplied ID, digest, or time when it builds one. Building or validating an intent fails for an empty eligible list, a repeated eligible or mandatory voter, a mandatory voter who is not eligible, an `AtLeast(n)` threshold that is zero or above the eligible count, a deadline that is not after the proposal time, and a digest that does not match the body. Call `validate()` before publishing an intent you built or changed outside the validated constructors. `Vote::cast` fails for a voter outside the eligible list. `decide` folds the votes with these rules: * It ignores votes from voters who are not eligible, votes for another digest or policy version, votes outside the window from the proposal time to the deadline, and votes stamped after `now`. * The same vote repeated counts once. A voter who changes their choice aborts the intent. * A mandatory voter's `block` or `abstain` aborts at once. Every mandatory voter must vote `allow` before the intent commits. * Under `All`, any vote that is not `allow` aborts at once. * It returns nothing while the outcome can still change. Otherwise it returns a `Decision` that is `committed` or `aborted`. After the deadline it always returns one. `authorizes(intent)` fails when the intent is invalid or the decision does not match the intent's ID, digest, and policy version. Otherwise it reports whether the outcome is `committed`. A voter name is a claim inside a record. Use signed principals or topic permissions when you must trust voter identity. ## Where it runs Role, binding, and history calls need a server that advertises the `authz` capability, as Laser Stack and LaserData Cloud do. Session listing and `sessions().get` need the `sessions` capability. `ActionGovernor`, `delegated_allow`, `grants_allow`, `over_budget()`, the workflow budget check, and intent records work on plain Apache Iggy. In Rust, governors, intents, and sessions need the `agent` feature, roles need `rbac`, and signing needs `sign`. `managed` includes `rbac` but not `agent` or `sign`. `authorize_edge` needs `a2a-bridge` or `mcp-bridge`. ## Related Source: https://docs.laserdata.com/laser-sdk/advanced/governance --- # Interop This page is the reference for the protocol bridges. For the agents behind them, start with [Agents](/laser-sdk/fabric). Bridges connect external A2A, MCP, and AG-UI clients to agents that speak [AGDX](/laser-sdk/advanced/agdx). A bridge maps the shared fields into an AGDX command and keeps the protocol's own data in the body. The agents behind a bridge do not change. Bridges work on plain Apache Iggy and need no model provider. ## Put a worker behind a bridge A bridge publishes an AGDX command on its request topic and waits for a correlated AGDX response on its reply topic. The worker is an ordinary agent that listens on the request topic and answers with `respond`. `respond` needs the agent's `respond_on` topic, which must be the bridge's reply topic. It answers an AGDX command with a typed response that carries the correlation and is addressed to the bridge. ```ts import { Agent, AgentId, AgentTopic, agentMessageBody } from "@laserdata/laser-sdk" const text = new TextEncoder() const decode = (bytes: Uint8Array) => new TextDecoder().decode(bytes) await using assistant = Agent.builder() .id(AgentId.new("assistant")) .listenOn(AgentTopic.Sessions) .respondOn(AgentTopic.Sessions) .handler({ handle: (message, ctx) => ctx.respond(text.encode(`handled: ${decode(agentMessageBody(message))}`)) }) .build() .spawn(laser) await assistant.ready() ``` ```rust use laser_sdk::prelude::full::*; struct Assistant; impl AgentHandler for Assistant { async fn handle(&self, message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { let text = String::from_utf8_lossy(message.body()).into_owned(); ctx.respond(format!("handled: {text}")).await } } let mut assistant = Agent::builder() .id("assistant".parse()?) .listen_on(AgentTopic::Sessions) .respond_on(AgentTopic::Sessions) .handler(Assistant) .build() .spawn(laser.clone()); assistant.ready().await?; ``` ```python async def assistant(ctx, message): await ctx.respond(b"handled: " + bytes(message.body())) worker = laser.spawn_agent( "assistant", ls.AgentTopic.Sessions, assistant, respond_on=ls.AgentTopic.Sessions, ) await worker.ready() ``` Bridge constructors take the request topic and the reply topic explicitly. The samples on this page use `agent.sessions` for both. Every topic lives on the stream of the `Laser` you pass, so pass `laser.with_default_stream(stream)` to run a bridge on another stream. ## A2A `A2aBridge` serves an internal agent through A2A JSON-RPC. `card()` returns its A2A v1.0 Agent Card. | A2A method | Mapping | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SendMessage` | Publishes an AGDX command on a new task conversation and returns `Submitted`. The task ID is the conversation ID. The A2A params ride in the body as JSON. | | `SendStreamingMessage` | Publishes the same command as `SendMessage`. Readers consume the stream from the log. The bridge does not send it as SSE. | | `GetTask` | Reads the reply topic and maps the response or error with the task's correlation to the A2A task. Returns `Working` until a reply arrives. | | `CancelTask` | Publishes an AGDX error terminal with code `Cancelled` and returns `Canceled`. | The bridge does not serve `ListTasks`, because it keeps no state outside the log. ```ts import { A2aBridge, AgentId, AgentTopic } from "@laserdata/laser-sdk" const bridge = new A2aBridge( laser, AgentId.new("a2a-gateway"), AgentTopic.Sessions, AgentTopic.Sessions ) const card = bridge.card() ``` ```rust use laser_sdk::prelude::full::*; use std::sync::Arc; let bridge = Arc::new(A2aBridge::new( laser.clone(), "a2a-gateway".parse::()?, AgentTopic::Sessions, AgentTopic::Sessions, )); let card = bridge.card(); let app = bridge.clone().router(); ``` ```python import laser_sdk as ls bridge = ls.A2aBridge( laser, "a2a-gateway", ls.AgentTopic.Sessions, ls.AgentTopic.Sessions, ) card = bridge.card() ``` The A2A bridge exposes `submit`, `task`, `cancel`, `card`, and `signed_card` (TypeScript: `signedCard`). `handle_rpc(request)` (TypeScript: `handleRpc`) serves a whole JSON-RPC request without HTTP. Rust's `a2a-http` feature adds `router()`, an Axum router with the JSON-RPC endpoint at `/` and the Agent Card at `/.well-known/agent-card.json`, which delegates to `handle_rpc`. `router()` consumes an `Arc`, so call it on a clone. TypeScript and Python have no router, so route HTTP requests to `handle_rpc` yourself. ```ts const response = await bridge.handleRpc({ jsonrpc: "2.0", id: 1, method: "GetTask", params: { id: taskId } }) ``` ```rust let response = bridge .handle_rpc(serde_json::json!({ "jsonrpc": "2.0", "id": 1, "method": "GetTask", "params": { "id": task_id }, })) .await; ``` ```python response = await bridge.handle_rpc( { "jsonrpc": "2.0", "id": 1, "method": "GetTask", "params": {"id": task_id}, } ) ``` What to expect from the A2A bridge: * `submit` takes the `SendMessage` params as raw JSON bytes in Rust. TypeScript and Python also accept a JSON string, which rides byte for byte, or a value they encode as JSON. Through `handle_rpc` and the router the params are parsed and encoded again, so key order and spacing can change. * `GetTask` and `CancelTask` do not check that the task exists. An unknown task ID reads `Working`. `CancelTask` always publishes and returns `Canceled`, but a later `GetTask` returns the first terminal it finds, so a completed task stays completed. * `GetTask` scans the reply topic from its start, up to 32 passes of up to 10,000 records per partition each. A reply deeper than that reads `Working`. * The task lookup needs a default stream on the `Laser`. * A Python task is a dictionary, such as `task["id"]`. TypeScript reports the task state as an object, such as `{ kind: "known", name: "Completed" }`. The JSON-RPC output is the same in all three. ### Sign and verify the Agent Card `with_capabilities` puts the agent's capability descriptors on the card as A2A skills. `signed_card(key)` adds a detached JWS signature to the card, so clients can check it before they trust it. `with_signing_key` signs the `CancelTask` terminal that the bridge publishes. It signs nothing else, and submitted commands are never signed. TypeScript uses `withCapabilities`, `withSigningKey`, and `signedCard`. Python passes `capabilities=` and `signing_key=` to `A2aBridge(laser, ..)` and calls `bridge.signed_card(key)`. The Rust router serves the unsigned card. To publish a signed card, serve `signed_card` yourself. A client checks a signed card with `verify_card`. It rebuilds the signed payload from the card itself, so a changed field fails verification. ```ts import { SigningKey, verifyCard } from "@laserdata/laser-sdk" const key = SigningKey.fromBytes(secret) const signed = bridge.signedCard(key) const [signature] = signed.signatures ?? [] if (signature !== undefined) { verifyCard(signed, signature, key.verifyingKey()) } ``` ```rust use laser_sdk::sign::{SigningKey, verify_card}; let key = SigningKey::from_bytes(&secret); let signed = bridge.signed_card(&key)?; let value = serde_json::to_value(&signed) .map_err(|error| LaserError::Codec(error.to_string()))?; verify_card(&value, &signed.signatures[0], &key.verifying_key())?; ``` ```python key = ls.SigningKey(secret) signed = bridge.signed_card(key) ls.verify_card(signed, signed["signatures"][0], key.verifying_key) ``` `secret` is a 32-byte Ed25519 seed. Rust needs the `sign` and `a2a-bridge` features for the card helpers. ## MCP `McpBridge` maps MCP JSON-RPC tool calls to AGDX commands. It waits for a response or error with the matching correlation and returns it as a tool result. | MCP method | Mapping | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `initialize` | Echoes the client's protocol version, `2025-11-25` when the client sends none. Always advertises `tools`, and advertises `resources` and `prompts` only when configured. | | `tools/list` / `tools/call` | Serves the tools added with `with_tool`. A call publishes an AGDX command with the tool name and returns the correlated reply as a tool result. | | `resources/list` / `resources/read` | Serves the text resources added with `with_resource`. | | `prompts/list` / `prompts/get` | Serves the prompts added with `with_prompt`. Arguments are listed but not substituted. | The bridge serves only these seven methods. Any other method, including notifications such as `notifications/initialized` and `ping`, gets an error response. JSON-RPC batch arrays are rejected. A tool call waits 30 seconds for its reply by default. Change it with `with_timeout` (TypeScript: `withTimeout(ms)`, Python: `timeout_ms=`). `with_memory_tools` adds `remember` and `recall` tools with fixed input schemas (TypeScript: `withMemoryTools`, Python: `memory_tools=True`). ```ts import { AgentId, AgentTopic, McpBridge } from "@laserdata/laser-sdk" const mcp = new McpBridge( laser, AgentId.new("mcp-gateway"), AgentTopic.Sessions, AgentTopic.Sessions, "ops-tools" ).withTool("ask", "ask the assistant", { type: "object" }) const tools = mcp.listTools() ``` ```rust let mcp = Arc::new( McpBridge::new( laser.clone(), "mcp-gateway".parse::()?, AgentTopic::Sessions, AgentTopic::Sessions, "ops-tools", ) .with_tool( "ask", Some("ask the assistant".into()), serde_json::json!({ "type": "object" }), )?, ); let tools = mcp.list_tools(); let app = mcp.clone().router(); ``` ```python mcp = ls.McpBridge( laser, "mcp-gateway", ls.AgentTopic.Sessions, ls.AgentTopic.Sessions, "ops-tools", tools=[ { "name": "ask", "description": "ask the assistant", "input_schema": {"type": "object"}, } ], ) tools = mcp.list_tools() ``` Rust and TypeScript add tools, resources, and prompts with `with_tool`, `with_resource`, and `with_prompt` (TypeScript: `withTool`, `withResource`, `withPrompt`). Python passes `tools=`, `resources=`, and `prompts=` lists of dictionaries. A resource dictionary has `uri`, `name`, `mime_type`, and `text`. A prompt dictionary has a `prompt` entry and a `messages` list of role and text pairs. Every language requires the input schema to be a JSON object. A missing schema, or one that is not an object, is an `InvalidError`, and Rust `with_tool` returns a `Result` for that reason. Every language also exposes `initialize`, `list_tools`, `call_tool`, `list_resources`, `read_resource`, `list_prompts`, and `get_prompt` (camelCase in TypeScript). `call_tool(name, params)` takes the arguments as raw JSON bytes in Rust. Python and TypeScript also take raw JSON text, sent as is, or a value they encode as JSON, such as `{ question: "status?" }`. Through `handle_rpc` the body is the whole MCP params object, with `name` and `arguments`. A tool that answers with an AGDX error returns a successful JSON-RPC result with `isError: true`. Each content item names its type in a `kind` field in Rust and TypeScript, and the JSON-RPC output and Python use `type`. Rust's `mcp-http` feature adds an Axum `router()`. The `mcp-bridge` feature alone has no HTTP dependency. `handle_rpc(request)` (TypeScript: `handleRpc`) dispatches one JSON-RPC request without HTTP. ## Address one agent A bridge call goes to every agent unless you name one. On the shared `agent.sessions` topic every listening agent receives an unaddressed task or tool call, so each of them would handle it. Name the agent, and only that agent takes the command as work. ```ts const worker = AgentId.new("assistant") const task = await bridge.submit( '{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}', { target: worker } ) const result = await mcp.callTool("ask", { question: "status?" }, { target: worker }) ``` ```rust let task = bridge .submit_to( "assistant".parse::()?, br#"{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}"#.to_vec(), ) .await?; let result = mcp .call_tool_to("assistant".parse::()?, "ask", br#"{"question":"status?"}"#.to_vec()) .await?; ``` ```python task = await bridge.submit( b'{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}', target="assistant", ) result = await mcp.call_tool("ask", b'{"question":"status?"}', target="assistant") ``` The Rust `submit_to` and `call_tool_to` take any agent ID that converts into the wire ID, so the prelude `AgentId` works directly. Give an inline `parse` its type, as in `"assistant".parse::()?`. Requests that arrive through the JSON-RPC routes are unaddressed, so put one worker on the request topic behind them. ## Run a bridge call as a child session A bridge call can run as a child [session](/laser-sdk/session) of a session you hold. The child gets a submitted start on `agent.sessions` with its parent and root before the command, and the command carries the same ancestry, so a session view rolls the call up under its parent. `submit_in` leaves the end to the agent that handles the task. If the command send fails after the start, the child stays open. `call_tool_in` ends the child by the result: completed on a tool result, failed on a tool error, a timeout, or a transport error. ```ts const parent = session.conversation const root = session.root ?? parent const task = await bridge.submitIn( parent, root, '{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}', { target: worker } ) const result = await mcp.callToolIn(parent, root, "ask", { question: "status?" }, { target: worker }) ``` ```rust let parent = session.conversation(); let root = session.root().unwrap_or(parent); let task = bridge .submit_in_to( "assistant".parse::()?, parent.into(), root.into(), br#"{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}"#.to_vec(), ) .await?; let result = mcp .call_tool_in_to( "assistant".parse::()?, parent.into(), root.into(), "ask", br#"{"question":"status?"}"#.to_vec(), ) .await?; ``` ```python parent = session.conversation root = session.root or parent task = await bridge.submit_in( parent, b'{"message":{"role":"user","parts":[{"kind":"text","text":"summarize"}]}}', root=root, target="assistant", ) result = await mcp.call_tool_in( parent, "ask", b'{"question":"status?"}', root=root, target="assistant" ) ``` Rust also has `submit_in` and `call_tool_in` without a target. Python and TypeScript make the target optional. Child lifecycle records carry neither the loop guard nor a signature. ## AG-UI AG-UI gives frontends state and events. `publish_state_snapshot` and `publish_state_delta` write the session state document of a conversation, the same records that `session.state()` writes, so every reader folds one state model. A snapshot must be a JSON object, and a delta must be an RFC 6902 JSON Patch array. `reconstruct_state` returns that document. It folds `agent.sessions` first, and reads the managed state view only when the lane no longer holds the document's baseline and the view is not behind the lane. It returns nothing until a state record exists. `agui_events` reads a conversation from the log and converts it into AG-UI events: * Chat chunks become `TEXT_MESSAGE_*` events, and reasoning chunks become `REASONING_MESSAGE_*` events. * Tool argument chunks become `TOOL_CALL_START`, `TOOL_CALL_ARGS`, and `TOOL_CALL_END`. A response or error that names a tool becomes `TOOL_CALL_RESULT`. * A task status of `Submitted` becomes `RUN_STARTED`. Completed or canceled status becomes `RUN_FINISHED`. Failed or rejected status becomes `RUN_ERROR`, with the status detail or `task failed`. Another error also becomes `RUN_ERROR`. Only status records with operation `task` render. * State records become `STATE_SNAPSHOT` and `STATE_DELTA`. AG-UI events with no AGDX source, such as `MESSAGES_SNAPSHOT`, `ACTIVITY_*`, `RAW`, and `CUSTOM`, are never produced. ```ts const conversation = ConversationId.new() await laser.publishStateSnapshot(AgentId.new("ui"), conversation, { count: 0 }) const events = await laser.aguiEvents(conversation, AgentTopic.Sessions) ``` ```rust let conversation = ConversationId::new(); laser .publish_state_snapshot("ui".parse::()?, conversation, &serde_json::json!({ "count": 0 })) .await?; let events = laser.agui_events(conversation, AgentTopic::Sessions).await?; ``` ```python conversation_id = ls.new_conversation_id() await laser.publish_state_snapshot("ui", conversation_id, {"count": 0}) events = await laser.agui_events(conversation_id, ls.AgentTopic.Sessions) ``` ## Pause for a human decision `request_input` publishes a prompt command and waits for the correlated response on the reply topic, up to the timeout. The approver answers with `respond_input`. If the approver rejects with an error, the wait fails with `LaserError::Rejected` in Rust and `RejectedError` in TypeScript and Python. Both calls use the existing command and response records and need no bridge. Name the approver so that only it receives the prompt. Without a target every agent on `agent.sessions` receives it, and any of them could answer. Rust uses `request_input_from(target, ..)`, Python `target=`, and TypeScript a `{ target }` option. ```ts const decision = await laser .agdx(AgentTopic.Sessions, AgentId.new("orchestrator"), conversation) .requestInput( AgentTopic.Sessions, new TextEncoder().encode("restart the metrics service on node-7?"), 300_000, { target: AgentId.new("approver") } ) ``` ```rust let decision = laser .agdx(AgentTopic::Sessions, "orchestrator".parse::()?, conversation.into()) .request_input_from( "approver".parse::()?, AgentTopic::Sessions, b"restart the metrics service on node-7?".to_vec(), Duration::from_secs(300), ) .await?; ``` ```python decision = await laser.agdx( ls.AgentTopic.Sessions, "orchestrator", conversation_id, ).request_input( ls.AgentTopic.Sessions, b"restart the metrics service on node-7?", timeout_ms=300_000, target="approver", ) ``` The timeout is required in every SDK and has no default. Python takes it as `timeout_ms=` in milliseconds. TypeScript also accepts a `signal` option, and an aborted signal throws `CancelledError`. The approver is an agent that listens on `agent.sessions`. Its handler answers with `respond_input`, which needs the handled message to be an AGDX command with a correlation and the agent to have an ID. An agent with a signing key signs the response, so a caller that verifies signatures accepts only that agent's decision. `request_input` does not sign the prompt. ```ts const approver = { handle: (_message: AgentMessage, ctx: AgentCtx) => ctx.respondInput(AgentTopic.Sessions, new TextEncoder().encode("approved")) } ``` ```rust struct Approver; impl AgentHandler for Approver { async fn handle(&self, _message: &AgentMessage, ctx: &AgentCtx<'_>) -> Result<(), LaserError> { ctx.respond_input(AgentTopic::Sessions, b"approved".to_vec()).await } } ``` ```python async def approver(ctx, message): await ctx.respond_input(ls.AgentTopic.Sessions, b"approved") ``` Inside a handler, `ctx.approval_gate(reply_topic, prompt, timeout)` (TypeScript: `approvalGate`) does the same from the handled conversation and returns the decision body. It always publishes the prompt unaddressed on `agent.sessions` and has no target option. ## Authorize edge requests The JSON-RPC bridges do not authenticate HTTP requests. Add authentication in your HTTP framework. Your HTTP layer decodes the token, and `authorize_edge` (TypeScript: `authorizeEdge`) checks the audience and scope of the decoded claims. The SDK does not parse tokens. ```ts import { authorizeEdge, edgeDenialChallenge } from "@laserdata/laser-sdk" const denial = authorizeEdge( { audience: ["agents.example"], scopes: ["tool:read"] }, "agents.example", "tool:write" ) if (denial !== undefined) { const challenge = edgeDenialChallenge(denial) } ``` ```rust use laser_sdk::edge_auth::{EdgeClaims, authorize_edge}; let claims = EdgeClaims { audience: vec!["agents.example".to_owned()], scopes: vec!["tool:read".to_owned()], }; if let Err(denial) = authorize_edge(&claims, "agents.example", "tool:write") { let challenge = denial.challenge(); } ``` ```python denial = ls.authorize_edge( ["agents.example"], ["tool:read"], "agents.example", "tool:write", ) if denial is not None: challenge = denial.challenge ``` A token for another audience is a hard reject with no challenge, which maps to `401`. A token for the right audience without the required scope returns the challenge `Bearer scope="tool:write"`, which maps to `403` with a step-up. Rust returns the denial as the error, with `expected` or `required_scope` on its variant. TypeScript returns the denial or `undefined`, with kind `wrongAudience` or `stepUp` and the fields `expected` and `requiredScope`. Python returns an `EdgeDenial` with `kind` (`wrong_audience` or `step_up`), `code`, `challenge`, `expected`, and `required_scope`, or `None`. In Rust, `authorize_edge` needs the `a2a-bridge` or `mcp-bridge` feature. [Governance](/laser-sdk/advanced/governance#act-for-a-user) covers delegated grants. `with_default_stream` (TypeScript: `withDefaultStream`) points a bridge at another stream on the same connection. Use separate credentials when Iggy permissions must keep the streams apart. ## Loop guard Each bridge stamps its own ID in the `bridge_hops` metadata of the commands and cancels it publishes. Use `with_bridge_hops(previous)` (TypeScript: `withBridgeHops`) to continue a path that a call already took through other bridges. A path that already holds the bridge's ID is rejected before anything is published. The guard never reads inbound records, so pass `previous` yourself. Rust's `with_bridge_hops` consumes the bridge and returns a result. TypeScript and Python change the bridge in place, and Python refuses while a call is in flight. ## Errors A failed bridge call still returns a JSON-RPC response envelope. Its `error` carries the application code `-32000` and a public message from a fixed set: `invalid request`, `unsupported operation`, `conflict`, `unauthenticated`, `forbidden`, `step-up authorization required`, `not found`, or `internal error`. A malformed request gets `invalid request`. A tool call timeout reads `internal error`. Over the Rust routers, a body that is not JSON is rejected with an HTTP 4xx before `handle_rpc` runs. With a key registry verifier on the `Laser` (Rust `sign` feature), `GetTask`, MCP calls, and `request_input` skip replies that fail verification. ## Standalone helpers These helpers need no connection and behave the same in every language. | Helper | Rust | TypeScript | Python | | -------------------------------- | ---------------------------------------- | ---------------------------------- | ------------------------------------ | | A2A request to AGDX command | `a2a::command_from_message_send` | `commandFromMessageSend` | `command_from_message_send` | | AGDX envelope to A2A task | `a2a::task_from_envelope` | `taskFromEnvelope` | `task_from_envelope` | | MCP tool call to AGDX command | `mcp::tool_call_from_request` | `toolCallFromRequest` | `tool_call_from_request` | | AGDX envelope to MCP tool result | `mcp::tool_result_from_envelope` | `toolResultFromEnvelope` | `tool_result_from_envelope` | | Bridge loop guard | `a2a::enter_bridge` | `enterBridge` | `enter_bridge` | | Edge token check | `edge_auth::authorize_edge` | `authorizeEdge` | `authorize_edge` | | Sign an Agent Card | `sign::sign_card_value` | `signCardValue` | `sign_card_value` | | Verify an Agent Card | `sign::verify_card` | `verifyCard` | `verify_card` | | Verify a signed delegation | `sign::verify_delegation` | `verifyDelegation` | `verify_delegation` | | Claim check in and out | `blob::check_in`, `blob::resolve_body` | `checkIn`, `resolveBody` | `check_in`, `resolve_body` | | Snapshot bytes | `snapshot::encode`, `snapshot::decode` | `encodeSnapshot`, `decodeSnapshot` | `encode_snapshot`, `decode_snapshot` | | Resume offsets of a snapshot | `agent::resume_offsets` | `resumeOffsets` | `resume_offsets` | | Reciprocal-rank fusion | `memory::fuse_reciprocal_rank` | `fuseReciprocalRank` | `fuse_reciprocal_rank` | | Clocks | `agent::SystemClock`, `agent::TestClock` | `SystemClock`, `TestClock` | `SystemClock`, `TestClock` | * The converters keep the original JSON bytes. Set the JSON content type when you publish the resulting envelope. * `enter_bridge` rejects an empty bridge ID and a path that already contains this bridge. In Rust it needs the `a2a-bridge` or the `mcp-bridge` feature. * `verify_delegation` checks that the envelope signature verifies against an enrolled key and returns the signer and the delegated user from the signed metadata, or nothing when the envelope carries no delegation. Authorize the pair with `delegated_allow`. * `resume_offsets` returns one past the last offset a snapshot folded for each partition, and stops at the unsigned 64-bit ceiling. The snapshot byte helpers read and write the named-field CBOR storage form, and fail with an invalid error on a bad snapshot. * `SystemClock` and `TestClock` count epoch microseconds as unsigned 64-bit values, and `advance` wraps around at the ceiling. A TypeScript `TestClock` needs a `bigint` start, and Python defaults the start to 0. Both raise `InvalidError` outside the range. * The card helpers need the `sign` and `a2a-bridge` features in Rust. The `sign` feature alone covers `verify_delegation`. ## Where it runs Bridges work on plain Apache Iggy, Laser Stack, and LaserData Cloud. `reconstruct_state` reads the managed state view only on a deployment that has one. The SDK does not host a model loop or a sandbox. A provider's event subscription is not an Iggy stream, so an adapter in front of one must map event IDs and cursors, record tool results with their correlation, and collect child results explicitly. In Rust, enable `a2a-bridge`, `mcp-bridge`, or `agui` for the adapters, and add `a2a-http` or `mcp-http` for the Axum routers. ## Related Source: https://docs.laserdata.com/laser-sdk/advanced/interop --- # AGDX This is the reference for implementers. To build agents with the SDK, start with [Agents](/laser-sdk/fabric) and [Sessions](/laser-sdk/session). AGDX (Agent Data Exchange Protocol) defines the records and rules for streaming, managed data, and agent coordination. Laser SDK implements it on Apache Iggy's durable partitioned log. Iggy supplies transport, retention, and consumer groups. The SDK records which model produced an output, but it never calls a model. The [specification](https://agdxprotocol.ai) and the [AGDX reference](https://github.com/laserdata/laser-sdk/blob/main/docs/agdx.md) in the laser-sdk repository define every field, code, limit, and byte layout. This page covers what an implementer or an application author needs to act on. ## One contract, five layers | Layer | Owns | Can be used alone | | --------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------ | | Substrate | Durable partitions, offsets, retention, consumer groups, pull-based reads | Laser SDK uses Apache Iggy | | Wire | Portable types, named-field CBOR, dictionaries, limits, capability shapes, fixtures | Yes, by an independent port | | Platform | Publish and consume, projections and query, key-value, forks, graph | Yes, with no agent concepts | | Fabric | Agent envelopes, sessions, reliable consumers, coordination, context, memory, governance | Yes, with no public edge protocol | | Edges | A2A, MCP, and AG-UI mappings | Only when an external client needs that contract | The log is the source of truth. Projections, indexes, state, graph data, session indexes, and agent registries all derive from its records. Applications do not keep separate stores in sync for these features. ## Envelope anatomy Each typed agent record carries a named-field CBOR `AgentEnvelope` as its payload. Routing information that a reader needs before decoding rides in Iggy headers outside the envelope. Each field has one authoritative location. One shared encoder writes exactly these headers on every agent record: | Header | Meaning | | --------------------------------------- | ------------------------------------------------- | | `agdx.ct` | Content type of the body | | `agdx.av` | Envelope version | | `gen_ai.conversation.id` | The conversation, which is also the partition key | | `agdx.parent_conv` and `agdx.root_conv` | Parent and root of a child session | | `gen_ai.agent.id` | The author | | `agdx.to` | The addressee, an agent ID or `*` for every agent | IDs ride in headers as canonical strings. Every record on a shared session topic carries `agdx.to`, and a reader never parses `*` as an agent ID. Headers have a soft limit of 1,024 bytes per record. Each value is at most 255 bytes, and each header costs 9 framing bytes toward the total. ```text command { kind: command record: 01J... conversation: 01J... # partition key and trace identity source: planner # a claim unless verified target: summarizer correlation: 01J... # request and reply pairing operation: summarize body: # decoded using agdx.ct signature: } ``` | Field family | Fields | Purpose | | ----------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Identity and routing | `kind`, `record`, `conversation`, `parent`, `root`, `source`, `target` | What this is, which session and session tree it belongs to, who claims to have sent it, and who it is for | | Causality | `cause`, `cause_at`, `correlation` | Parent record, optional native log position, request and reply pairing | | Chunk lifecycle | `channel`, `sequence`, `last`, `finish_reason` | Ordered streaming and deterministic reassembly | | Execution | `operation`, `tool`, `task_state`, `deadline_micros`, `idempotency_key` | What is happening and which safety rules apply | | Content | `body`, content type header | Opaque payload bytes and their codec | | Accounting and policy | `usage`, `metadata` | Advisory token usage and cost, and pinned policy or routing context | | Evolution and integrity | `must_understand`, `signature` | Strict feature handling and optional verified authorship | Machine IDs are 128-bit values, shown as 26-character Crockford base32 strings. CBOR encodes them as 16-byte big-endian byte strings. `cause` identifies the parent record portably, and the optional `cause_at` gives its Iggy log position for a direct local lookup. `must_understand` is a bitset of features a receiver must implement to process the record. A receiver rejects or dead-letters a record with a bit it does not know. No bits are defined yet, and `0`, the default, means ignore anything unknown. `usage.cost_micros` is the cost in millionths of the deployment's currency. Producers multiply a decimal amount by one million, round half up, and reject values outside the unsigned 64-bit range. Like all usage, it is advisory. ### Envelope limits | Limit | Value | | ------------------------------------------ | ----------------------------------------------- | | `operation`, `tool`, `finish_reason`, each | 256 bytes | | Idempotency key | 64 bytes | | Metadata entries, key, value, total | 32 entries, 256 bytes, 1,024 bytes, 8,192 bytes | | Body reference | 1,024 bytes | | Agent card capabilities | 64 | | Chunk body | 64 KiB | ### Pinned metadata keys | Key | Meaning | | ----------------------------------------------------------------------- | -------------------------------------------------------------- | | `role` | Chat role, such as `user`, `assistant`, `system`, or `tool` | | `bridge_hops` | The bridge loop guard, a list of bridge IDs | | `submitted` | Marks the first command of a submitted session | | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.provider.name` | The requested model, the model that answered, and the provider | | `duration_micros` | Duration of a model or tool call, measured by the application | | `on_behalf_of` | The user an agent acts for | | `purpose`, `data_classification`, `task_context`, `session_intent` | Policy inputs. Advisory unless the envelope is signed | ## Legal message shapes The message kind decides which fields are required and which are forbidden. The wire decoder, the SDK constructors, and receivers enforce the same rules. | Kind | Required core | Meaning | | ---------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `command` | `record`, `conversation`, `source`, `correlation`, `body` | Requests a reply or an effect. A command without a correlation is invalid | | `response` | `record`, `conversation`, `source`, `correlation`, `body` | Answers the command with the same correlation | | `event` | `record`, `conversation`, `source`, `body` | Announces something and expects no reply | | `chunk` | `conversation`, `source`, `correlation`, `channel`, `sequence`, `body` (the terminal chunk can be empty) | Carries one ordered part of `chat`, `reasoning`, or `tool_args` | | `status` | `record`, `conversation`, `source`, `operation` | Carries task, session, card, progress, quarantine, or unquarantine state | | `error` | `record`, `conversation`, `source`, `correlation`, `body` | Ends a request, and optionally a chunk channel, with a typed error | R means required, O optional, and X forbidden: | Field | `command` | `response` | `event` | `chunk` | `status` | `error` | | --------------------- | --------- | ---------- | ------- | ------------------------------- | ----------------------- | ------------------- | | `correlation` | R | R | O | R | O (R for `task`) | R | | `channel`, `sequence` | X | X | X | R | X | O (stream terminal) | | `last` | X | X | X | O | O | X | | `finish_reason` | X | O | X | O (with `last`) | X | X | | `idempotency_key` | O | O | O | X | X | X | | `deadline_micros` | O | X | X | O (opening chunk) | X | X | | `task_state` | X | O | X | X | R (`task` or `session`) | O | | `operation` | O | O | O | R on the opening chunk, X after | R | O | | `tool` | O | O | O | O | X | O | | `usage` | X | O | O | O (terminal chunk) | O | O | More rules: * A `status` with operation `task` needs a correlation and a task state. * A `status` with operation `session` needs a task state and a CBOR body: a start, a transition, or an end. Its `last` must equal whether the task state is terminal, so an end sets it and a start or transition must not. * The opening chunk, sequence `0`, carries the purpose `chat`, `reasoning`, or `tool_args`. Later chunks must not. * `root` needs `parent`, and neither may equal the record's own conversation. Use the typed SDK constructors instead of raw maps. They apply these rules before publishing. ## Streams of chunks Chunks share a `channel` and are ordered by `sequence` within one conversation partition. A stream ends with a chunk that sets `last` and a `finish_reason`, or with an `error` that names the channel. Readers reassemble with the same rules everywhere: * Apply chunks in order from sequence 0, once per sequence. * Drop duplicate sequences and count them. * On a gap, end the local stream with finish reason `gap`. * Accept only the first terminal record, and drop and count records after it. * Carry whole-stream usage once, on the terminal chunk. The reader creates the `gap` and `abandoned` outcomes locally, `abandoned` when the opening chunk's deadline passes. Neither is written back to the log, and replay returns the original records. Offsets let a reader resume. ## Produce the same command in every SDK ```ts import { AgentId, AgentTopic, ConversationId, CorrelationId, MintUlid } from "@laserdata/laser-sdk" const text = new TextEncoder() const conversation = ConversationId.new() const correlation = MintUlid.mint(CorrelationId) const record = await laser .agdx(AgentTopic.Sessions, AgentId.new("planner"), conversation) .command(correlation, text.encode("summarize incident 42")) .withOperation("summarize") .send() ``` ```rust use laser_sdk::prelude::full::*; use laser_sdk::types::MintUlid; use laser_sdk::wire::agent::{ConversationId as WireConversationId, CorrelationId}; let conversation = ConversationId::new(); let correlation = CorrelationId::mint(); let record = laser .agdx( AgentTopic::Sessions, "planner".parse::()?, WireConversationId::from(conversation), ) .command(correlation, b"summarize incident 42".to_vec()) .with_operation("summarize") .send() .await?; ``` ```python import laser_sdk as ls conversation = ls.new_conversation_id() correlation = ls.mint_ulid() record = await laser.agdx( ls.AgentTopic.Sessions, "planner", conversation, ).command( correlation, b"summarize incident 42", operation="summarize", ) ``` Each client publishes the same logical `command` envelope and returns the generated record ID. Rust `send()` returns `Option` and TypeScript returns `RecordId | undefined`. Python's verbs publish at once and return the ID, with no `send()`. Rust mints a correlation with `CorrelationId::mint()` from the `MintUlid` trait, TypeScript with `MintUlid.mint(CorrelationId)`, and Python with `ls.mint_ulid()`. In Rust, `WireConversationId` is `laser_wire::agent::ConversationId`, imported under that name. The worker answers with `respond` and the same correlation, or opens a chunk stream for incremental output. Send builders refine the envelope before it is published: | Refinement | Rust | TypeScript | Python keyword | | ------------------------------ | ------------------------------------ | ------------------------------ | --------------------------------------------------------------------------------- | | Cause record and position | `with_cause(record, Some(position))` | `withCause(record, position)` | `cause=` and `cause_at=ls.LogPosition(stream_id, topic_id, partition_id, offset)` | | Parent and root session | `with_ancestry(..)` | `withAncestry(..)` | `parent=` and `root=` | | Deadline in epoch microseconds | `with_deadline_micros(n)` | `withDeadlineMicros(n)` | `deadline_micros=` | | Idempotency key | `with_idempotency_key(key)` | `withIdempotencyKey(key)` | `idempotency_key=` | | Metadata entry | `with_metadata(key, value)` | `withMetadata(key, value)` | `metadata=` with a dict | | Tool name | `with_tool(name)` | `withTool(name)` | `tool=` | | Token usage | `with_usage(usage)` | `withUsage(usage)` | `usage=` | | Finish reason | `with_finish_reason(reason)` | `withFinishReason(reason)` | `finish_reason=` | | Target agent | `with_target(agent)` | `withTarget(agent)` | `target=` | | Operation name | `with_operation(name)` | `withOperation(name)` | `operation=` on `command`, `respond`, `emit`, and `fail`. Positional on `status` | | Content type | `content_type(type)` | `contentType(type)` | `content_type=` | | Claim check | `claim_check(&store, threshold)` | `claimCheck(store, threshold)` | `claim_check=(store, threshold_bytes)` | | Signing | `signed_by(&key)` | `signedBy(key)` | `signed_by=` per send, or `signing_key=` on `laser.agdx(..)` | Rust and TypeScript builders also have `with_task_state`, `with_correlation`, `last()`, and `body()` (camelCase in TypeScript). Python passes refinements as keywords to `command`, `respond`, `emit`, `status`, and `fail`. All five take `operation=`, `target=`, `task_state=`, `last=`, and a correlation, which is positional on `command`, `respond`, and `fail` and a keyword on `emit` and `status`. `status` keeps its operation positional in every SDK. `fail` takes no content type, because its body is the CBOR `AgentErrorBody`. Rust and TypeScript refuse an error envelope relabeled with another content type with an invalid error at send. Rust `fail` returns a `Result`, so it needs `?` before the builder calls. A cause position needs a cause record ID. The per-kind rules still apply, so a refinement the kind forbids fails the send. ### Large bodies `claim_check(store, threshold)` stores a large body outside the log and replaces it with a `BodyRef`. At or above the threshold, the store receives the body and the record carries the reference, size, and SHA-256 digest, with content type `ref`. Below it, the body stays inline. The SDK has no default blob store. A store implements `put` and `get`. The standalone `check_in(store, threshold, payload)` and `resolve_body(store, payload)` apply the same rule without a connection (TypeScript: `checkIn` and `resolveBody`). `resolve_body` checks the size and then the SHA-256 digest before it returns bytes, and a mismatch is an integrity error. For a received message, `message.resolve_body(store)` in Rust and Python and `agentMessageResolveBody(message, store)` in TypeScript start from the message body, which is the envelope body for an AGDX record. They fetch and check the stored bytes when the content type is `ref`, and return the body unchanged otherwise. A `BodyRef` reference is a URI, object key, or key-value key of at most 1,024 bytes, with a size, a 32-byte SHA-256 digest, and optional encryption details. A consumer must compare fetched bytes with the digest. ### Chunk writers Chunk writers stream incremental output under one correlation. The purpose is `chat`, `reasoning`, or `tool_args`, and it rides the opening chunk. A chunk body is at most 64 KiB. A writer can take a target, a content type, an opening deadline, and bounded buffering. `buffered(max_chunks, linger)` takes a `Duration` in Rust and milliseconds in TypeScript and Python. `flush()` publishes pending chunks, and the terminal call, `finish` or `fail`, always flushes. Linger is checked on writes, so an idle writer keeps its pending chunks until a write, a flush, or the terminal call. ```ts const stream = laser .agdx(AgentTopic.Streams, AgentId.new("summarizer"), conversation) .stream(correlation, "chat") .buffered(32, 20) await stream.write(text.encode("The incident ")) await stream.write(text.encode("is resolved.")) await stream.finish("stop") ``` ```rust use std::time::Duration; let mut stream = laser .agdx( AgentTopic::Streams, "summarizer".parse::()?, WireConversationId::from(conversation), ) .stream(correlation, "chat") .buffered(32, Duration::from_millis(20)); stream.write(b"The incident ".to_vec()).await?; stream.write(b"is resolved.".to_vec()).await?; stream.finish("stop", None).await?; ``` ```python stream = laser.agdx( ls.AgentTopic.Streams, "summarizer", conversation, ).stream(correlation, "chat").buffered(32, 20) await stream.write(b"The incident ") await stream.write(b"is resolved.") await stream.finish() # finish_reason defaults to "stop" ``` Rust `finish(reason, usage)` takes an optional usage. TypeScript `finish(reason, usage?)` and Python `finish(finish_reason="stop", usage=None)` match it. Signing works on the command, response, event, status, and error verbs. The `request_input` helper and the chunk writer do not sign records. ## Read an agent message A handler receives an `AgentMessage`. Routing and accounting fields live on `message.provenance`, the decoded envelope on `message.envelope`, and the envelope body behind `body()` (TypeScript: `agentMessageBody(message)`). `id` is the message's log position. The provenance carries the sender as `agent`, the conversation, correlation, and idempotency key, the deadline (`deadline` in Rust and Python, `deadlineMicros` in TypeScript), and advisory token usage as an `LlmUsage` in `usage`. Token counts are optional, and `bigint` in TypeScript. ```ts import { type AgentCtx, type AgentMessage, agentMessageBody } from "@laserdata/laser-sdk" const summarizer = { handle: async (message: AgentMessage, ctx: AgentCtx) => { const sender = message.provenance.agent?.asStr() ?? "unknown" const tokens = message.provenance.usage?.inputTokens ?? 0n const body = agentMessageBody(message) await ctx.respond( new TextEncoder().encode(`${body.byteLength} bytes from ${sender}, ${tokens} tokens`) ) } } ``` ```rust struct Summarizer; impl AgentHandler for Summarizer { async fn handle( &self, message: &AgentMessage, ctx: &AgentCtx<'_>, ) -> Result<(), LaserError> { let provenance = &message.provenance; let sender = provenance.agent.as_ref().map_or("unknown", |agent| agent.as_str()); let tokens = provenance .usage .as_ref() .and_then(|usage| usage.input_tokens) .unwrap_or(0); let body = message.body(); ctx.respond(format!("{} bytes from {sender}, {tokens} tokens", body.len())) .await } } ``` ```python async def summarize(ctx, message): provenance = message.provenance sender = provenance.agent or "unknown" tokens = (provenance.usage.input_tokens if provenance.usage else None) or 0 body = bytes(message.body()) await ctx.respond(f"{len(body)} bytes from {sender}, {tokens} tokens".encode()) ``` Python handlers receive `(ctx, message)`, in that order. ## Sessions A session is one conversation with a recorded lifecycle. Its ID is the conversation ID. [Sessions in depth](/laser-sdk/advanced/sessions) shows the SDK surface. The records are ordinary envelopes: | Record | Topic | Meaning | | ------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------- | | `status` with operation `session` and task state `Submitted` | `agent.sessions` | A session handed to an agent, with its start body | | `command` with metadata `submitted = true`, operation `invoke_agent` by default | `agent.sessions` | The work handed to the agent in a submission | | `status` with operation `session` and task state `Working` | `agent.sessions` | A start, or a pickup or resume with a transition body | | `status` with operation `session` and task state `Paused` | `agent.sessions` | A participant acknowledging a pause request | | `status` with operation `session` and a terminal task state | `agent.sessions` | The end, with a reason and an optional structured error | | `command` with `chat`, `text_completion`, or `generate_content` | `agent.sessions` | A model request, answered by a `response` or `error` with the same correlation | | `command` with `execute_tool` | `agent.sessions` | A tool call, answered the same way | | `event` with `state_delta` or `state_snapshot` | `agent.sessions` | A JSON Patch delta or a whole-document snapshot of session state | | `event` with `context_assembled`, `context_compacted`, or `context_retrieved` | `agent.sessions` | What a model call received, what a summary replaced, and what memory was recalled | | `event` with `policy_decision` | `agent.sessions` | A governance decision | | `status` with operation `progress` | `agent.heartbeats` | One process heartbeat per stream, listing the sessions it holds | | `command` with `session_pause`, `session_resume`, `session_cancel`, or `force_cancel` | `agent.control` | An operator request. The same request on any other topic is never applied | | `status` with operation `session`, task state `Canceled`, and reason `forced` | `agent.control` | An operator ending a session whose agent is gone | A start body names the agent, the SDK language and version, and optionally a label, namespace, parent and root, idle timeout, token and cost budget, and tags. A label is at most 256 bytes with no control characters. ### Lifecycle * The first terminal record on the session lane wins. A repeated identical terminal changes nothing. Records after it still count, and readers flag them `after_end`. * A session without a start record is implicit: active, shown idle after its timeout, never completed. * Readers map task states to statuses: `Submitted` to `submitted`, `Working`, `InputRequired`, and `AuthRequired` to `active`, `Paused` to `paused`, `Completed` to `completed`, `Canceled` to `canceled`, and `Failed` and `Rejected` to `failed`. * Idle and over budget are worked out at read time and are never terminal. An SDK that enforces a budget ends the session itself with reason `budget`. * A dead-lettered record never fails a session by itself unless the runtime is configured to. ### Heartbeats A process that holds a lease on a session lists it in a heartbeat on `agent.heartbeats`, keyed by process. One record lists at most 2,048 sessions. A process beats every 60 seconds by default, or at one fifth of the shortest idle timeout it holds when that is shorter. The default idle timeout is 5 minutes, and `agent.heartbeats` expires records after one hour. A process killed without a terminal record shows idle, never failed. ### Pause and resume An operator's `session_pause` request names its participants, the agents whose acknowledgments complete the pause. Each acknowledges with a `Paused` status naming the exact request. An agent holds work that arrives while the session is paused: it appends a `session_parked` event before it commits the source and never treats parked work as handled. After `session_resume`, each agent acknowledges with a `Working` status, handles each held record at least once before new work, and appends `session_unparked`. A cancel while paused ends the session with the held records unhandled. Agents follow every partition of `agent.control`, outside their consumer groups. Operators may sign control records and agents may sign terminal records, and the managed index names a verified signer as `verified_actor`. No capability advertises the pause runtime yet. ### Dispatch A reliable consumer classifies every record before its handler sees it. Only a command for an operation the handler serves, addressed to this agent, to every agent, or to no agent, is work. Events, replies, status records, and records for other agents are skipped and committed. A control request counts only on `agent.control`. The author never decides the class, so an agent can send work to itself. A reply answers a request only when it carries the request's correlation, belongs to the request's session, is a response or an error, and is addressed to the requester when the request named one. The request never answers its own wait. ### State Readers apply session state in lane order. A patch applies as a whole or not at all. A repeated delta applies once. A snapshot replaces the document only when it was written against the current revision, so a late snapshot cannot overwrite a newer change. A patch has at most 256 operations, a document at most 8 MiB, and JSON integers must fit the range that JavaScript represents exactly. ### Managed reads A deployment that registers a stream as a session source serves seven reads: list, get, events, state, links, sources, and changes. Every read names its stream first, and there is no list across streams. The HTTP routes sit under `/agdx/sessions/{stream}`. Replies carry the fold frontier of each source and report gaps, so a reader can tell a settled view from one still catching up. The change feed is a per-stream sequence that a reader polls with the last sequence it saw. List reads filter by status, agent, text, `root`, and `label_prefix`. A session summary reports `held` work and a `liveness_unknown` flag. Timelines use the display types `session.parked`, `session.unparked`, and `invalid` beside the others. The SDKs read the index through `get`, `list`, `events`, `state`, `links`, `sources`, `changes`, and `watch` on the session factory. A failed read surfaces as a typed session error. Access needs `session:read` on `stream:` and read permission on the whole stream. Registering a stream needs `session:admin` and the same stream-wide read permission. The server stamps the verified stream on every session request, and the managed backend refuses one without it. A session factory remembers the stream and lane topic generations and the partition count it saw. Before a lane write, the SDK compares them with the current source and returns a stale session error when they changed. ## Stream-scoped names A stream is the boundary of one tenant. Managed names that belong to a stream are written `stream:/`. A client with a default stream scopes key-value and memory namespaces, lease and fence namespaces, the key registry, graph names, projection and index IDs, query indexes, fork IDs, and change-feed index filters. A name that already starts with `stream:` passes through. Schema requests carry the stream, and the registry keys a schema by stream and ID. Range scans, bulk deletes, and graph reads carry the stream next to their conversation lens. The server stamps the verified stream on every request that names such a resource, refuses an unresolved stream, and refuses a request or batch that names two streams. A deployment in stream tenancy mode announces `stream_tenancy`, rejects unscoped names, refuses role grants that span streams, and keeps each stream's change records and dead letters on that stream's own topics. Change records then carry their `stream`. See [Stream-scoped resource names](/laser-sdk/connect#stream-scoped-resource-names). ## Identifiers and topics | Need | Rust | TypeScript | Python | | -------------------------- | ----------------------------------------------------------- | ------------------------------ | ------------------------- | | Agent ID as text | `agent.as_str()` | `agent.asStr()` | A plain `str` | | Agent ID for an envelope | `agent.wire_id()` | `agent.wireId()` | A plain `str` | | Fresh ULID-valued wire ID | `CorrelationId::mint()` with `MintUlid` | `MintUlid.mint(CorrelationId)` | `ls.mint_ulid()` | | Well-known topic | `AgentTopic::Sessions` | `AgentTopic.Sessions` | `ls.AgentTopic.Sessions` | | Any other topic | `AgentTopic::Custom(&identifier)` with an Iggy `Identifier` | `AgentTopic.Custom(name)` | The topic name as a `str` | | Invalid ID | `IdError` | `IdError` | `IdError` | | Invalid provenance headers | `ProvenanceError` | `ProvenanceError` | `ProvenanceError` | The well-known topics and their names: | Topic | Name | Holds | | ----------------- | ------------------------ | -------------------------------------------------------------------------------------- | | `Sessions` | `agent.sessions` | The session lane: work, replies, model and tool records, lifecycle, state, and context | | `Streams` | `agent.streams` | Chunk streams | | `Heartbeats` | `agent.heartbeats` | Process heartbeats, with a one-hour expiry | | `Control` | `agent.control` | Operator control requests, writable by operators only | | `Memory` | `agent.memory` | Memory records | | `Audit` | `agent.audit` | Policy evidence | | `Registry` | `agent.registry` | Agent cards and registry facts | | `WorkflowJournal` | `agent.workflow_journal` | Workflow step outcomes | | `Dlq` | `agent.dlq` | Dead-letter capsules | Both errors name the failure, such as an empty or invalid ID or a missing or oversized header. Rust reports it as the enum variant, TypeScript in the message, and Python in the error's `kind`. ## Delivery, ordering, and replay * Delivery is at least once. Commit the consumer offset after the handler succeeds. A crash before the commit causes a replay. * Agent records use the session, which is the conversation ID, as their partition key. Each session is ordered, and separate sessions run in parallel. * Lifecycle and state always stay on the session's partition of `agent.sessions`. * A stream picks one layout for agent work: shared (the default), a topic per agent, a partition per agent, or a single partition. In the per-agent partition layout, a command, response, error, or chunk addressed to a declared agent lands on that agent's partition. A command is keyed by its addressee and a reply by its requester. In the per-agent topic layout, the same records sent to `agent.sessions` for a declared agent go to that agent's declared topic, keyed by session, and a plain record with a target follows the same rule. Lifecycle, state, broadcast records, and records for undeclared agents stay on the lane in every layout. * Across partitions, readers order records by broker append time, then by stream, topic, partition, and offset. The producer's clock is never used for order. * On a shared topic, a server with filtered reads can deliver each agent only its own and broadcast records through the filter `agdx.to In [, "*"]`. A filter is not an access boundary. Only separate topics with separate grants keep agents from reading each other's records. * Exactly-once effects need an application idempotency key and durable processed-key storage. They are not a delivery mode. * An acknowledgment commits an offset. AGDX adds no ack, nack, visibility timeout, priority, or server-side retry protocol. * Dead-letter records on `agent.dlq` include the original record bytes, the source position, the reason, the attempt count, and optional detail. ## Trust boundary Agent-written fields are claims until they are authenticated. Routing information does not grant access. | Signal | Safe reading | | ---------------------------------------------- | ------------------------------------------------------------ | | `source`, target, usage, cost, policy metadata | Advisory on a shared unsigned topic | | Verified envelope signature | The enrolled principal signed the canonical envelope | | Signature context | Also binds the out-of-band content type and envelope version | | Write-exclusive Iggy topic with ACLs | Authorship established by topology | | Server-stamped user on managed commands | Trusted input to capability RBAC | | Fence token checked by the state store | Rejects a stale lease holder before an effect | `target` restricts routing without granting permission. If the registry has a signature verifier, quarantine and unquarantine records count only with a valid operator signature. Without a verifier, write access to the registry topic is the only gate. Delegated work stores `on_behalf_of` in envelope metadata. It is a claim unless the envelope is signed. An action must pass both the agent's grants and the user's grants. ### Signatures A signature uses Ed25519 (scheme 1) with an 8-byte key ID. The signed input is the domain `agdx.signature.v1`, the encoded context when present, and the canonical envelope with its signature absent. A context binds `agdx.ct` and `agdx.av`, so changing either header breaks the signature. Keys bind to an authenticated principal, not to the claimed `source`. With a verifier configured, an unsigned reply, an unknown key, an invalid signature, or the wrong signer never resolves a correlated wait, and key validity is judged at the time the server recorded. ### Enroll and verify signing keys A `KeyRegistry` maps principals to their Ed25519 verifying keys. Pass it to an agent as its `verifier` to accept only envelopes that an enrolled key signed. A `KeyRecord` binds one key to a principal, a kind (`agent` or `operator`), and an optional validity window. `from_verifying_bytes` (TypeScript: `fromVerifyingBytes`) builds one from the 32 public-key bytes a signer published, and `verifying` reads the key back. ```ts import { KeyKind, KeyRecord, KeyRegistry, SigningKey } from "@laserdata/laser-sdk" const key = SigningKey.fromBytes(secret) const registry = new KeyRegistry() registry.enrollRecord(KeyRecord.fromVerifyingBytes("planner", key.verifyingKey(), KeyKind.Agent)) const principal = registry.verify(envelope) ``` ```rust use laser_sdk::sign::{KeyKind, KeyRecord, KeyRegistry, SigningKey}; let key = SigningKey::from_bytes(&secret); let mut registry = KeyRegistry::new(); registry.enroll_record(KeyRecord::from_verifying_bytes( "planner", key.verifying_key().as_bytes(), KeyKind::Agent, )?); let principal = registry.verify(&envelope)?; ``` ```python key = ls.SigningKey(secret) registry = ls.KeyRegistry() registry.enroll_record(ls.KeyRecord.from_verifying_bytes("planner", key.verifying_key, "agent")) principal = registry.verify(envelope) ``` `secret` is a 32-byte Ed25519 seed that only the signer holds, and `envelope` is a decoded `AgentEnvelope`. `verify` returns the principal that signed the envelope and fails for an unsigned envelope, an unknown key, or a revoked key. `verify_at` (TypeScript: `verifyAt`) also checks the key's validity window at a time in epoch microseconds and returns a `VerifiedPrincipal` with the principal and its kind. `verify_observed_at` (TypeScript: `verifyObservedAt`) also checks the observed `agdx.ct` and `agdx.av` headers against the signed context. The managed key registry stores records in the key-value namespace `agent.keys` by default, keyed by the lowercase hex of the first 8 SHA-256 bytes of the verifying key. Enrollment and revocation use compare-and-swap. ## More than agent messages These data interfaces use standard authenticated Iggy transport. Filtered group readers also open dedicated coordinator and partition connections. | Surface | Operations | Source of truth | | ------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- | | Streaming | Publish, consume, typed envelopes, replay, batching | Apache Iggy log | | Consumer filters | Group-aware reads, fenced acknowledgments, sample tests, previews, group filter policies and their revisions | Apache Iggy log, with the policy catalog in the managed plane | | Materialized views | Projections, schemas, query, change feed, graph traversal | Views derived from log records | | Working state | Key-value, compare-and-swap, fenced writes, leases, copy-on-write forks | Ordered mutations recorded through the platform | Memory combines these interfaces instead of adding a wire command family. `remember` publishes a typed record. Recall reads an available view, graph operations manage relationships, and context supplies conversation scope. ## Capability negotiation The SDK sends `hello` when it connects and again when it refreshes capabilities. The reply reports managed support, operation versions, feature bits, backends, and optionally the names of the ops stream topics: control, dead letters, changes, and the managed mutation topics. Explicit client configuration wins over the reported names. Backend descriptors report resource identity, generation, readiness, and supported operations. A backend that is replaying does not enable managed operations. After startup or a restart, refresh capabilities or wait for readiness with a deadline. See [Managed data](/laser-sdk/advanced/managed-data). * An unavailable interface returns a typed unsupported error. * A mismatched operation version fails locally before sending. * Optional guarantees, such as stronger query consistency, need explicit advertised support. * The `sessions` flag is set only by a server that serves the managed session reads. A client starts with it off and never infers it from the SDK version. The former `agent_workflow` bit is retired and never set, and the retired run codes are never reused. * The `stream_tenancy` flag means the deployment scopes every managed name to one stream. * Features default to off. A server must not advertise a guarantee it cannot provide. * Consumer filters report native evaluation, group-aware reads, and the policy catalog separately. The LaserData Iggy server evaluates filters and serves group reads without the managed plane. Configuring a group's filter needs the catalog in the managed plane. Without it, a group with no catalog history reads unfiltered, and a group with catalog history, or a read that needs a minimum catalog position, returns `catalog_unavailable`. * Standalone Iggy supports streaming and agent services backed by the log. Managed operations need advertised capabilities, which Laser Stack and LaserData Cloud supply. Use `laser.capabilities()` to choose supported operations. Applications do not need to guess support from failed requests or supply a capability list by hand. ## Interop is an edge mapping Bridges translate public protocols into AGDX at the system boundary. Internal agents keep reading and appending durable records. A request passes through the external adapter, an AGDX command, the internal agent, an AGDX reply or stream, and the response adapter. | External contract | AGDX mapping | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | A2A message and task lifecycle | Command on a fresh task conversation, then response, error, and task-status records. `submit_in` runs the task as a child session | | MCP `tools/call` | Command carrying the tool name, and a correlated response or error. `call_tool_in` runs the call as a child session | | AG-UI chat, reasoning, and tool calls | Chunk streams rendered as frontend events | | AG-UI shared state | `state_snapshot` and RFC 6902 `state_delta` events | | Human approval | Ordinary command and response through `request_input` and `respond_input` | Map shared meaning into envelope fields and keep protocol-specific body bytes unchanged. The `bridge_hops` metadata key is the loop guard. `enter_bridge(bridge, previous)` (TypeScript: `enterBridge`) appends a bridge ID to a hop list and rejects an ID that is already present. In Rust it is `laser_sdk::a2a::enter_bridge` with the `a2a-bridge` feature and `laser_sdk::mcp::enter_bridge` with the `mcp-bridge` feature. Bridges in all three SDKs stamp the list on the records they publish, and `with_bridge_hops(previous)` (TypeScript: `withBridgeHops`) continues an existing path. See [Interop](/laser-sdk/advanced/interop). ## Encoding and conformance Every client must keep these rules: * Encode wire payloads through one named-field CBOR encoder. * Encode exactly one CBOR item. Reject trailing bytes, corrupt known fields, and wrong known types. * Ignore unknown fields for additive changes. Keep unknown dictionary codes without rejecting the whole record. * Omit absent optional fields instead of writing empty placeholders. * Encode envelope IDs as 16-byte big-endian values. Header IDs ride as canonical Crockford strings, and readers also accept the older typed `Uint128` conversation header. * Apply the message-kind rules before publishing and after decoding. * Keep accepted bytes and rejected shapes from the fixtures. Decoding, validating, and encoding again must produce identical bytes. The [`laser-wire`](https://github.com/laserdata/laser-sdk/tree/main/wire) crate defines the contract and the [golden fixtures](https://github.com/laserdata/laser-sdk/tree/main/wire/fixtures). It contains no network transport runtime, no envelope signing or verification, no clock, and no ID minting. Schema fingerprinting and content hashing are part of the wire crate. All clients use the same envelope rules and scenarios. Source: https://docs.laserdata.com/laser-sdk/advanced/agdx --- # Introduction Start with [your first Free deployment](/getting-started/quick-start). The guide covers signup, readiness, credentials, and your first messages, with Docker and native SDK paths. **LaserData Cloud is generally available.** Sign up at [laserdata.cloud](https://laserdata.cloud) to create a Free tier deployment on AWS or GCP. No credit card is required. Free deployments use one node on shared infrastructure. Paid deployments use three nodes with replication between nodes. Read more at [laserdata.com](https://laserdata.com) or write to [hey@laserdata.com](mailto:hey@laserdata.com). LaserData Cloud deploys and operates [Apache Iggy](https://iggy.apache.org), a persistent message streaming platform written in Rust. It manages infrastructure, upgrades, networking, and security. Choose [Managed](/deployments#managed), [Bring Your Own Cloud (BYOC)](/deployments#byoc-bring-your-own-cloud), or [On-Premise](/deployments#on-premise) according to where your infrastructure must run. ## Before You Sign Up Sign in with GitHub, Google, or Microsoft. The Console does not offer a separate email-and-password account. The profile form requests your name, Title / Position, an organization name when creating a workspace, and acceptance of the terms. For personal use, enter Student or Hobbyist as your title and a name for your project as the organization. The first eligible organization can create one Free deployment without a credit card. It uses one isolated slot on a shared host. Read the [current Free limits](/deployments/tiers-storage#free) before choosing it. A Pro organization badge does not turn a Free deployment into a paid deployment. Provisioning runs in the background. Wait for the deployment to become `initialized`, then follow the [connection steps](/getting-started/quick-start#2-wait-for-the-deployment). Setup time varies with available capacity and node startup. A disabled Stream UI button does not prevent you from reading the guide. ## Why LaserData Cloud Iggy supports TCP, QUIC, HTTP, and WebSocket. Performance depends on hardware, message size, batching, replication, and network distance. Benchmark figures for Iggy are not Free-tier throughput or latency promises. LaserData Cloud adds the services that operate it: * Create infrastructure, install upgrades, manage TLS certificates, and monitor deployments. * Use Managed, BYOC, or On-Premise deployments through the same interface. * Use [Laser SDK](/laser-sdk) from Rust, Python, or TypeScript. Agents and services exchange messages, share state and memory, and keep a replayable record of every session, over one connection. * Manage resources through the [Web Console](https://laserdata.cloud) or [`laser`](/cli), the command-line tool. Its [TUI dashboard](/cli/tui) provides an interactive terminal interface. * Control access with outbound management connections, signed tasks, scoped credentials, and client access rules. * Organize resources into Organizations, Divisions, Environments, and Deployments. APIs call an organization a tenant. Roles assign permissions at each level. * Read telemetry dashboards, audit logs, and current health reports. * Choose Standard, Performance, or Enterprise. Pricing starts from $199, $999, and $2,999 per month on AWS. Compute and Storage form the configuration price, with data transfer billed separately for actual usage. * Protect data with TLS, disk encryption, and optional custom key encryption. Immutable audit logs, GDPR data export, and deletion support compliance requirements. ## How You Manage It The [Console](https://laserdata.cloud) and [`laser` CLI](/cli) use the same APIs, authentication, and permission model. Both manage deployments, connectors, networking, telemetry, audit logs, and organization resources. The CLI is a single static binary for macOS and Linux. Use commands such as `laser deployment list -o json` in scripts, CI, and agents. Run `laser tui` to open the interactive dashboard. Install the CLI: ```bash curl -fsSL https://cli.laserdata.cloud/install.sh | sh ``` The [REST API](/getting-started#api-architecture) exposes every deployment and organization operation. The CLI includes tier selection and pricing previews for paid deployments. ## Deployment Models | Model | Infrastructure | Data Location | Best For | | --------------------------------------------------------------------- | -------------------- | -------------------- | ---------------------------------------------------------------------------------- | | [Managed](/deployments#managed) | LaserData AWS/GCP | LaserData AWS/GCP | Fully managed, no infrastructure setup required | | [Bring Your Own Cloud (BYOC)](/deployments#byoc-bring-your-own-cloud) | Your AWS/GCP account | Your AWS/GCP account | Enterprises needing data sovereignty | | [On-Premise](/deployments#on-premise) | Your infrastructure | Your infrastructure | Regulated industries and private infrastructure with outbound control-plane access | All three models use the same Warden agent, Console UI, and APIs. Warden is the agent that manages each node. The model determines where the infrastructure and data reside. ## Architecture at a Glance | Component | Role | Communication | | ------------ | -------------------------------------- | ---------------------------------------------------- | | Console UI | Web interface for managing deployments | Talks to Platform API | | Platform API | Central control plane | Receives connections from Warden | | Warden Agent | Lightweight agent on each node | Outbound HTTPS only - pulls config, tasks, and certs | | Iggy Server | Message streaming platform | Managed by Warden on each node | Warden starts each management connection from the node to the control plane, the services that manage deployments. These connections use outbound HTTPS. The control plane does not use inbound connections, SSH, or SSM to manage your infrastructure. ## Quick Links Source: https://docs.laserdata.com --- # Platform Overview LaserData Cloud operates Apache Iggy deployments and the services around them. You manage deployments, teams, networking, and data access through the Console, CLI, or API. This page explains how those parts fit together. ## Apache Iggy [Apache Iggy](https://iggy.apache.org) is a persistent message streaming platform written in Rust. Throughput and latency depend on hardware, configuration, and workload. General benchmark figures do not describe the Free tier. It supports TCP, QUIC, HTTP, WebSocket, consumer groups, replication, and optional encryption of message payloads. A cluster is a group of servers that work together and replicate data between nodes. ## Why LaserData Cloud LaserData Cloud manages deployment, scaling, networking, security, monitoring, and connectors for Apache Iggy. You choose where deployments run and who can use them. The platform handles the tasks that keep them running. ### Complete Isolation by Default Paid deployments need [access rules](/networking/access-rules) before clients can connect. Managed Free deployments include a global rule that you can restrict or replace. Warden starts management connections outbound, while application messages travel directly between clients and deployment nodes. ### Enterprise-Grade Security The platform protects management operations and connections: * [Warden](/deployments/warden) starts all management connections outbound. Management needs no inbound ports, SSH, or remote access. * The platform signs binaries and checks their signatures before execution. It signs operational tasks with Ed25519. * The platform issues and rotates TLS certificates. Connections use encryption from end to end. * Managed upgrades use signed binaries, controlled restarts, and rollback after failure. ### Organization & Access Control The [resource hierarchy](/organization) is Organization > Division > Environment > Deployment. APIs call an organization a tenant. For a first Free deployment, use the existing division and the starter environment instead of creating more groups. [RBAC](/organization/roles-permissions), role-based access control, assigns permissions at each level. Four system roles are available: `admin`, `developer`, `viewer`, and `billing`. You can also create custom roles and apply division or environment overrides. Teams, invitations, and [API keys](/security/api-keys) are available through the API. ### Full API Coverage The API supports every operation available in the Console. The [main API](/getting-started#api-architecture) manages resources, and the [deployment API](/getting-started#api-architecture) operates deployments. Both use the same API keys and RBAC model. You can use them in CI/CD pipelines, Terraform providers, and custom integrations. ### Built-in Connectors [Connectors](/connectors) move data between Iggy and external systems. They use compiled Rust plugins for systems such as PostgreSQL, Elasticsearch, Apache Iceberg, and Quickwit. Activate a connector in the Console, map its streams, and configure its instances. ### Metrics and logs [Monitoring](/observability) provides metrics, health reports, logs, and immutable [audit trails](/observability/audit-compliance), records that cannot be changed. You can send logs and traces to your own OpenTelemetry-compatible endpoint. ## Key Features The platform includes these tools and deployment choices: * [Stream UI](/deployments/warden#stream-ui) browses streams, topics, messages, and consumer groups. Its [Managed data](/deployments/warden#managed-data) area provides projections, queries, key-value storage, and forks when the managed data plane is enabled. The browser connects directly to Warden on the node, so Iggy data does not pass through the LaserData backend. * [Versioned configuration](/deployments/configuration) records changes to Iggy and connector configuration. You can create, activate, and restore versions. - [Pricing](/organization/billing) separates Compute, Storage, and Network charges across three paid managed tiers. There are no per-partition charges. * Deploy to AWS or GCP, or use your own infrastructure through On-Premise. * Choose [Standard, Performance, or Enterprise](/deployments/tiers-storage), with adjustable storage and throughput. Free deployments are available for development. * Paid deployments use three-node clusters with replication. Enterprise also supports Multi AZ, deployment across availability zones. ## Deployment Models Each model runs Iggy and [Warden](/deployments/warden). The difference is who owns the infrastructure and where it runs. The management interface stays the same. | Model | Infrastructure | Best For | | ------------------------------------------------ | -------------------- | --------------------------------------------------------- | | [Managed](/deployments) | LaserData's cloud | Fully managed, no infrastructure setup required | | [Bring Your Own Cloud (BYOC)](/deployments/byoc) | Your AWS/GCP account | Data sovereignty, your cloud bill | | [On-Premise](/deployments/on-premise) | Your servers (any) | Private infrastructure with outbound control-plane access | ## Connectors The [Connector Catalog](/connectors/catalog) lists the available sources and sinks. A source brings data into Iggy. A sink sends data from Iggy to another system. ## Deployment Tiers & Storage Paid managed deployments use Standard, Performance, or Enterprise. The tier sets starting monthly pricing, included features, and telemetry retention. Standard starts from $199, Performance from $999, and Enterprise from $2,999 per month on AWS, plus measured data transfer. Standard supports Network Drive. Performance and Enterprise also support Local NVMe. Throughput is a sizing and transfer-estimation input. Free deployments use one node on shared infrastructure for development and testing. Read [Tiers & Storage](/deployments/tiers-storage) for the available configurations. [Billing & Pricing](/organization/billing) explains monthly pricing and measured transfer. Serverless, tiered storage, and a Kafka gateway are planned features. They are marked as coming soon and are not available yet. ## Networking & Connectivity Deployments receive a custom subdomain, such as `your-cluster.laserdata.cloud`, with automatic TLS. Connections use encryption from end to end. The networking features control which clients can reach a deployment. | Feature | What It Does | | ---------------------------------------- | -------------------------------------------------------------------------------------------- | | Custom subdomain | Unique endpoint per deployment for connection strings, with automatic TLS | | [Access Rules](/networking/access-rules) | Allow specific IPs/CIDRs to reach deployment endpoints, per-protocol | | [VPC Peering](/networking/vpc-peering) | Private network path between your VPC and the deployment | | [PrivateLink](/networking/private-link) | Expose the deployment as a VPC endpoint service | | Public IP | Public (static Elastic IP with subdomain) or Private (no public IP, private networking only) | Paid deployments need access rules before clients can connect. Managed Free deployments start with a global rule that you can restrict or replace. Free deployments use a configured 100 KB/s network limit. Shared hosts apply it to outgoing slot traffic. Paid deployments have no broker throughput limit. ## Security The security model covers management access, client access, and stored data: * Paid deployments remain inaccessible to clients until you create [access rules](/networking/access-rules). Managed Free deployments include a global rule. * [Warden](/deployments/warden) starts management connections outbound. The control plane does not use inbound ports, SSH, or remote access. * The platform signs binaries and checks their signatures before execution. * Every operational task uses an Ed25519 signature. * The platform issues and rotates TLS certificates. * Upgrades use signed binaries, controlled restarts, and rollback after failure. * Application data travels directly between clients and deployment nodes. [Stream UI](/deployments/warden#stream-ui) uses a short-lived signed session with Warden's HTTP proxy. [Access rules](/networking/access-rules) apply to the browser's IP address. * GDPR support includes encryption of personal information at rest, data export, and the right to erasure. See [Security Architecture](/security) for the full model. ## Observability [Monitoring](/observability) provides these records for each deployment: * Metrics for CPU, memory, disk I/O, messages, and client connections, grouped by node and runtime. * Heartbeats, periodic health reports from managed runtimes. * Centralized logs that you can search by node, runtime, level, and time range. * Log and trace delivery to an OpenTelemetry-compatible endpoint. * [Audit logs](/observability/audit-compliance) that record every operation that changes state. ## Plans A deployment tier determines deployment pricing. Your account plan sets organization limits and feature access. New organizations start on Pro. | Feature | Basic | Pro | Enterprise | | -------------------- | ------ | --------- | ---------- | | Members | 10 | 100 | 1,000 | | Divisions | 2 | 5 | 10 | | Environments | 3 | 20 | 100 | | Custom roles | 2 | 20 | 100 | | Audit log retention | 7 days | 30 days | 365 days | | BYOC | - | Available | Available | | On-Premise | - | - | Available | | Cluster (multi-node) | - | Available | Available | | Multi-AZ | - | Available | Available | | Private networking | - | Available | Available | | Cross-region DR | - | - | Available | Enterprise limits can be customized for a tenant. [Contact us](mailto:hey@laserdata.com) to arrange a custom plan. See [Billing & Pricing](/organization/billing) for deployment tiers and account features. ## API Architecture LaserData Cloud exposes two API layers. Both are available through the Console or through [API keys](/security/api-keys). The API for an operation depends on its scope. ### Main API - api.laserdata.cloud The main API manages tenants, divisions, environments, members, roles, API keys, notifications, deployment creation, and connector activation. It also manages billing and plans. ### Supervisor API (Regional) The Supervisor API operates deployments within one cloud provider and geographic area. It manages configuration, networking, monitoring, connectors, tasks, and backups. Deployments in the same cloud and area share an endpoint. | Area | Cloud | Supervisor URL | | ---- | ----- | ----------------------------------- | | US | AWS | `supervisor-aws-us.laserdata.cloud` | | EU | AWS | `supervisor-aws-eu.laserdata.cloud` | | AP | AWS | `supervisor-aws-ap.laserdata.cloud` | | US | GCP | `supervisor-gcp-us.laserdata.cloud` | | EU | GCP | `supervisor-gcp-eu.laserdata.cloud` | | AP | GCP | `supervisor-gcp-ap.laserdata.cloud` | ### How It Works A deployment response includes `supervisor_url`, the endpoint for its cloud and area. Send operational requests for that deployment to this URL. The Console selects it automatically. ```json { "id": 12345, "name": "prod-cluster", "cloud": "aws", "area": "us", "region": "us-west-1", "supervisor_url": "https://supervisor-aws-us.laserdata.cloud", ... } ``` Both APIs use `ld-api-key` authentication and the same [RBAC permission model](/organization/roles-permissions). ### OpenAPI Schemas Each service publishes an OpenAPI 3.1 specification with a browser. Core, Audit, and Notifier share the dropdown at api.laserdata.cloud/docs. Each regional supervisor provides its own `/docs` endpoint. See [API Reference](/api#openapi-schema) for the supervisor URLs. ### Error Responses Both APIs return errors in an `application/problem+json` response, as defined by RFC 7807. Responses with `4xx` and `5xx` status codes use this structure: ```json { "type": "about:blank", "title": "Invalid Email", "code": "invalid_email", "reason": "Invalid email address", "instance": "8f4a2b6c9d1e4f3a8b5c7d9e0f1a2b3c", "field": "email", "field_issues": [ { "code": "invalid_email", "reason": "malformed address", "path": "email" } ], "status": 400, "retryable": false } ``` | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | RFC 7807 type URI. `about:blank` when no type is registered | | `title` | Short human-readable title derived from `code` | | `code` | Stable machine-readable error code (e.g. `invalid_email`, `tenant_not_found`, `insufficient_permissions`) | | `reason` | Long-form human explanation | | `instance` | Mirrors the `ld-request` response header. Quote this in support tickets | | `field` | Single field name when the error is bound to one field (legacy, still emitted for back-compat) | | `field_issues` | Array of structured per-field issues for validation errors. Each entry has `code`, `reason`, and an optional dotted `path`. Omitted on non-validation errors | | `status` | Mirror of the HTTP status code | | `retryable` | `true` for retryable conditions (`408`, `425`, `429`, `500`, `502`, `503`, `504`), `false` otherwise | Validation failures return `400 Bad Request` with one or more `field_issues` entries. Use those entries to show errors beside the relevant fields. Requests that change state (`POST`, `PUT`, `PATCH`) accept an optional `idempotency-key` header, which identifies repeated requests for safe retries. See [API Reference](/api) for its behavior. ### Naming Rules Tenants, divisions, environments, deployments, channels, API keys, and similar resources share these naming rules: * Use 1-100 characters. * Start with an alphanumeric character. * Use alphanumeric characters, `-`, `_`, `.`, `:`, or spaces. Resources in the tenant hierarchy also accept `,`. * Avoid consecutive special characters. For example, `a--b` is rejected and `a-b` is accepted. Hierarchy resources allow spaces beside a single special character. * Do not use leading or trailing whitespace. The API rejects invalid names with `400` and describes the affected fields in `field_issues`. ## Console The Console manages deployments, connectors, networking, monitoring, configuration, members, roles, and audit logs. Use the pages below for each part of the platform. Source: https://docs.laserdata.com/getting-started --- # Quick Start Create a Free deployment, connect to it, and read back your own messages. Docker is one option. The [native SDK example](/laser-sdk/quickstart#2-install-the-sdk) works with Python, Rust, or TypeScript on your machine and does not require Docker. If your deployment already says `initialized`, continue with [network access](#3-limit-network-access) and [credentials](#4-choose-your-credentials). You do not need to find the Quick Start card in the Console to follow this guide. ## 1. Create Your Free Deployment 1. Open [laserdata.cloud](https://laserdata.cloud) and sign in with GitHub, Google, or Microsoft. 2. Complete the profile form and accept the terms. 3. Select your organization and its existing division. 4. Open the Free deployment creation form and review the cloud and region. 5. Create the deployment and open its page. Signup requires a name and Title / Position. A new workspace also needs an Organization Name. For personal use, Student or Hobbyist is a valid title, and the organization can use your project name. If your employer blocks every supported sign-in provider, [contact LaserData](mailto:hey@laserdata.com). There is no separate email-and-password signup. An organization contains divisions, which contain environments. The starter flow creates `sandbox` when no existing environment is selected. Use these initial groups for your first deployment. Read [Organization Hierarchy](/organization) when you need more groups or members. Free eligibility, resources, and region availability are described in [Tiers & Storage](/deployments/tiers-storage#free). A Pro organization badge controls organization allowances, not the deployment price. Creating another organization does not provide another Free deployment. ## 2. Wait for the Deployment Creation starts a background provisioning task. Keep the deployment page open or return to it later. Setup time varies with capacity and startup, so use the status and recent activity rather than a fixed countdown. `initialized` means that the setup workflow completed. It does not prove that your browser or application can reach the endpoint. Check Heartbeats for recent node reports, then continue with the connection steps below. Stream UI also requires permission to read this deployment. If its button remains disabled after initialization, read [When the Console Blocks You](#when-the-console-blocks-you). Logs and Metrics remain useful for identifying the failed step. ## 3. Limit Network Access A network rule controls which source addresses can reach the deployment. A CIDR range describes a group of IP addresses. For one IPv4 address, use a `/32` range. Managed Free deployments start with a global `0.0.0.0/0` rule. This permits connections from every IPv4 address and is not a privacy feature. Authentication is still required. Before sending application data: 1. Open the deployment Access Rules tab. 2. Add a rule for your current public IP or trusted network. 3. Enable TCP for this example and HTTP if you will use Stream UI. 4. Replace or remove the global rule. Paid deployments need an access rule before clients can connect. Stream UI connects from your browser directly to the node, so its rule must permit your browser IP. See [Access Rules](/networking/access-rules). ## 4. Choose Your Credentials Open Credentials in the deployment page and use Show or Copy only when needed. Copy the deployment domain and the Iggy credentials. A Cloud API key manages Cloud resources and cannot replace an Iggy password or Personal Access Token. The initial Iggy administrator is named `root`. This is an Iggy account, not your operating-system root account or your Cloud organization role. Use the username shown in Credentials rather than substituting `admin`. For an application, use a dedicated Iggy user with the required stream and topic permissions and issue its Personal Access Token. The token inherits that user's permissions. Do not include real passwords or tokens in screenshots, reports, or shared logs. See [Authentication](/security/authentication#cloud-sign-in-and-iggy-credentials). ## 5. Send and Read Messages Choose the client that fits your machine: | Client | Requirements | Next step | | ---------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- | | Python | Python 3.10 or later | Open the [SDK Quickstart](/laser-sdk/quickstart#2-install-the-sdk) and select Python | | TypeScript | Node.js 22.14 or later | Open the [SDK Quickstart](/laser-sdk/quickstart#2-install-the-sdk) and select TypeScript | | Rust | The Rust version listed in the SDK guide | Open the [SDK Quickstart](/laser-sdk/quickstart#2-install-the-sdk) and select Rust | | Docker | Docker installed locally | Continue with the commands below | For a Cloud deployment, use its connection details and skip the SDK guide's local-server setup. You do not need to install Iggy or Laser Stack on your machine. ### Pull the Docker Image ```bash docker pull ghcr.io/laserdata/quickstart:latest ``` ### Run the Producer Use your deployment domain and Iggy credentials: ```bash docker run ghcr.io/laserdata/quickstart \ /producer {DOMAIN} \ -u {USERNAME} \ -p {PASSWORD} ``` The producer sends messages to `sample-stream` and `sample-topic`. To use a Personal Access Token, replace both `-u` and `-p` arguments with `-t {TOKEN}`. Do not combine the two authentication methods. ### Run the Consumer Open another terminal with the same domain and authentication method: ```bash docker run ghcr.io/laserdata/quickstart \ /consumer {DOMAIN} \ -u {USERNAME} \ -p {PASSWORD} ``` The consumer prints the messages that it reads. This send-and-read result is the connection test. You can also inspect `sample-stream` and `sample-topic` in Stream UI when browser access is available. ## Understand the First Metrics A fresh deployment can contain platform streams, topics, and connected service clients. The health prober writes to `_ld/prober` every five seconds by default. Managed services also maintain their own records. These counts are not evidence that another customer is using your deployment. Use Include prober in Metrics to distinguish probe traffic where the option is available. Inspect the topic used by your example to identify your own messages. The total counts vary with enabled services. Read [Monitoring](/observability#understand-a-new-deployment) for health labels, system traffic, and shared-host metrics. ## When the Console Blocks You | Symptom | Next step | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Stream UI is disabled | Make sure that status is `initialized` and your role includes `deployment:read` in this environment | | Streams says Access denied | Read [deployment permissions](/organization/roles-permissions#access-denied-after-creating-a-deployment). A network rule cannot grant a missing Cloud permission | | Backends requests deployment configuration access | This page requires `deployment:config:manage`. You do not need to configure a backend to send your first messages | | The browser or client times out | Make sure that the rule permits your current source IP and the protocol you are using | | A client rejects credentials | Use Iggy credentials or an Iggy Personal Access Token, not your SSO login or Cloud API key | | Health Probe says Degraded while nodes look healthy | Compare the timestamps and meanings in [Monitoring](/observability#which-health-status-to-use) | If you created the organization and are its only administrator, a deployment permission denial is not an instruction to find another administrator. Reload the Console and sign in again. If it persists, [contact LaserData](mailto:hey@laserdata.com) with the deployment ID, environment, time and timezone, and correlation ID if available. Do not send credentials. ## Finish the Trial To remove the deployment, open its Overview page and find Delete Deployment near the bottom. This action requires `deployment:manage`. If the action is missing for the organization owner, use the permission guidance above. Delete only when you no longer need the deployment. Deletion permanently removes its resources and data. Protected mode adds an emailed code before deletion. It does not pause the server or block client traffic, and turning it off does not delete anything. See [resource protection](/organization#resource-protection) and the [delete API](/api/deployments#delete-a-deployment). ## Connection Options | Option | Description | | ----------------------------------- | -------------------------------------------------------- | | `-t, --token ` | Use a Personal Access Token instead of username/password | | `--transport ` | Transport protocol (default: `tcp`) | | `--no-tls` | Disable TLS (local development only) | | `--messages-count ` | Number of messages to send or receive | | `--stream ` | Custom stream name | | `--topic ` | Custom topic name | Run `/producer --help` or `/consumer --help` inside the container to list all flags. ## Quick Start via API To create a Free-tier deployment through the API, use your [API key](/security/api-keys): ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/deployments/starter \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "region": "us-west-1" }' ``` Supply `cloud` and `region`. You can also supply `environment_name`, `environment_id`, or `deployment_name`. See [Create a Starter Deployment](/deployments#create-a-starter-deployment) for the full request. Source: https://docs.laserdata.com/getting-started/quick-start --- # Organization Hierarchy An organization owns your deployments, members, roles, and billing. APIs and some Settings or audit labels call it a tenant. These names refer to the same resource. A `TENANT CREATED` audit event means that an organization was created. For a first deployment, use the division already created for your organization. A division groups projects or teams. An environment groups deployments, such as development and production. The starter flow creates an environment named `sandbox` when you do not select an existing one. You do not need a larger hierarchy to run one Free deployment. Default Tenant and Default Division select preferred resources for your account. A default that is not set does not mean that the corresponding resource is absent. ## Organization Plan and Deployment Tier New organizations start with the Pro organization plan. It controls member, role, division, and environment allowances. The Free deployment tier controls the resources and cost of that deployment. A Pro badge on the organization is therefore compatible with a Free deployment and no saved card. Settings also shows account limits. Tenants Limit is the number of organizations that the user can own. Memberships Limit is the number of organizations that the user can belong to. Neither value is a node count or the number of Free deployments available. Read [Tiers & Storage](/deployments/tiers-storage#free) for Free eligibility. ## Hierarchy | Level | Represents | Example | | -------------------------------- | ------------------------------------------ | -------------------------------- | | Organization (tenant in the API) | Your organization | Acme Corp | | Division | Business unit or team | Platform Engineering | | Environment | Deployment stage | Production, Staging, Development | | Deployment | A LaserData deployment (one or more nodes) | `prod-us-west-1` | ### Tenant A tenant represents an organization. It owns billing, membership, roles, permissions, and isolated audit logs. A user can belong to multiple tenants. ### Division A division groups environments for a business unit, team, or project. For example, separate Platform and Data Engineering into divisions. Permissions can apply to one division without granting access to the others. ### Environment An environment groups deployments within a division. Common uses include Development, Staging, and Production. Each environment contains one or more deployments, and permissions can apply to it individually. ### Deployment A deployment runs [Apache Iggy](https://iggy.apache.org) on one or more nodes. [Warden](/deployments/warden) manages those nodes. ## Resource Protection Protection adds a code requirement before a tenant, division, environment, or deployment can be deleted. Enable it with `protected: true` in an update request, or during deployment creation. To delete a protected resource: 1. [Request a resource code](#request-resource-code). 2. Retrieve the time-based one-time code from the tenant's registered email address. 3. Supply it as the `code` query parameter in the delete request. Any member with the relevant manage permission can enable protection. Only the tenant owner can change `protected: true` back to `false`. Use protection for production resources. Deletion is irreversible. Deleting a deployment destroys its nodes, data, streams, topics, messages, configuration, and telemetry. Delete contained deployments before deleting a division, environment, or tenant. Deleting a division also removes its environments. Deleting a tenant removes its divisions, environments, members, roles, and API keys. ## Member Management Manage tenant membership through these actions: * Invite users by email. They join when they accept the invitation. * Assign or change [roles](/organization/roles-permissions). * Remove members to revoke their access immediately. ## Programmatic Access Use [API keys](/security/api-keys) for CI/CD, automation, and integrations. Their roles and permissions follow the same RBAC model as interactive users. RBAC is role-based access control. ## Plan Limits | Resource | Basic | Pro | Enterprise | | ------------------------- | ----- | --- | ---------- | | Divisions | 2 | 5 | 10 | | Environments (total) | 3 | 20 | 100 | | Environments per division | 2 | 3 | 5 | | Members | 10 | 100 | 1000 | | Invitations | 10 | 100 | 1000 | | Custom roles | 2 | 20 | 100 | | API keys | 3 | 10 | 100 | ## API Reference ### Get Tenant Retrieve tenant details, including plan features, subscription, and limits: ```bash curl https://api.laserdata.cloud/tenants/{tenant_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "name": "Acme Corp", "protected": false, "created_at": "2025-01-10T08:00:00Z", "updated_at": "2025-06-15T12:30:00Z", "description": "Main production tenant", "email": "admin@acme.com", "features": { "invitations_limit": 100, "members_limit": 100, "roles_limit": 20, "divisions_limit": 5, "environments_limit": 20, "deployment_tiers": [ { "tier": "free", "limit": 1 }, { "tier": "small", "limit": 3 }, { "tier": "medium", "limit": 3 }, { "tier": "large", "limit": 2 }, { "tier": "xlarge", "limit": 1 }, { "tier": "2xlarge", "limit": 1 } ], "deployment_access_rules_limit": 10, "deployment_configs_limit": 5, "deployment_backups_limit": 3, "deployment_snapshots_limit": 5, "private_connections_limit": 3, "private_endpoints_limit": 1, "byoc_enabled": true, "cluster_enabled": true, "on_premise_enabled": false, "private_networking_enabled": true, "multi_az_enabled": true, "dedicated_enabled": false, "backup_enabled": true, "cross_region_dr_enabled": false, "audit_retention_days": 30, "api_keys_limit": 10, "cloud_accounts_limit": 5, "notification_channels_limit": 5, "notification_subscriptions_limit": 10, "custom_domains_limit": 1, "backup_regions_limit": 3, "nodes_per_deployment_limit": 5, "backup_retention_days": 30, "snapshot_retention_days": 14, "pending_invitations_limit": 100, "api_key_allowed_ips_limit": 20, "divisions_per_role_limit": 3, "environments_per_division_limit": 3, "concurrent_deployments_limit": 2, "audit_export_enabled": true, "advanced_connectors_enabled": true, "customer_managed_keys_enabled": false, "advanced_notification_channels_enabled": true }, "subscription": { "id": 1, "plan": "pro", "active": true, "created_at": "2025-01-10T08:00:00Z", "valid_from": "2025-01-10T08:00:00Z", "valid_to": "2026-01-10T08:00:00Z" }, "starter_available": true, "has_payment_method": true } ``` `features` reports current limits and enabled capabilities. `deployment_tiers` lists the allowed tiers and their maximum deployment counts. `cloud_accounts_limit` limits [cloud accounts](/organization/cloud-accounts) across the tenant. `notification_channels_limit` and `notification_subscriptions_limit` limit [notification](/observability/notifications) channels and subscriptions per channel. `starter_available` reports whether another Free tier deployment can be created. Enterprise features can be customized for a tenant. ### Update Tenant ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Acme Corp", "description": "Updated description", "email": "admin@acme.com", "protected": true }' ``` Include only the fields that you want to change. Set `protected` to `true` to enable [resource protection](#resource-protection). Only the owner can set it back to `false`. A successful request returns `204 No Content`. ### Get Tenant Structure Retrieve divisions, environments, and deployments in one request: ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/structure \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "name": "Acme Corp", "divisions": [ { "id": 1, "name": "Platform Engineering", "environments": [ { "id": 1, "name": "production", "deployments": [ { "id": 1, "name": "prod-cluster", "cloud": "aws", "region": "us-west-1", "variant": "managed", "tier": "large", "supervisor_url": "https://supervisor-aws-us.laserdata.cloud" } ] } ] } ] } ``` ### Get Tenant Summary ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/summary \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "total_divisions": 2, "total_environments": 5, "total_deployments": 8 } ``` ### Create a Division ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Platform Engineering", "description": "Core platform team", "email": "platform@acme.com" }' ``` ### List Divisions ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/divisions \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "name": "Platform Engineering", "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Create an Environment ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "production", "description": "Production environment" }' ``` ### List Environments ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "name": "production", "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Update a Division ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Platform Engineering", "description": "Updated description", "email": "platform@acme.com", "protected": true }' ``` All fields are optional. Only the tenant owner can set `protected` back to `false`. A successful request returns `204 No Content`. ### Delete a Division Delete every deployment in the division first. If the division is protected, [request a resource code](#request-resource-code) and supply it in the delete request: ```bash curl -X DELETE "https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}?code={protection_code}" \ -H "ld-api-key: YOUR_API_KEY" ``` An unprotected division does not need `code`. Deletion permanently removes the division and all its environments. It cannot be undone. ### Update an Environment ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "production", "description": "Updated description", "protected": true }' ``` All fields are optional. Only the tenant owner can set `protected` back to `false`. A successful request returns `204 No Content`. ### Delete an Environment Delete every deployment in the environment first. If the environment is protected, [request a resource code](#request-resource-code) and supply it in the delete request: ```bash curl -X DELETE "https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}?code={protection_code}" \ -H "ld-api-key: YOUR_API_KEY" ``` An unprotected environment does not need `code`. Deletion is irreversible. ### Delete a Tenant Only the owner can delete a tenant. First delete all deployments and settle all billing. If the tenant is protected, [request a resource code](#request-resource-code): ```bash curl -X DELETE "https://api.laserdata.cloud/tenants/{tenant_id}?code={protection_code}" \ -H "ld-api-key: YOUR_API_KEY" ``` An unprotected tenant does not need `code`. Deletion permanently destroys the tenant, divisions, environments, members, roles, API keys, and other tenant data. It cannot be undone. ### Request Resource Code Request a one-time code before deleting a protected resource. The platform sends it to the tenant's registered email address. ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/request_code \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "action": { "action_type": "delete_tenant", "payload": { "tenant_id": 1 } } }' ``` A successful request returns `204 No Content`. The following action types select the resource: | Action Type | Payload Fields | | -------------------- | ------------------------------------------------------------- | | `delete_tenant` | `tenant_id` | | `delete_division` | `tenant_id`, `division_id` | | `delete_environment` | `tenant_id`, `division_id`, `environment_id` | | `delete_deployment` | `tenant_id`, `division_id`, `environment_id`, `deployment_id` | Requests are rate limited. Wait before requesting another code for the same resource. ### Get Tenant Summary Use the deployment API, `{supervisor_url}`, for an aggregate summary of deployment resources across the tenant: ```bash curl {supervisor_url}/tenants/{tenant_id}/summary \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "healthy_deployments": 5, "unhealthy_deployments": 0, "healthy_nodes": 8, "unhealthy_nodes": 0, "total_connectors": 3, "total_streams": 12, "total_topics": 24, "total_partitions": 48, "total_segments": 192, "total_messages": 15000000, "total_consumer_groups": 6, "total_clients": 10 } ``` ### Get Division Summary Retrieve the same summary for one division: ```bash curl {supervisor_url}/tenants/{tenant_id}/divisions/{division_id}/summary \ -H "ld-api-key: YOUR_API_KEY" ``` The response uses the tenant summary format. ## Related pages Source: https://docs.laserdata.com/organization --- # Roles & Permissions Role-based access control (RBAC) determines what a member can read and manage. A role contains permissions and the resources to which they apply. Members with multiple roles receive their combined permissions. ## Permission Hierarchy Permissions follow the [organization hierarchy](/organization): Tenant, Division, and Environment. Each level has its own permissions. Scope determines which resources receive those permissions. The Senior Developer example grants deployment management in Staging and read-only access in Production. It also grants only read access in Data Engineering. One role expresses these differences through division and environment overrides. ## How Scoping Works A role defines defaults for the whole tenant and overrides for selected resources. An override replaces that role's default at its scope. Permissions from separate roles still combine. ### Global Scope Three fields provide tenant-wide defaults: * `tenant` grants access to tenant information, members, roles, billing, and configuration. * `division` supplies the default permissions for every division. * `environment` supplies the default permissions for every environment in every division. For example, grant `deployment:read` in global `environment` to permit reading across all environments. A role with full grants at all three levels can manage the whole tenant. ### Per-Division Scope A division override replaces the role's defaults for that division. Other divisions retain the global defaults. For example, grant full access in Platform Engineering and read-only access in Data Engineering. ### Per-Environment Scope (within a Division) A division override can contain defaults for its environments and overrides for individual environments. An overridden environment uses its specific grants. Other environments in that division use the division's environment defaults. For example, permit deployment management in Production and only reading in Staging. ### Scoping Examples | Scenario | How to Configure | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Full access to everything | Global scope - set all permissions at tenant, division, and environment level | | Full access to one division only | Per-division override with full permissions. No default division permissions | | Deploy to Production, view Staging | Per-division override with per-environment overrides: `deployment:manage` on Production, `deployment:read` on Staging | | Billing only, no infrastructure access | Tenant-level `billing:manage` and `subscription:manage`. No division or environment permissions | | Read-only across all divisions | Global scope with `read` permissions at every level | ## Tenant Permissions Tenant permissions apply across the tenant's divisions and environments: | Permission | Read | Manage | | ------------- | ---------------------------------------------------------- | ------------------------------------------------ | | info | View tenant information | Update tenant information | | audit | View audit logs | - | | settings | View tenant settings | Update tenant settings | | role | View roles | Create, update, and delete roles | | member | View members | Invite, update, and remove members | | subscription | View subscription plan | Change subscription plan | | billing | View billing and payment | Update billing and payment | | division | View divisions | Create, update, and delete divisions | | api\_key | View [API keys](/security/api-keys) | Create and delete API keys | | notifications | View [notification channels](/observability/notifications) | Create, update, and delete notification channels | ## Division Permissions Global division grants apply to every division. A division-specific grant applies only to the named division. | Permission | Read | Manage | | ------------- | ------------------------------------------------------------------- | --------------------------------------------------------- | | info | View division information | Update division information | | audit | View division audit logs | - | | settings | View division settings | Update division settings | | role | View division roles | Create, update, and delete division roles | | member | View division members | Invite, update, and remove division members | | environment | View environments | Create, update, and delete environments | | api\_key | View division API keys | Create and delete division-scoped API keys | | notifications | View division [notification channels](/observability/notifications) | Create, update, and delete division notification channels | ## Environment Permissions Environment grants control deployments and their services. They can apply globally, to every environment in one division, or to one environment. | Permission | Read | Manage | | ---------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------- | | info | View environment information | Update environment information | | deployment | View deployments | Create, update, and delete deployments | | deployment:config | View deployment [configuration](/deployments/configuration) | Modify configuration, create versions, activate | | deployment:access | View [access rules](/networking/access-rules) | Create, update, and delete access rules | | deployment:network | View [VPC peering](/networking/vpc-peering) and [PrivateLink](/networking/private-link) | Create and manage network connections | | deployment:task | View deployment tasks | Execute deployment tasks | | deployment:telemetry | View [monitoring](/observability) data | Configure telemetry retention | | deployment:credentials | View deployment [credentials](/api/deployments#get-deployment-credentials) | - | | deployment:connector | View [connectors](/connectors) | Manage connector instances and configurations | Every permission has read and manage variants. Manage includes read. ## Built-in System Roles Every tenant includes four system roles. They cannot be deleted. Copy and customize them through [Custom Roles](#custom-roles). | Role | Tenant Permissions | Division Permissions | Environment Permissions | | ----------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `admin` | All | All | All | | `developer` | `info:read`, `member:read`, `role:read`, `division:read`, `api_key:read`, `notifications:read` | `info:read`, `environment:read`, `api_key:read` | `deployment:manage`, `deployment:config:manage`, `deployment:network:manage`, `deployment:task:manage`, `deployment:connector:read` + `:manage`, `deployment:credentials:read`, `deployment:telemetry:read` | | `viewer` | `info:read`, `role:read`, `member:read`, `division:read` | `info:read`, `environment:read`, `role:read`, `member:read` (no `settings`, no `api_key`, no `notifications`) | `info:read`, `deployment:read` | | `billing` | `info:read`, `subscription:read`/`manage`, `billing:read`/`manage` | None | None | System roles populate global `environment` by default. When a division is created, the platform also creates their per-division environment grants. Members therefore retain access in the new division without manual changes. The tenant owner is the user who founded the tenant. Ownership is stored on the tenant, separately from roles. The owner can disable protection and delete the tenant regardless of their assigned role. ## Access Denied After Creating a Deployment Cloud permissions, Iggy users, and managed-data grants are separate. An Admin label in account Settings is not the same field as an organization role. Iggy credentials do not grant permission to a Console page. The relevant Cloud permissions belong to the environment that contains the deployment: | Console action | Permission | | ------------------------------------------ | ----------------------------- | | Open Streams or Stream UI | `deployment:read` | | Open the deployment Backends configuration | `deployment:config:manage` | | Read credentials | `deployment:credentials:read` | | Change network access rules | `deployment:access:manage` | | Delete the deployment | `deployment:manage` | The built-in organization `admin` role includes these permissions. A newly created organization's founder receives that role. In Roles, inspect the Environment permissions and any division or environment overrides. Permission labels shown as words in the Console map to the API names above. If you are the owner and only administrator, an access denial is not resolved by asking a nonexistent second administrator. Reload the Console and sign in again to obtain a fresh session. If the denial remains, [contact LaserData](mailto:hey@laserdata.com) with the organization, environment, deployment ID, timestamp, and correlation ID if available. Do not provide passwords, tokens, or API keys. An access rule controls network reachability and cannot fix a Cloud permission denial. A higher organization plan or paid deployment is not a remedy for an incorrect role assignment. ## Custom Roles ### From the Console 1. Open the tenant's Roles page. 2. Click Create Role. 3. Enter a role name. 4. Set tenant permissions, which always apply globally. 5. Set default division permissions. 6. Set default environment permissions. 7. If a division needs different grants, select it and set its division and environment defaults. 8. If an environment needs different grants, add its override within that division. 9. Save the role. ### Assigning Roles Assign roles in an invitation or after a member joins. Members can hold multiple roles. Changes apply on their next request. ## Plan Limits | Resource | Basic | Pro | Enterprise | | ------------ | ----- | --- | ---------- | | Custom roles | 2 | 20 | 100 | | Members | 10 | 100 | 1000 | | Invitations | 10 | 100 | 1000 | ## API Reference ### Get Role Details ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/roles/{role_id} \ -H "ld-api-key: YOUR_API_KEY" ``` The response contains the role's permissions and scope overrides. ### Get Role Members ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/roles/{role_id}/members?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ### Invite a Member ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/invitations \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "email": "user@example.com", "roles": [123] }' ``` ### List Members ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/members?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "email": "user@example.com", "name": "Jane Smith", "active": true, "roles": ["admin"], "created_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Update a Member ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/members/{member_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "active": true, "roles": [123, 456] }' ``` ### Remove a Member ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/members/{member_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ### Create a Custom Role ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/roles \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "developer", "permissions": { "tenant": ["info:read", "member:read", "division:read"], "division": ["environment:read", "environment:manage"], "divisions": { "1": { "permissions": ["environment:read"], "environment": ["deployment:read", "deployment:manage"], "environments": { "2": ["deployment:read", "deployment:manage", "deployment:telemetry:read"] } } } } }' ``` ### List Roles ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/roles?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "name": "admin", "kind": "system" }, { "id": 2, "name": "developer", "kind": "custom" } ], "page": 1, "total_results": 2, "total_pages": 1 } ``` ### Assign Members to a Role ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/roles/{role_id}/members/assign \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "members": [1, 2, 3] }' ``` ### Revoke Members from a Role ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/roles/{role_id}/members/revoke \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "members": [1] }' ``` ### Delete a Role ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/roles/{role_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ### List Invitations ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/invitations?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ### Delete an Invitation ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/invitations/{invitation_id} \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/organization/roles-permissions --- # Billing & Pricing Choose Free on shared infrastructure or a paid managed tier: Standard, Performance, or Enterprise. Paid deployments start with three dedicated nodes. Monthly pricing covers compute, storage, operations, and tier features. Data transfer is additional and billed on actual usage. ## Managed Tiers | Tier | Starts from on AWS / month | Starts from on GCP / month | Storage choices | Telemetry retention | Connectivity and availability | | ----------- | -------------------------- | -------------------------- | --------------------------- | ------------------- | -------------------------------------------------------------- | | Free | $0 | $0 | Shared storage | 7 days | One node on shared infrastructure | | Standard | $199 | $199 | Network Drive | 14 days | Single AZ | | Performance | $999 | $1,399 | Local NVMe or Network Drive | 30 days | Single AZ, VPC peering | | Enterprise | $2,999 | $2,999 | Local NVMe or Network Drive | 90 days | Multi AZ, VPC peering, PrivateLink and Private Service Connect | Starting prices use `us-east-1` on AWS and `us-central1` on GCP. They cover the default configuration and exclude data transfer. GCP Performance has different compute and storage defaults, so its starting price is higher. Region, capacity, and agreed terms can change the price. The [pricing page](https://laserdata.com/pricing) and [public pricing API](/api/billing#public-pricing) show current prices. Use an estimate for your selected configuration before you create a deployment. Enterprise requires account approval and direct contact. Business support, SLAs, and other custom services are agreed separately. The advertised price does not include those services. Paid tiers include encryption in transit and at rest, provisioning, upgrades, and monitoring. They have no broker throughput limit. Partitions are dynamic, with no upfront count or per-partition charge. ## What You Pay For | Component | Covers | | --------- | --------------------------------------------------------- | | Compute | Reserved node capacity, operations, and tier features | | Storage | Provisioned storage across the nodes | | Network | Measured data transfer, plus applicable Multi AZ transfer | Compute plus Storage is the monthly configuration price. It remains payable while the deployment is active, even without traffic. Network is zero when there is no billable transfer. Larger compute or storage can increase monthly pricing. A smaller configuration does not reduce the price below the tier's starting amount. The throughput input estimates traffic and helps select suitable compute. It does not reserve transfer or require payment for unused throughput. With the same compute and storage, changing throughput changes only the transfer estimate. If the workload requires larger compute, the additional resources increase monthly pricing. Estimates assume 30 days of steady traffic at the selected rate in each direction. For AWS Standard at 0.1 MB/s in and out, the estimate is $199 per month plus about $15.55 for transfer. The estimated total is $214.55. Billing uses actual transfer instead of the estimate. Same-region transfer uses graduated rates: each price applies only to the volume within its band. Cross-region and cross-cloud traffic use separate rates. Multi AZ adds a charge for measured cross-zone transfer. Telemetry retention is included in the tier. ## Partial Months and Changes Billing follows UTC calendar months. Proration means charging for the active part of a month. The platform calculates that time by the second, from activation until billing ends. If a tier or resource upgrade takes effect during the month, each configuration is charged for its active period. Failed upgrades do not activate the proposed pricing. Reports follow recorded resources and their effective times. A deployment keeps the pricing version and commercial terms recorded for it. Publishing new rates does not change those terms automatically. Agreed terms can override published rates. Ordinary upgrades do not lower the existing monthly configuration price. An estimate or preview is not an invoice. Invoices use active time, measured transfer, and applicable credits. ## BYOC and On-Premise BYOC platform pricing starts from $1,999 per month. You pay AWS or GCP directly for infrastructure. LaserData adds charges for provisioned vCPU-hours, the allocated processor capacity multiplied by its active time. The public catalog reports the monthly starting amount. The BYOC estimator accepts `vcpus_per_node` and includes provisioned capacity in its total. Arrange capacity and commercial terms through sales. On-Premise pricing requires an agreement with LaserData. See [BYOC](/deployments/byoc) and [On-Premise](/deployments/on-premise) for deployment requirements. ## Account Plans A managed tier applies to one deployment. An account plan controls organization limits and access to features. The [tenant response](/organization#get-tenant) reports the plan as `basic`, `pro`, `enterprise`, or `custom`. Plans limit members, divisions, environments, roles, audit retention, and deployment counts by compute size. They also control access to BYOC, private networking, Multi AZ, and dedicated infrastructure. New organizations start on Pro. Selecting Enterprise for a deployment does not change the account plan. Enterprise also requires an approved Enterprise or Custom account plan. Contact LaserData to arrange that access. ## Spend Limits A spend limit is a monitoring threshold in USD. The status is `ok` below 80 percent, `warning` from 80 percent, `critical` from 90 percent, and `exceeded` at 100 percent. No limit, or a nonpositive limit, reports `ok`. This threshold does not stop billing or shut down the deployment. Set it during creation or through [Update Spend Limit](/api/deployments#update-spend-limit). ## Usage Reports The tenant's Billing page contains deployment reports and invoices. Each deployment has a monthly report. Report details include `compute_cost`, `storage_cost`, `network_cost`, and `total_cost`. Managed reports also provide `monthly_base_charge` for the configuration and `network_usage_cost` for transfer. These describe charges before negotiated adjustments. The configuration amount includes prorated reserved resources and any billable additional capacity. Do not add these explanatory amounts to the report total again. Current-month reports show usage to date. Previous-month reports can change while provider data arrives and reconciles. Invoicing waits when reconciliation or settlement checks remain unresolved. Preserve monetary values as decimal strings in API integrations. ## Invoices Invoices group deployment charges for a billing period. Automatically generated numbers follow `INV-{MM}-{YYYY}-{tenant_id}`. Payment is due 14 days after generation. Customer APIs expose `approved`, `paid`, `failed`, `refunded`, and `partially_refunded` invoices. These invoices support PDF downloads. Draft, pending, and cancelled invoices are not visible through customer APIs. ## Payments and Credits Add a payment method through the Console's Stripe setup flow. Paid creation and upgrades require valid payment information for the billing account. A linked marketplace payment provider can meet this requirement. AWS Marketplace and Google Cloud Marketplace subscribers can link an agreement to a tenant. The marketplace handles billing for that agreement's scope. Direct payment charges and retries follow the configured payment policy. Redeem a promo code through `POST /tenants/{tenant_id}/billing/promo_codes`. A tenant can redeem each code once. Its credit reduces later invoices until used or expired. Credits are consumed oldest first. A credit never takes an invoice below zero: an invoice fully covered by credits is marked paid, the invoice shows the applied amount on a Credits line, and the unused balance carries to the next invoice. List balances through `GET /tenants/{tenant_id}/billing/credits`. Promo codes do not apply to marketplace-billed tenants. Existing credits do not change metered usage or the deployment report amounts. ## API Reference The [Billing API](/api/billing) provides rates, estimates, billing details, reports, invoices, payment methods, and credits. Public pricing endpoints do not require authentication. Tenant billing reads require `billing:read`, and changes require `billing:manage`. Source: https://docs.laserdata.com/organization/billing --- # Cloud Accounts Save cloud account IDs, VPC details, and credentials once for reuse across the platform. [VPC Peering](/networking/vpc-peering), [PrivateLink](/networking/private-link), and [BYOC](/deployments/byoc) use saved accounts to fill setup fields. ## Overview Each saved cloud account belongs to a tenant. The platform encrypts provider-specific credentials at rest. You can filter accounts by provider or region. ## Creating a Cloud Account ### From the Console 1. Open the tenant's Settings page. 2. Open Cloud Accounts and click Add Cloud Account. 3. Select the provider, such as AWS. 4. Enter a unique name of 1-100 characters and the cloud account ID. 5. If needed, select a default region. 6. Enter provider-specific credentials, such as an AWS IAM role ARN, in Settings. 7. Add optional remarks of at most 500 characters. 8. Click Save. ### Supported Cloud Providers | Provider | Value | Status | | -------- | ----- | --------- | | AWS | `aws` | Available | | GCP | `gcp` | Available | ### AWS Settings Supply the AWS configuration in this format: ```json { "aws": { "identity_arn": "arn:aws:iam::123456789012:role/LaserDataRole", "external_id": "unique-external-id", "vpc_id": "vpc-0abc123def456", "vpc_cidr": "10.0.0.0/16" } } ``` | Field | Required | Description | | -------------- | -------- | ---------------------------------------------------- | | `identity_arn` | Yes | IAM role ARN that LaserData assumes for provisioning | | `external_id` | Yes | External ID for secure cross-account role assumption | | `vpc_id` | Yes | VPC ID where infrastructure will be provisioned | | `vpc_cidr` | No | CIDR block of the VPC (used for network planning) | ### GCP Settings Supply the GCP configuration in this format: ```json { "gcp": { "vpc_network": "my-vpc-network", "vpc_cidr": "10.128.0.0/20" } } ``` | Field | Required | Description | | ------------- | -------- | ------------------------------------------------- | | `vpc_network` | No | VPC network name for infrastructure provisioning | | `vpc_cidr` | No | CIDR block of the VPC (used for network planning) | The database encrypts this configuration at rest. ## Account Status | Status | Description | | ---------- | ------------------------------------------------- | | `active` | Account is active and can be used for deployments | | `inactive` | Account is inactive | | `locked` | Account is locked | | `deleted` | Account has been deleted | ## Permissions Managing cloud accounts requires tenant-level `settings:manage`. Reading them requires `settings:read`. See [Roles & Permissions](/organization/roles-permissions). ## Plan Limits | Resource | Basic | Pro | Enterprise | | -------------- | ----- | --- | ---------- | | Cloud accounts | 1 | 5 | 10 | ## API Reference ### Create a Cloud Account ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/cloud_accounts \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "name": "production-aws", "account_id": "123456789012", "region": "us-west-1", "settings": { "aws": { "identity_arn": "arn:aws:iam::123456789012:role/LaserDataRole", "external_id": "unique-external-id", "vpc_id": "vpc-0abc123def456", "vpc_cidr": "10.0.0.0/16" } }, "remarks": "Main production AWS account" }' ``` | Field | Required | Description | | ------------ | -------- | -------------------------------------------------------------- | | `cloud` | Yes | Cloud provider: `aws`, `gcp` | | `name` | Yes | Unique name (1-100 chars) | | `account_id` | Yes | Cloud provider account ID (max 256 chars) | | `region` | No | Default region for this account | | `settings` | No | Cloud-specific credentials (see [AWS Settings](#aws-settings)) | | `remarks` | No | Notes (max 500 chars) | A successful request returns `201 Created`. ### List Cloud Accounts ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/cloud_accounts?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` Use these query parameters to filter the list: | Parameter | Type | Description | | --------- | ------- | ------------------------------------------- | | `page` | integer | Page number (optional) | | `results` | integer | Results per page (optional) | | `name` | string | Filter by name (contains match, optional) | | `cloud` | string | Filter by cloud provider (optional) | | `region` | string | Filter by region (contains match, optional) | The response lists the newest accounts first: ```json { "total_pages": 1, "total_results": 2, "page": 1, "items": [ { "id": 1, "cloud": "aws", "name": "production-aws", "account_id": "123456789012", "region": "us-west-1", "validated_at": null, "status": "active", "created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z" } ] } ``` ### Get Cloud Account Details ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/cloud_accounts/{cloud_account_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "cloud": "aws", "name": "production-aws", "account_id": "123456789012", "region": "us-west-1", "status": "active", "created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z", "settings": { "aws": { "identity_arn": "arn:aws:iam::123456789012:role/LaserDataRole", "external_id": "unique-external-id", "vpc_id": "vpc-0abc123def456", "vpc_cidr": "10.0.0.0/16" } }, "remarks": "Main production AWS account" } ``` Details include `settings` and `remarks`. List responses omit these fields. ### Update a Cloud Account ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/cloud_accounts/{cloud_account_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "production-aws-updated", "region": "us-west-2", "status": "active" }' ``` Include only fields that you want to change. To clear an optional field, set it to `null`. | Field | Type | Description | | ------------ | -------------- | --------------------------------------------------------- | | `name` | string | New name (must be unique within the tenant) | | `account_id` | string | Updated cloud account ID | | `region` | string or null | Updated region, or `null` to clear | | `settings` | object or null | Updated credentials, or `null` to clear | | `remarks` | string or null | Updated notes, or `null` to clear | | `status` | string | Account status: `active`, `inactive`, `locked`, `deleted` | A successful request returns `204 No Content`. ### Delete a Cloud Account Deletion permanently removes the saved cloud account and cannot be undone. ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/cloud_accounts/{cloud_account_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/organization/cloud-accounts --- # Deployment Models Choose Managed, Bring Your Own Cloud (BYOC), or On-Premise according to where the infrastructure must run and who owns it. Each uses [Warden](/deployments/warden), the software that manages a node, plus the same Console and APIs. For your first Free deployment, follow the [Quick Start](/getting-started/quick-start). ## How It Works Each node runs an [Iggy server](https://iggy.apache.org) and Warden, the agent that manages it. Warden retrieves tasks and configuration from the LaserData control plane, the services that manage deployments. Warden starts management connections outbound over HTTPS. Management does not require inbound connections, SSH, or cloud-specific agents. Application clients connect to the deployment endpoints. ## Managed LaserData creates and operates the infrastructure in its AWS or GCP accounts. Use this model when you want managed infrastructure without configuring a cloud account. You can create a deployment in the Console and connect within minutes. Managed deployments include these services: * Infrastructure, networking, TLS certificates, upgrades, and monitoring. * A custom subdomain such as `your-cluster.laserdata.cloud`, with automatic TLS, for public deployments. * [VPC Peering](/networking/vpc-peering) for private access from your AWS or GCP VPC, a private cloud network. * [PrivateLink](/networking/private-link) to expose an AWS deployment as a VPC endpoint service. * [Private Service Connect](/networking/private-service-connect) to expose a GCP deployment through a service attachment. * Load-balanced endpoints for public or private access, with encryption from end to end. ## BYOC (Bring Your Own Cloud) With BYOC, LaserData manages the deployment in your AWS or GCP account. Nodes, storage, networking, and application data remain in your account. You pay the cloud provider for those resources. The Console, monitoring, upgrades, and tasks work as they do for Managed deployments. LaserData uses a scoped AWS IAM role or impersonates a GCP service account to provision infrastructure. Deployments run on compute instances and do not require Kubernetes. Provisioning permissions cover compute, networking, and storage. They do not grant access to object storage, secret managers, or application data. Follow the [BYOC Setup Guide](/deployments/byoc) to configure access. ## On-Premise On-Premise runs Iggy on physical servers, private cloud infrastructure, or virtual machines that you control. LaserData manages tasks through Warden's outbound connection. Management requires outbound HTTPS on port 443. Iggy continues to run when the control plane is unavailable. Pending tasks run after the connection returns. The LaserData team provisions On-Premise deployments and supplies the installation details. [Contact us](mailto:hey@laserdata.com) to start, or read the [On-Premise Setup Guide](/deployments/on-premise). ## Comparison | | Managed | BYOC | On-Premise | | -------------------- | ----------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------- | | Infrastructure owner | LaserData | You (AWS/GCP) | You (any) | | Data location | LaserData AWS/GCP | Your cloud account | Your infrastructure | | Cloud bill | Included in the monthly configuration price and measured transfer | Your cloud account | Your infrastructure | | Provisioning | Automatic | Automatic (IAM role on AWS, service account on GCP) | LaserData team ([contact us](mailto:hey@laserdata.com)) | | Networking | VPC Peering, PrivateLink (AWS), PSC (GCP) | Direct VPC access | Your network | | Upgrades | Automatic | Automatic | Pull-based via Warden | | Console & APIs | Full access | Full access | Full access | | Kubernetes required | No | No | No | ## What You Get with Every Deployment The following services apply across the deployment models. Some network features depend on whether the deployment has a public IP. ### Custom Subdomain Public deployments receive a subdomain such as `your-cluster.laserdata.cloud` for connection strings. The platform manages it and its TLS certificates automatically. Private deployments do not receive a subdomain. Client connections use TLS. ### Built-in Stream UI The [Console](https://laserdata.cloud) includes [Stream UI](/deployments/warden#stream-ui) for streams, topics, partitions, messages, and consumer groups. Its [Managed data](/deployments/warden#managed-data) area provides projections, queries, key-value storage, and forks when the managed data plane is enabled. Stream UI runs in the browser and connects directly to Warden's HTTP proxy on the node. It uses a short-lived signed session token. Messages travel between the browser and node, without passing through the LaserData backend. The browser IP needs an [access rule](/networking/access-rules) with `iggy_http: true`. ### Data Isolation The control plane manages tasks, configuration, and certificates. Application data and messages stay on deployment nodes and travel directly to application clients. This separation applies to all three models. ### Encryption Connections use TLS to encrypt data in transit. Cloud providers encrypt NVMe SSDs at the hardware level, and network disks such as EBS and Persistent Disk always use encryption. You can also enable custom key encryption when you create a deployment. Iggy then encrypts message data with a per-deployment key before writing it to disk. This adds encryption above the cloud provider's disk encryption. ### Monitoring & Telemetry [Warden](/deployments/warden) collects [metrics, heartbeats, and logs](/observability) on each node and sends them to the control plane. Standard retains telemetry for 14 days, Performance for 30 days, and Enterprise for 90 days. You can send logs to your own OpenTelemetry-compatible endpoint. See [Monitoring](/observability) for configuration. ## Creating a Deployment Free provisioning and first-message instructions are in the [Quick Start](/getting-started/quick-start). It also explains disabled Stream UI controls, initial system traffic, and deletion after testing. Select Standard, Performance, or Enterprise, then choose a cloud, region, and compute size. Set storage, estimated throughput, availability, and network scope. Enterprise requires account approval. Its public pricing estimate leads to direct contact. Use the [deployment preview](/api/deployments#preview-deployment-cost) to see the monthly configuration price and estimated total before you provision resources. [Tiers & Storage](/deployments/tiers-storage) explains the available configurations. ### From the Console 1. Open the Environment in the Console. 2. Click Create Deployment. 3. Select Managed or BYOC. For On-Premise, [contact the LaserData team](/deployments/on-premise). 4. Enter the deployment configuration shown below. | Setting | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Name | Human-readable name for the deployment | | Cloud | `aws` or `gcp` | | Region | Cloud region, for example `us-west-1` or `europe-west1` | | Tier | Standard, Performance, or Enterprise. Sets starting monthly pricing, the included features, and the telemetry retention | | Compute | Per-node size, such as Small or Large. Each tier has a default size and supports larger sizes | | Storage | Network Drive with an adjustable size, or Local NVMe with a fixed size per Compute size. Local NVMe starts at Performance | | Throughput | Estimated symmetric traffic. Selects suitable compute and estimates data transfer, which is billed for actual usage | | Network scope | Same region, cross region, or cross cloud. Sets the rates for measured transfer | | Availability | Single AZ or Multi AZ. Multi AZ spreads the three nodes across zones and requires Enterprise | | Encryption | Custom key encryption of message data on top of the always-on disk encryption | | Protected | [Resource protection](/organization#resource-protection). Deleting a protected deployment requires a one-time code sent to the organization email | | Public IP | Public with a static IP and a custom subdomain, or private with access only through private networking | | Retention | Telemetry retention for metrics, heartbeats, and logs, up to the tier's entitlement | | Spend limit | Optional monthly spend monitoring threshold in USD | 5. Click Deploy. Provisioning usually takes a few minutes. The status moves through `creating`, `deploying_nodes`, and `waiting_for_nodes` to `initialized`. ### Free Free deployments use one node on shared infrastructure for development and testing. They cost nothing. The first eligible organization can create one. Creating another organization does not grant another Free deployment. Free deployments have these limits and defaults: * Throughput is limited to 100 KB/s. * The initial access rule permits all sources, `0.0.0.0/0`. You can restrict or replace it. * The public IP can change after a restart. Paid deployments use a static IP. * A custom subdomain provides the connection address. * Only Single AZ is available. Private connectivity and Local NVMe are unavailable. The platform marks a Free deployment inactive after 14 days without traffic and deletes it to release its slot. It sends warning emails on days 10 and 12. Send traffic to keep the deployment active. Use the starter endpoint below or `laser deployment create-starter` to create one. ### Public IP | Mode | Behavior | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Public | Static IP that persists across restarts, plus a custom subdomain with automated TLS | | Private | No public IP. Access only through [VPC Peering](/networking/vpc-peering), [PrivateLink](/networking/private-link), or [Private Service Connect](/networking/private-service-connect) | A subdomain requires a public IP. Private deployments have no subdomain and accept connections only through private networking. VPC peering starts at Performance. PrivateLink and Private Service Connect require Enterprise. ### Regions Regions belong to US, EU, or AP geographic areas. Each cloud provider has a [Supervisor API](/getting-started#api-architecture) for each area. That API operates the deployments in those regions. US | Cloud | Region | Location | | ----- | ------------- | -------------- | | AWS | `us-east-1` | N. Virginia | | AWS | `us-east-2` | Ohio | | AWS | `us-west-1` | N. California | | AWS | `us-west-2` | Oregon | | GCP | `us-central1` | Iowa | | GCP | `us-east1` | South Carolina | | GCP | `us-east4` | N. Virginia | | GCP | `us-west1` | Oregon | EU | Cloud | Region | Location | | ----- | -------------- | --------- | | AWS | `eu-central-1` | Frankfurt | | AWS | `eu-west-1` | Ireland | | AWS | `eu-west-2` | London | | GCP | `europe-west1` | Belgium | | GCP | `europe-west2` | London | | GCP | `europe-west3` | Frankfurt | AP | Cloud | Region | Location | | ----- | ----------------- | --------- | | AWS | `ap-south-1` | Mumbai | | AWS | `ap-southeast-1` | Singapore | | AWS | `ap-southeast-2` | Sydney | | AWS | `ap-northeast-1` | Tokyo | | GCP | `asia-south1` | Mumbai | | GCP | `asia-southeast1` | Singapore | | GCP | `asia-northeast1` | Tokyo | Use [List Available Clouds](#list-available-clouds) and [List Regions](#list-regions) to retrieve the current choices for your tenant. ### Upgrading a Deployment You can increase Compute, grow a Network Drive, or move to a higher tier after creation. A tier-only upgrade changes commercial terms and telemetry retention without changing hardware. Storage cannot shrink. Local NVMe deployments cannot change Compute or Storage through self-service upgrades. The `can_upgrade` field reports whether the upgrade cooldown permits a change. See [Tiers & Storage](/deployments/tiers-storage#upgrades). ### Coming Soon These features are planned and are not available: * Serverless will provide dynamic streams and automatic scaling without a reserved dedicated cluster, through a lower-cost model shared by tenants. * Tiered storage will move sealed segments to object storage such as S3 to retain data beyond local disk capacity. * A Kafka gateway will let applications connect with their existing Kafka SDK or client. ## Plan Limits The tenant plan limits deployment counts by Compute size and the number of saved configurations. It also controls BYOC, Multi AZ, dedicated infrastructure, and private networking. The [tenant response](/organization#get-tenant) reports feature access in `features`. Its `deployment_tiers` array lists permitted Compute sizes and their maximum deployment counts. New organizations start on Pro. Contact the LaserData team to arrange Enterprise or Custom limits. ## API Reference Use the main API, `api.laserdata.cloud`, for creation, upgrades, retention, and spend limits. Use the Supervisor API, `{supervisor_url}`, for access rules, configuration, connectors, metrics, and logs. See [API Architecture](/getting-started#api-architecture). ### List Available Clouds ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/clouds \ -H "ld-api-key: YOUR_API_KEY" ``` ### List Regions ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/clouds/{cloud}/regions \ -H "ld-api-key: YOUR_API_KEY" ``` ### List Available Compute Sizes ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/clouds/{cloud}/regions/{region}/tiers \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "key": "free", "name": "Free", "description": "Perfect for getting started. Great for development, testing, and learning the platform.", "available": true, "limit": 1, "clusters": ["standalone"], "storages": ["network_balanced"], "rate_limit": "100 KB/s" }, { "key": "large", "name": "Large", "description": "Sized for ~10 MB/s workloads. Built for demanding production applications with dedicated isolated nodes.", "available": true, "limit": 2, "clusters": ["cluster"], "storages": ["local_ssd", "network_balanced"], "rate_limit": null } ] ``` The response lists Compute sizes available to your account and the remaining deployment count for each. It also reports supported cluster kinds, storage types, and per-node resources. ### List Available Storage Types ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/clouds/{cloud}/regions/{region}/storages \ -H "ld-api-key: YOUR_API_KEY" ``` These discovery endpoints return the choices available to your account and region. Use them to populate deployment forms. ### Create a Managed Deployment ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/managed \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "prod-cluster", "cloud": "aws", "region": "us-west-1", "managed_tier": "standard", "tier": "small", "cluster": "cluster", "storage": { "type": "network_balanced", "size": 250 }, "target_network_tput": 1000, "network_scope": "same_region", "availability_mode": "single_az", "protected": true, "encrypted": true, "public_ip_enabled": true, "subdomain_enabled": true, "retention": { "telemetry_days": 14 }, "spend_limit": 500.00 }' ``` | Field | Required | Values / Description | | -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Deployment name | | `cloud` | Yes | `aws` or `gcp` | | `region` | Yes | Cloud region, for example `us-west-1` or `europe-west1` | | `managed_tier` | Paid | `standard`, `performance`, or `enterprise`. Required for every paid deployment and omitted for Free | | `tier` | Yes | Compute size: `free`, `small`, `medium`, `large`, `xlarge`, `2xlarge`, `4xlarge`, `8xlarge`, or `16xlarge` | | `cluster` | Yes | `cluster` for paid deployments. Free uses `standalone` | | `storage.type` | No | `network_balanced` for Network Drive or `local_ssd` for Local NVMe. Defaults to 100 GB of Network Drive per node | | `storage.size` | No | Network Drive size in GB per node, from 100 to 30,000. Local NVMe size follows the Compute size | | `target_network_tput` | No | Throughput estimate in KB/s. `1000` is 1 MB/s. Defaults to the tier's default throughput estimate | | `network_scope` | No | `same_region` (default), `cross_region_same_cloud`, or `cross_cloud` | | `availability_mode` | No | `single_az` (default) or `multi_az`. Multi AZ requires Enterprise | | `protected` | No | Enable resource protection. Default `false` | | `encrypted` | No | Enable custom key encryption of message data. Default `false` | | `public_ip_enabled` | No | Assign a static public IP. Default `true` | | `subdomain_enabled` | No | Assign a custom subdomain. Requires a public IP. Default `true` | | `dedicated` | No | Dedicated infrastructure isolation. Requires the Enterprise plan entitlement. Default `false` | | `retention.telemetry_days` | No | Telemetry retention in days. Defaults to the tier entitlement. Values above it are rejected | | `spend_limit` | No | Monthly spend monitoring threshold in USD | A successful request returns `202 Accepted`. The `ld-environment` and `ld-deployment` headers contain the new resource IDs. Paid deployments require valid payment information. ### Create a BYOC Deployment BYOC uses the same Compute and Storage fields, with an `aws` or `gcp` credentials object. Do not use the managed commercial fields `managed_tier`, `network_scope`, or `dedicated`. See the [BYOC Setup Guide](/deployments/byoc) for the request and setup procedure. ### Create a Starter Deployment Create a Free deployment for testing: ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/deployments/starter \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "region": "us-west-1" }' ``` | Field | Required | Description | | ------------------ | -------- | ---------------------------------------------------------------------------------------- | | `cloud` | Yes | `aws` or `gcp` | | `region` | Yes | Cloud region | | `environment_id` | No | Existing environment to deploy into | | `environment_name` | No | Name for a new environment. Defaults to `sandbox` when neither an ID nor a name is given | | `deployment_name` | No | Deployment name. Generated when omitted | A successful request returns `202 Accepted` with `ld-environment` and `ld-deployment` headers. ### List Deployments ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/deployments --- # Tiers & Storage Choose Free on shared infrastructure or a paid managed tier: Standard, Performance, or Enterprise. New paid deployments start with three dedicated nodes. Monthly pricing covers compute, storage, operations, and tier features. Data transfer is estimated separately and billed on actual usage. ## Managed Tiers | Tier | Starts from on AWS / month | Starts from on GCP / month | Default storage per node | Storage choices | Default availability | Telemetry | | ----------- | -------------------------- | -------------------------- | --------------------------------------- | --------------------------- | -------------------- | --------- | | Standard | $199 | $199 | 100 GB Network Drive | Network Drive only | Single AZ | 14 days | | Performance | $999 | $1,399 | 400 GB Local NVMe on AWS, 375 GB on GCP | Local NVMe or Network Drive | Single AZ | 30 days | | Enterprise | $2,999 | $2,999 | 900 GB Local NVMe on AWS, 750 GB on GCP | Local NVMe or Network Drive | Multi AZ | 90 days | Starting prices exclude transfer. They use `us-east-1` on AWS and `us-central1` on GCP. Catalog presets use 0.1 MB/s for Standard, 5 MB/s for Performance, and 10 MB/s for Enterprise. These inputs estimate traffic rather than reserve transfer. Larger compute or storage can increase monthly pricing. A smaller configuration cannot go below the tier's starting amount. * Standard supports Network Drive only. * Performance adds optional Local NVMe and VPC peering. Network Drive remains available. * Enterprise adds Multi AZ, AWS PrivateLink, and GCP Private Service Connect. It requires an approved Enterprise or Custom account plan and direct contact. Business support, SLAs, and extra commercial features are agreed separately. Every paid tier includes encryption in transit and at rest, provisioning, upgrades, and monitoring. Paid deployments have no broker throughput limit. See the [pricing page](https://laserdata.com/pricing), [public pricing API](/api/billing#public-pricing), and [billing guide](/organization/billing). The table describes catalog presets. If API creation omits storage, it uses 100 GB of Network Drive per node. If it omits availability, it uses Single AZ, including for Enterprise. Send the preset's compute, storage, and availability fields explicitly to match it. See [Create a Managed Deployment](/api/deployments#create-a-managed-deployment). ## Free Free runs one isolated deployment slot on a shared host. Use it to learn the platform, develop applications, or run tests. The slot is not a dedicated virtual machine. The first eligible organization can run one Free deployment. Creating another organization does not grant another Free deployment. The current default allocation is: | Resource | Free allocation | | ------------------------ | --------------------------------------- | | Deployments | One for the first eligible organization | | Nodes | One node on shared infrastructure | | Memory | 512 MiB for the slot | | Disk | 20 GiB quota on shared block storage | | Network rate-limit value | 100 KB/s, or 800 kbit/s | | Client connections | 100 open connections for the slot | | Partitions | 100 across all streams and topics | MiB and GiB are binary units. Console labels can use MB and GB for these values. Memory and disk figures describe the slot allowance, not the host total. The shared host applies the configured network ceiling to outgoing traffic. General Iggy benchmark figures, such as messages per second or sub-millisecond latency, describe particular test workloads. They are not guarantees for a Free slot. Network distance also affects latency. Its configuration includes these limits and defaults: * Throughput is limited to 100 KB/s. * A connection past 100 open client connections is refused, and creating partitions past 100 fails with `PartitionsLimitReached`. * Telemetry remains available for 7 days. * The public IP can change after a restart. The deployment still receives a custom subdomain. * A default access rule opens all Iggy protocols. You can restrict or replace it. * Local NVMe, Multi AZ, and private connectivity are unavailable. By default, the platform deletes a Free deployment after 14 days without traffic to release its slot. Warning emails arrive on days 10 and 12. ### Region Availability Choose from the regions offered for Free in the creation form. The list depends on the cloud and enabled regions, and allocation also needs free capacity in a shared host. A default selection such as `us-west1` does not mean that every account must use that region. If the nearest region is unavailable, contact LaserData before relying on a distant region for latency-sensitive work. ## Clusters New paid managed deployments start with three dedicated nodes. A quorum is the number of replicas needed to agree on a write. For three nodes, that number is two. The cluster can keep serving after one node fails. Single AZ puts the nodes in one availability zone. Multi AZ spreads the three-node configuration across three zones. See [Server & Durability](/deployments/server) for write guarantees. The public extension endpoint does not currently add nodes. Use supported compute and storage upgrades instead. ## Compute Compute sets the size of each node. AWS offers Small through 8XLarge with Network Drive and Large through 16XLarge with Local NVMe. GCP offers Small through 8XLarge with Network Drive and Large through 8XLarge with Local NVMe. Published profiles define the supported storage and throughput for each size. The deployment form and [tiers discovery endpoint](/api/deployments#discovery) list sizes available to your account in the selected cloud and region. They also provide throughput guidance for sizing. Partitions are dynamic, with no upfront count or per-partition charge. Storage needs depend on data volume, retention, and workload. Capacity metadata in the [deployment preview](/api/deployments#preview-deployment-cost) describes storage assumptions, not a guaranteed partition limit. ## Storage Network Drive and Local NVMe both encrypt data at rest. Storage is provisioned and billed on all three nodes. Replicas protect the data but do not increase the capacity for unique messages. Network Drive uses cloud block storage, such as EBS or Persistent Disk, rather than object storage. It supports 100 GB to 30,000 GB per node and can grow after creation. All paid managed tiers support it. Its API value is `network_balanced`. Local NVMe attaches storage directly to the node. Capacity follows the Compute size. AWS starts Performance at 400 GB per node and Enterprise at 900 GB. GCP starts them at 375 GB and 750 GB. Larger Compute sizes provide more capacity. Local NVMe is available from Performance. Network Drive remains available on both Performance and Enterprise. Its API value is `local_ssd`. Replication protects against node loss within the cluster's fault tolerance. [Backups](/deployments/backups) provide separate protection for supported storage types. Retention currently depends on provisioned disk capacity. Tiered object storage is planned, as described in [Coming Soon](#coming-soon). ## Network Throughput in MB/s is an input for estimating data transfer and selecting suitable compute. Self-service estimates range from 0.1 to 100 MB/s. Published compute and Local NVMe profiles impose additional sizing constraints. The input does not reserve transfer or impose a paid broker limit. Measured transfer is billed using the network scope and availability mode. At a fixed resource configuration, a larger throughput estimate changes the estimated transfer amount without changing monthly pricing. If it requires larger compute, the additional capacity changes that configuration price. Network scope describes where traffic travels: | Scope | API value | Meaning | | ----------------------- | ------------------------- | ------------------------------------------------------- | | Same region | `same_region` | Traffic stays in the selected cloud region | | Cross region | `cross_region_same_cloud` | Traffic crosses regions in the same cloud | | Cross cloud or internet | `cross_cloud` | Traffic crosses a cloud boundary or the public internet | Multi AZ adds a cross-zone transfer charge to the selected scope. ## Telemetry Retention Metrics, heartbeats, and logs share a retention period. Standard includes 14 days, Performance 30 days, and Enterprise 90 days. Free includes 7 days. You can select a shorter period. The platform rejects a period above the tier's allowance. A tier-only upgrade applies the new retention directly. If an upgrade changes resources, the new retention takes effect after successful completion. ## Upgrades You can increase resources without recreating the deployment: * Increase Compute to a larger size. * Grow a Network Drive. Storage cannot shrink. * Move to a higher tier, such as Standard to Performance or Performance to Enterprise. A tier-only upgrade changes commercial terms and telemetry retention without changing hardware. Moving from Network Drive to Local NVMe requires a planned migration with LaserData. Local NVMe deployments cannot change Compute or Storage through self-service upgrades. Upgrades require valid payment information. The `can_upgrade` field reports whether the cooldown permits an upgrade. Arrange tier downgrades with LaserData. See [Upgrade a Deployment](/api/deployments#upgrade-a-deployment) for the request. ## Coming Soon These features are planned and are not available: * Serverless will provide dynamic streams and automatic scaling without a reserved dedicated cluster, through a lower-cost model shared by tenants. * Tiered storage will move sealed segments to object storage such as S3. It will retain data beyond disk capacity, with Storage charges per GB-month. * A Kafka gateway will let applications connect with their existing Kafka SDK or client. Source: https://docs.laserdata.com/deployments/tiers-storage --- # Server & Durability Every LaserData deployment runs [Apache Iggy](https://iggy.apache.org), a persistent message streaming server written in Rust. Iggy uses a thread-per-core architecture with `io_uring`. Paid deployments, BYOC, and On-Premise use VSR (Viewstamped Replication Revisited) for consensus and replication. Shared-host Free deployments run a single node. This page explains write completion, topic durability, and server configuration. Durability defines what must survive after a successful write. You choose retention, segment size, and durability when you create a topic. The platform manages the server configuration. ## Understand the Version Badges A version ending in `-ld` identifies a LaserData build of the Iggy fork. The fork adds the managed AGDX extensions used by LaserData while retaining standard Iggy streaming behavior. Its artifact version identifies the deployed binary, and release notes identify the upstream revision it includes. Iggy Server, Connectors, and the client SDKs have separate version numbers. An older badge in a screenshot does not identify the current release. Do not assume that matching numbers across those components are required, or that an SDK supports every server change because a basic connection succeeds. ## Replication and Availability A cluster commits an operation when the required quorum accepts it. A quorum is the minimum required group of replicas. In a three-node cluster, both replication and view changes require two replicas. The cluster can keep serving after one replica fails and restore that replica from its peers when it returns. Single AZ places all three replicas in one availability zone. Enterprise also supports Multi AZ across three zones, so one zone failing does not remove the quorum. See [Tiers & Storage](/deployments/tiers-storage). Warden waits for a healthy roster of all nodes before it issues credentials. A node can start before it finishes recovery, partition repair, or replay of managed state. Read [node readiness](/api/observability#readiness) and replication metrics before sending production traffic. A heartbeat shows that the process is alive, not that it caught up. Clients use advertised addresses and follow leadership changes automatically. Warden configures those addresses and the roster. You do not edit them. TCP-TLS and secure WebSocket connections use the configured server shards for handshakes and encrypted traffic. A shard is a unit of work assigned to a core. Encrypted connections can therefore use the node's allocated cores. Heartbeat freshness follows the current view, a numbered leadership period. A new primary can start with a lower heartbeat counter. Replayed heartbeats from the same view cannot keep a failed primary marked live. ## Topic Durability Message durability and consumer-offset durability are independent. A consumer offset records a reader's position in a partition. Select both policies when you create a topic. | Option | Default | Controls | | ---------------------------- | ------------ | ------------------------------------------- | | `durability` | `replicated` | Message production | | `consumer_offset_durability` | `replicated` | Explicit consumer-offset stores and deletes | Both policies accept `replicated` or `persisted`. Both write data to disk. They differ in what must finish before the server reports success. | Policy | Required before success | | ------------ | ------------------------------------------------------------------------------------------------------------------ | | `replicated` | Quorum commit and local application, with no additional stable-storage barrier | | `persisted` | Quorum commit backed by recoverable copies on stable storage at the required quorum, followed by local application | With `replicated`, acknowledged messages can remain in a replica's in-memory journal until a flush trigger fires. Replication protects those messages while enough replicas retain their copies. A failure that destroys the memory copies on a quorum, such as shared power loss, can lose acknowledged writes. A Free deployment has one node, so its quorum of one provides no second copy. With `persisted`, acknowledged operations are recoverable while the required stable-storage copies remain intact and storage honors synchronization. Clustered partitions use a bounded on-disk prepare log during replication. Messages do not need to reach segment files before completion. This policy does not protect against retention, deletion, or loss of every durable copy. The two policies default independently to `replicated`. Selecting `persisted` for messages does not change the consumer-offset policy. To persist both messages and explicit offset changes, create the topic with both policies: ```bash iggy topic create my-stream my-topic 1 none \ --durability persisted \ --consumer-offset-durability persisted ``` These policies are create-only. `UpdateTopic` cannot change them on existing topics. The server rejects the removed `enforce_fsync` topic field. Persisted completion adds synchronization. Its latency and throughput depend on storage, batching, and concurrency. Local NVMe, available from Performance, offers the lowest read and write latency for persisted topics. ## Acknowledgements and Offset Commits Wait for the server to complete a write. Adding a message to a producer's local queue does not prove server completion. An awaited HTTP write returns `201 Created` with an `Iggy-Durability` header that names the topic policy. With `?ack=none`, HTTP returns `202 Accepted` and `Iggy-Durability: none` after dispatch, without waiting for commit or persistence. The value `ack=replicated` selects the awaited path, including for persisted topics. It does not weaken their policy. Poll auto-commit runs asynchronously, without waiting for offset completion. A successful poll does not prove that its offset committed or became durable, even with persisted consumer offsets. If processing needs an explicit completion boundary, process the message, then store the offset and wait for success. Delivery can repeat after failover, so downstream effects must tolerate repeated messages. A timed-out poll does not prove rejection. With auto-commit, the server can advance the offset before the response reaches the client. Retrying with `next` can then skip messages from that missing response. Keep a checkpoint for each partition, one past the last message processed in order. After a missing response, poll from that explicit offset. Advance the checkpoint only after processing the returned messages. Use each message's offset rather than the partition's current offset. ## Backpressure Each partition limits the produce and offset operations that wait for commit. If its prepare and request queues fill, the server returns `TransientNotAccepted`. This response proves that the request was not admitted, so it permits a retry without a duplicate write. The Rust SDK retries automatically. For HTTP produce and offset writes, the server retries this response against other cluster nodes until the retry deadline. It then returns the error if no attempt succeeds. `TransientNotCommitted` means that the operation can still commit. The server does not retry that response for you. ## Topic Storage Options | Option | Default | Purpose | | ----------------------------------- | --------- | ------------------------------------------------------------- | | `segment_size` | 1 GiB | Soft segment limit, from 1 MiB to 1 GiB in 512-byte multiples | | `messages_required_to_save` | 1,024 | Message-count trigger for ordinary segment writes | | `size_of_messages_required_to_save` | 1 MiB | Byte-count trigger for ordinary segment writes | | `preallocate_segments` | `false` | Reserve segment storage in advance | | `message_expiry` | Never | Age-based retention of sealed segments | | `max_topic_size` | Unlimited | Size-based retention of sealed segments | Flush thresholds control batching and I/O frequency. Lower thresholds do not give `replicated` the guarantees of `persisted`. Persisted completion can flush below ordinary thresholds, so `messages_required_to_save` does not need to be one. Retention removes sealed segments and leaves the active segment intact. Every partition keeps at least one segment on disk. Storage capacity and segment size therefore bound partition counts. The [deployment preview](/api/deployments#preview-deployment-cost) returns those bounds. After creation, only `compression_algorithm`, `message_expiry`, and `max_topic_size` can change. Use `GET /options/topic` or `iggy options topic` to list supported topic configuration and defaults for the running version. ## Server Configuration The server merges one TOML file over defaults embedded in the binary. Each key supports an `IGGY_` environment override. Main sections include `node`, `cluster`, `sharding`, `partition`, `metadata`, `http`, `tcp`, `quic`, `websocket`, `telemetry`, `logging`, `encryption`, and `data_maintenance`. Sections belong at the file's top level. Unknown keys and the removed `system` table prevent startup. Warden generates and applies this configuration on LaserData Cloud. You change permitted values through the deployment's versioned [configuration](/deployments/configuration). Its schema lists types, defaults, and rules. The platform owns listener addresses, TLS material, the cluster roster, shard placement, and secrets. These managed fields affect operation: * `partition.wal_bytes_max` limits the on-disk prepare log for a persisted partition. Its default is 256 MiB. Capacity pressure causes checkpointing and backpressure without weakening durability. * `partition.wal_group_commit_delay_micros` permits a wait for more prepares before one synchronization step applies to the group. The default is 0, with a maximum of 10,000 microseconds. It trades acknowledgement latency for fewer device writes without changing durability. An idle partition does not wait. * `message_bus.reconnect_period` defaults to `1 s` and also limits each outbound TCP dial between replicas. An unanswered dial cannot hold the reconnect sweep indefinitely. Stored overrides remain effective after a binary upgrade, including the connection deadline. * `message_bus.connections_max` caps the open client sockets of a node across TCP, WebSocket, and HTTP. QUIC connections do not count. Left unset, the server uses half of the process file descriptor limit, and `0` disables the cap. * `metadata.partitions_max` caps partitions across all streams and topics of a node. The default `0` means no cap. Past the cap, creating a topic or partitions fails with `PartitionsLimitReached`. * `consumer_group.session_timeout` drops a group member that stops reporting, `30 s` by default. `consumer_group.heartbeat_interval` sets how often active sessions are reported, `5 s` by default. The timeout must exceed three intervals plus 6.2 seconds, and both values must match on every node. * The HTTP metrics endpoint requires authentication. Warden reads it for platform [telemetry](/observability). ## On-Premise Notes On-Premise uses the same binaries and Warden configuration. Review host service limits, memory policy, CPU placement, and shutdown grace periods. If the grace period expires and the process is killed, shutdown did not finish. Only a completed graceful shutdown forces a final flush of committed `replicated` messages. Upgrades are forward-compatible. The current server writes an extended on-disk superblock, a record of durable server state. To return a node to an older release, wipe its data directory and let it recover from peers. The platform manages server upgrades for you. ## Coming Soon Tiered storage and a Kafka gateway are planned and are not available. Tiered storage will move sealed segments to object storage such as S3 for retention beyond disk capacity. A Kafka gateway will let applications connect with their existing Kafka SDK or client. Source: https://docs.laserdata.com/deployments/server --- # BYOC Setup BYOC (Bring Your Own Cloud) runs a LaserData-managed [Apache Iggy](https://iggy.apache.org) deployment in your AWS or GCP account. Your account owns the infrastructure and application data. You pay the cloud provider for its resources. ## Architecture For AWS, LaserData assumes an IAM role, permissions that a service can temporarily use. The role covers EC2, networking, and EBS provisioning. Once the nodes run, Warden starts management connections outbound, as it does for Managed and On-Premise deployments. Your application data stays in your AWS account. ## Prerequisites For AWS setup, prepare these resources: * An AWS account. * A VPC, a private cloud network, in the target region. The default VPC is sufficient. * BYOC access through a Pro or Enterprise plan. ## Step 1: Generate BYOC Setup Start a BYOC deployment in the Console and select the cloud and region. LaserData generates these items: * An IAM trust policy that permits supervisors in the LaserData AWS Organization to assume your role. It uses `aws:PrincipalOrgID` and `sts:ExternalId` to restrict access. * An IAM permissions policy for the resources that LaserData manages. * An external ID that prevents another customer from misusing the service's access to your account. * The LaserData AWS Organization ID, `laserdata_org_id`. Its format is `o-` followed by 10-32 alphanumeric characters. Trust applies to the organization. New supervisor regions receive access without a separate trust grant from you. ## Step 2: Create IAM Role In your AWS account: 1. Open IAM, then Roles, then Create role. 2. Select Custom trust policy. 3. Paste the trust policy from LaserData. 4. Create a policy with the supplied permissions policy. 5. Attach the policy to the role. 6. Name the role, for example `LaserDataByocRole`. 7. Copy the Role ARN, the AWS resource identifier. ### Using AWS CLI ```bash aws iam create-role \ --role-name LaserDataByocRole \ --assume-role-policy-document file://trust-policy.json aws iam put-role-policy \ --role-name LaserDataByocRole \ --policy-name LaserDataByocPermissions \ --policy-document file://permissions-policy.json ``` ## Step 3: Complete Deployment Enter your AWS Account ID, Role ARN, and external ID in the Console. The trust policy has this structure: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:PrincipalOrgID": "o-xxxxxxxxxx", "sts:ExternalId": "your-external-id" } } } ] } ``` LaserData then provisions the deployment: 1. It assumes the role through STS, the AWS service for temporary credentials. 2. It creates a subnet in your VPC and selects an available CIDR address range. 3. It creates security groups, route tables, and an internet gateway when required. 4. It starts EC2 instances with Elastic IPs. 5. It installs [Warden](/deployments/warden) on the nodes for outbound management connections. ## IAM Scope The role grants these permissions: | Category | Operations | | -------------- | ----------------------------------------------------------------------------------------- | | EC2 | Launch, terminate, start, stop, describe instances | | Networking | VPC, subnets, security groups, route tables, internet gateways, NAT gateways, elastic IPs | | EBS | Create, delete, attach volumes and snapshots | | Load Balancing | Create and manage NLBs and target groups | | IAM | Create `LaserNode-*` roles (for Cluster fencing only) | It excludes S3, Secrets Manager, CloudWatch, and SSM. LaserData receives no access to application data or secrets. ## Cleanup When you delete a BYOC deployment, LaserData removes its resources: 1. It terminates EC2 instances. 2. It releases Elastic IPs. 3. It deletes security groups, subnets, and route tables. 4. It removes IAM instance profiles and roles created for the deployment. Internet gateways remain because other resources can share them. ## GCP BYOC GCP BYOC uses a service account, an identity for software. LaserData impersonates that account to provision resources in your project. Your application data stays in the project. ### Prerequisites For GCP setup, prepare these resources: * A GCP project. * A VPC network in the target region. * BYOC access through a Pro or Enterprise plan. ### Setup 1. Start a BYOC deployment in the Console and select GCP. 2. Read the generated instructions for your project. 3. Create a service account with the IAM roles below. 4. Grant LaserData `roles/iam.serviceAccountTokenCreator` on that account. 5. Enter your Project ID, service account email, and VPC network name. LaserData uses the supplied account to provision the deployment in your project. ### IAM Roles Grant these roles to the service account on the project: | Role | Purpose | | -------------------------------- | ----------------------------------------------------- | | `roles/compute.instanceAdmin.v1` | VM management (create, start, stop, delete instances) | | `roles/compute.networkAdmin` | Networking (VPC, subnets, firewall rules, routes) | | `roles/compute.securityAdmin` | Firewall rules and SSL certificates | | `roles/iam.serviceAccountUser` | Attach service accounts to instances | | `roles/resourcemanager.tagAdmin` | Create and manage resource tags | | `roles/resourcemanager.tagUser` | Bind tags to resources | Grant LaserData the role needed to impersonate the account: | Role | Purpose | | -------------------------------------- | ---------------------------------------------------------------- | | `roles/iam.serviceAccountTokenCreator` | Allows LaserData to generate credentials for the service account | The permissions exclude Cloud Storage, Secret Manager, and Cloud Logging. LaserData receives no access to application data or secrets. ## API Reference ### Validate BYOC Credentials ```bash curl -X POST {supervisor_url}/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/byoc/validate \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "region": "us-west-1", "account_id": "123456789012", "identity_arn": "arn:aws:iam::123456789012:role/LaserDataByocRole", "external_id": "unique-external-id-123", "vpc_id": "vpc-12345678" }' ``` ### Generate BYOC Setup ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/byoc/setup \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "region": "us-west-1" }' ``` The AWS response includes `laserdata_org_id`, `external_id`, `permissions_policy`, and `trust_policy`. The organization ID uses `o-` followed by 10-32 alphanumeric characters. The trust policy uses `aws:PrincipalOrgID` and `sts:ExternalId` so supervisors in that organization can assume the role. The legacy field name `laserdata_account_id` remains an accepted alias. ### Create a BYOC Deployment ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/byoc \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "byoc-prod", "cloud": "aws", "tier": "large", "cluster": "cluster", "region": "us-west-2", "protected": true, "encrypted": true, "storage": { "type": "network_balanced", "size": 500 }, "availability_mode": "multi_az", "subdomain_enabled": true, "aws": { "account_id": "123456789012", "identity_arn": "arn:aws:iam::123456789012:role/LaserDataByocRole", "external_id": "your-external-id", "vpc_id": "vpc-0abc123def456", "vpc_cidr": "10.0.0.0/16" } }' ``` Use the Compute and Storage fields from [managed deployments](/deployments#create-a-managed-deployment), plus an `aws` credentials object. The fields `managed_tier`, `network_scope`, and `dedicated` do not apply. A successful request returns `202 Accepted` with `ld-environment` and `ld-deployment` headers. ### Create a GCP BYOC Deployment ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/byoc \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "byoc-gcp-prod", "cloud": "gcp", "tier": "large", "cluster": "cluster", "region": "europe-west1", "protected": false, "encrypted": false, "storage": { "type": "network_balanced", "size": 100 }, "availability_mode": "single_az", "public_ip_enabled": true, "subdomain_enabled": true, "gcp": { "project_id": "my-gcp-project-123", "service_account_email": "laserdata-byoc@my-gcp-project-123.iam.gserviceaccount.com", "vpc_name": "default" } }' ``` GCP returns the same response as AWS: `202 Accepted` with `ld-environment` and `ld-deployment` headers. Source: https://docs.laserdata.com/deployments/byoc --- # On-Premise Setup On-Premise runs [Apache Iggy](https://iggy.apache.org) on servers that you control. The LaserData control plane manages monitoring, configuration, and tasks. It uses the same management services as Managed and BYOC deployments. The LaserData team provisions On-Premise deployments. [Contact us](mailto:hey@laserdata.com) to discuss requirements and receive the deployment configuration and Warden installation instructions. ## How It Works Use physical servers, private cloud infrastructure, or virtual machines that you control. After LaserData provisions the deployment, each node runs Iggy and [Warden](/deployments/warden). Warden is the agent that manages the node and communicates with the control plane. ## Architecture Warden starts management connections outbound. Management requires no inbound firewall rule, but application clients need access to the deployment's listeners. Iggy keeps running if the control plane becomes unavailable. Pending tasks run after the connection returns. | Direction | What Flows | Protocol | | -------------------- | ---------------------------------------------- | -------------- | | Node → Control Plane | Heartbeats, metrics, task results | Outbound HTTPS | | Node ← Control Plane | Config, tasks, certificates (pulled by Warden) | Outbound HTTPS | | Control Plane → Node | Nothing - no inbound connections | N/A | ## Setup Flow ### 1. Contact LaserData Ask the LaserData team to create an On-Premise deployment. Provide these details: * The tenant, division, and environment. * Standalone for one node, or Cluster for multiple nodes with replication. * Hostnames, public and private IPs, and ports for each node. * Whether the deployment needs custom key encryption or deletion protection. ### 2. Deployment Provisioning LaserData creates the deployment, registers its nodes, and prepares TLS certificates and Warden credentials. The team supplies these installation materials: * A Warden installation script or manual instructions. * An API key and authentication token for each node's Warden agent. * TLS certificates signed by the deployment's CA, its certificate authority. ### 3. Install Warden On each node, use the installation details and credentials from LaserData: ```bash curl -fsSL https://artifacts.laserdata.com/scripts/install.sh | sudo bash -s -- \ --api \ --secret ``` The script installs dependencies, downloads binaries, and checks their signatures. It then configures services and starts Warden. For manual installation, follow the supplied instructions on each node: 1. Install the system dependencies. 2. Create the required directories. 3. Download binaries from the LaserData CDN and make sure that their signatures are valid. 4. Configure Warden with the node secret and control plane URL. 5. Start the Warden service. Warden then manages the Iggy lifecycle. ### 4. Verify After Warden starts, it registers with the control plane and retrieves configuration. Open the deployment in the Console to read its status. Nodes normally report healthy within a few minutes. ## Ongoing Operations Operational changes use the task system. Warden retrieves and runs each task on the node. Failed upgrades return to the previous version automatically. | Operation | How It Works | | --------------------- | -------------------------------------------------------------------------------------- | | Configuration changes | Update via Console or API - Warden pulls and applies automatically | | Upgrades | Trigger via Console or API - Warden downloads, verifies, and swaps binaries atomically | | Certificate rotation | Automatic - Warden pulls new certificates before expiry | | Monitoring | Warden pushes [heartbeats, metrics, and logs](/observability) to the control plane | ## Adding or Removing Nodes [Contact the LaserData team](mailto:hey@laserdata.com) to change the nodes in a deployment. The team updates the deployment configuration and supplies credentials and TLS certificates for new nodes. A deployment supports up to 100 nodes. ## System Requirements Each node needs these: * Ubuntu 22.04 or later. * Outbound HTTPS on port 443 to the LaserData control plane. * Root or sudo access for installation. Source: https://docs.laserdata.com/deployments/on-premise --- # Configuration Configuration controls the Iggy server and active [connectors](/connectors) in a deployment. Each saved change creates a version. You can activate a version or return to a compatible earlier version. The platform encrypts configuration data at rest. Each saved configuration records the runtime release used to validate it in `runtime_version`. After an upgrade, use this field to identify configurations that need review before reuse. ## Configuration Kinds | Kind | What It Configures | | -------------------- | --------------------------------------------------------------------------------------------- | | Iggy | Server limits, transport settings, runtime maintenance, and telemetry | | Connectors | Global connectors runtime settings | | Individual connector | Per-connector instance settings - stream/topic mappings, plugin configuration, batch settings | ## How It Works A deployment starts with configuration for its runtime and topology, the arrangement of its nodes. To change it: 1. Save new values under the existing configuration kind name. For a connector instance, use its instance key as the name. 2. Activate the version that you want to use. 3. Create a reconfigure task, as described in [Apply Configuration Changes](#apply-configuration-changes). 4. Wait for [Warden](/deployments/warden) to retrieve the task and apply the active version. 5. If the change fails, activate a compatible earlier version and reconfigure the deployment. Topic retention, segment size, and durability belong to topic configuration. See [Server & Durability](/deployments/server#server-configuration) for their relationship to server configuration. Warden, plane, and shared connector-runtime configuration require administrative access. ## Configuration Schemas A schema defines the available fields, types, defaults, and rules. Retrieve the schema before you create a configuration. It shows which values you can change. ### From the Console 1. Open your deployment's Configuration tab. 2. Select Iggy, Connectors, or a connector instance. 3. Read the form's current values, defaults, and descriptions. 4. Change the values that you need. 5. Click Save to create a version. 6. Click Activate to select that version for the deployment. ## Managing Configurations The Configuration tab provides these actions: * Read named configurations and their version history. * Find the primary configuration, the active version. * Compare two versions side by side. * Activate a saved version. * Delete configurations that you no longer need. Use `deployment:config:manage` to create, activate, or delete configuration. Use `deployment:config:read` to read it. ## API Reference ### Get Configuration Schema This endpoint returns the fields, types, and defaults for a configuration kind. It groups entries into sections with names and descriptions. ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy/schema \ -H "ld-api-key: YOUR_API_KEY" ``` An entry from the TCP section looks like this: ```json { "tcp": { "name": "TCP", "description": "TCP listener configuration.", "schema": [ { "key": "IGGY_TCP_ENABLED", "name": "TCP server enabled", "description": "Determines if the TCP server is active.", "default_value": true, "kind": "bool", "editable": true, "secret": false, "requirements": [], "rules": [] } ] } } ``` Section keys use snake\_case. Examples include `tcp`, `http`, `quic`, `websocket`, `partition`, `sharding`, `metadata`, `encryption`, and `data_maintenance`. | Field | Description | | --------------- | -------------------------------------------------------------------------------------- | | `key` | Configuration key (e.g. `IGGY_TCP_ENABLED`) | | `name` | Human-readable name | | `description` | What the setting controls | | `default_value` | Default value | | `kind` | Field type - `string`, `bool`, `int`, `float`, `size`, `duration`, `path`, `url`, etc. | | `editable` | Whether the value can be changed | | `secret` | Whether the value is masked in responses | | `requirements` | Dependencies on other config values | | `rules` | Validation rules (min, max, etc.) | Connector schemas also include `sink`, `source`, and `plugin_config` sections. Configuration responses include `runtime_version`, a SemVer release number. After an Iggy, Warden, or Connectors upgrade, compare it with the installed release before you reuse the configuration. ### Create a Configuration ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/configs/iggy \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "iggy", "values": { "IGGY_DATA_MAINTENANCE_MESSAGES_INTERVAL": "1m" }, "activate": false }' ``` A successful request returns `201 Created`. The `ld-config` header contains the new configuration ID. Set `"activate": true` to activate it at creation. ### Get Active Configuration ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy/primary \ -H "ld-api-key: YOUR_API_KEY" ``` ### List All Configurations ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get a Specific Configuration ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy/{config_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Version History ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy/{config_name}/versions \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get a Specific Version ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/iggy/{config_name}/versions/{version} \ -H "ld-api-key: YOUR_API_KEY" ``` ### Activate a Version ```bash curl -X PUT {supervisor_url}/deployments/{deployment_id}/configs/iggy/{config_name}/activate/{version} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. ### Delete a Configuration ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/configs/iggy/{config_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. ### Apply Configuration Changes After you activate a version, create a reconfigure task for the deployment: ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/tasks \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "type": "iggy:reconfigure" }' ``` Warden on each node retrieves the task and applies the active configuration. For connectors, use `connectors:reconfigure`. These tasks require `deployment:task:manage`. ### List Tasks ```bash curl {supervisor_url}/deployments/{deployment_id}/tasks \ -H "ld-api-key: YOUR_API_KEY" ``` The response includes task types and statuses for the deployment. Use it to follow reconfiguration, upgrades, and other operations. Source: https://docs.laserdata.com/deployments/configuration --- # Backups Backups are experimental. Endpoints, request and response formats, and behavior can change during development. A backup captures deployment storage at a point in time. Use backups to protect data or save the current state before an upgrade. LaserData creates EBS snapshots of the selected storage volumes. Backups require an AWS deployment with network storage and a Pro or Enterprise plan. NVMe SSD deployments do not support backups. ## How It Works Select a node to back up its storage volumes, or omit the node to back up all nodes. The backup runs in the background. Its status moves from Pending to In Progress to Completed. ## Backup Types | Type | API Value | Description | | ----------- | ------------- | ------------------------------------------------ | | Manual | `manual` | Created on demand from the Console or API | | Scheduled | `scheduled` | Created automatically on a recurring schedule | | Pre-upgrade | `pre_upgrade` | Automatically created before deployment upgrades | ## Backup Status | Status | Meaning | | ----------- | ---------------------------------- | | Pending | Backup requested, waiting to start | | In Progress | Snapshot creation in progress | | Completed | Backup finished successfully | | Failed | Backup encountered an error | | Deleting | Backup is being removed | Only one backup can run for a deployment at a time. ## Creating a Backup ### From the Console 1. Open your deployment in the Console. 2. Open the Backups tab. 3. Click Create Backup. 4. Enter a name. 5. If you want automatic expiry, enter the number of days. 6. Click Create. ## Managing Backups The Backups tab lists each backup's status, size, and creation time. Delete unused backups to release snapshot storage. Reading backups requires `deployment:read`. Creating or deleting them requires `deployment:manage`. Wait for a backup to finish before you delete it. You cannot delete a backup with pending or in-progress status. ## Plan Limits | Resource | Basic | Pro | Enterprise | | ---------------------- | ------ | ------- | ---------- | | Backups per deployment | - | 3 | 5 | | Backup retention | 7 days | 30 days | 365 days | | Backup regions | 1 | 3 | 10 | ## API Reference ### List Backups ```bash curl {supervisor_url}/deployments/{deployment_id}/backups \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "deployment_id": 611298765432109056, "node_id": 1, "name": "pre-upgrade-backup", "backup_type": "manual", "status": "completed", "snapshot_ids": ["snap-0abc123def456789a"], "volume_ids": ["vol-0abc123def456789a"], "size_bytes": 53687091200, "region": "us-west-1", "remarks": null, "started_at": "2025-01-15T10:30:00Z", "completed_at": "2025-01-15T10:45:00Z", "expires_at": "2025-02-14T10:30:00Z", "created_at": "2025-01-15T10:30:00Z" } ] ``` ### Create a Backup ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/backups \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "before-migration", "remarks": "Snapshot before schema migration", "expires_in_days": 30 }' ``` | Field | Required | Description | | ----------------- | -------- | ----------------------------------------------- | | `name` | Yes | Name for the backup | | `node_id` | No | Specific node to back up (all nodes if omitted) | | `backup_type` | No | `manual` (default), `scheduled`, `pre_upgrade` | | `remarks` | No | Optional description | | `expires_in_days` | No | Auto-delete after this many days | A successful request returns `204 No Content`. ### Restore a Backup A restore replaces the deployment's storage volumes with the backup snapshots. The deployment stops during the restore and restarts automatically. Only one restore can run for a deployment at a time. Before you restore, make sure that you can discard data written after the backup. The restore replaces the current storage volumes and loses that data. You can create a new backup first. ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/backups/{backup_id}/restore \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. ### Delete a Backup ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/backups/{backup_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/deployments/backups --- # Snapshots A snapshot records diagnostic information from a deployment node. [Warden](/deployments/warden) collects 30 or more categories, including system state, runtime health, certificates, and network configuration. It packages the results in a self-contained HTML report that you can search and share.
Warden Snapshot report - dark theme Warden Snapshot report - light theme
## What Is Captured Warden runs diagnostic commands in parallel and limits how long each command can run. It removes secrets from the output. The HTML report groups the results into these categories. ### System & Hardware | Category | What's Collected | | ------------------ | ---------------------------------------------------------------------------------- | | System Information | OS version, kernel, hostname, cloud metadata (AWS/GCP) | | CPU | Per-core usage, load average, CPU count, architecture | | Memory | Usage breakdown, huge pages, NUMA, top consumers, OOM activity | | Disk & Storage | Filesystem usage, inodes, NVMe health, DRBD status, I/O stats | | Network | Interfaces, routes, listening ports, iptables, traffic shaping, TCP states | | Processes | Top consumers by CPU/memory, LaserData process details (PID, threads, FDs, uptime) | ### Runtimes & Services | Category | What's Collected | | ---------------------- | ------------------------------------------------------------------------------------- | | Installed Versions | Warden, Iggy, Connectors, Prober binary versions and capabilities | | Runtime Telemetry | Per-runtime heartbeats, CPU, memory, uptime, connector sources/sinks | | Iggy Server Health | Authenticated stats - messages, streams, topics, clients, partitions, consumer groups | | Systemd Services | Service states, failed units, restart history (24h), timers | | Warden Process Metrics | CPU/memory/disk/network I/O, file descriptors, thread count | ### Security & Certificates | Category | What's Collected | | ------------------ | ------------------------------------------------------------------------------------ | | Certificates & TLS | Full chain validation, expiry warnings, CA trust, Let's Encrypt status, ACME renewal | | Credentials Status | File permissions, ages, PAT freshness (values redacted) | | Security Context | AppArmor/SELinux, ASLR, entropy, kernel hardening | ### Kernel & Stability | Category | What's Collected | | ---------------- | ------------------------------------------------------- | | Sysctl & Tuning | TCP buffer sizes, memory settings, mimalloc overrides | | Kernel Modules | Loaded modules, interrupts, softirqs, ECC memory errors | | System Stability | Reboot history, vmstat, swap activity, zombie detection | | Coredump History | Crash records (last 30 days) with binary identification | ### Logs & Connectivity | Category | What's Collected | | --------------------- | -------------------------------------------------------------------------------- | | Service Logs | Last 2000 lines per runtime (Warden, Iggy, Connectors, Prober) + errors filtered | | System Logs | Kernel errors, dmesg, boot log errors, journal storage | | Outbound Connectivity | ACME endpoint, port 80/443 reachability | | Time Synchronization | NTP/chrony status, time drift | ### Interactive Report The report includes these tools: * A dashboard with CPU, memory, disk, I/O throughput, and Iggy statistics. * Search across sections, with matching text highlighted. * Dark and light themes, selected in the header. * Controls to expand or collapse sections and find errors. * A copy button for each entry. * A layout for printing. ## How It Works A snapshot follows this sequence: 1. You request it through the Console or API. 2. The Supervisor sends a snapshot task to each node. 3. Warden runs diagnostic commands in parallel, with a 30s timeout for each command. 4. Warden removes secrets from the output. 5. Warden creates the HTML report. 6. If requested, it includes an Iggy snapshot of the data directory as a ZIP. 7. It uploads the report to encrypted cloud storage. 8. You download the ZIP with the report and optional Iggy snapshot. ## Creating a Snapshot ### From the Console 1. Open your deployment in the Console. 2. Open the Snapshots tab. 3. Click Create Snapshot. The status changes from Processing to Completed after all nodes finish. ### Options | Option | Default | Description | | ---------------- | ------- | ------------------------------------------------------------- | | `redact_secrets` | `true` | Mask sensitive values (passwords, keys, tokens) in the report | | `include_iggy` | `true` | Include Iggy server data snapshot as a separate ZIP | ## Snapshot Status | Status | Meaning | | ---------- | --------------------------------------------------------- | | Processing | Snapshot tasks dispatched, waiting for nodes to complete | | Completed | All nodes have uploaded their reports - ready to download | Only one snapshot can run for a deployment at a time. ## Plan Limits | Resource | Basic | Pro | Enterprise | | ------------------------ | ------ | ------- | ---------- | | Snapshots per deployment | 3 | 5 | 20 | | Snapshot retention | 7 days | 14 days | 90 days | Reading and downloading snapshots require `deployment:read`. Creating or deleting them requires `deployment:manage`. ## API Reference ### Create a Snapshot ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/snapshots \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "redact_secrets": true, "include_iggy": true }' ``` Both fields are optional and default to `true`. A successful request returns `204 No Content`. ### List Snapshots ```bash curl "{supervisor_url}/deployments/{deployment_id}/snapshots?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "tenant_id": 1, "division_id": 1, "environment_id": 1, "deployment_id": 1, "node_id": 610809900976570889, "deployment_name": "my-cluster", "name": "snapshot-20260601-103000", "status": "completed", "created_at": "2026-06-01T10:30:00Z", "completed_at": "2026-06-01T10:31:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Download a Snapshot ```bash curl {supervisor_url}/deployments/{deployment_id}/snapshots/{snapshot_id}/download \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "url": "https://storage.example.com/snapshots/...?signature=..." } ``` The response contains a presigned URL, a temporary authorized download link, for the snapshot ZIP. ### Delete a Snapshot ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/snapshots/{snapshot_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/deployments/snapshots --- # Activities Activities record actions on deployment nodes, such as start, stop, restart, and reconfigure. Each node reports its own status. The record shows the action, its time, and the person who requested it. ## Runtime Actions You can request these actions: | Action | Description | | ----------- | ------------------------------------ | | start | Start a stopped runtime | | stop | Stop a running runtime | | restart | Restart a runtime | | reconfigure | Apply a specific configuration by ID | ### Supported Runtimes | Runtime | Supported Actions | | ------------ | --------------------------------- | | `iggy` | start, stop, restart, reconfigure | | `connectors` | start, stop, restart, reconfigure | ## Activity Status | Status | Meaning | | ---------- | ----------------------------------------------- | | processing | Action dispatched, waiting for node to complete | | completed | Action finished successfully | | rejected | Action failed or was rejected | ## How It Works An action follows this sequence: 1. You request the action through the Console or API. 2. The Supervisor sends a task to each target node. 3. Warden on each node runs the task. 4. The node reports completion or rejection, and the activity status changes. You can target selected nodes or the entire deployment. Each target receives a separate activity entry. ## API Reference ### Execute a Runtime Action ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/activities/{runtime}/{action} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{}' ``` To select the target nodes, include their IDs: ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/activities/iggy/restart \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "nodes": [610809900976570889] }' ``` Set `runtime` to `iggy` or `connectors`. Set `action` to `start`, `stop`, `restart`, or `reconfigure:{config_id}`. The optional `nodes` array selects nodes. Omit the body or send `{}` to target all nodes. The endpoint requires `deployment:manage` and returns `202 Accepted`. ### List Activities ```bash curl "{supervisor_url}/deployments/{deployment_id}/activities?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "tenant_id": 1, "division_id": 1, "environment_id": 1, "deployment_id": 1, "author_id": 608123456789012345, "runtime": "iggy", "action": "restart", "node_id": 610809900976570889, "status": "completed", "created_at": "2026-03-16T12:00:00Z", "finished_at": "2026-03-16T12:00:05Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` This endpoint requires `deployment:read`. Source: https://docs.laserdata.com/deployments/activities --- # Warden Agent Warden runs on every deployment node. It manages local runtimes and exchanges tasks, configuration, and telemetry with the LaserData control plane. A runtime is a managed process, such as Iggy or Connectors. ## Why Pull-Based Warden retrieves tasks instead of accepting inbound management connections. It starts outbound HTTPS connections on port 443. The control plane does not need inbound ports, SSH, AWS SSM, or a bastion host to manage a node. Warden still runs authorized tasks from the control plane. The identity of that service and its task-signing keys remain part of the security boundary. Outbound-only connections do not remove that trust requirement. Application traffic connects directly to deployment endpoints. Paid deployments need [access rules](/networking/access-rules) first. Managed Free deployments include an initial global rule. ## Architecture ## What Warden Manages Warden manages these services on each node: | Responsibility | How It Works | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | Iggy server lifecycle | Start, stop, restart, and manage the Iggy process | | Connectors runtime | Start and manage the optional [Connectors](/connectors) runtime | | HTTP proxy for Stream UI | Authenticated reverse proxy in front of the Iggy HTTP API, consumed by the browser-side Console [Stream UI](#stream-ui) | | Configuration | Pull and apply [configuration changes](/deployments/configuration) from the control plane | | Upgrades | Download, verify, and swap new versions of Iggy, Connectors, and itself | | TLS certificates | Pull and rotate certificates automatically before expiry | | Telemetry | Push [metrics, heartbeats, and logs](/observability) to the control plane | ## Stream UI Stream UI browses streams, topics, partitions, messages, consumer groups, and connected clients. It loads through the [Console](https://laserdata.cloud) and runs in your browser. Application data travels directly between the browser and deployment node. ### Data isolation Stream UI separates data access from platform management: * The browser calls Warden's HTTP proxy on the deployment node directly. * The LaserData backend serves static application files. It does not receive stream contents, message bodies, or query results. * Stream UI runs in a sandboxed iframe with its own origin, isolated from the surrounding Console. The surrounding Console cannot read its data. * Each user receives a signed session token for one deployment. Tokens expire in minutes. Warden makes sure that the signature is valid on every request, using the Ed25519 key pair that also signs tasks. ### Network requirements Warden's HTTP proxy uses the Iggy HTTP API ports, 80 and 443. The [`iggy_http`](/networking/access-rules) toggle therefore controls browser access. Add the browser IP to an access rule with `iggy_http: true` on the deployment. LaserData does not proxy these requests, so the rule needs no LaserData-owned IP. ### Managed data The data plane is the service that turns records into queryable views and state. A backend is a storage or query engine that this service uses. The Backends page configures those engines and requires `deployment:config:manage`. You do not need to open this page to send and read ordinary Iggy messages. If the deployment runs the managed data plane, Stream UI also exposes its supported tools. Requests use the same direct path through Warden. They do not pass through the LaserData backend. The tools include these operations: * Create, replace, and drop projections, definitions that extract events into queryable tables. Bind them to source topics and targets. * Query a projection with the structured JSON query language. It supports filters, sorting, paging, aggregation, and vector search rather than SQL input. * Register and drop writer schemas in the schema registry. * Read and edit namespaced keys and values. An optional TTL sets each key's lifetime. * Create forks, copy-on-write branches of a read model. Promote a fork onto the trunk or discard it. The Managed data area appears only when the deployment advertises support. Individual tools appear according to its capabilities. ## Task System Configuration changes, upgrades, and certificate rotation follow the same signed task sequence: 1. A user or automated process requests an action through the Console or API. 2. The control plane creates and signs a task. 3. Warden retrieves the task on its next polling cycle. 4. Warden makes sure that the signature is valid before it runs the task. 5. Warden reports success or failure to the control plane. Warden rejects invalid signatures. This protects tasks against changes in transit and authenticates their origin at the control plane. ## Cluster Readiness and Shared Hosts Warden follows local Iggy leadership and node readiness. Provisioning waits for a healthy cluster roster before it creates credentials. Source connectors run only on the healthy local leader. If leadership is unknown, they remain disabled with their configuration intact. Managed data also depends on plane replay, ownership, and backend health. Read [readiness and metrics](/api/observability) during recovery. A recent heartbeat alone does not prove that a node can serve requests. Free deployments can share a host with separate runtime processes and CPU quotas. Host metrics and runtime metrics remain separate. The fleet upgrade workflow updates shared-host binaries in place. ## Upgrades and Rollbacks Runtime upgrades can interrupt client connections. Clients must reconnect after an interruption. Warden follows this sequence: 1. A published version becomes available, and Warden receives an upgrade task. 2. Warden downloads the binary and makes sure that its signature is valid. 3. Warden replaces the binary atomically, as one indivisible operation. 4. If the new binary fails to start, Warden restores the previous version. Warden can upgrade itself without stopping Iggy or Connectors. Those services continue to run during its upgrade. Warden rejects any binary with an invalid signature. Signatures apply to Warden, Iggy, and Connectors binaries. ## Credential Scope Each Warden token grants access to one node. A compromised token remains limited to that node's permissions: | Can Do | Cannot Do | | ------------------------------- | ---------------------------------- | | Pull tasks for that single node | Access the VM shell | | Push telemetry for that node | Read customer data | | | Execute arbitrary commands | | | Access other nodes | | | Modify Iggy configuration directly | ## Network Requirements | Direction | What | Port | | --------- | ------------------------------------ | ---- | | Outbound | HTTPS to the LaserData control plane | 443 | | Inbound | Nothing | None | Management requires outbound HTTPS. It does not require inbound firewall rules, SSH, SSM, or bastion hosts. Application listeners still require the client access described above. Source: https://docs.laserdata.com/deployments/warden --- # Connectors Connectors move data between Iggy and external systems without application code. The [Apache Iggy connectors runtime](https://github.com/apache/iggy/tree/master/core/connectors) loads compiled Rust plugins for each integration. The [Connector Catalog](/connectors/catalog) lists 16 sinks and 5 sources with their configuration. The plugins run as native code. Their data path uses no JVM or garbage collector. ## How It Works Each plugin implements the `Source` or `Sink` trait from the [Apache Iggy Connectors SDK](https://github.com/apache/iggy/tree/master/core/connectors/sdk). The runtime loads plugins at startup and manages configuration, execution, monitoring, and shutdown. [Warden](/deployments/warden) manages the runtime as a separate process on the same node. The runtime connects to Iggy through TCP with TLS. Cluster routing can direct it to another deployment node. External sources and destinations use their own network endpoints. ## Connector Types | Type | Direction | What It Does | | ------ | ---------------------- | ------------------------------------------------------------------------------ | | Source | External system → Iggy | Produces messages into Iggy streams from an external system | | Sink | Iggy → External system | Consumes messages from Iggy streams and pushes them to an external destination | Multiple instances can run together. For example, one source can read from an external system while several sinks write to different destinations. ## Activating a Connector ### From the Console 1. Open the deployment's Connectors tab. 2. Browse the catalog or filter it by source or sink. 3. Click Activate for the plugin that you need. 4. If needed, set a custom instance name and key. The platform provisions the instance on every node with the Connectors runtime. Its status moves from Pending to Active after the nodes process the activation task. ### Instance Naming Each instance has a unique key within the deployment. Without a custom key, the platform generates `{connector}-{type}-{operation_id}`. Save the returned key as the name for the instance's configuration. You can activate a plugin more than once. For example, two PostgreSQL sink instances can write to different databases. ## Connector Lifecycle | Status | Meaning | | -------- | --------------------------------------------------------- | | Pending | Instance created, waiting for nodes to process activation | | Active | Running and processing messages | | Inactive | Disabled but configuration preserved | | Failed | Encountered errors - check logs for details | Activation installs the plugin with its initial configuration disabled. To start processing, create an enabled configuration under the instance key. In a cluster, sources run only on the healthy local Iggy leader. Followers and nodes with unknown leadership keep sources disabled while retaining their desired configuration. Sinks coordinate consumption through consumer groups. ### Monitoring The runtime reports these metrics for each instance: | Metric | Description | | ------------------- | ------------------------------------------------------------ | | messages\_produced | Total messages produced (source connectors) | | messages\_consumed | Total messages consumed (sink connectors) | | messages\_sent | Messages sent to Iggy by sources | | messages\_processed | Messages processed by sink plugins | | messages\_filtered | Messages intentionally removed by transforms | | errors | Error count | | status | Runtime status (Starting, Running, Stopping, Stopped, Error) | Runtime metrics also include CPU, memory, and the total sources and sinks running. Read them in the Console's Metrics tab or through the [Monitoring API](/observability). ### Deleting a Connector Instance On the Connectors tab, click Delete for the instance. This removes it from all nodes and deletes its configuration. Activation, configuration, and deletion require `deployment:connector:manage`. Reading instances requires `deployment:connector:read`. ## API Reference Use the Main API to browse the catalog and activate plugins. Use the Supervisor API to manage running instances. ### List Available Connectors ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/{deployment_id}/connectors?type=sink&page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` The response includes each connector's availability and permission status for the deployment. ### Activate a Connector ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/{deployment_id}/connectors/activate \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "connector_key": "postgres", "connector_type": "sink", "instance_name": "Telemetry to Postgres", "instance_key": "telemetry-pg-sink" }' ``` The endpoint returns `202 Accepted`. The instance moves from Pending to Active after the nodes process its task. ### List Connector Instances ```bash curl {supervisor_url}/deployments/{deployment_id}/connectors/instances \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "deployment_id": 611298765432109056, "connector_type": "sink", "connector_key": "postgres", "name": "Telemetry to Postgres", "key": "telemetry-pg-sink", "status": "active" } ] ``` ### Delete a Connector Instance ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/connectors/instances/{instance_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/connectors --- # Connector Catalog The catalog lists source and sink plugins from the [Apache Iggy connectors runtime](https://iggy.apache.org). LaserData builds and signs the Rust plugins. Warden delivers them to deployment nodes. Their data path uses no JVM or garbage collector. The tables list each plugin's key, purpose, and `plugin_config` fields. API responses mask fields marked as secrets. Each instance also uses the shared [pipeline configuration](/connectors/configuration). Use the deployment's [catalog endpoint](/api/connectors#list-available-connectors) to find enabled plugins for your account. Use its [schema endpoint](/connectors/configuration#get-config-schema) for the fields, types, and defaults of the running version. ## Sink Connectors Sinks read Iggy streams and write to external systems. Most support `batch_size`, `include_metadata`, `include_checksum`, `include_origin_timestamp`, `payload_format`, `max_retries`, `retry_delay`, and `timeout`. The table lists plugin-specific configuration. | Plugin | Key | What it does | Main settings | | --------------- | --------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PostgreSQL | `postgres` | Writes messages to a PostgreSQL table | `connection_string` (secret), `target_table`, `auto_create_table`, `max_connections`, `payload_format` such as `bytea` | | ClickHouse | `clickhouse` | Inserts messages into a ClickHouse table | `url`, `database`, `username`, `password` (secret), `table`, `insert_format` such as `json_each_row`, `timeout_seconds` | | Amazon Redshift | `redshift` | Loads messages into Redshift through S3-staged Parquet files | `connection_string` (secret), `target_table`, `s3_bucket`, `s3_prefix`, `s3_endpoint`, `aws_region`, `aws_access_key_id` and `aws_secret_access_key` (secrets) or `aws_iam_role`, `archive`, `payload_format` such as `varbyte` | | Amazon S3 | `s3` | Writes messages to S3 or S3-compatible object storage as rotated files | `bucket`, `prefix`, `region`, `path_template` with `{stream}`, `{topic}`, `{date}`, and `{hour}` placeholders, `file_rotation`, `max_file_size`, `output_format` such as `json_lines`, `include_headers`, `max_attempts` | | Elasticsearch | `elasticsearch` | Indexes messages into an Elasticsearch index | `url`, `index`, `create_index_if_not_exists`, `timeout_seconds`, `max_retries`, `retry_delay`, `retry_max_delay` | | RabbitMQ | `rabbitmq` | Publishes messages to RabbitMQ exchanges over AMQP | `amqp_url` (secret), `exchange`, `exchange_type`, `routing_key`, `durable_exchange`, `delivery_mode`, `include_metadata`, `max_retries`, `retry_delay_secs`, `max_retry_delay_secs`, `timeout_secs` | | Quickwit | `quickwit` | Indexes messages into a Quickwit index | `url`, `index` | | Meilisearch | `meilisearch` | Indexes messages as documents in Meilisearch | `url`, `index`, `primary_key`, `document_action` such as `replace`, `create_index_if_not_exists`, `wait_for_tasks` | | MongoDB | `mongodb` | Writes messages to a MongoDB collection | `connection_uri` (secret), `database`, `collection`, `auto_create_collection`, `payload_format` such as `binary` | | InfluxDB | `influxdb` | Writes messages as points to InfluxDB | `version`, `url`, `org`, `bucket`, `token` (secret), `measurement`, `precision`, `include_stream_tag`, `include_topic_tag`, `include_partition_tag` | | SurrealDB | `surrealdb` | Writes messages to a SurrealDB table | `endpoint`, `namespace`, `database`, `table`, `username`, `password` (secret), `auth_scope`, `use_tls`, `auto_define_table`, `define_indexes`, `include_headers`, `query_timeout`, `max_retry_delay` | | Apache Iceberg | `iceberg` | Writes messages to Iceberg tables through a catalog | `tables`, `catalog_type` such as `rest`, `uri`, `warehouse`, `dynamic_routing` with `dynamic_route_field`, and the object store settings `store_url`, `store_class`, `store_region`, `store_path_style_access`, `store_access_key_id` and `store_secret_access_key` (secrets) | | Delta Lake | `delta` | Appends messages to a Delta Lake table on local, S3, Azure, or GCS storage | `table_uri`, `storage_backend_type`, and the credential keys of the chosen backend. For S3, set both `aws_s3_access_key` and `aws_s3_secret_key` (secret), or leave both empty to use the AWS SDK credential chain | | Apache Doris | `doris` | Loads messages into Doris tables through Stream Load | `fe_url`, `database`, `table`, `username`, `password` (secret), `label_prefix`, `timeout` | | HTTP | `http_generic` | POSTs message batches to an HTTP endpoint with retries | `url`, `method`, `batch_mode` such as `ndjson`, `max_payload_size_bytes`, `success_status_codes`, `health_check_enabled` and `health_check_method`, `retry_backoff_multiplier`, `max_retry_delay`, `max_connections`, `tls_danger_accept_invalid_certs`, and a `headers` table for values such as `Authorization` | | Stdout | `stdout` | Prints messages to the runtime's standard output for debugging | `print_payload` | The HTTP sink uses catalog key `http_generic` and runtime artifact name `http`. ## Source Connectors Sources bring data from external systems into an Iggy stream and topic. The runtime tracks progress and saves checkpoints, records of completed work, to resume after restart. See [Source Checkpoints and Failover](/connectors/configuration#source-checkpoints-and-failover). | Plugin | Key | What it does | Main settings | | ------------- | --------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PostgreSQL | `postgres` | Polls tables and produces new rows as messages | `connection_string` (secret), `mode` such as `polling`, `tables`, `tracking_column`, `initial_offset`, `poll_interval`, `batch_size`, `max_connections`, `snake_case_columns`, `include_metadata` | | Elasticsearch | `elasticsearch` | Polls an index for new documents by timestamp | `url`, `index`, `timestamp_field`, `polling_interval`, `batch_size` | | HTTP | `http_generic` | Receives webhook POSTs on an embedded HTTP listener and produces the bodies as raw messages | `listen_addr`, `admin_listen_addr`, `topic_path`, `auth_bearer_token` (secret), `management_token` (secret), `max_body_size_bytes` (at most 64,000,000, and empty bodies are rejected), `buffer_capacity`, `max_batch_size`, `include_http_metadata`, `forward_headers`, `endpoints` with per-endpoint `auth_type` such as `hmac-sha256` | | InfluxDB | `influxdb` | Runs a Flux query on an interval and advances a cursor | `version`, `url`, `org`, `token` (secret), `query` with `$cursor` and `$limit` placeholders, `cursor_field`, `initial_offset`, `poll_interval`, `batch_size`, `payload_format`, `include_metadata`, `max_retries`, `retry_delay`, `timeout` | | Random | `random` | Generates random messages for development and load testing | `interval`, `max_count`, `messages_range`, `payload_size` | The HTTP source needs a reachable listener on the active source node. Cloud connector configuration does not open an ingress port or expose the admin listener. Static `endpoints` are masked as one list because their URL IDs act as credentials. To change them, supply a complete replacement list. ## Transforms A transform changes message fields before a sink writes them or a source produces them. Instances can apply an ordered list of transforms: | Transform | Purpose | | --------------------- | ----------------------------------------------------------- | | `add_fields` | Add fields with static or computed values | | `delete_fields` | Remove fields | | `filter_fields` | Keep only the listed fields | | `update_fields` | Change existing field values | | `proto_convert` | Convert to or from Protocol Buffers | | `flat_buffer_convert` | Convert to or from FlatBuffers | | `avro_convert` | Convert to or from Avro using a schema | | `unwrap_envelope` | Unwrap a nested payload envelope into the top-level message | Transform configuration uses JSON values. The same structure applies when the surrounding plugin configuration uses JSON, YAML, or TOML. Source: https://docs.laserdata.com/connectors/catalog --- # Connector Configuration After [activation](/connectors), configure each instance through the deployment's [Configuration](/deployments/configuration) system. Each instance has independent versions that you can save, activate, or restore. The [Connector Catalog](/connectors/catalog) lists plugin-specific fields. ## Sink Configuration | Setting | Description | | --------------- | ----------------------------------------------------------------------------- | | enabled | Whether the connector instance is active | | streams | Which Iggy streams and topics to consume from | | schema | Message format - `json`, `raw`, `text`, `proto`, `flat_buffer`, or `avro` | | batch\_length | Number of messages to batch before sending | | poll\_interval | How often to poll for new messages | | consumer\_group | Optional consumer group for coordinated consumption | | plugin\_config | Plugin-specific settings (connection strings, credentials, table names, etc.) | | transforms | Optional data transformations before sending | ## Source Configuration | Setting | Description | | -------------- | ------------------------------------------------------------------------- | | enabled | Whether the connector instance is active | | streams | Which Iggy stream and topic to produce into | | schema | Message format - `json`, `raw`, `text`, `proto`, `flat_buffer`, or `avro` | | batch\_length | Number of messages to batch before producing | | linger\_time | Maximum time to wait before flushing a batch | | plugin\_config | Plugin-specific settings (source connection, polling interval, etc.) | | transforms | Optional data transformations before producing | ## Runtime and Plugin Settings Before upgrading the runtime, make sure that source destination topics use `persisted` durability. The runtime creates missing topics with that policy. It rejects existing destinations that use `replicated`. Instance configuration separates pipeline fields, such as `streams`, `enabled`, and transforms, from `plugin_config`. Shared `connectors` configuration controls the runtime's Iggy connection, HTTP API, telemetry, logging, and checkpoint storage. Instance fields also include `plugin_config_format`, `verbose`, and `benchmark`. Formats are `json`, `yaml`, `toml`, and `text`. Cloud configuration accepts plugin values according to the schema. The platform owns `type`, `key`, `version`, `name`, and the library `path`. For Avro, set `schema: "avro"`. Supply `avro_schema_json` or a schema file available on the runtime node. An inline schema avoids a file dependency on every node. ### Source Checkpoints and Failover A checkpoint records completed source progress. File checkpoints remain on the node's disk. HTTP checkpoints can survive node replacement if the service that stores them is shared and durable. The runtime saves a checkpoint only after Iggy acknowledges its batch. If sending or checkpoint storage fails, the connector receives the batch again on its next poll. This repeats work instead of skipping it. The HTTP source first queues webhook requests in memory. Its success response proves queue admission, not a durable Iggy write. Restart can lose queued requests, and sender retries can duplicate them. Use durable retries at the sender and include a delivery ID so downstream systems can remove duplicates. HTTP checkpoint storage uses conditional writes with ETags, identifiers for stored versions, and an idempotency key for each logical save. A missing checkpoint returns `404`. Other load failures must not restart ingestion from the beginning. Version conflicts or revoked authorization stop checkpoint writes until the connector restarts. Checkpoint configuration requires administrative access. Use `IGGY_CONNECTORS_STATE_STORAGE` and the schema's `IGGY_CONNECTORS_STATE_HTTP_*` fields. Retrieving configuration over HTTP and storing checkpoints over HTTP are separate features. Warden enables cluster sources only on the healthy local Iggy leader. Other nodes keep sources disabled while preserving the saved desired configuration. Leadership changes can still repeat work. Use durable checkpoints and downstream operations that are safe to repeat. ## Data Transforms Apply transforms in order to change message fields: | Transform | Description | | --------------------- | ----------------------------------------------------------- | | `add_fields` | Add new fields to messages | | `delete_fields` | Remove fields from messages | | `filter_fields` | Keep only the listed fields | | `update_fields` | Modify existing field values | | `proto_convert` | Convert to or from Protocol Buffers | | `flat_buffer_convert` | Convert to or from FlatBuffers | | `avro_convert` | Convert to or from Avro using a schema | | `unwrap_envelope` | Unwrap a nested payload envelope into the top-level message | To build a custom transform in Rust, implement `Transform`. ## Examples ### PostgreSQL Sink ```json { "name": "telemetry-pg-sink", "values": { "enabled": true, "streams": [ { "stream": "telemetry", "topics": ["readings", "alerts"], "schema": "json", "batch_length": 100, "poll_interval": "1s", "consumer_group": "pg-sink-telemetry" } ], "plugin_config": { "connection_string": "postgres://user:pass@host:5432/telemetry", "target_table": "reading_events" } }, "activate": true } ``` ### Random Source (Development) ```json { "name": "random-source", "values": { "enabled": true, "streams": [ { "stream": "test_stream", "topic": "test_topic", "schema": "json", "batch_length": 1000, "linger_time": "5ms" } ], "plugin_config": { "interval": "3000ms", "max_count": 1000000, "messages_range": [1, 5], "payload_size": 200 } }, "activate": true } ``` ## Plugin Schema Validation Each plugin schema defines types, defaults, and secrets. Creation and updates follow these rules: * Unknown `plugin_config` keys are rejected. * Values must match their field types, such as string, integer, boolean, duration, or array. * Fields marked `secret`, such as passwords and connection strings, appear as `***` in API responses. Before creating configuration, retrieve [Get Config Schema](#get-config-schema) for the available fields. ## Configuration Flow To configure an instance: 1. Retrieve its schema for fields, defaults, and rules. 2. Save values under the instance key to create a version. 3. Activate that version, or use `"activate": true` during creation. Versions belong to individual instances. Changing one instance does not select a version for another. ## Secret Masking API responses show `***` for secrets such as passwords, connection strings, and credentials. An update preserves masked values. Send only the fields that you want to change. ## API Reference A connector kind follows `connector:{type}:{plugin_key}`, for example `connector:sink:postgres`. Its configuration `name` is the activated instance key, such as `telemetry-pg-sink`. Use this name for activation and history. An arbitrary name does not create an instance. ### Get Config Schema Retrieve fields, types, defaults, and rules in the [section format](/deployments/configuration#get-configuration-schema) used by Iggy configuration: ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/schema \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "sink": { "name": "Sink", "description": "Connector sink pipeline settings.", "schema": [] }, "plugin_config": { "name": "Plugin Config", "description": "Connector plugin-specific settings.", "schema": [ { "key": "connection_string", "name": "Connection String", "description": "PostgreSQL connection string", "default_value": "", "kind": "string", "editable": true, "secret": true, "requirements": [], "rules": [] } ] } } ``` The `sink` or `source` section contains pipeline fields, such as enabled state, streams, and batching. `plugin_config` contains plugin fields, such as connection strings and table names. Fields with `"secret": true` appear as `***` in responses. ### Create a Config ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "telemetry-pg-sink", "values": { "enabled": true, "streams": [ { "stream": "telemetry", "topics": ["readings", "alerts"], "schema": "json", "batch_length": 100, "poll_interval": "1s", "consumer_group": "pg-sink-telemetry" } ], "plugin_config": { "connection_string": "postgres://user:pass@host:5432/telemetry", "target_table": "reading_events" }, "transforms": {} }, "activate": true }' ``` Supply these fields: * Set `name` to the activated instance key. * Put schema-compatible configuration in `values`. * Set `activate` to `true` to select the new version as primary immediately. A successful request returns `201 Created` with the new configuration ID in `ld-config`. An existing name receives another version automatically. ### Get Active Config ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/primary \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "kind": "connector:sink:postgres", "name": "telemetry-pg-sink", "primary": true, "initialized": true, "version": 2, "created_at": "2026-03-16T12:00:00Z", "updated_at": "2026-03-16T12:05:00Z", "values": { "enabled": true, "streams": [], "plugin_config": { "connection_string": "***", "target_table": "reading_events" }, "transforms": {} } } ``` ### List Config Versions ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/telemetry-pg-sink/versions \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get a Specific Version ```bash curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/telemetry-pg-sink/versions/2 \ -H "ld-api-key: YOUR_API_KEY" ``` ### Activate a Specific Version Activation selects the primary version and starts reconfiguration on every node: ```bash curl -X PUT {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/telemetry-pg-sink/activate/2 \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. ### Delete a Config ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/{config_id} \ -H "ld-api-key: YOUR_API_KEY" ``` Creation, activation, and deletion require `deployment:config:manage`. Reading configuration or schemas requires `deployment:config:read`. Source: https://docs.laserdata.com/connectors/configuration --- # Networking Overview Networking configuration controls which clients can reach a deployment and how their traffic travels. Paid deployments need access rules before clients can connect. Free deployments start with a global rule. VPC peering requires Performance or Enterprise. PrivateLink and Private Service Connect require Enterprise. The account plan must also enable private networking. Standard does not include these features. The deployment tier and account plan impose separate requirements. ## Connectivity Options | Feature | Scope | Availability | Purpose | | -------------------------------------------------------------- | ---------- | ----------------------------------------- | ----------------------------------------------------------------- | | [Access Rules](/networking/access-rules) | All clouds | Every deployment | Allow specific IPs/CIDRs to reach deployment endpoints | | [VPC Peering](/networking/vpc-peering) | AWS, GCP | Performance and Enterprise (Managed only) | Private network path between your VPC and the deployment VPC | | [PrivateLink](/networking/private-link) | AWS | Enterprise (Managed only) | Expose the deployment as a VPC endpoint service in your account | | [Private Service Connect](/networking/private-service-connect) | GCP | Enterprise (Managed only) | Expose the deployment as a PSC service attachment in your project | ## Default Posture Paid deployments start without client access rules. Create a rule before you connect. Warden's outbound management connection operates independently of these rules. Free deployments include `0.0.0.0/0`, a rule that opens all Iggy protocols to all IPv4 addresses. It also permits [Stream UI](/deployments/warden#stream-ui) in the browser to reach Warden's HTTP proxy. You can delete or replace this rule. ## Required Permissions Networking permissions apply within an environment: | Permission | Grants | | --------------------------- | ------------------------------------------------------------------------------- | | `deployment:access:read` | View access rules | | `deployment:access:manage` | Create and delete access rules (implies read) | | `deployment:network:read` | View VPC peering connections, PrivateLink services, and PSC service attachments | | `deployment:network:manage` | Create and delete VPC peering, PrivateLink, and PSC (implies read) | ## Plan Limits VPC Peering requires Performance or Enterprise. PrivateLink and Private Service Connect require Enterprise. All require private networking access in the account plan. The plan also limits the number of rules and connections per deployment. | Resource | Basic | Pro | Enterprise | | ----------------------------------------- | ----- | --------- | ---------- | | Access rules per deployment | 3 | 10 | 20 | | VPC peering connections per deployment | - | 3 | 5 | | PrivateLink / PSC services per deployment | - | 1 | 5 | | Private networking | - | Available | Available | ## Network Info Retrieve a Managed deployment's VPC ID and CIDR range before you configure peering or firewall rules. A VPC is a private cloud network. A CIDR range describes a group of IP addresses. ```bash curl {supervisor_url}/deployments/{deployment_id}/network/info \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": "vpc-0abc123def456789", "cidr": "10.0.0.0/16" } ``` The request requires `deployment:network:read`. This endpoint is unavailable for BYOC. Your cloud account already contains the VPC details. Source: https://docs.laserdata.com/networking --- # Access Rules Access rules permit selected IP addresses and CIDR ranges to reach deployment endpoints. A CIDR range describes a group of IP addresses. Rules apply to Managed, BYOC, and On-Premise deployments on every plan. Paid deployments need a rule before clients can connect. Managed Free deployments include the global rule `0.0.0.0/0`. It permits every IPv4 address on the enabled protocols, including browser access when HTTP is enabled. Replace it with trusted source ranges before sending application data. Network access and Cloud permissions are separate. A rule cannot fix a Console `deployment:read` denial. If your organization owner account is blocked, use the [permission troubleshooting steps](/organization/roles-permissions#access-denied-after-creating-a-deployment). ## Why Access Rules Use rules to control the source addresses, protocols, and duration of client access: * Permit known office or VPN address ranges. * Open TCP for applications and HTTP for monitoring or [Stream UI](/deployments/warden#stream-ui) independently. * Grant temporary access that expires automatically, for example during contractor work or debugging. * Limit Stream UI access to trusted operators' browser IPs. ## Concepts ### Protocol Rules Select the protocols that a rule permits for its CIDR ranges. You can enable any combination from the table. A rule without enabled protocols opens no ports, even when it contains address ranges. | Protocol | API Field | What It Opens | | -------------- | ---------------- | ----------------------------------------- | | Iggy HTTP | `iggy_http` | Iggy HTTP API endpoint (ports 80 and 443) | | Iggy TCP | `iggy_tcp` | Iggy TCP transport | | Iggy WebSocket | `iggy_websocket` | Iggy WebSocket transport | | Iggy UDP | `iggy_udp` | Iggy QUIC/UDP transport | [Stream UI](/deployments/warden#stream-ui) loads from the [Console](https://laserdata.cloud) and runs in the browser. It connects directly to Warden's HTTP proxy on ports 80 and 443. Enable `iggy_http` for the browser's IP address. LaserData does not add its own IP to make this connection. ### CIDR Blocks Each rule needs at least one IPv4 CIDR block. Supported forms include these: * `203.0.113.5/32` permits one host. * `10.0.0.0/16` permits a subnet. * `0.0.0.0/0` permits every IPv4 address to use the selected protocols. Subnet masks must range from `/0` to `/32`. ### Rule Expiry A rule can have a future expiration date. After that date, the rule no longer permits traffic from its address ranges. For example, use expiry to grant a partner 30 days of access without a separate removal task. ## Creating an Access Rule ### From the Console 1. Open your deployment's Access Rules tab. 2. Click Add Rule. 3. Enter a name that is unique within the deployment. 4. Add at least one CIDR range. 5. Select Iggy TCP, HTTP, WebSocket, UDP, or a combination. 6. If access is temporary, set an expiration date. 7. Add remarks if you need to explain the rule. 8. Click Create. LaserData applies the rule through AWS security groups or GCP firewall rules. ### Validation A new rule must meet these requirements: * Its name is unique within the deployment, without regard to letter case. * It contains at least one valid IPv4 CIDR block. * An expiry date, when supplied, is in the future. * The deployment remains within its plan's access-rule limit. ## Managing Access Rules The Access Rules tab lists address ranges, enabled protocols, expiry status, and creation times. Delete a rule to remove it and its corresponding cloud infrastructure configuration. Other active rules retain their permissions. ## Plan Limits | Resource | Basic | Pro | Enterprise | | --------------------------- | ----- | --- | ---------- | | Access rules per deployment | 3 | 10 | 20 | ## Audit The [audit log](/observability/audit-compliance) records rule creation, updates, and deletion. Creation records identify the requester, CIDRs, and protocols. Update records preserve previous values. Deletion records identify who removed the rule and when. ## API Reference Use [API keys](/security/api-keys) for programmatic access. Creating and deleting rules require `deployment:access:manage`. Listing rules requires `deployment:access:read`. ### Create a Rule ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/access_rules \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "production-api-access", "cidr_blocks": ["10.0.0.0/16", "172.16.0.0/12"], "rules": { "iggy_tcp": true, "iggy_http": true }, "valid_to": "2026-12-31T23:59:59Z", "remarks": "Production API servers" }' ``` ### List Rules ```bash curl {supervisor_url}/deployments/{deployment_id}/access_rules \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "name": "production-api-access", "remarks": "Production API servers", "rules": { "iggy_http": true, "iggy_tcp": true, "iggy_websocket": false, "iggy_udp": false }, "cidr_blocks": ["10.0.0.0/16", "172.16.0.0/12"], "valid_to": "2026-12-31T23:59:59Z", "created_at": "2025-01-15T10:30:00Z" } ] ``` ### Delete a Rule ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/access_rules/{rule_id} \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/networking/access-rules --- # VPC Peering VPC Peering connects your private cloud network to a LaserData Managed deployment. Traffic uses private IP addresses within the cloud provider's network. It does not cross the public internet. VPC Peering supports Managed deployments on AWS and GCP. It requires Performance or Enterprise and private networking enabled in the account plan. BYOC already runs in your VPC and does not need peering. Standard does not include private connectivity. ## Why VPC Peering Public Managed endpoints use [access rules](/networking/access-rules). Peering provides a private path with these properties: * Traffic stays on the provider's network and does not use the public internet. * Applications do not need public IP addresses to use the connection. * Access rules can restrict traffic to the peered VPC's CIDR, its IP address range. Use a VPC CIDR that does not overlap the deployment subnet or another active peering on the deployment. If ranges overlap, the platform rejects the request and identifies the conflicting CIDRs. ## AWS VPC Peering ### Prerequisites Prepare these before you create the connection: * A running Managed deployment on AWS. * An AWS VPC in the same region or another region. * Your 12-digit AWS Account ID, shown in the AWS Console's top-right area. * Your VPC ID, which starts with `vpc-` and appears in the AWS VPC Console. * Your VPC CIDR block. ### Setup 1. Open your deployment's Networking tab. 2. Click Add VPC Peering. 3. Enter a connection name. 4. Enter the VPC ID, AWS Account ID, and VPC CIDR. 5. If the VPC is in another region, enter that region. 6. Click Create. LaserData makes sure that the inputs are valid and creates the AWS peering request. It configures routes and security groups on the deployment side. The connection starts in Pending Acceptance. ### Accept the Peering Request In the AWS account that owns your VPC: 1. Open the AWS VPC Console in the VPC's region. 2. Open Peering Connections. 3. Select the pending request from LaserData. 4. Open Actions and select Accept Request. ### Configure Your VPC After you accept the request, configure your side of the connection: 1. Open Route Tables in the AWS VPC Console. 2. Select the route table for your VPC subnets. 3. Open Edit routes and select Add route. 4. Set Destination to the deployment subnet CIDR shown in LaserData Console. 5. Set Target to the peering connection ID, `pcx-...`. 6. Save the route. 7. Permit traffic to and from the deployment CIDR in your security groups. ### Connection Status | Status | API Value | Meaning | Action | | ------------------ | -------------------- | ----------------------------------------- | ---------------------------------------------------------------------- | | Initiating Request | `initiating_request` | LaserData is creating the peering request | Wait for it to proceed | | Pending Acceptance | `pending_acceptance` | Waiting for you to accept in AWS | Accept in the AWS VPC Console | | Provisioning | `provisioning` | AWS is setting up the connection | Wait for it to complete | | Active | `active` | Peering established, traffic can flow | No action needed | | Rejected | `rejected` | You rejected the request | Delete and recreate if needed | | Expired | `expired` | Request was not accepted in time | Delete and recreate | | Failed | `failed` | Peering failed | Make sure that VPC ID and Account ID are correct. Recreate the peering | | Deleting | `deleting` | Peering is being removed | Wait for deletion to complete | | Deleted | `deleted` | Peering has been removed | No action needed | ## GCP VPC Peering ### Prerequisites Prepare these resources and identifiers: * A running Managed deployment on GCP. * A GCP VPC network. * Your Project ID, with 6-30 lowercase letters, digits, or hyphens. * Your VPC network name, with at most 63 lowercase letters, digits, or hyphens. * Your VPC CIDR block. ### Setup 1. Open your deployment's Networking tab. 2. Click Add VPC Peering. 3. Enter a connection name. 4. Enter your Project ID, VPC network name, and VPC CIDR. 5. Click Create. LaserData creates its side of the connection. The peering remains Inactive until you create the matching connection in your project. ### Create the Reciprocal Peering GCP requires peering configuration on both sides. After LaserData creates its connection: 1. In GCP Console, open VPC network, then VPC network peering. 2. Click Create peering connection. 3. Enter a peering name. 4. Select your VPC network. 5. Enter the LaserData project ID and network name from the LaserData Console instructions. 6. Click Create. The peering detail page in LaserData Console provides instructions for its current status. Use those instructions for your connection. ### Configure Firewall Rules After both sides are connected, configure your GCP project: 1. Open VPC network, then Firewall. 2. Create an ingress rule that permits traffic from the deployment CIDR. 3. Create an egress rule that permits traffic to the deployment CIDR. ### Connection Status | Status | API Value | Meaning | Action | | -------- | ---------- | --------------------------------------------- | --------------------------------- | | Inactive | `inactive` | Waiting for reciprocal peering from your side | Create the peering in GCP Console | | Active | `active` | Peering established, traffic can flow | No action needed | | Deleted | `deleted` | Peering has been removed | Recreate if needed | LaserData periodically reads the peering status from GCP. The displayed status changes after GCP recognizes the reciprocal connection. ## Deleting a Peering Connection Delete a peering from the Networking tab. LaserData removes its connection and routing configuration. Remove the corresponding peering, routes, and firewall or security-group entries from your own VPC. ## Plan Limits | Resource | Basic | Pro | Enterprise | | -------------------------------------- | ----- | --------- | ---------- | | VPC peering connections per deployment | - | 3 | 5 | | Private networking | - | Available | Available | ## Audit The [audit log](/observability/audit-compliance) records peering creation and deletion. Creation records include the requester, VPC, and CIDR. Deletion records include the requester and time. ## API Reference Use [API keys](/security/api-keys) with the endpoints for the deployment's cloud provider. Creation and deletion require `deployment:network:manage`. Listing connections and reading instructions require `deployment:network:read`. ### AWS #### Create a Peering Connection ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/network/aws/vpc_peering \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "app-to-iggy", "peer_vpc_id": "vpc-0abc123def456789a", "peer_owner_id": "123456789012", "peer_vpc_cidr": "172.16.0.0/16", "peer_region": "us-west-2", "remarks": "Application VPC to deployment" }' ``` A successful request returns `204 No Content`. #### List Peering Connections ```bash curl {supervisor_url}/deployments/{deployment_id}/network/aws/vpc_peering \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "name": "app-to-iggy", "peering_connection_id": "pcx-0abc123def456789a", "requester_vpc_id": "vpc-deployment", "requester_cidr": "10.0.0.0/16", "accepter_vpc_id": "vpc-0abc123def456789a", "accepter_cidr": "172.16.0.0/16", "requester_region": "us-west-1", "accepter_region": "us-west-2", "requester_owner_id": "987654321098", "accepter_owner_id": "123456789012", "route_table_ids": ["rtb-0abc123def456789a"], "status": "active", "expiry_at": null, "remarks": "Application VPC to deployment", "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:35:00Z" } ] ``` #### Get Setup Instructions The response provides instructions for the current peering status: ```bash curl {supervisor_url}/deployments/{deployment_id}/network/aws/vpc_peering/{peering_id}/instructions \ -H "ld-api-key: YOUR_API_KEY" ``` #### Delete a Peering Connection ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/network/aws/vpc_peering/{peering_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. ### GCP #### Create a Peering Connection ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/network/gcp/vpc_peering \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "app-to-iggy", "peer_vpc_name": "my-vpc-network", "peer_project_id": "my-gcp-project", "peer_vpc_cidr": "172.16.0.0/16", "remarks": "Application VPC to deployment" }' ``` A successful request returns `204 No Content`. | Field | Required | Description | | ----------------- | -------- | -------------------------------------------------------------------- | | `name` | Yes | Name for the peering connection | | `peer_vpc_name` | Yes | Your GCP VPC network name (lowercase, digits, hyphens, max 63 chars) | | `peer_project_id` | Yes | Your GCP project ID (6-30 chars, lowercase, digits, hyphens) | | `peer_vpc_cidr` | Yes | Your VPC CIDR block (must not overlap with deployment subnet) | | `remarks` | No | Optional description | #### List Peering Connections ```bash curl {supervisor_url}/deployments/{deployment_id}/network/gcp/vpc_peering \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "name": "app-to-iggy", "peering_name": "laser-peering-12345", "local_vpc_name": "ld-vpc-deployment-611298765432109056", "peer_vpc_name": "my-vpc-network", "peer_project_id": "my-gcp-project", "peer_vpc_cidr": "172.16.0.0/16", "state": "active", "state_details": null, "remarks": "Application VPC to deployment", "created_at": "2026-03-20T10:30:00Z", "updated_at": "2026-03-20T10:35:00Z" } ] ``` #### Get Setup Instructions For an `inactive` peering, the response explains how to create its reciprocal connection in GCP Console. Instructions follow the current peering status. ```bash curl {supervisor_url}/deployments/{deployment_id}/network/gcp/vpc_peering/{peering_id}/instructions \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "peering_name": "laser-peering-12345", "state": "inactive", "local_vpc_name": "ld-vpc-deployment-611298765432109056", "peer_vpc_name": "my-vpc-network", "peer_project_id": "my-gcp-project", "steps": [ "Open the Google Cloud Console...", "Navigate to VPC network peering...", "..." ] } ``` #### Delete a Peering Connection ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/network/gcp/vpc_peering/{peering_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/networking/vpc-peering --- # PrivateLink AWS PrivateLink exposes a LaserData Managed deployment as a VPC endpoint service. Consumers create interface endpoints, private entry points in their own VPCs. Traffic stays within AWS and does not cross the public internet. PrivateLink requires a Managed AWS deployment on Enterprise and private networking enabled in the account plan. BYOC runs in your VPC and does not need PrivateLink. ## Why PrivateLink PrivateLink exposes one service rather than connecting two entire VPCs. It has these properties: * Consumer and deployment address ranges can overlap. * You can permit other AWS accounts without sharing VPCs. * Consumers connect to the deployment. The connection does not let the deployment reach into consumer VPCs. * Multiple consumers can connect independently. ## How It Works A connection follows this sequence: 1. You create an endpoint service for the deployment. 2. LaserData creates the AWS service behind the deployment's Network Load Balancer. 3. Consumers create interface endpoints in their VPCs with the service name. 4. Traffic passes privately through AWS from those endpoints to the deployment. ## Prerequisites You need a running Managed AWS deployment with a Network Load Balancer. The deployment must use Enterprise. The account plan must enable private networking. ## Creating an Endpoint Service ### From the Console 1. Open your deployment's Networking tab. 2. Click Add PrivateLink. 3. Enter a service name that is unique within the deployment. 4. Choose whether new connections require acceptance. Acceptance is required by default. 5. If access is limited to selected principals, enter their AWS IAM ARNs. 6. Click Create. An allowed principal is an AWS identity permitted to request access. For example, `arn:aws:iam::123456789012:root` identifies an account. If the list is empty, any AWS account can discover the service and request a connection. Disable required acceptance only if you trust every allowed principal. LaserData returns a service name such as `com.amazonaws.vpce.us-west-1.vpce-svc-0abc123def...`. Consumers use that name to create endpoints. ### What Gets Created LaserData creates and configures these resources: * An AWS VPC Endpoint Service linked to the deployment's Network Load Balancer. * The acceptance policy and allowed principals. * A service name for consumers to use. ## Connecting as a Consumer After the service exists, configure an interface endpoint in the consumer's AWS account. ### Step 1 - Create the VPC Endpoint In the consumer account: 1. Open AWS VPC Console. 2. Open Endpoints and select Create Endpoint. 3. Select Other endpoint services. 4. Enter the service name from the service owner. 5. Click Verify service to make sure that the name resolves. 6. Select the endpoint's VPC, subnets, and security groups. ### Step 2 - Accept the Connection (if required) If `acceptance_required` is enabled, the owner must approve the request. Open the service's pending connections in LaserData Console. Accept the consumer connection to permit traffic. ### Step 3 - Connect After activation, applications use the endpoint's private DNS name or ENI IP addresses. ENIs are network interfaces attached to the endpoint. Consumers do not need an internet gateway or NAT for this connection. ## Managing Endpoint Services The Networking tab lists service names, acceptance policies, allowed principals, connected endpoints, and their statuses. Deleting a service removes the underlying AWS VPC Endpoint Service. Connected endpoints then stop working. Consumers must remove their endpoint resources separately. ## Plan Limits | Resource | Basic | Pro | Enterprise | | -------------------------------- | ----- | --------- | ---------- | | Endpoint services per deployment | - | 1 | 5 | | Private networking | - | Available | Available | ## Audit The [audit log](/observability/audit-compliance) records service creation, configuration details, and the requester. It also records who deleted a service and when. ## API Reference Use [API keys](/security/api-keys) for these requests. Creating and deleting services require `deployment:network:manage`. Listing them requires `deployment:network:read`. ### Create an Endpoint Service ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/network/aws/private_link \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "iggy-endpoint-service", "acceptance_required": true, "allowed_principals": [ "arn:aws:iam::123456789012:root" ], "remarks": "PrivateLink for production consumers" }' ``` ### List Endpoint Services ```bash curl {supervisor_url}/deployments/{deployment_id}/network/aws/private_link \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "name": "iggy-endpoint-service", "vpc_endpoint_service_id": "vpce-svc-0abc123def456789a", "service_name": "com.amazonaws.vpce.us-west-1.vpce-svc-0abc123def456789a", "service_type": "Interface", "network_load_balancer_arns": ["arn:aws:elasticloadbalancing:us-west-1:987654321098:loadbalancer/net/ld-nlb/abc123"], "availability_zones": ["us-west-1a", "us-west-1b"], "acceptance_required": true, "allowed_principals": ["arn:aws:iam::123456789012:root"], "private_dns_name": null, "state": "available", "remarks": "PrivateLink for production consumers", "created_at": "2025-01-15T10:30:00Z", "updated_at": "2025-01-15T10:30:00Z" } ] ``` Service states are `pending`, `available`, `deleting`, `deleted`, and `failed`. ### Delete an Endpoint Service ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/network/aws/private_link/{service_id} \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/networking/private-link --- # Private Service Connect GCP Private Service Connect (PSC) exposes a Managed deployment through a service attachment, a private service connection for consumers. Consumers in your project or other permitted projects create PSC endpoints in their VPCs. Traffic stays within Google's network and does not cross the public internet. PSC requires a Managed GCP deployment on Enterprise and private networking enabled in the account plan. BYOC runs in your VPC and does not need PSC. ## Why Private Service Connect PSC provides the GCP equivalent of AWS PrivateLink. It exposes a service rather than connecting two entire VPCs. The connection has these properties: * Consumer and deployment address ranges can overlap. * You can permit other GCP projects without sharing VPCs. * Consumers connect to the deployment. The connection does not let the deployment reach into consumer VPCs. * Multiple consumers can connect independently. ## How It Works A connection follows this sequence: 1. You create a service attachment for the deployment. 2. LaserData creates it behind the deployment's internal load balancer. 3. Consumers create PSC endpoints with the attachment URI. 4. Traffic passes privately through Google's network from those endpoints to the deployment. ## Prerequisites You need a running Managed GCP deployment on Enterprise. The account plan must enable private networking. ## Creating a Service Attachment ### From the Console 1. Open your deployment's Networking tab. 2. Click Add Private Service Connect. 3. Enter an attachment name that is unique within the deployment. 4. Choose manual or automatic acceptance. Manual acceptance is the default. 5. If access is limited to selected projects, add their IDs to the consumer accept list. 6. If the connection header needs the original client IP, enable proxy protocol. 7. Click Create. Manual acceptance requires approval before traffic can flow. Automatic acceptance approves connections without a separate action. With an empty consumer accept list, any project can request access under the selected policy. LaserData returns the attachment URI, such as `projects/ld-prod/regions/us-central1/serviceAttachments/my-attachment`. Consumers use this URI to create endpoints. ### What Gets Created LaserData creates and configures these resources: * A GCP PSC Service Attachment linked to the deployment's internal load balancer. * NAT subnets for the attachment. * The acceptance policy and consumer accept lists. * An attachment URI for consumers to use. ## Connecting as a Consumer After the attachment exists, configure an endpoint in the consumer's GCP project. ### Step 1 - Create the PSC Endpoint In the consumer project: 1. Open Google Cloud Console. 2. Open Network services, then Private Service Connect. 3. Click Connect to a published service. 4. Enter the attachment URI from its owner. 5. Select a subnet and IP address in your VPC. 6. Click Add endpoint. ### Step 2 - Accept the Connection (if manual) With `accept_manual`, a connection remains pending until the owner accepts it. Open the attachment's pending connections in LaserData Console. Accept the connection to permit traffic. ### Step 3 - Connect When the status is `accepted`, applications use the endpoint's assigned private IP address. The consumer does not need an internet gateway or NAT for this connection. ## Managing Service Attachments The Networking tab lists attachment URIs, acceptance policies, consumer accept lists, NAT subnets, and connection statuses. Deleting an attachment removes the GCP service attachment. Connected endpoints then stop working. Consumers must remove their endpoint resources separately. ## Plan Limits | Resource | Basic | Pro | Enterprise | | ---------------------------------- | ----- | --------- | ---------- | | Service attachments per deployment | - | 1 | 5 | | Private networking | - | Available | Available | ## Audit The [audit log](/observability/audit-compliance) records attachment creation, configuration details, and the requester. It also records who deleted an attachment and when. ## API Reference Use [API keys](/security/api-keys) for these requests. Creating and deleting attachments require `deployment:network:manage`. Listing attachments, instructions, and connections requires `deployment:network:read`. ### Create a Service Attachment ```bash curl -X POST {supervisor_url}/deployments/{deployment_id}/network/gcp/psc \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "iggy-psc-attachment", "connection_preference": "accept_manual", "consumer_accept_lists": [ "my-gcp-project-123" ], "enable_proxy_protocol": false, "remarks": "PSC for production consumers" }' ``` | Field | Required | Description | | ----------------------- | -------- | ------------------------------------------------------ | | `name` | Yes | Unique name for the service attachment | | `connection_preference` | No | `accept_manual` (default) or `accept_automatic` | | `consumer_accept_lists` | No | GCP project IDs allowed to connect | | `enable_proxy_protocol` | No | Include original client IP in header (default `false`) | | `remarks` | No | Optional description | A successful request returns `204 No Content`. ### List Service Attachments ```bash curl {supervisor_url}/deployments/{deployment_id}/network/gcp/psc \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "name": "iggy-psc-attachment", "attachment_id": "psc-abc123def456", "service_attachment_uri": "projects/ld-prod/regions/us-central1/serviceAttachments/iggy-psc-attachment", "target_service": "projects/ld-prod/regions/us-central1/backendServices/ld-backend", "connection_preference": "accept_manual", "consumer_accept_lists": ["my-gcp-project-123"], "nat_subnets": ["projects/ld-prod/regions/us-central1/subnetworks/psc-nat-subnet"], "enable_proxy_protocol": false, "state": "active", "remarks": "PSC for production consumers", "created_at": "2026-03-20T10:30:00Z", "updated_at": "2026-03-20T10:30:00Z" } ] ``` Attachment states are `pending`, `active`, and `closed`. Connection statuses are `pending`, `accepted`, `rejected`, `closed`, and `needs_attention`. ### Get Setup Instructions The response explains how a consumer connects to the attachment: ```bash curl {supervisor_url}/deployments/{deployment_id}/network/gcp/psc/{attachment_id}/instructions \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "service_attachment_uri": "projects/ld-prod/regions/us-central1/serviceAttachments/iggy-psc-attachment", "connection_preference": "accept_manual", "instructions": [ "In your GCP project, navigate to Network services → Private Service Connect", "Click 'Connect to a published service' and enter the target service: projects/ld-prod/regions/us-central1/serviceAttachments/iggy-psc-attachment", "Select a subnet and IP address in your VPC for the PSC endpoint", "Click 'Add endpoint' to create the PSC connection", "The connection preference is ACCEPT_MANUAL - your connection will be pending until accepted by the service producer", "Once the connection status is ACCEPTED, use the assigned private IP to reach the service" ] } ``` ### List Connections The response lists the attachment's current PSC endpoint connections: ```bash curl {supervisor_url}/deployments/{deployment_id}/network/gcp/psc/{attachment_id}/connections \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "id": 1, "connection_id": "psc-conn-xyz789", "consumer_project_id": "my-gcp-project-123", "consumer_network": "projects/my-gcp-project-123/global/networks/default", "consumer_forwarding_rule": "projects/my-gcp-project-123/regions/us-central1/forwardingRules/psc-fr-1", "status": "accepted", "error_info": null, "created_at": "2026-03-21T14:00:00Z", "updated_at": "2026-03-21T14:05:00Z" } ] ``` ### Delete a Service Attachment ```bash curl -X DELETE {supervisor_url}/deployments/{deployment_id}/network/gcp/psc/{attachment_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. Source: https://docs.laserdata.com/networking/private-service-connect --- # Security Architecture Warden starts outbound HTTPS connections to the LaserData control plane, the services that manage deployments. Signed tasks, signed binaries, and scoped credentials protect that management path. Application clients connect directly to deployment endpoints through TLS and access rules. ## Security Model Thick arrows show requests initiated by Warden. Dashed arrows show telemetry sent outbound from Warden. The control plane does not initiate inbound connections to nodes. [Stream UI](/deployments/warden#stream-ui) loads through the [Console](https://laserdata.cloud), then connects from the browser to Warden's HTTP proxy. Its short-lived Ed25519-signed session is limited to one deployment and user. Iggy payloads do not pass through the LaserData backend. Stream UI is isolated from the surrounding Console. An [access rule](/networking/access-rules) must permit the browser IP, as for any other Iggy HTTP client. ## Pull-Based Architecture [Warden](/deployments/warden) starts every management connection from the node. The control plane supplies tasks through those outbound connections. It does not open an inbound connection to your infrastructure. | Data Flow | Direction | Description | | ------------------------------ | ---------------- | ----------------------------------------------------------------- | | Config, tasks, certificates | Pulled by Warden | Warden polls the control plane over HTTPS | | Heartbeats, metrics | Pushed by Warden | Warden reports node health outbound | | Inbound management connections | None | Warden polls outbound. Client listeners are controlled separately | This model has three requirements: * Warden trusts authorized tasks from the control plane. Task-signing keys and control-plane identity remain security boundaries. * Nodes need outbound HTTPS for management. Client traffic needs separate inbound access. * Management requires no SSH keys, cloud-specific management agents, or bastion hosts. ## Network Isolation | Property | Managed | BYOC | On-Premise | | ------------------------------------------------ | ------- | ---- | ---------- | | Control plane can push commands | No | No | No | | SSH access | None | None | None | | SSM access | None | None | None | | Inbound management connection from control plane | None | None | None | | Customer data leaves infrastructure | N/A | No | No | | LaserData has network access to endpoints | No | No | No | Paid deployments need [access rules](/networking/access-rules) before clients can connect. Managed Free deployments include a global rule. Application messages travel directly between clients and deployment nodes. ## Encryption | What | How | | --------------------- | ------------------------------------------------------------------------------ | | In transit | TLS on all connections - Warden to control plane, client to Iggy | | NVMe SSD at rest | Encrypted at the hardware level by the cloud provider | | Network disks at rest | Encryption always enabled (EBS on AWS, Persistent Disk on GCP) | | Custom key encryption | Optional per-deployment encryption with a custom key on top of disk encryption | | Certificate lifecycle | Automated issuance and rotation - no manual intervention | | Audit data | Encrypted at rest, including actor names and event payloads | ## Binary Integrity Warden, Iggy, and Connectors binaries use cryptographic signatures. Before it executes a downloaded binary, Warden makes sure that its signature matches the LaserData public key. It rejects a binary with an invalid signature, including an unsigned replacement from a compromised download channel. ## Task Signing The control plane signs each operational task with Ed25519. Warden makes sure that the signature is valid before execution. This authenticates the task's origin and detects changes in transit. ## Credential Scope ### Warden Tokens Each Warden agent uses an Ed25519-signed token limited to one node. This token grants management API permissions. It is separate from Iggy client credentials used to read or write application data. See [Warden Agent](/deployments/warden). ### Provisioning Credentials #### BYOC Deployments On AWS, LaserData assumes an IAM role for provisioning. The role grants these permissions: | Permission Scope | Purpose | | ---------------- | ---------------------------------- | | EC2 lifecycle | Provisioning and maintenance | | Networking | VPC, subnets, security groups, NLB | | EBS | Storage management | It does not include S3, Secrets Manager, CloudWatch, or SSM. On GCP, LaserData impersonates a service account with these permissions: | Permission Scope | Purpose | | ----------------- | ------------------------------------ | | Compute instances | Provisioning and maintenance | | Networking | VPC, subnets, firewall rules, routes | | IAM | Service account binding to instances | It does not include Cloud Storage, Secret Manager, or Cloud Logging. Neither provider's provisioning permissions grant access to application data. #### Managed Deployments Managed nodes use minimal cloud credentials. Warden uses credentials created for the node during provisioning. Nodes do not receive broad cloud API access. ## Multi-Cloud Consistency Warden uses HTTPS on cloud instances or physical servers. The management model remains the same across providers. It does not depend on cloud-specific management agents. Source: https://docs.laserdata.com/security --- # Authentication Interactive users sign in through an identity provider and receive a session. Sessions support immediate revocation and permission changes. LaserData Cloud stores no user passwords. ## Sign-In Flow To sign in: 1. Click Sign In and select Google, GitHub, or Microsoft. 2. Authenticate with the provider. 3. Return to the Console through the redirect. The provider authenticates your identity. LaserData then creates the session. GitHub, Google, and Microsoft are the available interactive sign-in methods. There is no separate LaserData password account. If you cannot use an allowed provider, [contact LaserData](mailto:hey@laserdata.com) before creating a deployment. The profile form also requests your name, Title / Position, and an organization name when creating a workspace. Student or Hobbyist can describe personal use. The organization name can identify your project. Review the profile form's terms checkbox even when the sign-in page also displays a terms notice. If a welcome email button is unreadable, open [laserdata.cloud](https://laserdata.cloud) directly and sign in. Invitations and protected-resource codes still require their specific invitation link or code. The welcome email is not an alternative credential. ## Cloud Sign-In and Iggy Credentials Your provider account signs you into the Cloud Console. Cloud roles control management access. Cloud API keys authenticate management API calls. Iggy usernames, passwords, and Personal Access Tokens authenticate clients that send or read deployment data. The initial Iggy administrator is named `root`. This name does not grant access to your computer or identify a Cloud organization role. Use the exact username displayed in the deployment Credentials tab. For applications, create an Iggy user with only the required permissions and issue a token for that user. Sign-in accounts are managed through account Settings. AWS and GCP accounts for BYOC belong under [Organization Cloud Accounts](/organization/cloud-accounts). If Console search does not find an account page, use the corresponding navigation entry. ## Sign-Up and Workspace Claim The first user with a custom-domain email, such as `alice@acme.com`, claims the domain for their tenant. The platform stores `acme.com` in the immutable `email_domain` field, which cannot change afterward. Later sign-ups with `*@acme.com` go to that tenant. Its [`join_policy`](/organization#get-tenant) determines the next step. | `join_policy` | Effect on a same-domain signup | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `invite_only` (default) | The user is created but not added as a member. They land on an "ask your admin for an invite" screen. | | `request_to_join` | The user is created and a pending join request is published. The tenant owner is notified. The user lands on a "request sent" screen. | | `open` | The user is auto-joined as a member with the system `viewer` role. The tenant owner is notified. The user lands on the dashboard. | Public email domains, including `gmail.com`, `outlook.com`, and `proton.me`, do not claim a workspace. Those users receive personal tenants. Two protection flags apply when `email_domain` is set. Both are off by default. With `block_external_invitations`, other tenants cannot invite users from the domain, and requests return `400 invitee_domain_locked`. With `enforce_domain_only_invitations`, invitations can target only the tenant's domain or a claimed division subdomain. External invitations then return `400 invitee_domain_not_allowed`. The sign-up response can include `workspace`, which identifies the screen that the UI must show next: | `workspace` value | When it fires | What follows | | --------------------- | ------------------------------------------------- | ----------------------------------------------------------------- | | `joined` | Same-domain signup, policy `open` | The user is a tenant member. Redirect to the dashboard. | | `join_requested` | Same-domain signup, policy `request_to_join` | The user is signed up but not a member. Show "request sent". | | `awaiting_invitation` | Same-domain signup, policy `invite_only` | The user is signed up but not a member. Show "ask for an invite". | | (field absent) | Standard signup (public-email or no domain match) | Existing flow: create personal tenant or follow invitation link. | Sign-up uses the user-session endpoint `POST /account/sign_up`. API keys cannot call it because they belong to an existing tenant. Manage the flags under Tenant Settings, then Workspace, or through the [Tenant Config API](/api/organization#get-tenant-config). ## Session Security Sessions use these protections: | Protection | What It Does | | ----------------- | --------------------------------------------------------------- | | HttpOnly cookie | Prevents JavaScript from reading the session token | | Secure flag | Cookie is only sent over HTTPS | | SameSite policy | Prevents cross-site cookie sending | | CSRF protection | Server-side token validation on all mutating requests | | Encrypted storage | Session data is encrypted at rest | | Token hashing | Raw session tokens are never persisted - only hashes are stored | | Absolute lifetime | Sessions expire after a maximum time regardless of activity | | Sliding expiry | Sessions also expire after a period of inactivity | ### IP Binding (Optional) IP binding limits a session to its original IP address and User-Agent. Users can enable it in their account configuration. It suits stable networks but can interfere with VPN or mobile connections. ## Session Management ### Revoke Sessions Revoke all active sessions through the Console or API. The server deletes their session data immediately. Revoked sessions no longer grant access. ### Session Limits The platform limits concurrent sessions per user. If you reach the limit, revoke existing sessions before creating more. ## Programmatic Access Use [API keys](/security/api-keys) for CI/CD, CLI tools, Terraform, and other software integrations. They use the same [permission model](/organization/roles-permissions) as interactive users. Keys also support IP allowlists and expiry. Source: https://docs.laserdata.com/security/authentication --- # API Keys API keys authenticate software that calls the LaserData Cloud API. Use them for CI/CD, CLI tools, Terraform providers, and other integrations. They follow the same [permission model](/organization/roles-permissions) as interactive sessions. ## How It Works A key follows this sequence: 1. You create it with a role and optional IP restrictions. 2. The platform shows its generated secret once. 3. You send the secret in the `ld-api-key` header on each request. 4. The platform evaluates the key, rate limit, IP allowlist, and role permissions. Save the secret when you create the key. The platform cannot display it again. ## Creating an API Key ### From the Console 1. Open the tenant's API Keys page. 2. Click Create API Key. 3. Enter a key name. 4. Select an existing role, or enter permissions to create a dedicated role for the key. 5. Set an expiration date no more than 365 days away. 6. Click Create. 7. Copy the secret before you leave the page. ### With Inline Permissions Inline permissions define the tenant, division, and environment grants within the request. The platform creates a dedicated role and ties it to the key. ```json { "name": "monitoring-key", "expiry_at": "2026-06-01T00:00:00Z", "permissions": { "tenant": ["info:read", "member:read"], "division": ["environment:read"], "environment": ["deployment:read", "deployment:telemetry:read"] } } ``` `permissions.environment` applies to every environment in every division. Use `permissions.divisions[id].environment` for a division's defaults. Use `permissions.divisions[id].environments[env_id]` for an environment override. [Roles & Permissions](/organization/roles-permissions#how-scoping-works) explains precedence. ## Security | Property | Description | | ------------------ | -------------------------------------------------------- | | High entropy | Long random secret - infeasible to brute-force | | One-way storage | Only the hash is stored - the secret cannot be recovered | | Required expiry | Maximum 365 days, enforced at creation | | Rate limiting | Per-key rate limiter prevents abuse | | IP allowlisting | Optional - restrict the key to specific IP addresses | | Instant revocation | Deleting the key blocks access immediately | ## IP Allowlisting An allowlist restricts a key to selected source IPs. Other addresses receive `403`. You can change an existing key's IP restrictions without recreating it. The account plan limits allowed IPs per key: | Plan | Allowed IPs per key | | ---------- | ------------------- | | Basic | 5 | | Pro | 20 | | Enterprise | 100 | ## Managing API Keys Use the API Keys page for these actions: * Read each key's name, role, expiry, and creation date. * Change its IP allowlist. * Delete it to revoke access immediately. Creation, updates, and deletion require `api_key:manage`. Listing requires `api_key:read`. The [audit log](/observability/audit-compliance) records all API key operations. ## API Reference ### Create an API Key with a Role ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/api_keys \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "ci-deploy-key", "role_id": 67890, "division_id": 123, "expiry_at": "2026-06-01T00:00:00Z", "validate_ip": false }' ``` Supply `role_id` for an existing role or `permissions` for a dedicated role. Do not supply both. The optional `division_id` limits the key to one division. ### Create an API Key with Inline Permissions ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/api_keys \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "monitoring-key", "expiry_at": "2026-06-01T00:00:00Z", "validate_ip": true, "allowed_ips": ["10.0.0.1"], "permissions": { "tenant": ["info:read", "member:read"], "division": ["environment:read"] } }' ``` ### List API Keys ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/api_keys \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "id": 1, "division_id": 123, "division_name": "production", "user_id": 1, "role_id": 67890, "role_name": "deployer", "name": "ci-deploy-key", "validate_ip": false, "allowed_ips": [], "expiry_at": "2026-06-01T00:00:00Z", "created_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Update Security Settings ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/api_keys/{api_key_id}/security \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "validate_ip": true, "allowed_ips": ["10.0.0.1", "192.168.1.0"] }' ``` ### Delete an API Key ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/api_keys/{api_key_id} \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/security/api-keys --- # Monitoring [Warden](/deployments/warden) collects metrics, heartbeats, and logs from each deployment node. It sends telemetry, the records of system activity, to the control plane. You can read it in the Console or through the API. Telemetry goes to LaserData by default. You can send logs and traces to your own OpenTelemetry-compatible endpoint instead. ## Understand a New Deployment Platform services create resources before you send application data. The prober, a service that tests broker writes, sends a message to `_ld/prober` every five seconds by default. Warden and managed services can also appear as connected clients. Stream, topic, client, and message counts therefore do not start at zero or have one fixed initial value. Use Include prober in Metrics where available to separate health traffic from application counts. For your first test, inspect the stream and topic named by the producer. Do not delete platform resources to make a dashboard count zero. ## Which Health Status to Use The cards measure different things. `initialized` is a provisioning state. A recent heartbeat shows that a service reported its condition. All nodes serving summarizes node readiness. The Health Probe card measures the age of the last observed probe write. The current Health Probe card uses these thresholds: | Label | Meaning | | ---------------------- | ---------------------------------------------------------------------- | | Healthy | The last probe is less than 15 seconds old | | Degraded | The last probe is at least 15 seconds old and less than 60 seconds old | | Unhealthy | The last probe is at least 60 seconds old | | Probe data unavailable | Required probe timestamps are missing | A node can report Healthy while the latest probe record is delayed. Read the probe time, node readiness, and logs together. The card calculates age from the browser clock, so make sure that your computer time is correct. If probe age continues increasing, test a send and read through your client and report the result with the deployment ID. ### Shared-Host Capacity and Network Counters On Free deployments, memory and disk values describe your slot limits. They do not describe the entire physical host. Compare them with [Free allocations](/deployments/tiers-storage#free), not a dedicated VM size. Slot telemetry currently reports network RX and TX as zero instead of exposing the shared host counters. A zero Network I/O card therefore does not prove that no messages moved. Broker message counts and process disk writes measure different activity and can increase at the same time. ### Times and Reports Some Console dates use month-first formatting and local time. Do not infer date order from a value such as `9/9/2026`. When reporting a problem, use `YYYY-MM-DD`, include the time and timezone, and include the detailed record timestamp when available. ## Metrics The metrics dashboard reports each node and runtime continuously. A runtime is a managed process, such as Iggy or Connectors. ### What Is Collected | Category | Metrics | | ------------ | ------------------------------------------------------------------------------------------------------ | | System | CPU usage, total CPU usage, memory usage, available memory | | Process | Process ID, run time, start time | | Disk I/O | Bytes read, bytes written | | Iggy | Messages count, messages size, streams count, topics count, partitions count, segments count | | Clients | Connected clients count, consumer groups count | | Iggy runtime | Open file count and limit, thread count, free and total data disk space, message cache hits and misses | Runtime names are `iggy`, `warden`, `connectors`, `plane`, and `connector:{id}` for individual connector instances. On dedicated nodes, `host` reports the virtual machine. Shared-host deployments expose slot-scoped values instead. Runtime-specific fields appear directly in each sample. ### Cluster and Managed Data Health | Area | Signals | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Iggy replication | `role`, `view`, expected and reachable voters, quorum, metadata and partition gaps, repairing and transferring groups | | Managed data plane | Projection bindings and routes, decode failures, KV entries, open forks, projector lag, replay barrier and lag | | Ownership and recovery | Owned, expected, and standby partitions, fence expiry and rejections, destination watermark, restore and synchronization state | | Host | VM CPU and memory, load, swap, disk capacity and I/O, network receive and transmit counters | | Shared-host quota | `cpu_quota_cores`, `cpu_usage_pct_of_quota`, `cpu_throttled_periods`, `cpu_throttled_usec` | | Connector pipeline | Produced, sent, consumed, processed, filtered messages, and errors | A heartbeat shows recent liveness. It does not prove that a node caught up or can serve managed requests. Read `agent_ready`, `iggy_ready`, `plane_ready`, `last_probe_at`, and `serving` from the node readiness endpoint. If a runtime omits an optional metric, treat its value as unknown. For plane `disk_free_bytes`, the maximum unsigned 64-bit value means that no disk sample exists. It does not mean unlimited capacity. Read host and process utilization separately. On shared hosts, increasing throttling counters and high quota utilization can explain latency even with low usage across the whole machine. ### Viewing Metrics 1. Open your deployment in the Console. 2. Open the Metrics tab. 3. Select the node and runtime. 4. Select the time range. ## Heartbeats Warden sends periodic health reports for managed runtimes such as Iggy and Connectors. You can read them by node, runtime, and historical time range. Missing reports indicate a possible runtime problem. ## Logs Iggy logs include internal replication and connection events. Messages about groups, views, or disconnected consensus clients do not by themselves prove that application data failed. Select the affected node, runtime, level, and time range. Compare repeated warnings or errors with probe age and a client send-and-read test. Warden collects logs from all runtimes and nodes. The platform stores them for searches. Large result sets use pages. ### Searching Logs Filter logs by these fields: | Filter | Description | | ---------- | --------------------------------------------------- | | Node | Filter to a specific node | | Runtime | Filter by runtime (Iggy, Connectors, Warden) | | Level | Filter by log level (any, debug, info, warn, error) | | Message | Pattern match on log message content | | Scope | Filter by component | | Time range | Start and end timestamps | ### Viewing Logs 1. Open your deployment in the Console. 2. Open the Logs tab. 3. Select filters for the records that you need. 4. Read further pages for additional results. ## Log Redirection (OpenTelemetry) LaserData stores logs by default. To keep them in your own system, configure an OpenTelemetry-compatible destination through the Console or support. This can support your privacy, compliance, or monitoring requirements. Warden sends logs with `ExportLogsServiceRequest`. Compatible collectors and storage systems include Grafana Loki, Datadog, Elastic, and custom pipelines. Traces use `ExportTracesServiceRequest`. ## Monitoring Alerts The platform watches deployment health and sends alerts through [notification channels](/observability/notifications). An alert starts when a threshold is crossed. A resolution event follows when the condition clears. | Alert | Threshold | Description | | ------------------- | ------------------------ | ---------------------------------- | | `high_cpu_usage` | CPU ≥ 80% (5-min window) | Sustained high CPU usage on a node | | `high_memory_usage` | Memory ≥ 90% | Memory usage approaching capacity | | `high_disk_usage` | Disk ≥ 80% | Disk usage approaching capacity | | `node_unreachable` | No heartbeat for 120s | Node stopped reporting heartbeats | Resolution events include `cpu_usage_resolved`, `memory_usage_resolved`, `disk_usage_resolved`, and `node_reachable`. The platform avoids repeated alerts for a condition that remains active. To receive alerts, create a [channel](/observability/notifications) and subscribe to its event types. ## Telemetry Retention Metrics, heartbeats, and logs share a retention period. Managed deployments include these periods: | Deployment | Included retention | | ----------- | ------------------ | | Free | 7 days | | Standard | 14 days | | Performance | 30 days | | Enterprise | 90 days | New deployments use the tier's included period. The platform rejects requests above that allowance and does not charge for extra telemetry days. For BYOC and deployments without managed tiers, the allowance follows Compute. Free and Small include 7 days, Medium and Large 30 days, and XLarge or larger 90 days. Set `retention.telemetry_days` through the [deployment API](/api/deployments#update-retention). Audit, snapshot, and backup retention are separate. Reading telemetry requires `deployment:telemetry:read`. Changing retention requires `deployment:telemetry:manage`. ## API Reference ### Get Deployment Metrics ```bash curl {supervisor_url}/deployments/{deployment_id}/metrics \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Node Metrics (by runtime) ```bash curl "{supervisor_url}/deployments/{deployment_id}/nodes/{node_id}/metrics/iggy?page=1&results=10&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Deployment Heartbeats ```bash curl {supervisor_url}/deployments/{deployment_id}/heartbeats \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Node Heartbeats (by runtime) ```bash curl "{supervisor_url}/deployments/{deployment_id}/nodes/{node_id}/heartbeats/iggy?page=1&results=10&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Deployment Logs ```bash curl "{supervisor_url}/deployments/{deployment_id}/logs/iggy?page=1&results=10&level=any&message=*&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \ -H "ld-api-key: YOUR_API_KEY" ``` Source: https://docs.laserdata.com/observability --- # Notifications Notifications send deployment events, resource changes, and operational alerts to Slack, webhooks, or email. A channel names a destination. Subscriptions choose which events it receives. ## How It Works A channel belongs to a tenant or division. Its subscriptions filter event types and resource scope, the resources that an event concerns. When an enabled channel has a matching subscription, the platform sends the event to its destination. ## Notification Channels Each channel has a name, delivery kind, and destination URL or email address. The platform encrypts destinations at rest. ### Channel Kinds | Kind | Destination Format | Description | | --------- | ------------------ | ------------------------------------------------------------------------ | | `slack` | HTTPS webhook URL | Sends a formatted message to a Slack channel via incoming webhook | | `webhook` | HTTPS URL | Sends a JSON payload (`{"subject": "...", "body": "..."}`) via HTTP POST | | `email` | Email address | Sends an HTML email with subject and body | Slack and webhook destinations must use HTTPS. They cannot target private or internal IP addresses. This prevents the notification service from accessing private network targets through supplied URLs. ### Channel Scoping Tenant channels apply across the organization. Division channels belong to one division. Subscriptions can further limit the resources whose events reach a channel. ### Channel Settings Slack and webhook channels support these optional fields. Slack: ```json { "slack": { "channel": "#alerts", "username": "LaserData Bot", "icon_emoji": ":bell:" } } ``` Webhook: ```json { "webhook": { "headers": { "Authorization": "Bearer token123" }, "method": "POST" } } ``` ## Notification Subscriptions A subscription selects one or more event types. Its optional scope filters limit the resources that can trigger delivery. ### Event Types | Event Type | Description | | --------------------------------- | ------------------------------------------------------------------------------ | | Deployments | | | `deployment_created` | A new deployment was created | | `deployment_initialized` | A deployment finished initializing and is ready | | `deployment_upgraded` | A deployment was upgraded to a new tier or storage | | `deployment_deleted` | A deployment was deleted | | `deployment_certificates_rotated` | Deployment TLS certificates were rotated | | `deployment_secrets_rotated` | Deployment secrets were rotated | | Invitations | | | `invitation_created` | A new team invitation was sent | | `invitation_accepted` | A team member accepted an invitation | | `invitation_rejected` | A team member rejected an invitation | | Tenant | | | `tenant_config_updated` | Workspace settings (join policy, invitation locks) were changed | | `tenant_join_requested` | A same-domain user requested to join the tenant under `request_to_join` policy | | `member_joined` | A same-domain user auto-joined the tenant under `open` policy | | Organization | | | `division_created` | A division was created | | `division_updated` | A division was updated | | `division_deleted` | A division was deleted | | `environment_created` | An environment was created | | `environment_updated` | An environment was updated | | `environment_deleted` | An environment was deleted | | Health Alerts | | | `high_cpu_usage` | CPU usage exceeded threshold on a node | | `high_memory_usage` | Memory usage exceeded threshold on a node | | `high_disk_usage` | Disk usage exceeded threshold on a node | | `node_unreachable` | A deployment node or connector instance is unreachable | | `cpu_usage_resolved` | CPU usage returned to normal | | `memory_usage_resolved` | Memory usage returned to normal | | `disk_usage_resolved` | Disk usage returned to normal | | `node_reachable` | A previously unreachable node or connector instance is back online | | Other | | | `certificate_expiring` | A TLS certificate is about to expire | | `billing_limit_reached` | A deployment's spend limit was reached | `node_unreachable` and `node_reachable` apply to Warden, Iggy, the Connectors host, and individual connector instances. Connector events include `connector_instance_name` and `connector_key`. The subject therefore uses `connector '' (, instance ID: )`. Heartbeats for a deleted connector instance do not raise false `node_unreachable` alerts. ### Scope Filtering An event must match every scope level that a subscription specifies. The levels combine with AND. An empty or omitted array matches every resource at that level. | Scope | Description | | ----------------------- | ----------------------------------- | | `scope_tenant_ids` | Only events from these tenants | | `scope_division_ids` | Only events from these divisions | | `scope_environment_ids` | Only events from these environments | | `scope_deployment_ids` | Only events from these deployments | For example, `scope_deployment_ids: [611298765432109056]` selects deployment `611298765432109056`. With no other scope filters, its matching event types are delivered regardless of division or environment. ### Subscription Behavior Delivery follows these rules: * A channel without subscriptions receives all events by default. * A channel with subscriptions receives events that match at least one subscription. * Throttling applies per channel, event type, and resource to limit repeated notifications. ## Permissions | Scope | Read | Manage | | -------- | -------------------- | ---------------------- | | Tenant | `notifications:read` | `notifications:manage` | | Division | `notifications:read` | `notifications:manage` | `manage` includes `read`. See [Roles & Permissions](/organization/roles-permissions). ## Plan Limits | Resource | Basic | Pro | Enterprise | | ---------------------------------- | ----- | --- | ---------- | | Notification channels (per tenant) | 1 | 3 | 20 | | Subscriptions per channel | 3 | 10 | 20 | ## API Reference ### Create a Channel (Tenant) ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/channels \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "channel": "slack", "name": "production-alerts", "destination": "https://hooks.slack.com/services/T00/B00/xxx", "settings": { "slack": { "channel": "#alerts", "username": "LaserData" } }, "remarks": "Primary alerting channel" }' ``` | Field | Required | Description | | ------------- | -------- | ---------------------------------------------------------------------- | | `channel` | Yes | Channel kind: `slack`, `webhook`, `email` | | `name` | Yes | Unique name (1-100 chars, alphanumeric with `-`, `_`, `.`, `:`, space) | | `destination` | Yes | Target URL or email (1-1000 chars) | | `settings` | No | Channel-specific settings (see [Channel Settings](#channel-settings)) | | `remarks` | No | Notes (max 500 chars) | A successful request returns `201 Created`. ### Create a Channel (Division) ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/channels \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "channel": "webhook", "name": "ops-webhook", "destination": "https://example.com/webhooks/laserdata" }' ``` Use the same body as for a tenant channel. ### List Channels ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/channels?page=1&results=10&channel=slack" \ -H "ld-api-key: YOUR_API_KEY" ``` Filter the list with these query parameters: | Parameter | Type | Description | | --------- | ------- | ------------------------------------------------------ | | `page` | integer | Page number (optional, default 1) | | `results` | integer | Results per page (optional, default 10, max 100) | | `channel` | string | Filter by kind: `slack`, `webhook`, `email` (optional) | The response has this format: ```json { "total_pages": 1, "total_results": 2, "page": 1, "items": [ { "id": 1, "owner_kind": "tenant", "owner_id": 100, "channel": "slack", "name": "production-alerts", "enabled": true, "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ] } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels`. ### Get Channel Details ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "owner_kind": "tenant", "owner_id": 100, "channel": "slack", "name": "production-alerts", "enabled": true, "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z", "destination": "https://hooks.slack.com/services/T00/B00/xxx", "settings": { "slack": { "channel": "#alerts", "username": "LaserData" } }, "remarks": "Primary alerting channel" } ``` Details include `destination`, `settings`, and `remarks`. List responses omit these fields. For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. ### Update a Channel ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id} \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "updated-channel-name", "destination": "https://hooks.slack.com/services/T00/B00/new", "enabled": false }' ``` Include only fields that you want to change. Set `enabled` to `false` to disable delivery without deleting the channel. A successful request returns `204 No Content`. For a division, use `PUT /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. ### Delete a Channel Deleting a channel also permanently deletes its subscriptions. This cannot be undone. ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. For a division, use `DELETE /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. ### Test a Channel Send a test message to make sure that the destination receives notifications: ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/test \ -H "ld-api-key: YOUR_API_KEY" ``` A channel permits one test per 10 seconds. A successful request returns `204 No Content`. For a division, use `POST /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/test`. ### Create a Subscription ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/subscriptions \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "message_types": ["deployment_created", "deployment_deleted", "node_unreachable"], "scope_division_ids": [1], "scope_environment_ids": [10, 20] }' ``` | Field | Required | Description | | ----------------------- | -------- | -------------------------------------------------------------- | | `message_types` | Yes | Non-empty array of [event types](#event-types) to subscribe to | | `scope_tenant_ids` | No | Filter to events from specific tenants | | `scope_division_ids` | No | Filter to events from specific divisions | | `scope_environment_ids` | No | Filter to events from specific environments | | `scope_deployment_ids` | No | Filter to events from specific deployments | Scope IDs must refer to resources in the tenant. A successful request returns `201 Created`. For a division, use `POST /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### Set Subscriptions (Replace All) Replace all channel subscriptions in one transaction: ```bash curl -X PUT https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/subscriptions \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "subscriptions": [ { "message_types": ["deployment_created", "deployment_deleted"], "scope_division_ids": [1] }, { "message_types": ["high_cpu_usage", "high_memory_usage", "high_disk_usage"], "scope_deployment_ids": [611298765432109056, 611298765432109057] } ] }' ``` The supplied list replaces every existing subscription for the channel. Its size must stay within `notification_subscriptions_limit`. A successful request returns `204 No Content`. For a division, use `PUT /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### List Subscriptions ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/subscriptions?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` Filter the list with these query parameters: | Parameter | Type | Description | | -------------- | ------- | ------------------------------------------------ | | `page` | integer | Page number (optional, default 1) | | `results` | integer | Results per page (optional, default 10, max 100) | | `message_type` | string | Filter by event type (optional) | The response has this format: ```json { "total_pages": 1, "total_results": 2, "page": 1, "items": [ { "id": 1, "channel_id": 100, "message_types": ["deployment_created", "deployment_deleted"], "scope_tenants": [{ "id": 1, "name": "Acme Corp" }], "scope_divisions": [{ "id": 10, "name": "Platform Eng" }], "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ] } ``` Empty scope arrays are omitted. Each scope entry includes `id` and `name` from the database. For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### Get Subscription Details ```bash curl https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/subscriptions/{subscription_id} \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "id": 1, "channel_id": 100, "message_types": ["deployment_created", "deployment_deleted"], "scope_tenants": [{ "id": 1, "name": "Acme Corp" }], "scope_divisions": [{ "id": 10, "name": "Platform Eng" }], "scope_environments": [{ "id": 100, "name": "Production" }], "scope_deployments": [{ "id": 611298765432109056, "name": "prod-cluster" }], "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions/{subscription_id}`. ### Delete a Subscription ```bash curl -X DELETE https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/subscriptions/{subscription_id} \ -H "ld-api-key: YOUR_API_KEY" ``` A successful request returns `204 No Content`. For a division, use `DELETE /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions/{subscription_id}`. ### Get Notification Types Retrieve the available event types: ```bash curl https://api.laserdata.cloud/notifications/types \ -H "ld-api-key: YOUR_API_KEY" ``` ```json [ { "type": "deployment_created", "name": "Deployment created" }, { "type": "deployment_initialized", "name": "Deployment initialized" }, { "type": "high_cpu_usage", "name": "High CPU usage" }, { "type": "node_unreachable", "name": "Node unreachable" } ] ``` ### Browse Notifications Read a channel's notification history: ```bash curl "https://api.laserdata.cloud/tenants/{tenant_id}/channels/{channel_id}/notifications?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` Filter the history with these query parameters: | Parameter | Type | Description | | -------------- | ------- | ------------------------------------------------ | | `page` | integer | Page number (optional, default 1) | | `results` | integer | Results per page (optional, default 10, max 100) | | `message_type` | string | Filter by event type (optional) | ```json { "total_pages": 1, "total_results": 5, "page": 1, "items": [ { "id": 1, "tenant_id": 100, "division_id": 10, "environment_id": 1, "deployment_id": 611298765432109056, "message_type": "deployment_initialized", "content": "Deployment prod-cluster has been initialized", "created_at": "2025-06-01T10:30:00Z" } ] } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/notifications`. Source: https://docs.laserdata.com/observability/notifications --- # Audit & Compliance Audit logs record changes to platform resources. Access logs record reads of sensitive data. Together with security events and data-protection controls, they support investigations and compliance work. ## Audit Logging Each operation that changes state creates an immutable audit record. Users cannot change or delete these records. Audit data remains separate for each tenant. ### What Is Logged Audit records include these details: * Each create, update, or delete operation. * The actor's user ID and name. * The tenant, division, environment, and deployment involved. * Previous and new values for updates. * The time of the operation. ### Viewing Audit Logs 1. Open the tenant or deployment in the Console. 2. Open the Audit tab. 3. Browse or search its history. ### Tenant Isolation The platform stores each tenant's audit data separately. One tenant cannot read another tenant's records. ### Encryption The platform encrypts audit data at rest, including actor names, resource names, and event payloads. ## Access Logs Access logs identify reads of sensitive information. They record the request ID, actor, action, IP address, and User-Agent. The recorded reads include these: * User profiles. * Member lists. * Invitations. * Audit logs. * Personal data exports. ## Security Events Permission-denied events record attempts to perform unauthorized actions. Authentication-failure events record failed sign-ins, their reasons, and request details. Use these events to investigate unusual activity and incidents. ## Data Protection (GDPR) ### Encryption at Rest The platform encrypts personally identifiable information, data that identifies a person, at rest. This includes emails, names, identity-provider external IDs, invitation emails, and personal information in audit fields. One-way email hashes support account lookups. The platform can find an account without decrypting every record. ### Data Export Users can export their personal data as JSON through the Console or API. The export contains these records: * Profile information. * Identity-provider connections. * Active sessions. * Tenant memberships. * Pending invitations. * Account configuration. ### Right to Erasure Deleting a user account removes its identities, memberships, and invitations. The platform retains audit logs for security under legitimate interest, as permitted by GDPR. ### Audit Retention | Resource | Basic | Pro | Enterprise | | ------------------- | ------ | ------- | ---------- | | Audit log retention | 7 days | 30 days | 365 days | Reading audit logs requires tenant-level `audit:read`. ## API Reference ### Get Audit Event Types Retrieve the event types available for filtering: ```bash curl https://api.laserdata.cloud/audit/types \ -H "ld-api-key: YOUR_API_KEY" ``` ### Get Tenant Audit Logs ```bash curl "https://api.laserdata.cloud/audit/tenants/{tenant_id}?page=1&results=10" \ -H "ld-api-key: YOUR_API_KEY" ``` ```json { "items": [ { "type": "deployment_created", "name": "Deployment Created", "author": { "id": 608123456789012345, "name": "Jane Smith" }, "api_key": { "id": 67890, "name": "ci-deploy-key" }, "division": { "id": 615380456123456790, "name": "Platform Engineering" }, "environment": { "id": 1, "name": "production" }, "deployment": { "id": 611298765432109056, "name": "events-prod" }, "timestamp": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` Each entry identifies a person in `author`. Automated calls also identify their `api_key`. Both use `{id, name}` pairs. Filter with `&author={user_id}` for a person or `&api_key={api_key_id}` for a key. See the [API reference](/api/audit) for the complete response and filters. User activity history and personal data export require a user session. Open them from account configuration in the Console. Tenant-scoped API keys cannot read user-account data. Source: https://docs.laserdata.com/observability/audit-compliance --- # Official LaserData CLI `laser` is the official LaserData CLI. It manages deployments, networking, roles, and monitoring from a terminal. The static binary supports macOS arm64 and Linux x86\_64 or arm64. Commands run headless by default, without an interactive interface. Examples include `laser tenant get` and `laser deployment list`. They return exit code `0` on success and a non-zero code on failure. Use `laser tui` or `--interactive` for the [Ratatui](https://ratatui.rs) terminal dashboard. It adds mouse and keyboard control, a command palette, and saved history. Both modes share configuration and credentials. The CLI uses a tenant-scoped [API key](/security/api-keys), rather than a user session. Give that key the [roles and permissions](/organization/roles-permissions) needed for its work. Use [laser-sdk](/laser-sdk) from Rust, Python, or TypeScript for application data, such as publishing, queries, and agents. ## When Agents Should Use laser Use `laser` when an agent needs a command, JSON output, and a process exit status. It suits deployment lists, operational status, reviewed configuration changes, and repeatable CI tasks. Use the [REST API](/api) for many calls, custom retry or concurrency behavior, or direct model function calls. Use [laser-sdk](/laser-sdk) for publishing, queries, semantic memory, and agent runtime features. For automation, supply `LD_API_KEY` and `LD_TENANT_ID`. Select `--output json` or `-o json` and inspect the exit code. Avoid interactive commands such as `laser tui`. Start with read-only commands. Obtain explicit approval before creating, changing, or deleting resources. The API contract is available at [`laserdata.com/openapi.json`](https://laserdata.com/openapi.json). ## Install ```bash curl -fsSL https://cli.laserdata.cloud/install.sh | sh ``` The installer detects the operating system and architecture. It downloads the matching archive from the [release repository](https://github.com/laserdata/laser-cli-releases) and makes sure that its SHA-256 checksum matches. It installs `laser` in `$HOME/.local/bin`, or `$LD_PREFIX` when set. Run it again to upgrade. ### Installer Flags Pass installer flags with `sh -s --`: | Flag | Effect | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `--to ` | Install dir (default `$HOME/.local/bin`). Alternative to `LD_PREFIX`. | | `--version ` | Pin a release tag (default `latest`). Alternative to `LD_VERSION`. | | `--with-cc-skills` | Also install/update the Claude Code skill pack from `cli.laserdata.cloud/claude.sh`. See [Claude Code Skills](/cli/claude-skills). | | `--help` | Show usage. | ### Environment Variables | Var | Effect | | ------------ | --------------------- | | `LD_VERSION` | Pin release tag. | | `LD_PREFIX` | Override install dir. | ### Common Recipes ```bash # Pin a version. curl -fsSL https://cli.laserdata.cloud/install.sh | sh -s -- --version v0.2.1 # System-wide install dir. curl -fsSL https://cli.laserdata.cloud/install.sh | sh -s -- --to /usr/local/bin # Bundle Claude Code skills in the same step. curl -fsSL https://cli.laserdata.cloud/install.sh | sh -s -- --with-cc-skills # All three at once. curl -fsSL https://cli.laserdata.cloud/install.sh \ | sh -s -- --version v0.2.1 --to /usr/local/bin --with-cc-skills ``` Repeated installation replaces the binary and skill pack in place, using `cp -f` for skills. You can use it in CI or dotfiles. ### In-Place Upgrade ```bash laser update ``` This downloads the latest release and replaces the binary atomically, as one operation. ### Verify ```bash laser version ``` ## Sign In ```bash laser auth login --tenant-id ``` Login prompts for the API key with masked input. It stores the secret in the OS keyring, the operating system's secret store. macOS uses Keychain, and Linux uses Secret Service. Only the account name is stored on disk outside the keyring. The CLI calls `GET /tenants//api_keys/context` and displays the resolved role on success. A non-2xx response indicates a revoked, expired, or wrong-tenant key. The first login needs a tenant ID. Resolution uses `--tenant-id `, then `LD_TENANT_ID`, then the saved context's `tenant_id`. Login fails if none is available. For agents, scripts, and CI, supply credentials through the environment: ```bash export LD_API_KEY=ld_pat_... export LD_TENANT_ID=615380456123456789 ``` `LD_API_KEY` takes precedence over the keyring, so CI needs no separate login. Use `laser tenant key context -o json` to inspect the active key's role and permissions. ### Local State State uses XDG application directories on Linux or the platform equivalent on macOS: | Path | Purpose | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | | `$XDG_CONFIG_HOME/laser/config.toml` | Named contexts (mode `0600`, atomic writes). Stores account names only. Secrets stay in the OS keyring. | | `$XDG_STATE_HOME/laser/active` | Pointer to the currently active context. | | `$XDG_DATA_HOME/laser/history` | TUI command history. Capped at 1000 entries with atomic writes. | | `$XDG_DATA_HOME/laser/logs/laser-YYYY-MM-DD.log` | Rolling daily debug log. Written for every invocation, including `--silent` and `--quiet` runs. | Set file-log verbosity with `LD_LOG_FILE` and standard-error verbosity with `LD_LOG`. Both accept `tracing` directives, such as `LD_LOG_FILE=trace`. ## Contexts A context saves an optional `tenant_id` and an API key reference in the keyring. Use contexts to switch tenants without entering credentials again: ```bash laser context create prod laser context list laser context switch prod # alias: laser ctx use prod laser context current laser context show prod laser context set tenant_id 615380456123456789 laser context rename prod prod-eu laser context delete prod-eu ``` For one command, select a context with `--context NAME` or `LD_CONTEXT=NAME`. ## Output Formats Choose a format with `-o` or `--output`: | Format | Default for | Notes | | ------- | -------------- | -------------------------------------------------------------------------------- | | `table` | TTY | Aligned ASCII table. Colored unless `NO_COLOR` is set or `--no-color` is passed. | | `json` | Pipe / non-TTY | Machine-readable. Stable schema. | | `yaml` | - | YAML 1.2. | | `name` | - | IDs and names only. Useful in `xargs` pipelines. | The default is `table` on a TTY, an interactive terminal, and `json` when output is piped. Use `-o` to override it for one call. ## Global Flags | Flag | Env | Purpose | | --------------------- | ------------ | --------------------------------------------------------------------------------------- | | `--context NAME` | `LD_CONTEXT` | Use a specific named context for this invocation. | | `--api-key KEY` | `LD_API_KEY` | Override the context API key. | | `-o, --output FORMAT` | - | `table`, `json`, `yaml`, or `name`. | | `--no-color` | `NO_COLOR` | Disable colored output. | | `-q, --quiet` | - | Suppress non-essential output. | | `--debug` | - | Enable debug logging to stderr. | | `--silent` | - | Suppress success output entirely. Errors still go to stderr. Exit code preserved. | | `--interactive` | - | Run the verb inside the TUI dashboard with rendered output. | | `--config PATH` | - | Use an alternate config file. | | `--yes` | - | Skip confirmation prompts for destructive operations. | | `--generate SHELL` | - | Print shell completion script (`bash`, `zsh`, `fish`, `powershell`, `elvish`) and exit. | ## Quick Start ```bash # install + login curl -fsSL https://cli.laserdata.cloud/install.sh | sh laser auth login --tenant-id # explore your tenant laser tenant get laser tenant structure -o json # launch the dashboard laser tui # spin up a starter deployment (AWS or GCP) laser deployment create-starter --cloud aws --region us-east-1 laser deployment create-starter --cloud gcp --region us-central1 # follow it to ready laser deployment watch --deployment-id ``` ## Related pages Source: https://docs.laserdata.com/cli --- # Command Reference This page lists `laser` commands. Use `laser --help` for flags. The setup commands `auth`, `context`, `version`, `update`, and `tui` work without an active context. Other commands require one. ## Auth ```bash laser auth login [--context NAME] [--api-key KEY] [--tenant-id ID] laser auth logout [--context NAME] [--all] laser auth purge # destructive: drops every context and keyring entry ``` Without `--api-key`, `login` prompts for a masked secret. It authenticates the key with the platform and saves it in the OS keyring, the operating system's secret store. ## Context ```bash laser context create NAME [--api-key KEY] [--tenant-id ID] [--division-id ID] [--environment-id ID] [--deployment-id ID] [--activate] laser context list laser context current laser context show [NAME] laser context switch NAME # alias: laser ctx use NAME laser context delete NAME laser context rename FROM TO laser context set FIELD VALUE # tenant_id | division_id | environment_id | deployment_id ``` `ctx` is an alias for `context`. A context stores the tenant and credential reference used by commands. ## Tenant `tn` is the tenant alias. Commands use the active context's tenant unless you supply `--tenant-id`. ```bash laser tenant get laser tenant structure # full tree: divisions -> environments -> deployments laser tenant summary # aggregate counts laser tenant update [--name NAME] [--description TEXT] [--email EMAIL] [--protected true|false] ``` ### Workspace Config ```bash laser tenant config get laser tenant config update [--join-policy invite_only|open|request_to_join] [--block-external-invitations true|false] [--enforce-domain-only-invitations true|false] ``` This displays the join policy, invitation restrictions, and claimed email domain when present. The join policy controls sign-ups that match the tenant's domain. Invitation restrictions apply only when `email_domain` is set. ### API Keys ```bash laser tenant key list [--name SUBSTRING] [--page N] [--results N] laser tenant key get --api-key-id ID laser tenant key create [--name NAME] [--role-id ID] [--division-id ID] [--expires-in-days N] [--validate-ip true|false] [--allowed-ip IP ...] laser tenant key context laser tenant key update-security --api-key-id ID [--validate-ip true|false] [--allowed-ip IP ...] laser tenant key delete --api-key-id ID ``` `tenant key get` returns a key's details and resolved permissions. It requires `api_key:read`. `tenant key context` returns the same structure for the calling key, without another permission. Repeat `--allowed-ip` for multiple addresses. On an interactive terminal, omitted `--name`, `--role-id`, and `--expires-in-days` values prompt for input. Save a new key's secret when it appears. It is displayed only once. ### Cloud Accounts ```bash laser tenant cloud-account list [--page N] [--results N] laser tenant cloud-account get --cloud-account-id ID laser tenant cloud-account create [--cloud aws|gcp] [--name NAME] [--account-id ID] [--region REGION] [--remarks TEXT] laser tenant cloud-account update --cloud-account-id ID [--name NAME] [--account-id ID] [--region REGION] [--remarks TEXT] [--status active|inactive] laser tenant cloud-account delete --cloud-account-id ID ``` `update` changes only the flags you pass. ### Members and Invitations ```bash laser tenant member list # active only laser tenant member all # include inactive laser tenant member permissions # list available permissions laser tenant member update --member-id ID [--active true|false] [--role-id ID ...] laser tenant member delete --member-id ID laser tenant invitation list # alias: inv laser tenant invitation create [--email EMAIL] [--message TEXT] [--role-id ID ...] laser tenant invitation cancel --invitation-id ID ``` Repeat `--role-id` to assign multiple roles in one call. ### Roles ```bash laser tenant role list [--page N] [--results N] laser tenant role get --role-id ID laser tenant role create --name NAME [--description TEXT] \ [--tenant-permission KEY ...] \ [--division-permission KEY ...] \ [--environment-permission KEY ...] laser tenant role update --role-id ID [--name NAME] [--description TEXT] \ [--tenant-permission KEY ...] \ [--division-permission KEY ...] \ [--environment-permission KEY ...] laser tenant role delete --role-id ID laser tenant role members --role-id ID laser tenant role assign --role-id ID --member-id ID laser tenant role revoke --role-id ID --member-id ID ``` Each `--*-permission` flag is repeatable and maps to an API scope: * `--tenant-permission` sets `permissions.tenant`. * `--division-permission` sets `permissions.division` for every division. * `--environment-permission` sets `permissions.environment` for every environment in every division. The CLI does not expose nested overrides in `permissions.divisions`. Use the [API](/api/members#create-role) for those. Run `laser tenant member permissions` to list valid permission keys. ### Billing ```bash laser tenant billing info laser tenant billing reports [--page N] [--results N] laser tenant billing report --report-id ID laser tenant billing invoices [--page N] [--results N] laser tenant billing invoice --invoice-id ID laser tenant billing invoice-pdf --invoice-id ID --output FILE.pdf laser tenant billing update-info [--name NAME] [--company NAME] [--address TEXT] [--city CITY] [--state STATE] \ [--postal-code CODE] [--country CC] [--tax-id ID] [--email EMAIL] [--phone PHONE] laser tenant billing payment-method laser tenant billing remove-payment-method [--yes] laser tenant billing credits laser tenant billing redeem --code CODE laser tenant marketplace agreements --cloud aws|gcp ``` `update-info` changes only the flags you pass, and needs a name or a company. A redeemed promo code becomes a credit. Credits reduce later invoices, oldest first, and an invoice total never drops below zero. Adding a card, and registering or linking a marketplace purchase, need the [console](https://laserdata.cloud) because they run in a browser. ## Division `div` is the division alias. ```bash laser division list [--page N] [--results N] laser division get --division-id ID laser division summary --division-id ID laser division create [--name NAME] [--description TEXT] [--email EMAIL] laser division update --division-id ID [--name NAME] [--description TEXT] [--email EMAIL] [--protected true|false] laser division delete --division-id ID [--code CODE] # protected divisions need --code ``` For `get`, `update`, and `delete`, an omitted `--division-id` uses the context's `division_id`. ## Environment `env` is the environment alias. ```bash laser environment list --division-id ID [--page N] [--results N] laser environment get --environment-id ID laser environment create --division-id ID [--name NAME] [--description TEXT] laser environment update --environment-id ID [--name NAME] [--description TEXT] [--protected true|false] laser environment delete --environment-id ID [--code CODE] ``` Where applicable, omitted `--division-id` and `--environment-id` values use the active context. ## Deployment `dep` is the deployment alias. Commands cover provisioning, lifecycle, configuration, snapshots, backups, networking, and monitoring. Subcommands accept `--tenant-id`, `--division-id`, `--environment-id`, and `--deployment-id`. An omitted flag uses the corresponding context value. ### Lifecycle ```bash laser deployment list [--environment-id ID] [--page N] [--results N] laser deployment get [--deployment-id ID] laser deployment update [--deployment-id ID] [--description TEXT] [--protected true|false] laser deployment delete [--deployment-id ID] [--code CODE] laser deployment extend [--deployment-id ID] --add-nodes N [--add-nodes N ...] laser deployment upgrade [--deployment-id ID] [--managed-tier TIER] [--tier SIZE] [--storage-type TYPE] [--storage-size-gb N] laser deployment retention [--deployment-id ID] --telemetry-days N laser deployment spend-limit [--deployment-id ID] [--spend-limit USD | --clear] laser deployment watch [--deployment-id ID] [--interval-secs N] [--timeout-mins N] ``` The CLI accepts `extend`, but the backend currently rejects node additions with `cluster_extension_not_allowed`. Use supported compute or storage upgrades instead. `watch` polls until the deployment is ready or fails. Its default interval is 5 seconds, limited to 2-60 seconds. Its default timeout is 60 minutes. ### Provisioning ```bash laser deployment preview \ [--cloud aws|gcp] [--region REGION] [--managed-tier standard|performance|enterprise] [--tier SIZE] \ [--storage-type network_balanced|local_ssd] [--storage-size-gb N] [--target-network-tput KBPS] \ [--network-scope same_region|cross_region_same_cloud|cross_cloud] [--availability-mode single_az|multi_az] laser deployment create-managed \ [--name NAME] [--cloud aws|gcp] [--region REGION] \ [--managed-tier standard|performance|enterprise] [--tier SIZE] [--cluster standalone|cluster] \ [--storage-type network_balanced|local_ssd] [--storage-size-gb N] [--target-network-tput KBPS] \ [--network-scope same_region|cross_region_same_cloud|cross_cloud] [--availability-mode single_az|multi_az] \ [--protected true|false] [--encrypted true|false] [--dedicated true|false] \ [--public-ip-enabled true|false] [--subdomain-enabled true|false] \ [--telemetry-days N] [--spend-limit USD] laser deployment create-byoc \ [--name NAME] [--cloud aws|gcp] [--region REGION] [--tier SIZE] [--cluster standalone|cluster] \ [--storage-type network_balanced|local_ssd] [--storage-size-gb N] [--availability-mode single_az|multi_az] \ [--protected true|false] [--encrypted true|false] \ [--public-ip-enabled true|false] [--subdomain-enabled true|false] \ [--telemetry-days N] [--spend-limit USD] \ [--aws-account-id ID] [--aws-identity-arn ARN] [--aws-external-id ID] [--aws-vpc-id ID] [--aws-vpc-cidr CIDR] \ [--gcp-project-id ID] [--gcp-service-account-email EMAIL] [--gcp-vpc-name NAME] laser deployment create-starter \ [--cloud aws|gcp] [--region REGION] \ [--environment-id ID | --environment-name NAME] [--deployment-name NAME] ``` `preview` evaluates the configuration and returns monthly pricing, estimated transfer, and the total without creating resources. Its technical capacity metadata is not a guaranteed partition limit. Paid `create-managed` requests require `--managed-tier` and `--cluster cluster`. `--tier` selects Compute size, such as `small` or `large`. `--target-network-tput` sets the throughput estimate in KB/s. Omitted values use tier defaults. See [Tiers & Storage](/deployments/tiers-storage). On an interactive terminal, missing required flags prompt for input. Use `create-starter` to create a Free deployment for testing. ### Configs ```bash laser deployment config list [--deployment-id ID] --kind iggy|plane|connectors|warden|connector laser deployment config primary [--deployment-id ID] --kind iggy|plane|connectors|warden|connector laser deployment config schema [--deployment-id ID] --kind iggy|plane|connectors|warden|connector laser deployment config versions [--deployment-id ID] --kind ... [--name NAME] laser deployment config version [--deployment-id ID] --kind ... [--name NAME] --version N laser deployment config get [--deployment-id ID] --kind ... --config-id ID laser deployment config activate [--deployment-id ID] --kind ... [--name NAME] --version N laser deployment config create [--deployment-id ID] --kind ... [--name NAME] --file FILE.json [--activate] laser deployment config delete [--deployment-id ID] --kind ... --config-id ID ``` ### Snapshots Snapshots contain per-node HTML diagnostics for the system, runtimes, certificates, network, kernel parameters, and recent logs. ```bash laser deployment snapshot list [--deployment-id ID] laser deployment snapshot create [--deployment-id ID] [--redact-secrets true|false] [--include-iggy true|false] [--node-id ID ...] laser deployment snapshot delete [--deployment-id ID] --snapshot-id ID laser deployment snapshot download [--deployment-id ID] --snapshot-id ID ``` ### Backups Backups capture storage volumes at a point in time. They require AWS Network Drive storage and backup access in the account plan. ```bash laser deployment backup list [--deployment-id ID] laser deployment backup create [--deployment-id ID] --name NAME [--node-id ID] [--remarks TEXT] [--expires-in-days N] laser deployment backup delete [--deployment-id ID] --backup-id ID laser deployment backup restore [--deployment-id ID] --backup-id ID ``` Repeat `--node-id` on `snapshot create` to cover only some nodes. `backup create` backs up every node unless `--node-id` is set. ### Access Rules (Firewall) `rules` is the access-rule alias. ```bash laser deployment access-rule list [--deployment-id ID] laser deployment access-rule add [--deployment-id ID] --name NAME \ --cidr CIDR [--cidr CIDR ...] \ [--iggy-tcp] [--iggy-http] [--iggy-websocket] [--iggy-udp] \ [--valid-to RFC3339] [--remarks TEXT] laser deployment access-rule delete [--deployment-id ID] --rules-id ID ``` Repeat `--cidr` for multiple address ranges. Select protocols with `--iggy-tcp`, `--iggy-http`, `--iggy-websocket`, and `--iggy-udp`. Set `--valid-to` for automatic expiry. ### Networking Peering and private endpoints follow the deployment's cloud. The CLI picks the AWS or GCP route itself. Peering needs Performance or Enterprise. Private endpoints need Enterprise. ```bash laser deployment peering list [--deployment-id ID] laser deployment peering create [--deployment-id ID] --name NAME --peer-vpc-cidr CIDR \ [--peer-vpc-id ID --peer-owner-id ID [--peer-region REGION]] \ [--peer-vpc-name NAME --peer-project-id ID] [--remarks TEXT] laser deployment peering instructions [--deployment-id ID] --peering-id ID laser deployment peering delete [--deployment-id ID] --peering-id ID laser deployment private-link list [--deployment-id ID] laser deployment private-link create [--deployment-id ID] --name NAME \ [--acceptance-required true|false] [--allowed-principal ARN ...] \ [--connection-preference accept_manual|accept_automatic] \ [--consumer-accept-list PROJECT ...] [--enable-proxy-protocol true|false] [--remarks TEXT] laser deployment private-link delete [--deployment-id ID] --service-id ID laser deployment private-link instructions [--deployment-id ID] --service-id ID laser deployment private-link connections [--deployment-id ID] --service-id ID ``` AWS peering takes `--peer-vpc-id` and `--peer-owner-id`, GCP peering `--peer-vpc-name` and `--peer-project-id`. `--acceptance-required` and `--allowed-principal` apply to AWS PrivateLink, the other private endpoint flags and the `instructions` and `connections` commands to GCP Private Service Connect. ### Connectors, Backends, Runtimes, and Tasks ```bash laser deployment connector list [--deployment-id ID] laser deployment connector catalog [--deployment-id ID] [--connector-type source|sink] [--page N] [--results N] laser deployment connector activate [--deployment-id ID] --connector-key KEY --connector-type source|sink \ [--instance-name NAME] [--instance-key KEY] laser deployment connector delete [--deployment-id ID] --instance-id ID laser deployment backend list [--deployment-id ID] laser deployment backend configure-embedded [--deployment-id ID] [--turso-sync-url URL] [--turso-auth-token TOKEN] [--turso-sync-interval-secs N] laser deployment backend configure-velodb [--deployment-id ID] --dsn DSN --fe-http-url URL [...] laser deployment backend enable [--deployment-id ID] --backend-id ID laser deployment backend disable [--deployment-id ID] --backend-id ID laser deployment runtime start|stop|restart [--deployment-id ID] --runtime iggy|connectors|connector:TYPE:KEY [--node-id ID ...] laser deployment task list [--deployment-id ID] ``` ### Observability ```bash laser deployment logs [--deployment-id ID] --runtime iggy|connectors|connector:ID|warden|plane|mcp [--page N] [--results N] [--message SUBSTRING] [--level debug|info|warn|error] [--node ID] [--scope-filter NAME] [--from RFC3339] [--to RFC3339] laser deployment activity [--deployment-id ID] [--page N] [--results N] laser deployment metrics [--deployment-id ID] laser deployment heartbeats [--deployment-id ID] laser deployment readiness [--deployment-id ID] --node-id ID laser deployment credentials [--deployment-id ID] laser deployment network [--deployment-id ID] ``` Log commands return pages. Run a command again to refresh the results. For live logs, use the [TUI](/cli/tui) Logs tab. ### Stream Data `deployment stream` works with a deployment's own data: Iggy streams and messages, and the managed data plane. It connects as the deployment's admin user, or as `--username` and `--password` when given. Definitions too rich for flags are read from a JSON file with `--file`, in the shape [laser-sdk](/laser-sdk) sends. | Surface | Commands | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Streams | `streams`, `topics`, `partitions`, `messages`, `users`, `user-get`, `clients`, `capabilities`, `whoami` | | KV | `kv-namespaces`, `kv-scan`, `kv-get`, `kv-set`, `kv-cas`, `kv-delete`, `kv-delete-many` | | Queries | `query`, `query-status`, `query-page`, `query-cancel` | | Schemas | `schemas`, `schema-get`, `schema-register`, `schema-drop`, `schema-decode` | | Projections | `projections`, `projection-get`, `projection-register`, `projection-drop`, `binding-apply`, `binding-remove` | | Graphs | `graphs`, `graph-get`, `graph-register`, `graph-drop`, `graph-query`, `graph` | | Forks | `forks`, `fork-create`, `fork-put-row`, `fork-promote`, `fork-delete` | | Runs | `runs`, `run-get`, `run-submit`, `run-cancel` | | Governance | `roles`, `role-get`, `role-define`, `role-delete`, `user-bind-roles` | | Destinations | `destinations`, `destination-get`, `destination-status`, `destination-checkpoint`, `destination-retention-gap`, `destination-prepared-attempt`, `destination-table`, `destination-table-schema`, `destination-current-snapshot`, `destination-snapshots`, `destination-snapshot`, `destination-files`, `destination-metrics`, `destination-create`, `destination-enable`, `destination-disable`, `destination-operation`, `query-routes` | | Consumer filters | `filters`, `filter-get`, `filter-revisions`, `filter-bindings`, `filter-binding`, `filter-operation`, `filter-metrics`, `filter-validate`, `filter-test`, `filter-preview`, `filter-register`, `filter-revise`, `filter-describe`, `filter-pause`, `filter-resume`, `filter-bind`, `filter-unbind`, `filter-archive`, `filter-drop` | ```bash laser deployment stream kv-set --namespace fleet --key sat-7 --value '{"mode":"safe"}' laser deployment stream kv-cas --namespace fleet --key sat-7 --value '{"mode":"nominal"}' --expect-version 3 laser deployment stream schema-register --kind avro --file telemetry.avsc --name telemetry --schema-version 1 laser deployment stream filter-test --filter safe-mode.json --payload '{"after":{"mode":"safe"}}' laser deployment stream filter-preview --filter safe-mode.json --stream fleet --topic changes --max-records 20 laser deployment stream filter-register --name sats-safe-mode --filter safe-mode.json laser deployment stream filter-bind --stream fleet --topic changes --group ground-station --id 5 --revision 1 ``` Catalog changes to [consumer filters](/laser-sdk/consumer-filters) print their operation ID while pending. Read the outcome with `filter-operation`, or rerun the command with the same `--operation-id`. Destination reads use the local view unless `--consistency linearizable` is passed. `filter-metrics` needs an Iggy user with `manage_servers`. Deletes, drops, pauses, and role changes ask for confirmation unless `--yes` is passed. ## Channel `chan` is the alias for tenant [notification channels](/observability/notifications). Channels support Slack, generic webhooks, and email. ```bash laser channel list [--page N] [--results N] laser channel get --channel-id ID laser channel create [--kind slack|webhook|email] [--name NAME] [--destination URL_OR_ADDRESS] [--remarks TEXT] laser channel update --channel-id ID [--name NAME] [--destination URL] [--enabled true|false] [--remarks TEXT] laser channel delete --channel-id ID laser channel test --channel-id ID laser channel notifications --channel-id ID [--page N] [--results N] ``` `--kind` determines whether `--destination` takes a Slack webhook URL, generic webhook URL, or email address. On an interactive terminal, `create` prompts for omitted required values. ## Cloud Catalog `cl` is the cloud alias. Discovery reads the active tenant's plan and regional availability. ```bash laser cloud list # clouds available laser cloud regions --cloud aws|gcp laser cloud clusters --cloud aws|gcp --region REGION laser cloud storages --cloud aws|gcp --region REGION laser cloud tiers --cloud aws|gcp --region REGION laser pricing get [--rates] laser pricing estimate managed --managed-tier TIER --cloud aws|gcp --compute-profile-id SIZE \ --storage-profile-id PROFILE --storage-gb-per-node N --throughput-mb-per-second MBPS laser pricing estimate byoc --vcpus-per-node N ``` Pricing commands need no login. `pricing get` lists each managed tier per cloud with its starting price, compute, storage, availability, telemetry retention, and private networking. `--rates` adds compute and Local NVMe prices per node, the Network Drive price per GB, and the data transfer bands. Use `-o json` for the catalog as the [API](/api/billing) returns it. Use `laser cloud tiers` for current tier availability, limits, Compute specifications, cluster and storage modes, and plan-dependent rate limits. ## Audit ```bash laser audit [--page N] [--results N] [--from RFC3339] [--to RFC3339] [--user USER_ID] [--author USER_ID] [--api-key API_KEY_ID] [--division ID] [--environment ID] [--deployment ID] [--types TYPE,TYPE] [--correlation-id UUID] ``` The tenant-wide [audit log](/observability/audit-compliance) contains immutable records of changes. `--user` filters the action's subject. `--author` filters the person who performed it. `--api-key` filters the key used for an automated call. ## System ```bash laser version # print CLI version laser update # in-place upgrade to the latest release laser tui # alias: laser ui (launches the dashboard) ``` ## Aliases | Long form | Alias | | ------------------------ | ------------------ | | `context` | `ctx` | | `tenant` | `tn` | | `division` | `div` | | `environment` | `env` | | `deployment` | `dep` | | `channel` | `chan` | | `cloud` | `cl` | | `tui` | `ui` | | `tenant invitation` | `tenant inv` | | `deployment access-rule` | `deployment rules` | ## Shell Completion ```bash laser --generate bash > /etc/bash_completion.d/laser laser --generate zsh > "${fpath[1]}/_laser" laser --generate fish > ~/.config/fish/completions/laser.fish laser --generate powershell > $PROFILE.laser.ps1 laser --generate elvish > ~/.config/elvish/lib/laser-completions.elv ``` Source: https://docs.laserdata.com/cli/commands --- # TUI Dashboard Run `laser tui`, or its alias `laser ui`, to open the interactive [Ratatui](https://ratatui.rs) dashboard. It supports keyboard and mouse input, a command palette, and saved history. TUI means terminal user interface. It shares the CLI binary, authentication, configuration, and backend. ## Launch ```bash laser tui ``` To display a command's result inside the dashboard, use interactive mode: ```bash laser deployment list --interactive laser tenant get --interactive ``` `--interactive` runs the selected command and displays its result beside the navigation panes. ## Layout The dashboard follows the [organization hierarchy](/organization): * Welcome / Dashboard shows the account, active context, environment overview, and starter shortcut. * Tenant shows members, roles, invitations, API keys, cloud accounts, and billing. * Division lists its environments. * Environment lists its deployments. * Deployment provides tabs for overview, credentials, configuration, heartbeats, metrics, logs, access rules, snapshots, activity, and connectors. * Resource tables provide paginated lists with filters based on the current location. The status bar shows the active context, polling state, and screen. Welcome also lists detected environment overrides, including `LD_API_KEY` and `LD_CONTEXT`. ## Key Bindings Direct bindings cover common navigation without conflicting with terminal conventions. Other actions use the command palette. Type on a navigation screen to open it, or press `:`. ### Navigation | Key | Action | | ------------------- | ----------------------------------------------------------------- | | `↑` / `↓` | Move up / down in lists, scroll panes | | `←` / `→` | Back / forward, or pane focus on the dashboard | | `Enter` | Drill into the selected resource, or execute the typed command | | `Esc` | Back one screen. On the Logs tab also clears the filter | | `Tab` | Next deployment tab. On the dashboard toggles tree / detail focus | | `Shift+Tab` | Previous deployment tab | | `1`...`9` / `0` | Jump directly to deployment tab N (`0` maps to tab 10) | | `Ctrl+C` / `Ctrl+D` | Quit | ### Function and Modifier Keys | Key | Action | | --------------------- | ---------------------------------------------------------------------------- | | `F5` | Force-refresh the active screen (works regardless of auto-poll) | | `F4` | Cycle the runtime filter (iggy / connectors / warden) on the deployment view | | `Ctrl+P` | Toggle background auto-poll on / off | | `PageUp` / `PageDown` | Page-scroll the active pane | | `Ctrl+B` / `Ctrl+F` | Page-scroll up / down (vim-style alternative) | | `<` / `>` | Page-scroll up / down (modifier-free alternative) | ### Command Palette The input appears at the bottom of the screen. Commands with a `:` prefix act on the current screen, such as `:create`, `:upgrade`, and `:refresh`. Commands without the prefix, such as `deployment list` and `tenant get`, use the CLI parser and run inside the dashboard. | Key (palette open) | Action | | ------------------ | ------------------------------------------------------- | | `Enter` | Execute the line (or accept the highlighted suggestion) | | `Tab` | Accept / cycle the next candidate | | `↑` / `↓` | Cycle command history (up = older) | | `Esc` | Reset the input. Second press exits | #### TUI verbs | Verb | Effect | | ----------------------------------- | ------------------------------------------------------------------------------- | | `:auth` / `:login` | Open the auth wizard | | `:logout` | Sign out of the active context | | `:purge` | Delete every context and keyring entry (confirmation required) | | `:whoami` | Show the active context and tenant | | `:nav` / `:browse` | Open the navigator from the welcome screen | | `:dashboard` / `:overview` | Jump to the dashboard | | `:welcome` / `:home` / `:hello` | Return to the welcome screen | | `:help` / `:?` | Show the help panel | | `:use ` | Switch the active context | | `:contexts` / `:ctxs` | List all contexts | | `:show` | Show details for the selected resource | | `:format ` | Change the output format used by inline command results | | `:new` / `:create` | Open the create wizard for the current scope | | `:edit` | Edit the selected resource | | `:delete` / `:rm` | Delete the selected resource (with confirmation) | | `:upgrade` | Open the upgrade wizard for the current deployment | | `:managed` / `:byoc` / `:starter` | Open the matching deployment provisioning wizard | | `:reveal` | Reveal masked deployment credentials | | `:activate` | Activate the selected config version | | `:download` | Generate a download URL for the selected snapshot | | `:filter` | Focus the filter input on logs / metrics tabs | | `:copy` / `:select` | Toggle terminal mouse-capture so you can drag-select text | | `:save` | Save the current command output to a file | | `:raw` | Show the raw response payload for the last command | | `:clear` | Clear the message log | | `:refresh` / `:reload` | Force-refresh the active screen (same as `F5`) | | `:poll` | Toggle auto-poll (same as `Ctrl+P`) | | `:permissions` / `:perms` | Show the active API key's role and scopes | | `:settings` / `:config` | Show the tenant settings | | `:types ` | Filter an audit or activity table by event types. No argument clears the filter | | `:stream` | Open the stream screen of the current deployment, see below | | `:stream-whoami` | Show the connected Iggy user's effective grants | | `:exit` / `:quit` / `:q` | Exit the TUI | History is stored at `$XDG_DATA_HOME/laser/history`. It keeps at most 1000 entries and uses atomic writes to prevent partial updates. ## Stream Screen From a deployment screen, `:stream` opens the deployment's own data: Iggy streams and the managed data plane. Each `:stream-` verb opens the same screen on a tab. `Tab`, `Shift+Tab`, and the digit keys move between tabs, `Enter` drills into a row, and `F5` refetches. | Tab | Verb | Shows | | ------------ | ---------------------- | --------------------------------------------------------------------------- | | streams | `:stream` | Streams, then topics, partitions, and messages | | kv store | `:stream-kv` | KV namespaces, then keys and values | | memory | `:stream-memory` | The same KV data as agent memory, by conversation | | schemas | `:stream-schemas` | Registered writer schemas | | forks | `:stream-forks` | Forks of the materialized state | | projections | `:stream-projections` | Projections and their bindings | | graphs | `:stream-graphs` | Registered graphs | | destinations | `:stream-destinations` | Materialization destinations and their status | | filters | `:stream-filters` | Saved consumer filters, then one filter with its revisions and bound groups | | query | `:stream-query` | A query form for a materialized index, in DSL or SQL mode | | runs | `:stream-runs` | Agent runs | | agents | `:stream-agents` | The agent registry | | users | `:stream-users` | Iggy users with their permissions and governance roles | | governance | `:stream-governance` | Governance roles and the users bound to them | | capabilities | `:stream-capabilities` | The AGDX capabilities of the deployment | On the stream screen, `s` hides or shows system-managed rows, `v` cycles a payload view between auto, JSON, text, and hex, `[` and `]` page through messages, and `g` jumps to an offset. ## Wizards Forms guide creation, upgrades, and deletion, with errors shown beside invalid fields: * The Managed deployment form asks for cloud, region, tier, cluster mode, storage, availability, encryption, retention, and spend limit. It retrieves choices from `laser cloud tiers` and `laser cloud storages`. * The BYOC form adds AWS account ID, role ARN, external ID, and VPC details. For GCP, it asks for project, service account, and VPC details. * The Starter form creates a Free deployment on one screen. Its region defaults are `us-east-1` for AWS and `us-central1` for GCP. * The Upgrade form retrieves valid tier and storage changes from the API. * Destructive actions require typed approval. Protected resources also require a one-time code. ## Mouse Mouse input uses `crossterm`. Left-click rows, tabs, or panes to select them and move focus. Use the wheel to scroll lists and details. If the terminal disables mouse support, use the keyboard equivalents. ## Polling The dashboard refreshes from the backend every five seconds. Toggle automatic polling with `Ctrl+P` or `:poll`. Refresh immediately with `F5` or `:refresh`. The status bar shows whether polling is active. ## Theming The dashboard uses a monochrome, high-contrast palette. `NO_COLOR` or `--no-color` disables color output. Alternate-screen mode preserves terminal scrollback when you exit. ## When to Use the TUI vs Headless Verbs | Use the TUI when | Use headless verbs when | | ---------------------------------- | ------------------------------ | | Exploring a new tenant | Scripting, CI, or cron jobs | | Creating deployments interactively | Composing pipelines | | Tailing live logs or metrics | Returning JSON to another tool | | Triaging an incident | Embedding in agents | Both modes use the same authentication, context, and backend. Choose either mode for each task. Source: https://docs.laserdata.com/cli/tui --- # Claude Code Skills The `laser-cli-claude` skill pack lets [Claude Code](https://docs.anthropic.com/claude/docs/claude-code) translate requests into `laser` commands. It includes `/laser-deploy`, `/laser-troubleshoot`, and `/laser-snapshot`. Source is available at [github.com/laserdata/laser-cli-claude](https://github.com/laserdata/laser-cli-claude). ## Install ### Bundled with the CLI (one shot) If `laser` is not installed, install the binary and skills together: ```bash curl -fsSL https://cli.laserdata.cloud/install.sh | sh -s -- --with-cc-skills ``` The installer runs `cli.laserdata.cloud/claude.sh` after installing the binary. Re-running updates both in place, with `cp -f` for skill files. See [installer flags](/cli#installer-flags) for the full list. ### Marketplace (recommended for skills-only) ``` /plugin marketplace add laserdata/laser-cli-claude /plugin install cli@laser ``` ### Script (skills only) ```bash curl -fsSL https://cli.laserdata.cloud/claude.sh | sh ``` The script downloads the latest bundle and copies its `*.md` files to `~/.claude/skills/`. Run it again to update existing files. Use these environment variables to change the destination: ```bash LASER_CLAUDE_DEST=~/my-skills ./claude.sh LASER_CLAUDE_REF=v0.0.1 ./claude.sh # pin to a tag ``` ### Manual ```bash git clone https://github.com/laserdata/laser-cli-claude.git cp -R laser-cli-claude/skills/* ~/.claude/skills/ ``` ## Prerequisites ```bash curl -fsSL https://cli.laserdata.cloud/install.sh | sh laser auth login --tenant-id ``` Each skill runs `laser` commands. It stops with an error if the binary is missing from `$PATH` or credentials are unavailable. Agents and CI can supply `LD_API_KEY` and `LD_TENANT_ID` instead of running `auth login`. ## Skills | Slash command | Purpose | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/laser-onboard` | First-run setup. Installs the binary, walks through sign-in, and sanity-checks the active context. | | `/laser-deploy` | Translates a request like "spin up Iggy in `eu-west-1` on the Standard tier" into the matching `laser deployment create-managed`, `create-starter`, or `create-byoc` invocation, plus `preview` for cost estimates, the two-step BYOC helpers `byoc setup` + `byoc validate`, and the full lifecycle: `upgrade`, `extend`, `retention`, `spend-limit`, `update`, `delete`. Cloud, region, tier, and storage are discovered from the API rather than guessed. | | `/laser-troubleshoot` | Pulls metrics, heartbeats, activity, recent runtime logs, network, tasks, and access rules for a deployment in trouble. Returns a structured health report with a concrete next step, including per-runtime restart on a single node when that is the right call. | | `/laser-config` | Manages versioned deployment configs (Iggy, connectors, Warden, individual connector instances): list, view, create new versions from a JSON file, activate, delete. Plus connector-instance lifecycle. | | `/laser-snapshot` | Manages diagnostic HTML snapshots covering system state, runtimes, certificates, network, kernel parameters, and logs. | | `/laser-backup` | Manages point-in-time storage volume backups. Available for AWS Network Drive deployments when backups are enabled on your account plan. | | `/laser-access` | Manages access rules: list, add CIDR allowlists with per-protocol toggles (Iggy TCP/HTTP/WebSocket/UDP), and delete. | | `/laser-channel` | Manages tenant notification channels (Slack, webhook, email): create, list, update, delete, test, and inspect delivered notifications. | | `/laser-iam` | Manages tenant identity and access: tenant config (join policy + invitation rules), API keys, members, roles, invitations, and cloud-account registrations. All scoped to the tenant. | | `/laser-billing` | Reads the tenant billing surface: subscription and customer info, billing reports, invoices, and invoice PDF downloads. Read-only on the CLI today. | | `/laser-credentials` | Reads deployment connection credentials safely. Output is masked in chat and never persisted. | | `/laser-context` | Manages named CLI contexts (api-url + tenant scope + key store): list, switch, create, rename, and delete. | | `/laser-audit` | Queries the tenant audit log with filters for time window, division, environment, deployment, subject user, author, types, or correlation id. Useful for incident forensics and compliance. | | `/laser-debug` | Reads the local debug log and surfaces recent failures grouped by target. Also documents the platform headers (`ld-request`, `idempotency-key`, `idempotent-replayed`, `link`, `retry-after`) and the `application/problem+json` envelope (RFC 7807) so failure traces map back to API behavior. | ## Built-in Guardrails Each skill follows these rules: * Retrieve current tiers, regions, and account details through `laser cloud tiers`, `laser cloud regions`, and `laser tenant get`. * Display the exact command before any change and wait for your approval. * Use `-o json` and parse structured responses. * Read credentials through `laser` and its keyring support instead of placing API keys in commands. * Require typed approval for deletion, restoration, and spend-limit changes. Protected resources also require a one-time code. * Show `/laser-credentials` values once on screen. Do not save them to files or shell history. ## Examples To create a starter deployment, ask: > "Set up a starter Iggy for me to play with." `/laser-deploy` asks for AWS or GCP and presents the default region, `us-east-1` or `us-central1`. It shows the `laser deployment create-starter` command and waits for approval. It then follows the deployment until it is ready. To investigate a deployment, ask: > "`events-prod` looks unhealthy." `/laser-troubleshoot events-prod` finds the deployment by name. It reads heartbeats, metrics, the last 200 log lines, recent activity, and active access rules. It presents a summary and the most likely cause on one screen. To add a firewall rule, ask: > "Allow my home IP to hit `events-prod`." `/laser-access` finds your public IP. It displays `laser deployment access-rule add --deployment-id 611298765432109056 --cidr /32 --description "home"` and waits for approval before applying it. Source: https://docs.laserdata.com/cli/claude-skills --- # API Reference LaserData Cloud provides a main API and regional Supervisor APIs. Programmatic requests use an `ld-api-key` header with a key whose role permits the operation. These APIs manage resources, configuration, and monitoring. For application messages, queries, and agents, use [laser-sdk](/laser-sdk) in Rust, Python, or TypeScript. ## Base URLs | API | Base URL | Purpose | | -------------- | ----------------------------- | --------------------------------------------------------------------------------------------- | | Main API | `https://api.laserdata.cloud` | Organization, deployments, API keys, billing, notifications, and audit records | | Supervisor API | `{supervisor_url}` | Deployment operations: configs, networking, metrics, logs, diagnostic snapshots, data backups | [List Deployments](/api/deployments#list-deployments) and [Get Deployment](/api/deployments#get-deployment-main-api) return each deployment's `supervisor_url`. An example is `https://supervisor-aws-us.laserdata.cloud`. ## OpenAPI Schema Each public service provides an OpenAPI 3.1 specification and browser, with request and response examples. Agents and generators can start at [`https://laserdata.com/openapi.json`](https://laserdata.com/openapi.json), the Core management API specification. Individual service specifications are also available: | Service | Machine-readable schema | | -------- | -------------------------------------------------------------------------------------------- | | Core | [`https://laserdata.com/openapi/core.json`](https://laserdata.com/openapi/core.json) | | Audit | [`https://laserdata.com/openapi/audit.json`](https://laserdata.com/openapi/audit.json) | | Notifier | [`https://laserdata.com/openapi/notifier.json`](https://laserdata.com/openapi/notifier.json) | Each operation has a stable, unique `operationId`, description, typed parameters, and typed responses. For an LLM function tool, use `operationId` as its name. Use the description as guidance and request parameters or body schema as inputs. Send credentials only to the declared LaserData service base URL. ### Main, Audit, Notifier Open api.laserdata.cloud/docs and use the dropdown to select Core, Audit, or Notifier. ### Supervisor (per region) Each cloud and area has a supervisor specification at `/docs`: | Cloud | US | EU | AP | | ----- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | AWS | supervisor-aws-us | supervisor-aws-eu | supervisor-aws-ap | | GCP | supervisor-gcp-us | supervisor-gcp-eu | supervisor-gcp-ap | Specifications declare `ld_api_key` and `session_cookie`. User-session endpoints explicitly require cookies and reject API keys. See [Authentication](/api/authentication#api-keys-vs-console-sessions). ## Authentication For API-key authentication, include `ld-api-key`: [Authentication](/api/authentication) explains scopes, security, and error responses. ## Pagination List endpoints accept these pagination parameters: | Parameter | Default | Description | | --------- | ------- | ------------------------ | | `page` | `1` | Page number (1-indexed) | | `results` | `10` | Items per page (max 100) | Paginated bodies include `items`, `page`, `total_results`, and `total_pages`. The RFC 8288 `Link` header provides `rel="first"`, `prev`, `next`, and `last` URLs. Clients can follow those links without parsing pagination fields. ## Request and Response Headers | Header | Direction | Purpose | | ------------------------------------------------------------------------ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `ld-api-key` | request | API key bearer credential (see [Authentication](/api/authentication)) | | `idempotency-key` | request | Optional client-supplied key (max 255 chars) that makes `POST`, `PUT`, and `PATCH` requests safe to retry. See [Idempotency](#idempotency) | | `ld-request` | response | Correlation ID for this request. Mirrored as `instance` in problem+json error bodies. Include it in support requests | | `idempotent-replayed` | response | `true` when the response was served from the idempotency cache instead of re-running the handler | | `link` | response | Pagination links (`first`, `prev`, `next`, `last`) on paged list responses | | `retry-after` | response | Seconds to wait before retrying. Sent on `429` and on transient `5xx` | | `ld-tenant`, `ld-division`, `ld-environment`, `ld-deployment`, `ld-role` | response | Created-resource ID headers, set on the matching `POST` endpoint when the new resource is created | ## Idempotency `POST`, `PUT`, and `PATCH` accept optional `idempotency-key` for safe retries. The platform caches the first response for each `(api_key, idempotency-key)` pair for 10 minutes. Repeating the same body with the same key returns that response with `idempotent-replayed: true`. The handler does not run again. * This feature applies to API-key authentication. Console sessions do not use it. * A repeated key with a different body returns `409 Conflict` and `code: idempotency_error`. * A retry while the original request runs returns `409 Conflict` and `code: idempotent_request_in_progress`. Wait briefly before retrying. * `DELETE` is excluded because repeated deletion already has the same effect. * Audit has no endpoints that change state and does not advertise idempotency support. Use the same request identity after network failures, CI retries, or queue redelivery, including when creating deployments. ## Quick Start Create a Free single-node deployment, then retrieve its details and credentials: # ld-deployment: `} /> ## HTTP Status Codes | Code | Meaning | | --------------------------- | -------------------------------------------------------- | | `200 OK` | Request succeeded | | `202 Accepted` | Request accepted. Operation will complete asynchronously | | `400 Bad Request` | Invalid parameters or request body | | `401 Unauthorized` | Missing or invalid API key | | `403 Forbidden` | API key valid but lacks required permission | | `404 Not Found` | Resource does not exist | | `409 Conflict` | Resource already exists or state conflict | | `429 Too Many Requests` | Rate limit exceeded | | `500 Internal Server Error` | Unexpected server error | ## API Sections Source: https://docs.laserdata.com/api --- # Authentication Use `ld-api-key` for programmatic access to tenant and deployment APIs. The Console also supports user sessions. Public pricing and schema discovery do not require a key. ## Request Header Every request authenticates the key. Missing, expired, or invalid keys return `401 Unauthorized`. A valid key without the required permission returns `403 Forbidden`. ## Key Format API keys are randomly generated secrets. The server stores only their hashes and cannot recover the original values. Copy the secret when the key is created. ## Scopes and Permissions A key receives permissions through a role. Roles grant access at these levels: | Scope | Description | | -------- | --------------------------------------------------------------------------------------------------------------- | | Tenant | Cross-organization permissions, for example `member:read`, `api_key:manage` | | Division | Resource permissions scoped to specific environments, for example `deployment:manage`, `deployment:config:read` | See [Roles & Permissions](/organization/roles-permissions) for scope and precedence. Assign an existing role or supply inline permissions to create a dedicated role for the key. ## Rate Limiting Each key has its own rate limit. Exceeding it returns `429 Too Many Requests`, with the reset time in `Retry-After`. ## IP Allowlisting An optional IP allowlist restricts where a key can be used. Other IPs receive `403 Forbidden`, even when the key is valid. Change an existing key's allowlist through [Update API Key Security](/api/api-keys#update-security-settings). Recreating the key is not required. ## Expiry Keys must expire within 365 days. Expired keys return `401 Unauthorized`. Create and distribute a replacement before the current key expires. ## Error Reference | Status | Cause | | --------------------------- | ---------------------------------------------------------------------------- | | `400 Bad Request` | Validation failed. Response includes `field_issues[]` with per-field details | | `401 Unauthorized` | Key missing, malformed, expired, or revoked | | `403 Forbidden` | Key valid but missing the required permission, or request from a blocked IP | | `404 Not Found` | Resource does not exist or is not visible to this key | | `429 Too Many Requests` | Per-key rate limit exceeded. Check `Retry-After` | | `500 Internal Server Error` | Unexpected server failure | ### Error Envelope Non-`2xx` responses use `application/problem+json`, based on [RFC 7807](https://datatracker.ietf.org/doc/html/rfc7807): ```json { "type": "about:blank", "title": "Invalid Email", "code": "invalid_email", "reason": "Invalid email address", "instance": "8f4a2b6c9d1e4f3a8b5c7d9e0f1a2b3c", "field": "email", "field_issues": [ { "code": "invalid_email", "reason": "malformed address", "path": "email" } ], "status": 400, "retryable": false, "resolution": "Correct the invalid fields or request body, then retry." } ``` `instance` matches `ld-request`, a UUID written as 32 hexadecimal characters without separators. | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | RFC 7807 type URI for the error class. `about:blank` when no type is registered | | `title` | Short human-readable title derived from `code` (acronyms capitalised) | | `code` | Stable machine-readable error code (e.g. `invalid_email`, `tenant_not_found`, `insufficient_permissions`) | | `reason` | Long-form human explanation of this occurrence | | `instance` | Mirrors the `ld-request` response header. Quote this when filing a support ticket | | `field` | Single field name when the error is bound to one field. Still emitted for back-compat | | `field_issues` | Array of per-field issues for validation errors. Each entry has `code`, `reason`, and an optional dotted `path`. Omitted on non-validation errors | | `status` | Mirror of the HTTP status code so agents can branch on the body without re-reading the response status | | `retryable` | `true` for retryable conditions (`408`, `425`, `429`, `500`, `502`, `503`, `504`), `false` otherwise | | `resolution` | Concrete recovery guidance for an automated client or operator | Validation failures return `400` with `field_issues`. Use these entries to show errors beside the relevant fields. ## API Keys vs Console Sessions OpenAPI declares tenant-scoped `ld_api_key` and browser-session `session_cookie`. Most endpoints accept either. User-scoped endpoints reject API keys, even keys created by the same user. These requests return `403 Forbidden` with `code: api_key_not_allowed`. Session-only operations include account reads and exports, sign-out, own-session lists, invitation acceptance or rejection, own invitations, and user activity. Tenant creation, leaving, and deletion also require a Console-issued cookie session. ## Security Best Practices * Store secrets in AWS Secrets Manager, HashiCorp Vault, GitHub Secrets, or another secret store. Keep them out of source code and logs. * Grant only required permissions. Read-only automation can use `deployment:read` instead of an administrator role. * Use short expiry periods, such as 30-90 days, for CI keys. * Enable IP restrictions for long-lived keys used from fixed infrastructure. * Rotate without interruption by creating a replacement, updating consumers, and deleting the old key after the change. * Read the [audit trail](/api/audit) for API key operations. Source: https://docs.laserdata.com/api/authentication --- # Organization The resource hierarchy is Tenant, Division, Environment, and Deployment. These endpoints manage the first three levels through `https://api.laserdata.cloud`. ## Tenant ### Get Tenant ```json { "id": 1, "name": "Acme Corp", "description": "Main production tenant", "email": "admin@acme.com", "protected": false, "created_at": "2025-01-10T08:00:00Z", "updated_at": "2025-06-15T12:30:00Z", "features": { "invitations_limit": 100, "members_limit": 100, "roles_limit": 20, "divisions_limit": 5, "environments_limit": 20, "deployment_tiers": [ { "tier": "free", "limit": 1 }, { "tier": "small", "limit": 3 }, { "tier": "medium", "limit": 3 }, { "tier": "large", "limit": 2 }, { "tier": "xlarge", "limit": 1 }, { "tier": "2xlarge", "limit": 1 } ], "deployment_access_rules_limit": 10, "deployment_configs_limit": 5, "deployment_backups_limit": 3, "deployment_snapshots_limit": 5, "private_connections_limit": 3, "private_endpoints_limit": 1, "byoc_enabled": true, "cluster_enabled": true, "on_premise_enabled": false, "private_networking_enabled": true, "multi_az_enabled": true, "dedicated_enabled": false, "backup_enabled": true, "cross_region_dr_enabled": false, "audit_retention_days": 30, "api_keys_limit": 10, "cloud_accounts_limit": 5, "notification_channels_limit": 5, "notification_subscriptions_limit": 10, "custom_domains_limit": 1, "backup_regions_limit": 3, "nodes_per_deployment_limit": 5, "backup_retention_days": 30, "snapshot_retention_days": 14, "pending_invitations_limit": 100, "api_key_allowed_ips_limit": 20, "divisions_per_role_limit": 3, "environments_per_division_limit": 3, "concurrent_deployments_limit": 2, "audit_export_enabled": true, "advanced_connectors_enabled": true, "customer_managed_keys_enabled": false, "advanced_notification_channels_enabled": true }, "subscription": { "id": 1, "plan": "pro", "active": true, "created_at": "2025-01-10T08:00:00Z", "valid_from": "2025-01-10T08:00:00Z", "valid_to": "2026-01-10T08:00:00Z" }, "starter_available": true, "has_payment_method": true, "has_custom_email_domain": true } ``` `features` reports plan limits and capabilities. `deployment_tiers` reports allowed tiers and deployment counts. `starter_available` reports whether another Free deployment can be created. `has_payment_method` is `true` for a valid card or invoice agreement. `has_custom_email_domain` is `true` for tenants that claimed a corporate domain during sign-up. It is `false` for public email providers such as Gmail and Outlook. This field controls access to domain-based join policy and invitation restrictions in [Tenant Config](#get-tenant-config). This request requires `info:read`. ### Update Tenant Include only fields that you want to change. `protected: true` enables protection, which requires a one-time code for deletion. A successful request returns `204 No Content`. ### Get Organization Structure ### Get Tenant Summary ### Get Tenant Config ```json { "join_policy": "invite_only", "block_external_invitations": false, "enforce_domain_only_invitations": false, "email_domain": "acme.com", "email_domain_verified_at": "2026-05-05T12:00:00Z" } ``` | Field | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `join_policy` | What happens when a new user signs up with an email matching `email_domain`. One of `invite_only` (default), `open`, `request_to_join`. | | `block_external_invitations` | When `true`, other tenants cannot invite users on this domain. Requires `email_domain` to be set. Defaults to `false`. | | `enforce_domain_only_invitations` | When `true`, this tenant can only invite users whose email matches `email_domain` (or one of its claimed division subdomains). Requires `email_domain` to be set. Defaults to `false`. | | `email_domain` | The registrable domain the tenant claimed at signup. Omitted when the tenant was created with a public-email provider (Gmail, Outlook, etc.). Immutable. | | `email_domain_verified_at` | When the claim was verified. For now, set to the signup timestamp (OAuth proves ownership). Omitted when no claim exists. | This request requires `settings:read`. ### Update Tenant Config A successful request returns `204 No Content`. Repeating unchanged configuration publishes no event. A lock flag set to `true` without `email_domain` returns `400 tenant_has_no_email_domain`. * `open` joins matching-domain users with the system `viewer` role and notifies the owner. * `request_to_join` creates a pending request and notifies the owner, who must approve it. * `invite_only`, the default, leaves matching-domain users waiting for an invitation without notifying the owner. A change publishes the `tenant_config_updated` audit event. This request requires `settings:manage`. ### Request Resource Code In `action`, `action_type` selects the operation and `payload` supplies its resource IDs: | Action Type | Payload Fields | | -------------------- | ------------------------------------------------------------- | | `delete_tenant` | `tenant_id` | | `delete_division` | `tenant_id`, `division_id` | | `delete_environment` | `tenant_id`, `division_id`, `environment_id` | | `delete_deployment` | `tenant_id`, `division_id`, `environment_id`, `deployment_id` | The endpoint returns `204 No Content` and emails the code. Supply it as `code` on the matching delete request. Requests are rate limited per resource. ### Delete Tenant Only the tenant owner can delete it. Deletion is irreversible. *** ## Divisions ### Create Division | Field | Required | Description | | ------------- | -------- | ---------------------------------------- | | `name` | Yes | Division name | | `description` | No | Optional description | | `email` | No | Optional contact email for this division | ### List Divisions ### Update Division ### Delete Division *** ## Environments ### Create Environment ### List Environments ### Update Environment ### Delete Environment *** ## Supervisor: Org Summaries These Supervisor endpoints summarize runtime data within one region. ### Tenant Summary (Supervisor) ### Division Summary (Supervisor) Source: https://docs.laserdata.com/api/organization --- # Members & Roles Members are users who can access a tenant. Roles define their permissions. Send these requests to `https://api.laserdata.cloud`. Listing members requires `member:read`. Invitations and removal require `member:manage`. Reading roles requires `role:read`, and creating or deleting them requires `role:manage`. ## Members ### List Members ```json { "items": [ { "id": 1, "email": "alice@example.com", "name": "Alice", "active": true, "roles": ["developer"], "created_at": "2025-01-15T10:00:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Update Member Role ### Remove Member *** ## Invitations ### Invite a Member The [tenant configuration](/api/organization#get-tenant-config) determines these domain-restriction errors: | Status / Code | Cause | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `400 invitee_domain_locked` | The invitee's email domain belongs to another tenant that has set `block_external_invitations: true`. | | `400 invitee_domain_not_allowed` | This tenant has `enforce_domain_only_invitations: true` and the invitee's email domain is outside the tenant's claimed `email_domain` (and any division subdomains). | ### List Invitations ### Cancel Invitation *** ## Roles ### List Roles ```json { "items": [ { "id": 1, "name": "admin", "kind": "system" }, { "id": 2, "name": "developer", "kind": "custom" } ], "page": 1, "total_results": 2, "total_pages": 1 } ``` Role `kind` identifies how the role was created: * `system` identifies `admin`, `developer`, `viewer`, or `billing`. These four built-in roles cannot be deleted. Tenant ownership is stored separately. * `custom` identifies a role created through the API or Console. * `api_key` identifies a dedicated role created from inline key permissions instead of an existing `role_id`. ### Get Role ### List Role Members ### Create Role | Field | Required | Description | | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Unique role name within the tenant | | `permissions.tenant` | No | Tenant-level permission strings (e.g. `info:read`, `member:manage`) | | `permissions.division` | No | Default division permissions applied to all divisions | | `permissions.environment` | No | Default environment permissions applied to every environment in every division. The blanket "all environments" knob: spares you per-division overrides for read-only or developer-style roles | | `permissions.divisions` | No | Per-division overrides keyed by division ID. Each entry takes its own `permissions`, `environment` (default for that division's environments), and `environments` map (per-environment overrides) | Environment permissions use the most specific grant: the environment override, division `environment` default, then global `permissions.environment`. Without a matching grant, access is denied. Division permissions use per-division `permissions`, then global `permissions.division`, then none. ### Assign Role to Members ### Revoke Role from Members ### Delete Role Source: https://docs.laserdata.com/api/members --- # Deployments Use `api.laserdata.cloud` for deployment creation, upgrades, and deletion. Use `{supervisor_url}` for runtime status, nodes, and credentials. See [API Architecture](/getting-started#api-architecture). Reading requires `deployment:read`. Creation, changes, and deletion require `deployment:manage`. ## Discovery Before creating a deployment, retrieve the configurations available to your plan and region. ### List Available Clouds ### List Regions ### List Available Tiers ```json [ { "key": "free", "name": "Free", "description": "Perfect for getting started.", "available": true, "limit": 1, "clusters": ["standalone"], "storages": ["network_balanced"], "rate_limit": "100 KB/s" }, { "key": "large", "name": "Large", "available": true, "limit": 2, "clusters": ["cluster"], "storages": ["local_ssd", "network_balanced"], "rate_limit": null } ] ``` ### List Available Storage Types *** ## Create Deployments ### Preview Deployment Cost ```bash curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/preview \ -H "ld-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cloud": "aws", "region": "us-east-1", "managed_tier": "standard", "tier": "small", "storage": { "type": "network_balanced", "size": 100 }, "target_network_tput": 100, "network_scope": "same_region", "availability_mode": "single_az" }' ``` A paid preview requires `managed_tier`, `cloud`, `region`, and `tier`. A Free preview omits `managed_tier` and returns zero cost. Preview accepts storage, throughput, network scope, and availability without creating resources. The product determines node count, so the request has no node-count field. The response reports `cloud`, `region`, `tier`, `nodes`, `compute_profile_id`, `included_telemetry_days`, `pricing_version`, `monthly_base_usd`, and `monthly_total_usd`. `breakdown` contains `compute_usd`, `storage_usd`, and `network_usd`. `capacity` reports `shards_per_node`, `default_segment_bytes`, `min_segment_bytes`, `partitions_per_node_at_default_segment`, and `partitions_per_node_at_min_segment`. `target_network_tput` uses KB/s. `100` means 0.1 MB/s, and `1000` means 1 MB/s. Omit it to use the tier's default throughput estimate. If storage is omitted, preview uses 100 GB of Network Drive per node. If availability is omitted, preview uses Single AZ. Preview requires `deployment:read`. `monthly_base_usd` excludes transfer. `monthly_total_usd` includes estimated transfer, which is billed for actual usage. Segment-based capacity fields are diagnostic values, not guaranteed partition limits. ### Create a Managed Deployment | Field | Required | Values / Description | | -------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Deployment name | | `cloud` | Yes | `aws` or `gcp` | | `region` | Yes | Cloud region, for example `us-west-1` or `europe-west1` | | `managed_tier` | Paid | `standard`, `performance`, or `enterprise`. Required for every paid deployment and omitted for Free | | `tier` | Yes | Compute size: `free`, `small`, `medium`, `large`, `xlarge`, `2xlarge`, `4xlarge`, `8xlarge`, or `16xlarge` | | `cluster` | Yes | `cluster` for paid deployments. Free uses `standalone` | | `storage.type` | No | `network_balanced` for Network Drive or `local_ssd` for Local NVMe. Defaults to 100 GB of Network Drive per node | | `storage.size` | No | Network Drive size in GB per node, from 100 to 30,000. Local NVMe size follows the Compute size | | `target_network_tput` | No | Throughput estimate in KB/s. Defaults to the tier's default throughput estimate. Used for compute sizing and estimating transfer, never a transfer commitment or paid broker rate limit | | `network_scope` | No | `same_region` (default), `cross_region_same_cloud`, or `cross_cloud` | | `availability_mode` | No | `single_az` (default) or `multi_az`. Multi AZ requires Enterprise | | `protected` | No | Enable [resource protection](/organization#resource-protection). Default `false` | | `encrypted` | No | Enable custom key encryption of message data. Default `false` | | `public_ip_enabled` | No | Assign a static public IP. Default `true` | | `subdomain_enabled` | No | Assign a custom subdomain. Requires a public IP. Default `true` | | `dedicated` | No | Dedicated infrastructure isolation. Requires the Enterprise plan entitlement. Default `false` | | `retention.telemetry_days` | No | Telemetry retention in days. Defaults to 14 days for Standard, 30 for Performance, or 90 for Enterprise. Free uses 7 days. Higher values are rejected | | `spend_limit` | No | Monthly spend monitoring threshold in USD | Catalog presets do not override omitted creation fields. To create an Enterprise preset with Local NVMe and Multi AZ, send its storage and availability explicitly. The endpoint returns `202 Accepted` with resource IDs in `ld-environment` and `ld-deployment`. Valid payment information is required. Local NVMe requires Performance or Enterprise. Multi AZ requires Enterprise. Unsupported tier features return `400` with `field_issues`. ### Generate BYOC Setup AWS setup returns `laserdata_org_id`, `external_id`, `trust_policy`, and `permissions_policy`. GCP setup returns service-account instructions. ### Validate BYOC Credentials ### Create a BYOC Deployment (AWS) A successful request returns `202 Accepted` with `ld-environment` and `ld-deployment`. See [BYOC Setup](/deployments/byoc) for IAM configuration. ### Create a BYOC Deployment (GCP) A successful request returns `202 Accepted` with `ld-environment` and `ld-deployment`. ### Create a Starter Deployment | Field | Required | Description | | ------------------ | -------- | -------------------------------------------------- | | `cloud` | Yes | `aws`, `gcp` | | `region` | Yes | Cloud region | | `environment_id` | No | Existing environment ID to deploy into | | `environment_name` | No | Name for a new environment (defaults to `sandbox`) | | `deployment_name` | No | Deployment name (auto-generated if omitted) | *** ## Read Deployments ### List Deployments ```json { "items": [ { "id": 1, "name": "prod-cluster", "code": "abc123", "variant": "managed", "domain": "prod-cluster-abc123.laserdata.cloud", "cloud": "aws", "region": "us-west-1", "tier": "large", "cluster": "cluster", "supervisor_url": "https://supervisor-aws-us.laserdata.cloud", "created_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ### Get Deployment (Main API) The Main API adds `code`, `protected`, `description`, and `upgrades`. Managed deployments with pricing terms include `managed_tier` and `monthly_base`. These describe their active terms and can change after an upgrade or an agreed terms update. `target_network_tput` reports the throughput estimate. `can_upgrade` reports whether the cooldown permits a change. ### Get Deployment (Supervisor API) The Supervisor adds `status`, `network_mode`, `cidr`, `configs`, `nodes`, and `aws` or `gcp` network details. Request both responses in parallel for the full deployment details. ### Get Deployment Credentials ```json { "username": "iggy", "password": "your-deployment-password" } ``` *** ## Modify Deployments ### Upgrade a Deployment The endpoint returns `202 Accepted`. Warden applies hardware changes asynchronously. Storage cannot shrink. This endpoint cannot change Compute or Storage on Local NVMe deployments. A tier-only upgrade sends `{"managed_tier":"performance"}`. It changes commercial terms and telemetry retention without changing hardware. Monthly pricing changes from the effective time, with charges calculated for each active period. This endpoint rejects downgrades, which require agreed terms. Enterprise requires approved account features. Every upgrade requires valid payment information. ### Extend a Deployment The request has a `nodes` array, such as `[{"count": 2}]`, but an authorized call for an existing deployment returns `cluster_extension_not_allowed`. Use supported Compute or Storage upgrades instead. ### Update Spend Limit A successful request returns `200 OK`. It requires environment-level `deployment:manage`. ### Update Retention A successful request returns `200 OK`. ### Update Deployment ### Delete a Deployment The endpoint returns `202 Accepted`. Deletion permanently destroys all nodes, data, configuration, backups, and telemetry. Source: https://docs.laserdata.com/api/deployments --- # Configuration The Supervisor API, `{supervisor_url}`, manages versioned configuration. Iggy, Warden, plane, and shared connector-runtime names follow their kind. Connector instances use their activated instance key. Each configuration keeps its own history, and all values are encrypted at rest. `runtime_version` records the release used to validate a saved configuration. After an upgrade, compare it with the installed release before reusing old values. Creation, activation, and deletion require `deployment:config:manage`. Reading values or schemas requires `deployment:config:read`. ## Configuration Scope Topic retention, segment size, and durability belong to topic configuration. They are not arbitrary server overrides. See [Server & Durability](/deployments/server#server-configuration). Use the deployed runtime's schema before changing values. Warden, plane, and shared connector-runtime changes require administrative access. ## Iggy Configuration ### Get Config Schema The TCP section includes entries such as this: ```json { "tcp": { "name": "TCP", "description": "TCP listener configuration.", "schema": [ { "key": "IGGY_TCP_ENABLED", "name": "TCP server enabled", "description": "Determines if the TCP server is active.", "default_value": true, "kind": "bool", "editable": true, "secret": false, "requirements": [], "rules": [] } ] } } ``` Each entry contains `key`, `name`, `description`, `default_value`, `kind`, `editable`, `secret`, `requirements`, and `rules`. `kind` identifies the type, and `rules` defines constraints such as minimums and maximums. ### Create a Configuration | Field | Required | Description | | ---------- | -------- | ---------------------------------------------------------------------------------- | | `name` | No | Omit or use `iggy` for Iggy. Connector updates must use the activated instance key | | `values` | Yes | Key-value pairs validated against the schema | | `activate` | No | `true` to immediately promote this version as primary (default `false`) | A successful request returns `201 Created` with the new configuration ID in `ld-config`. ### Get Active Configuration ### List All Configurations ### Get a Specific Configuration ### List Version History ### Get a Specific Version ### Activate a Config Version A successful request returns `204 No Content`. ### Delete a Configuration A successful request returns `204 No Content`. *** ## Connector Configuration Connector configuration uses the same versioning model. Kinds follow `connector:{type}:{key}`, such as `connector:sink:postgres` or `connector:source:random`. ### Get Connector Config Schema ### Create a Connector Config A successful request returns `201 Created` with `ld-config`. Saving an existing name creates another version. ### Get Active Connector Config ### List Connector Config Versions ### Activate a Connector Config Version A successful request returns `204 No Content`. ### Delete a Connector Config *** ## Tasks After activation, create a reconfigure task to apply the version on deployment nodes. ### Apply Configuration Changes For connectors, use `"type": "connectors:reconfigure"`. This task requires `deployment:task:manage`. ### List Tasks Source: https://docs.laserdata.com/api/configuration --- # Connectors Connectors run compiled Rust plugins in deployments. Use the Main API for activation and the Supervisor API for instances. Activation and deletion require `deployment:connector:manage`. Listing requires `deployment:connector:read`. ## Activation (Main API) ### List Available Connectors | Query Parameter | Description | | --------------- | ---------------------------- | | `type` | Filter by `source` or `sink` | ### Activate a Connector | Field | Required | Description | | ---------------- | -------- | ------------------------------------------------------------------------------------------------- | | `connector_key` | Yes | Connector identifier (e.g. `postgres`, `elasticsearch`, `iceberg`) | | `connector_type` | Yes | `source` or `sink` | | `instance_name` | No | Human-readable name for this instance | | `instance_key` | No | Unique key within the deployment (auto-generated if omitted, e.g. `postgres-sink-{operation_id}`) | The endpoint returns `202 Accepted` and creates disabled initial configuration. Enable it under the returned instance key. Instances of one plugin keep separate histories. See [Connector Configuration](/connectors/configuration). *** ## Instances (Supervisor API) An instance is a copy of a connector running on a deployment node. ### List Connector Instances ```json [ { "id": 1, "deployment_id": 611298765432109056, "connector_type": "sink", "connector_key": "postgres", "name": "Telemetry to Postgres", "key": "telemetry-pg-sink", "status": "active" } ] ``` Instance statuses are `pending`, `active`, `inactive`, and `failed`. ### Delete a Connector Instance A successful request returns `204 No Content`. Source: https://docs.laserdata.com/api/connectors --- # Networking VPC peering requires Performance or Enterprise. PrivateLink and Private Service Connect require Enterprise. The tenant plan must also enable private networking. Standard does not include these features. Tier and account requirements apply separately. Use the deployment's `supervisor_url` for networking requests. The regional Supervisor API is shown as `{supervisor_url}` below. Network creation and deletion require `deployment:network:manage`. Reading lists or instructions requires `deployment:network:read`. Access rules use `deployment:access:manage` and `deployment:access:read`. ## Network Info ### Get Network Info *** ## AWS VPC Peering AWS Managed deployments support this feature on Performance or Enterprise, with private networking enabled in the account plan. ### Create VPC Peering | Field | Required | Description | | --------------- | -------- | ---------------------------------------------------------------- | | `name` | Yes | Name for the peering connection | | `peer_vpc_id` | Yes | Your VPC ID (e.g. `vpc-0abc123...`) | | `peer_owner_id` | Yes | Your 12-digit AWS Account ID | | `peer_vpc_cidr` | Yes | Your VPC CIDR block. Must not overlap with the deployment subnet | | `peer_region` | No | Your VPC region if different from the deployment region | | `remarks` | No | Optional description | ### List VPC Peerings ```json [ { "id": 1, "name": "app-to-iggy", "peering_connection_id": "pcx-0abc123def456789a", "requester_vpc_id": "vpc-deployment", "requester_cidr": "10.0.0.0/16", "accepter_vpc_id": "vpc-0abc123def456789a", "accepter_cidr": "172.16.0.0/16", "requester_region": "us-west-1", "accepter_region": "us-west-2", "requester_owner_id": "987654321098", "accepter_owner_id": "123456789012", "status": "pending_acceptance", "remarks": "Application VPC to deployment", "created_at": "2025-01-15T10:30:00Z" } ] ``` ### Get Peering Setup Instructions ### Delete VPC Peering A successful request returns `204 No Content`. *** ## AWS PrivateLink PrivateLink exposes an AWS Managed deployment as a VPC Endpoint Service. It requires Enterprise and private networking access in the account plan. ### Create Endpoint Service | Field | Required | Description | | --------------------- | -------- | ---------------------------------------------------------------------------- | | `name` | Yes | Unique name for the endpoint service | | `acceptance_required` | No | Require manual approval of each new endpoint connection (default `true`) | | `allowed_principals` | No | AWS IAM ARNs permitted to create endpoints. Empty = any account can request. | | `remarks` | No | Optional description | ### List Endpoint Services ```json [ { "id": 1, "name": "iggy-endpoint-service", "vpc_endpoint_service_id": "vpce-svc-0abc123def456789a", "service_name": "com.amazonaws.vpce.us-west-1.vpce-svc-0abc123def456789a", "acceptance_required": true, "allowed_principals": ["arn:aws:iam::123456789012:root"], "state": "available", "remarks": "PrivateLink for production consumers", "created_at": "2025-01-15T10:30:00Z" } ] ``` ### Delete Endpoint Service *** ## GCP VPC Peering GCP Managed deployments support this feature on Performance or Enterprise, with private networking enabled in the account plan. ### Create GCP VPC Peering | Field | Required | Description | | ----------------- | -------- | -------------------------------------------------------------------- | | `name` | Yes | Name for the peering connection | | `peer_vpc_name` | Yes | Your GCP VPC network name (lowercase, digits, hyphens, max 63 chars) | | `peer_project_id` | Yes | Your GCP project ID (6-30 chars) | | `peer_vpc_cidr` | Yes | Your VPC CIDR block. Must not overlap with the deployment subnet | | `remarks` | No | Optional description | The endpoint returns `204 No Content`. Peering remains `inactive` until you create its reciprocal connection in GCP Console. ### List GCP VPC Peerings Peering states are `inactive`, `active`, and `deleted`. ### Get GCP Peering Instructions ### Delete GCP VPC Peering A successful request returns `204 No Content`. *** ## GCP Private Service Connect PSC exposes a GCP Managed deployment through a service attachment. It requires Enterprise and private networking access in the account plan. ### Create Service Attachment | Field | Required | Description | | ----------------------- | -------- | -------------------------------------------------------------------- | | `name` | Yes | Unique name for the service attachment | | `connection_preference` | No | `accept_manual` (default, requires approval) or `accept_automatic` | | `consumer_accept_lists` | No | GCP project IDs allowed to connect. Empty = any project can request. | | `enable_proxy_protocol` | No | Include original client IP in connection header (default `false`) | | `remarks` | No | Optional description | A successful request returns `204 No Content`. ### List Service Attachments ```json [ { "id": 1, "name": "iggy-psc-attachment", "service_attachment_uri": "projects/ld-prod/regions/us-central1/serviceAttachments/iggy-psc-attachment", "connection_preference": "accept_manual", "consumer_accept_lists": ["my-gcp-project-123"], "state": "active", "remarks": "PSC for production consumers", "created_at": "2026-03-20T10:30:00Z" } ] ``` Attachment states are `pending`, `active`, and `closed`. ### Get PSC Setup Instructions ### List PSC Connections ```json [ { "id": 1, "connection_id": "psc-conn-xyz789", "consumer_project_id": "my-gcp-project-123", "consumer_network": "projects/my-gcp-project-123/global/networks/default", "consumer_forwarding_rule": "projects/my-gcp-project-123/regions/us-central1/forwardingRules/psc-fr-1", "status": "accepted", "created_at": "2026-03-21T14:00:00Z" } ] ``` ### Delete Service Attachment A successful request returns `204 No Content`. *** ## Access Rules Access rules permit source IP ranges. Paid deployments need a rule before clients can connect. Managed Free deployments include a global rule. Free deployments use `0.0.0.0/0` initially. You can delete or replace it. ### Create Access Rule | Field | Required | Description | | ------------- | -------- | --------------------------------------------------------------------------------- | | `name` | Yes | Unique name within the deployment | | `cidr_blocks` | Yes | Array of IPv4 CIDR blocks (at least one required) | | `rules` | Yes | Object of protocol toggles: `iggy_tcp`, `iggy_http`, `iggy_websocket`, `iggy_udp` | | `valid_to` | No | Optional expiry timestamp. Rule is no longer enforced after this time. | | `remarks` | No | Optional description | `iggy_http` also controls Warden's HTTP proxy for [Stream UI](/deployments/warden#stream-ui). Permit the browser IP with `iggy_http: true`. The browser connects directly, so no LaserData-owned IP is required. ### List Access Rules ```json [ { "id": 1, "name": "production-api-access", "remarks": "Production API servers", "rules": { "iggy_http": true, "iggy_tcp": true, "iggy_websocket": false, "iggy_udp": false }, "cidr_blocks": ["10.0.0.0/16", "172.16.0.0/12"], "valid_to": "2026-12-31T23:59:59Z", "created_at": "2025-01-15T10:30:00Z" } ] ``` ### Delete Access Rule Source: https://docs.laserdata.com/api/networking --- # Observability Warden collects telemetry on each node. The platform retains it for the period included in the deployment tier. Use `{supervisor_url}` for these Supervisor API requests. Reading requires `deployment:telemetry:read`. Changing retention requires `deployment:telemetry:manage`. ## Runtime and Response Shapes Runtime paths accept `iggy`, `warden`, `connectors`, `plane`, and available `connector:{id}` instances. Latest deployment metrics use a `nodes` map keyed by node ID. Each node maps runtime names to arrays of samples. A `host` array can contain virtual-machine metrics. Historical requests return paginated samples. Runtime fields appear directly in each sample. Iggy can report leadership, view, voters, quorum, replication gaps, and repair state. Plane metrics include projector lag, decoding counters, replay readiness, ownership, fencing, and backend synchronization. See [Monitoring](/observability#cluster-and-managed-data-health). ## Metrics ### Get Deployment Metrics (Latest) ### Get Node Metrics (Latest) ### Get Node Metrics (Historical) | Query Parameter | Description | | --------------- | ---------------------------------------- | | `page` | Page number (default `1`) | | `results` | Items per page (default `10`, max `100`) | | `from` | Start time (ISO 8601, inclusive) | | `to` | End time (ISO 8601, inclusive) | *** ## Readiness Readiness reports `observed_at`, `agent_ready`, `iggy_ready`, `plane_ready`, `last_probe_at`, and `serving`. Iggy and plane readiness can be unknown. A running process or recent heartbeat alone does not prove readiness to serve. ## Heartbeats Heartbeats report recent liveness. Use them to track uptime and detect restarts. ### Get Deployment Heartbeats (Latest) ### Get Node Heartbeats (Latest) ### Get Node Heartbeats (Historical) *** ## Logs ### Get Deployment Logs | Query Parameter | Description | | --------------- | ------------------------------------------------- | | `page` | Page number | | `results` | Items per page (max `100`) | | `level` | Log level: `any`, `info`, `warn`, `error` | | `message` | Glob filter on message text (e.g. `*connection*`) | | `from` | Start time (ISO 8601) | | `to` | End time (ISO 8601) | You can also send logs to an OpenTelemetry-compatible endpoint. See [Monitoring](/observability) for configuration. Source: https://docs.laserdata.com/api/observability --- # Activities Activities record runtime actions sent by the Supervisor. Each target node receives an entry with its own status. Use the Supervisor API, `{supervisor_url}`, for these requests. ## Supported Runtimes and Actions | Runtime | Actions | | ------------ | ----------------------------------------------------- | | `iggy` | `start`, `stop`, `restart`, `reconfigure:{config_id}` | | `connectors` | `start`, `stop`, `restart`, `reconfigure:{config_id}` | ## Activity Status | Status | Meaning | | ------------ | ----------------------------------------------- | | `processing` | Action dispatched, waiting for node to complete | | `completed` | Action finished successfully | | `rejected` | Action failed or was rejected by the node | ## Trigger a Runtime Action | Path Parameter | Values | | -------------- | ----------------------------------------------------- | | `runtime` | `iggy`, `connectors` | | `action` | `start`, `stop`, `restart`, `reconfigure:{config_id}` | To target all nodes, omit the body or send `{}`: To select nodes, supply their IDs: | Field | Required | Description | | ------- | -------- | ------------------------------------------------------------------- | | `nodes` | No | Array of node IDs to target. Omit or send `{}` to target all nodes. | A successful request returns `202 Accepted`. This action requires `deployment:manage`. *** ## List Activities ```json { "items": [ { "id": 1, "tenant_id": 1, "division_id": 1, "environment_id": 1, "deployment_id": 1, "author_id": 608123456789012345, "runtime": "iggy", "action": "restart", "node_id": 610809900976570889, "status": "completed", "created_at": "2026-03-16T12:00:00Z", "finished_at": "2026-03-16T12:00:05Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` This request requires `deployment:read`. Source: https://docs.laserdata.com/api/activities --- # Snapshots Diagnostic snapshots run 30+ node diagnostics covering the operating system, runtimes, certificates, network, kernel configuration, and logs. Results form a self-contained interactive HTML report. A ZIP of the Iggy data directory is optional. Snapshots help diagnose node problems. They do not protect against data loss. For point-in-time recovery, use [Backups](/api/backups). Use the Supervisor API, `{supervisor_url}`, for these requests. Listing and downloading require `deployment:read`. Creation and deletion require `deployment:manage`. All plans support snapshots. Basic permits 3 with 7-day retention, Pro 5 with 14-day retention, and Enterprise 20 with 90-day retention. *** ## Create a Snapshot | Field | Default | Description | | ---------------- | ------- | ------------------------------------------------------ | | `redact_secrets` | `true` | Mask passwords, keys, and tokens in the report | | `include_iggy` | `true` | Bundle the Iggy data directory state as a separate ZIP | The endpoint returns `204 No Content`. Status changes from `processing` to `completed` after every node finishes. ## List Snapshots ```json { "items": [ { "id": 1, "tenant_id": 615380456123456789, "division_id": 1, "environment_id": 1, "deployment_id": 611298765432109056, "node_id": 610809900976570889, "deployment_name": "events-prod", "name": "snapshot-20260601-103000", "status": "completed", "created_at": "2026-06-01T10:30:00Z", "completed_at": "2026-06-01T10:31:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` ## Download a Snapshot ```json { "url": "https://storage.example.com/snapshots/...?signature=..." } ``` Download the archive promptly because the presigned URL expires. ## Delete a Snapshot A successful request returns `204 No Content`. Source: https://docs.laserdata.com/api/snapshots --- # Backups Backups are experimental. Endpoints, request and response formats, and behavior can change without notice. A backup captures deployment storage volumes at a point in time through AWS EBS snapshots. Restore it to recover data or return to an earlier state after an upgrade. [Diagnostic snapshots](/api/snapshots) capture runtime information for troubleshooting. Backups capture disk data for later restoration. Use the Supervisor API, `{supervisor_url}`, for these requests. Backups require AWS network storage and a Pro or Enterprise plan. Local NVMe SSD deployments do not support them. Listing requires `deployment:read`. Creation, restoration, and deletion require `deployment:manage`. Only one backup can run for a deployment at a time. *** ## List Backups ```json [ { "id": 1, "deployment_id": 611298765432109056, "node_id": 1, "name": "pre-migration", "backup_type": "manual", "status": "completed", "snapshot_ids": ["snap-0abc123def456789a"], "volume_ids": ["vol-0abc123def456789a"], "size_bytes": 53687091200, "region": "us-west-1", "remarks": null, "started_at": "2025-01-15T10:30:00Z", "completed_at": "2025-01-15T10:45:00Z", "expires_at": "2025-02-14T10:30:00Z", "created_at": "2025-01-15T10:30:00Z" } ] ``` `snapshot_ids` and `volume_ids` identify the underlying AWS resources for traceability. ## Create a Backup | Field | Required | Description | | ----------------- | -------- | ---------------------------------------------------- | | `name` | Yes | Human-readable backup name | | `node_id` | No | Specific node to back up. Omit to back up every node | | `backup_type` | No | `manual` (default), `scheduled`, or `pre_upgrade` | | `remarks` | No | Optional free-form description | | `expires_in_days` | No | Auto-delete after this many days | The endpoint returns `204 No Content`. The background operation progresses from `pending` to `in_progress` to `completed`. ## Restore from a Backup The endpoint returns `204 No Content`. Only one restore can run for a deployment at a time. Before restoring, save a fresh backup if later writes must be kept. Restoration replaces current volumes and loses data written after the selected backup. ## Delete a Backup A successful request returns `204 No Content`. Source: https://docs.laserdata.com/api/backups --- # Notifications Notification channels deliver platform and deployment events to external destinations. Channels belong to a tenant or division. Send requests to `https://api.laserdata.cloud`. Reading requires `notifications:read`. Creation, updates, and deletion require `notifications:manage`. ## Notification Types ### List Notification Types ```json [ { "type": "deployment_created", "name": "Deployment created" }, { "type": "deployment_initialized", "name": "Deployment initialized" }, { "type": "high_cpu_usage", "name": "High CPU usage" }, { "type": "node_unreachable", "name": "Node unreachable" } ] ``` *** ## Channels A channel selects a destination through `slack`, `webhook`, or `email`. Slack uses an incoming webhook. Generic webhooks receive HTTPS POST requests with JSON. Destinations are encrypted at rest. ### Create a Tenant Channel | Field | Required | Description | | ------------- | -------- | ------------------------------------------------------------------------- | | `channel` | Yes | Channel kind: `slack`, `webhook`, `email` | | `name` | Yes | Unique name (1-100 chars) | | `destination` | Yes | Target URL or email address | | `settings` | No | Channel-specific settings (Slack channel/username, webhook headers, etc.) | | `remarks` | No | Notes (max 500 chars) | A successful request returns `201 Created`. ### Create a Division Channel ### List Channels ```json { "total_pages": 1, "total_results": 1, "page": 1, "items": [ { "id": 1, "owner_kind": "tenant", "owner_id": 100, "channel": "slack", "name": "production-alerts", "enabled": true, "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ] } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels`. | Query Parameter | Description | | --------------- | ------------------------------------------- | | `channel` | Filter by kind: `slack`, `webhook`, `email` | | `page` | Page number | | `results` | Items per page (max 100) | ### Get Channel ```json { "id": 1, "owner_kind": "tenant", "owner_id": 100, "channel": "slack", "name": "production-alerts", "enabled": true, "destination": "https://hooks.slack.com/services/T00/B00/xxx", "settings": { "slack": { "channel": "#alerts", "username": "LaserData" } }, "remarks": "Primary alerting channel", "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. ### Update Channel Use `enabled: false` to disable delivery without deleting the channel. A successful request returns `204 No Content`. For a division, use `PUT /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. ### Test Channel A successful request returns `204 No Content`. For a division, use `POST /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/test`. ### Delete Channel A successful request returns `204 No Content`. For a division, use `DELETE /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}`. *** ## Subscriptions Subscriptions select event types and optional resource scopes. Without subscriptions, a channel receives every event. ### Create Subscription | Field | Required | Description | | ----------------------- | -------- | ------------------------------------- | | `message_types` | Yes | Non-empty array of event type strings | | `scope_tenant_ids` | No | Limit to specific tenants | | `scope_division_ids` | No | Limit to specific divisions | | `scope_environment_ids` | No | Limit to specific environments | | `scope_deployment_ids` | No | Limit to specific deployments | An event must match every supplied scope level. Empty or omitted levels match all resources. A successful request returns `201 Created`. For a division, use `POST /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### Set Subscriptions (Replace All) The supplied list replaces every existing subscription. A successful request returns `204 No Content`. For a division, use `PUT /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### List Subscriptions ```json { "total_pages": 1, "total_results": 1, "page": 1, "items": [ { "id": 1, "channel_id": 100, "message_types": ["deployment_created", "deployment_deleted"], "scope_tenants": [{ "id": 1, "name": "Acme Corp" }], "scope_divisions": [{ "id": 10, "name": "Platform Eng" }], "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ] } ``` Responses omit empty scope arrays. Each scope entry contains `id` and `name`. For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions`. ### Get Subscription ```json { "id": 1, "channel_id": 100, "message_types": ["deployment_created", "deployment_deleted"], "scope_tenants": [{ "id": 1, "name": "Acme Corp" }], "scope_divisions": [{ "id": 10, "name": "Platform Eng" }], "scope_environments": [{ "id": 100, "name": "Production" }], "scope_deployments": [{ "id": 611298765432109056, "name": "prod-cluster" }], "created_at": "2025-06-01T10:00:00Z", "updated_at": "2025-06-01T10:00:00Z" } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions/{subscription_id}`. ### Delete Subscription A successful request returns `204 No Content`. For a division, use `DELETE /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/subscriptions/{subscription_id}`. *** ## Notification History ### Browse Notifications ```json { "total_pages": 1, "total_results": 5, "page": 1, "items": [ { "id": 1, "tenant_id": 100, "division_id": 10, "environment_id": 1, "deployment_id": 611298765432109056, "message_type": "deployment_initialized", "content": "Deployment prod-cluster has been initialized", "created_at": "2025-06-01T10:30:00Z" } ] } ``` For a division, use `GET /tenants/{tenant_id}/divisions/{division_id}/channels/{channel_id}/notifications`. Source: https://docs.laserdata.com/api/notifications --- # Audit Audit in the Console records tenant activity. Use these endpoints through the main API at `https://api.laserdata.cloud` for programmatic access. Records are appended without changing earlier entries. These requests require `audit:read`. ## Event Types ### List Audit Event Types *** ## Audit Log ### Query Tenant Audit Log ```json { "items": [ { "type": "deployment_created", "name": "Deployment Created", "author": { "id": 608123456789012345, "name": "Jane Smith" }, "api_key": { "id": 67890, "name": "ci-deploy-key" }, "user": null, "division": { "id": 615380456123456790, "name": "Platform Engineering" }, "environment": { "id": 615380456123456791, "name": "production" }, "deployment": { "id": 611298765432109056, "name": "events-prod" }, "data": { "cloud": "aws", "area": "us", "region": "us-east-1", "variant": "managed", "tier": "large", "cluster_kind": "standalone", "public_ip": "static", "storage_type": "network_balanced", "storage_size": 500, "target_network_tput": 10000, "encrypted": true, "protected": true, "retention": { "telemetry": { "logs_days": 90, "metrics_days": 90, "heartbeats_days": 90 } }, "code": "abc123", "name": "events-prod", "domain": "events-prod-abc123.laserdata.cloud", "nodes": [ { "id": 610809900976570889, "name": "node-1", "image_id": "ami-0abc123", "variant": "ubuntu_x64", "storage_type": "network_balanced" } ] }, "correlation_id": "8f4a2b6c9d1e4f3a8b5c7d9e0f1a2b3c", "timestamp": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` Entries use the SDK's `TenantAuditInfo` structure: | Field | Description | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | Event type slug (e.g. `deployment_created`, `member_invited`, `access_rule_added`) | | `name` | Human-readable label for the event type | | `author` | The human who triggered the action. `{id, name}` or `null` for system events. Always set when an interactive session or an API key is involved (the user that owns the key). | | `api_key` | The API key that authored the call when the action came from automation. Omitted for actions taken from an interactive Console session. `{id, name}`. Lets you tell direct user activity apart from key-driven activity for SOC 2 / ISO machine-identity tracking. | | `user` | The user the action was performed on (member operations, role assignments). `null` when not applicable | | `division` / `environment` / `deployment` | Resource scope. Each is `{id, name}` or `null` depending on what the event touches | | `data` | Event-type-specific payload. Shape varies per event (see below). Can be `null` for events without an extra payload | | `correlation_id` | UUID linking this event to others triggered by the same request or workflow. `null` if untracked | | `timestamp` | ISO 8601 time when the event was recorded | `data` contains event-specific details: * `deployment_created` records `{cloud, area, region, variant, tier, cluster_kind, public_ip, storage_type, storage_size, target_network_tput?, encrypted, protected, retention, code, name, domain?, nodes[]}`. Each node includes `id, name, image_id, variant, storage_type`. * `deployment_upgraded` records `{from_tier, to_tier, from_storage, to_storage}`. * `deployment_deleted` records `{name, code, reason?}`. * `access_rule_added` and `access_rule_deleted` record `{cidrs, description?, rules}`, including protocol toggles. * `member_invited` records `{email, roles}`. * `api_key_created` and `api_key_deleted` record `{name, role_id, division_id?}`. * `config_activated` records `{config_id, kind, name, version}`. * `connector_activated` and `connector_deleted` record `{connector_type, connector_key, instance_key, instance_id, version}`. `connector_type` is `source` or `sink`. `connector_key` identifies the catalog plugin, and `instance_key` is its deployment-specific hyphenated name. `instance_id` is numeric, and `version` is the plugin SemVer. The owning `deployment`, `environment`, and `division` appear in the top-level scope fields. Use [`/audit/types`](#list-audit-event-types) to retrieve the complete event-type list for your tenant. | Query Parameter | Description | | ---------------- | -------------------------------------------------------- | | `page` | Page number | | `results` | Items per page (max `100`) | | `from` | Start time (ISO 8601) | | `to` | End time (ISO 8601) | | `user` | Filter by subject user id | | `author` | Filter by acting user id | | `api_key` | Filter by API key id (only events authored via that key) | | `division` | Filter by division id | | `environment` | Filter by environment id | | `deployment` | Filter by deployment id | | `types` | Comma-separated list of event types to include | | `correlation_id` | Find all events sharing a correlation UUID | Source: https://docs.laserdata.com/api/audit --- # API Keys Use API keys for CI/CD, CLI tools, Terraform providers, and other software integrations. These endpoints use `https://api.laserdata.cloud`. Listing keys requires `api_key:read`. Creation, updates, and deletion require `api_key:manage`. ## Create an API Key ### With an Existing Role ### With Inline Permissions | Field | Required | Description | | ------------- | -------- | ----------------------------------------------------------------- | | `name` | Yes | Human-readable name | | `expiry_at` | Yes | Expiration timestamp (ISO 8601, max 365 days) | | `role_id` | No\* | Existing role to assign | | `permissions` | No\* | Inline permissions object (creates a dedicated role) | | `division_id` | No | Scope key to a specific division | | `validate_ip` | No | Enable IP allowlisting (default `false`) | | `allowed_ips` | No | IP addresses or CIDRs to allowlist (requires `validate_ip: true`) | \*Supply `role_id` or `permissions`, but not both. Copy the returned secret immediately. It cannot be retrieved again. ```json { "id": 1, "name": "ci-deploy-key", "secret": "ld_api_key_abcdef...", "role_id": 67890, "expiry_at": "2026-06-01T00:00:00Z", "created_at": "2025-01-15T10:30:00Z" } ``` *** ## List API Keys ```json { "items": [ { "id": 1, "name": "ci-deploy-key", "role_id": 67890, "role_name": "deployer", "division_id": 123, "division_name": "production", "validate_ip": false, "allowed_ips": [], "expiry_at": "2026-06-01T00:00:00Z", "created_at": "2025-01-15T10:30:00Z" } ], "page": 1, "total_results": 1, "total_pages": 1 } ``` *** ## Get API Key ```json { "id": 1, "division_id": null, "division_name": null, "user_id": 608123456789012345, "role_id": 67890, "role_name": "deployer", "name": "ci-deploy-key", "validate_ip": false, "allowed_ips": [], "expiry_at": "2026-06-01T00:00:00Z", "created_at": "2025-01-15T10:30:00Z", "permissions": { "tenant": ["info:read"], "division": ["environment:read"], "environment": ["deployment:read"], "divisions": { "...": "..." } } } ``` The response uses the `ApiKeyDetails` structure from [Self-Introspection](#self-introspection) for the selected tenant key. It requires tenant-level `api_key:read`. *** ## Self-Introspection ```json { "id": 1, "division_id": null, "division_name": null, "user_id": 608123456789012345, "role_id": 67890, "role_name": "deployer", "name": "ci-deploy-key", "validate_ip": false, "allowed_ips": [], "expiry_at": "2026-06-01T00:00:00Z", "created_at": "2025-01-15T10:30:00Z", "permissions": { "tenant": ["info:read"], "division": ["environment:read"], "environment": ["deployment:read"], "divisions": { "...": "..." } } } ``` Use this endpoint to authenticate a key and read its scope. It accepts API-key sessions only and does not return `403` for permission failures. A `403` from another endpoint cannot distinguish an invalid key from insufficient permission. *** ## Update Security Settings *** ## Delete an API Key Source: https://docs.laserdata.com/api/api-keys --- # Cloud Accounts Saved cloud accounts hold AWS IAM roles, VPC IDs, GCP network details, and related credentials. BYOC, VPC Peering, and PrivateLink reuse them. Send these requests to `https://api.laserdata.cloud`. Reading requires `settings:read`. Creation, updates, and deletion require `settings:manage`. ## Create a Cloud Account | Field | Required | Description | | ------------ | -------- | ------------------------------------------------------------ | | `cloud` | Yes | Cloud provider: `aws` or `gcp` | | `name` | Yes | Unique name (1-100 chars) | | `account_id` | Yes | Cloud provider account ID (e.g. AWS 12-digit account number) | | `region` | No | Default region for this account | | `settings` | No | Cloud-specific credentials (see below) | | `remarks` | No | Notes (max 500 chars) | AWS configuration: | Field | Required | Description | | -------------- | -------- | ---------------------------------------------------- | | `identity_arn` | Yes | IAM role ARN that LaserData assumes for provisioning | | `external_id` | Yes | External ID for secure cross-account role assumption | | `vpc_id` | Yes | VPC ID where infrastructure will be provisioned | | `vpc_cidr` | No | CIDR block of the VPC (used for network planning) | GCP configuration: | Field | Required | Description | | ------------- | -------- | ------------------------------------------------ | | `vpc_network` | No | VPC network name for infrastructure provisioning | | `vpc_cidr` | No | CIDR block of the VPC | A successful request returns `201 Created`. *** ## List Cloud Accounts ```json { "total_pages": 1, "total_results": 1, "page": 1, "items": [ { "id": 1, "cloud": "aws", "name": "production-aws", "account_id": "123456789012", "region": "us-west-1", "validated_at": null, "status": "active", "created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z" } ] } ``` Account statuses are `active`, `inactive`, `locked`, and `deleted`. | Query Parameter | Description | | --------------- | --------------------------------- | | `name` | Filter by name (contains match) | | `cloud` | Filter by provider: `aws`, `gcp` | | `region` | Filter by region (contains match) | *** ## Get Cloud Account ```json { "id": 1, "cloud": "aws", "name": "production-aws", "account_id": "123456789012", "region": "us-west-1", "status": "active", "settings": { "aws": { "identity_arn": "arn:aws:iam::123456789012:role/LaserDataRole", "external_id": "unique-external-id", "vpc_id": "vpc-0abc123def456", "vpc_cidr": "10.0.0.0/16" } }, "remarks": "Main production AWS account", "created_at": "2026-06-01T10:00:00Z", "updated_at": "2026-06-01T10:00:00Z" } ``` *** ## Update Cloud Account | Field | Description | | ------------ | --------------------------------------------------------- | | `name` | New name (must be unique within the tenant) | | `account_id` | Updated cloud account ID | | `region` | Updated region, or `null` to clear | | `settings` | Updated credentials, or `null` to clear | | `remarks` | Updated notes, or `null` to clear | | `status` | Account status: `active`, `inactive`, `locked`, `deleted` | A successful request returns `204 No Content`. *** ## Delete Cloud Account A successful request returns `204 No Content`. Source: https://docs.laserdata.com/api/cloud-accounts --- # Billing The Billing API provides rates, estimates, contact details, usage reports, invoices, payment methods, and credits. Send requests to `https://api.laserdata.cloud`. Tenant billing reads require `billing:read`. Changes require `billing:manage`. The public pricing endpoints below do not require authentication. ## Public Pricing The response contains `currency`, `nodes`, `managed.aws`, `managed.gcp`, and `byoc`. Managed clouds contain `tiers`, `compute`, `network_drive`, `local_nvme`, `network`, and `throughput`. `tiers` describes Standard, Performance, and Enterprise. Each record includes these fields: | Fields | Meaning | | ----------------------------------------------------------------- | ------------------------------------------- | | `monthly_base` | Starting monthly configuration price | | `compute_profile_id`, `storage_profile_id`, `storage_gb_per_node` | Default compute and storage configuration | | `throughput_mb_per_second` | Default traffic estimate | | `availability_mode` | Default availability for the preset | | `included_telemetry_days` | Managed-tier retention: 14, 30, or 90 days | | `vpc_peering`, `private_link` | Access to VPC peering and private endpoints | Managed telemetry retention comes from the tier record, not the compute profile. `network.minimum_monthly` describes networking operations already included in `monthly_base`. Do not add it to the transfer estimate again. Prices use USD under the current published pricing version. ### Estimate Pricing ```bash curl -X POST https://api.laserdata.cloud/pricing/estimate \ -H "Content-Type: application/json" \ -d '{ "variant": "managed", "managed_tier": "standard", "cloud": "aws", "region": "us-east-1", "compute_profile_id": "small", "storage_profile_id": "network_drive", "storage_gb_per_node": 100, "throughput_mb_per_second": "0.1", "network_scope": "same_region", "availability_mode": "single_az" }' ``` All shown fields except `region` are required for managed estimates. Omit `region` to use `us-east-1` for AWS or `us-central1` for GCP. `managed_tier` accepts `standard`, `performance`, or `enterprise`. `compute_profile_id` selects a size such as `small` or `large`. `storage_profile_id` accepts `network_drive` or `local_nvme`. Network scopes are `same_region`, `cross_region_same_cloud`, and `cross_cloud`. Local NVMe requires Performance or Enterprise, while Multi AZ requires Enterprise. Free is excluded because it has no charge. Throughput uses decimal MB/s, with a self-service range of 0.1 to 100 MB/s. Compute sizing and Local NVMe impose further bounds. Network Drive supports 100 to 30,000 GB per node. Read current limits from the rate card instead of fixing them in application code. The response includes `monthly_base`, `monthly_total`, `currency`, `pricing_version`, and these objects: | Object | Fields | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `breakdown` | `compute`, `storage`, `network` | | `usage` | `estimated_customer_gb_per_month`, `provisioned_storage_gb_total`, `partitions_per_node_at_default_segment`, `partitions_per_node_at_min_segment` | | `deployment` | `nodes`, `compute_profile_label`, `storage_profile_label`, `availability_mode`, `included_telemetry_days` | | `capacity` | `shards_per_node`, `default_segment_bytes`, `min_segment_bytes` | `monthly_base` is the monthly configuration price for compute, storage, and tier features. `monthly_total` adds estimated data transfer from `breakdown.network`. Transfer is billed on actual usage, not the selected throughput. Estimates use 30 days of symmetric traffic. For AWS Standard at 0.1 MB/s, the response gives $199 per month plus $15.552 estimated transfer. The total is $214.552, displayed as $214.55. See [Billing & Pricing](/organization/billing#what-you-pay-for). Partitions are dynamic and require no upfront count. The segment-based capacity fields are diagnostic storage calculations, not guaranteed active-partition limits. For BYOC, send `{"variant":"byoc","vcpus_per_node":4}`. The estimate uses three nodes for a full 720-hour month. Billing uses actual active capacity and agreed terms. ## Billing Info ### Get Billing Info ```json { "name": "John Doe", "company": "Acme Corp", "address": "123 Main St", "city": "San Francisco", "state": "CA", "postal_code": "94105", "country": "US", "tax_id": "US123456789", "email": "billing@acme.com", "phone": "+1-555-0100" } ``` ### Update Billing Info A successful request returns `204 No Content`. *** ## Usage Reports ### List Usage Reports ### Get Usage Report Detailed reports contain `compute_cost`, `storage_cost`, `network_cost`, and `total_cost`. Managed reports also expose `monthly_base_charge` and `network_usage_cost` before negotiated adjustments. The first includes prorated reserved resources and any billable additional capacity. The second describes measured transfer. Do not add these explanatory amounts to `total_cost` again. ## Credits and Promo Codes Redeem a code with `{"code":"YOUR_PROMO_CODE"}`. Its credit reduces later invoices until used or expired. Marketplace-billed tenants cannot redeem promo codes. *** ## Invoices ### List Invoices ```json { "items": [ { "id": 1, "tenant_id": 100, "division_id": null, "number": "INV-01-2026-100", "period_start": "2026-01-01T00:00:00Z", "period_end": "2026-02-01T00:00:00Z", "status": "paid", "currency": "USD", "total": "289.22", "created_at": "2026-02-01T00:00:00Z" } ], "page": 1, "total_pages": 1, "total_results": 1 } ``` Customer endpoints do not return draft, pending, or cancelled invoices. Payment is due 14 days after invoice generation. ### Get Invoice ### Download Invoice PDF *** ## Payment Methods ### Get Payment Method ### Create Payment Intent ### Delete Payment Method *** ## Spend Limits Configure a deployment threshold through [Update Spend Limit](/api/deployments#update-spend-limit). Source: https://docs.laserdata.com/api/billing