Managed data

This page covers the managed plane that serves Queries and views, Key-value state, and the rest of the managed features. Read it when you run a managed deployment in production.

Streaming and managed calls share one authenticated Iggy connection. Iggy forwards managed requests, such as queries, key-value calls, and destination changes, to the managed plane. Projection and binding changes are published to the control log instead. Filtered group readers and the dedicated fenced-lease client open their own connections. Laser Stack runs both services locally, and LaserData Cloud runs them for you.

Check backend readiness

A connection to Iggy does not mean the managed backend is ready. A running backend can still be replaying or restoring data, and its managed operations stay unavailable until it reports ready. After startup, failover, or a backend restart, refresh the capabilities or wait for readiness with a deadline:

import { unreadyBackends } from "@laserdata/laser-sdk"

const caps = await laser.waitUntilReady(30_000)
console.log(caps.query.available, caps.kv.fencedLeases, unreadyBackends(caps).length)

use std::time::Duration;

let caps = laser.wait_until_ready(Duration::from_secs(30)).await?;
println!(
    "{} {} {}",
    caps.query.available,
    caps.kv.fenced_leases,
    caps.unready_backends().count()
);

caps = await laser.wait_until_ready(30_000)
print(caps.query.available, caps.kv.fenced_leases, len(caps.unready_backends()))

refresh_capabilities() (TypeScript: refreshCapabilities()) reads a fresh snapshot without waiting. capabilities() returns the held snapshot, and probes again by itself when that snapshot has no managed plane and its last probe is at least one second old. The wait returns an unsupported error when the server has no managed backend descriptors. See Connect for the connection side.

Check the feature you need on its own. Query support does not imply fenced leases, destinations, or every consistency level. The capability groups are objects in all three SDKs, with camelCase fields in TypeScript:

The flags managed, graph, forks, a2a_gateway, sessions, watch, authz, and stream_tenancy sit directly on the capabilities, beside versions, backends, and hello. Every feature is off until the server reports it.

Backend descriptors

Each backend descriptor names a resource ID, a mode (operational or lakehouse), a label, the implementation kind and version, the observed backend generation, the runtime_configuration_revision, the desired and observed state, the readiness state with reason codes, and the materialization, query, schema, maintenance, and limit support. A descriptor reports what the deployment actually serves. A destination type that exists in the protocol is not a promise that every backend implements it.

Rust returns iterators from the first two. Python returns lists.

Operational indexes

A projection names the fields to extract. Its binding selects a source and an index. An optional backend pins a resource ID and generation. Without it, the binding uses replica-local operational storage. notify adds the binding to the change feed. An optional retention sets how long rows live, independent of the source topic. Without it, the deployment default applies, which mirrors the log unless an operator changed it: rows go when Iggy removes the messages that produced them.

The serialized form of a binding, as Python and the protocol use it:

{
  "source": { "stream": "telemetry", "topic": "readings" },
  "allowed_projections": ["readings_v1.v1"],
  "default_projection": "readings_v1.v1",
  "index": "readings_v1",
  "notify": true
}

TypeScript uses the same fields in camelCase. All three SDKs also have a binding builder. See Queries and views in depth.

Destinations and query routes

A materialization destination writes a projection to a lakehouse table, such as Parquet or Iceberg v2. Each destination generation is immutable. It fixes the source scope, the partition recreation policy, the projection version, the schema fingerprint, the backend generation, the physical table, the start and new-partition behavior, the error policy, and the desired state. Progress and ownership live in checkpoint status records: prepared and completed attempts, repairs, retention gaps, blocking errors, and the current lease.

A query names its target explicitly. laser.query(index) reads an operational index. laser.query_lakehouse(destination, generation) (TypeScript: queryLakehouse) reads one destination generation. A query route gives a name to one of those targets. Generations and source incarnations tell a recreated table or topic apart from an older one with the same name.

laser.destinations() exposes the destination and checkpoint operations in all three SDKs, with camelCase names in TypeScript. Every mutation takes the expected global state revision first, plus the revisions that the operation checks.

accept_retention_gap, record_repair, and supersede_generation need a signed supervisor assertion. Rust and TypeScript send any mutation with one through mutate_with_supervisor_assertion. Python passes supervisor_assertion= to mutate. Reads take a consistency of linearizable or potentially_stale.

When a mutation fails on a revision conflict, read the current state before you try another change. Destination lease acquisition uses an expected lease sequence and epoch, which are different rules from ordinary key-value leases. Destination lease durations are microseconds in all three SDKs (lease_duration_micros, TypeScript leaseDurationMicros as a bigint), while key-value lease TTLs are milliseconds in Python and TypeScript.

Source retention can remove records before a destination reads them. A recorded retention gap blocks progress until a repair or an explicit acceptance. It never counts as successful ingestion.

Typed query results

A result contains fields, rows, page, and context. The values in each row follow the order of fields. Each value is a tagged typed value that keeps booleans, integers, longs, floats and doubles, decimals, dates in days, times and timestamps in microseconds, UUIDs, fixed and variable binary values, strings, lists, maps, structs, and nulls apart. A vector or lexical query also returns a score on each row.

field_index(name) (TypeScript: queryResultFieldIndex(result, name)) returns a field's position. typedValueDiagnosticText(value) in TypeScript and ls.typed_value_diagnostic_text(value) in Python render any typed value as text.

Use typed accessors for logic and text accessors for display. The client rejects a result whose rows have the wrong number of values, or a value that does not match its field type. A result, its later pages, its status, and its cancellation stay bound to the execution ID of the original query. context records the engine, the target, backend identity and generations, the requested and delivered consistency, and resource use. Lakehouse results add the destination generation, table, and snapshot.

Retry and recovery

Every managed mutation carries a stable operation identity, so the plane can return the saved outcome when the same mutation arrives twice. Reads can reconnect and retry. Mutations retry only with their original identity, and a deterministic rejection, such as invalid input, is never retried. A lost reply does not prove that the mutation failed.

Lease acquisition is the exception that needs care. The SDK never repeats a lease acquisition after an uncertain disconnect. Ordinary key-value lease calls wait out the requested lease lifetime and then report the uncertainty. Until that lifetime has passed, do not assume that no lease exists.

The fenced-lease client, FencedLeaseClient in all three SDKs, prepares each acquisition, renewal, release, or fenced write once. The prepared request keeps its operation ID and exact bytes. Repeat it only when its recovery action allows that: renew and release can repeat, an uncertain acquisition waits out the lease lifetime, and a fenced write needs to be checked against its target. Each attempt times out after 10 seconds by default. A renewal keeps the fencing token, and a release revokes it at once. Before planning a protected write, wait for a read that includes the lease's commit position. See Key-value state in depth.