# Warden Agent

Warden runs on every deployment node. It manages local runtimes and exchanges tasks, configuration, and telemetry with the LaserData control plane. A runtime is a managed process, such as Iggy or Connectors.

## Why Pull-Based

Warden retrieves tasks instead of accepting inbound management connections. It starts outbound HTTPS connections on port 443. The control plane does not need inbound ports, SSH, AWS SSM, or a bastion host to manage a node.

Warden still runs authorized tasks from the control plane. The identity of that service and its task-signing keys remain part of the security boundary. Outbound-only connections do not remove that trust requirement.

Application traffic connects directly to deployment endpoints. Paid deployments need [access rules](/networking/access-rules) first. Managed Free deployments include an initial global rule.

## Architecture

```mermaid
graph LR
  Clients[Your Applications]
  Browser[Console Stream UI<br/>in your browser]

  subgraph CP[LaserData Control Plane]
    Tasks[Tasks, Configs, TLS Certs]
    Telemetry[Heartbeats, Metrics, Logs]
  end

  subgraph Node[Deployment Node]
    Warden[Warden Agent]
    Iggy[Iggy Server]
    Connectors[Connectors]
  end

  Clients -->|TLS| Iggy
  Browser -->|HTTPS, signed session| Warden
  Tasks ==>|PULL| Warden
  Warden -.->|PUSH| Telemetry
  Warden -->|manages| Iggy
  Warden -->|manages| Connectors
  Warden -->|proxies HTTP to| Iggy
```

## What Warden Manages

Warden manages these services on each node:

| Responsibility | How It Works |
| --- | --- |
| Iggy server lifecycle | Start, stop, restart, and manage the Iggy process |
| Connectors runtime | Start and manage the optional [Connectors](/connectors) runtime |
| Kafka gateway | Start and manage the optional [Kafka gateway](/kafka-gateway) on each node where it is enabled |
| HTTP proxy for Stream UI | Authenticated reverse proxy in front of the Iggy HTTP API, consumed by the browser-side Console [Stream UI](#stream-ui) |
| Configuration | Pull and apply [configuration changes](/deployments/configuration) from the control plane |
| Upgrades | Download, verify, and swap new versions of Iggy, Connectors, and itself |
| TLS certificates | Pull and rotate certificates automatically before expiry |
| Telemetry | Push [metrics, heartbeats, and logs](/observability) to the control plane |
| Metadata sync | Push stream and topic changes so the Console lists them without connecting to the deployment |

## Partition ownership

In a cluster, every partition has its own primary node, and only the primary's plane writes that partition to a shared destination. Warden tells its local plane every few seconds whether the node is the metadata primary, which owns table-wide work and connector state. The plane finds the partitions its node leads itself. It looks up the primary of every partition it writes to a shared destination, and only those, against its own replica over the standard Iggy TCP client.

A scan runs up to 16 lookups at once, each with a 5 second limit, spread evenly over an interval of 5 to 60 seconds that follows the number of partitions against a budget of 200 lookups per second by default. A failed lookup is retried once in the same scan. A lookup result keeps a node eligible for 3 minutes, so one failed lookup never drops a live owner. Every 5 seconds the plane also checks that its replica reaches a majority of the cluster. A node cut off from the majority gives up every partition and table-wide work until the majority is back, so an isolated former primary stops writing.

## Partition replication

On a cluster, Warden reads three partition replication gauges from its node's Iggy along with its other metrics: partitions under-replicated, partitions below write quorum, and partitions owing retirement, each summed over the node's shards. The platform stores them with the node's other [metrics](/observability#cluster-and-managed-data-health) and raises an [alert](/observability#monitoring-alerts) when one stays above zero for 5 minutes. Before a Compute or Storage upgrade, a configuration rollout, or a runtime stop, restart, or reconfigure disrupts the next node, it waits while any node leads partitions that are under-replicated or below write quorum. Partitions owing retirement do not hold it up. See [Upgrading a Deployment](/deployments#upgrading-a-deployment).

## Stream UI

Stream UI browses streams, topics, partitions, messages, consumer groups, and connected clients. It loads through the [Console](https://laserdata.cloud) and runs in your browser. Application data travels directly between the browser and the deployment node, unless [data access through LaserData](#data-access-through-laserdata) is on.

### Data isolation

Stream UI separates data access from platform management:

* With data access off, the browser calls Warden's HTTP proxy on the deployment node directly.
* The LaserData backend serves static application files. With data access off, it does not receive stream contents, message bodies, or query results.
* Stream UI runs in a sandboxed iframe with its own origin, isolated from the surrounding Console. The surrounding Console cannot read its data.
* Each user receives a signed session token for one deployment. Tokens expire in minutes. Warden makes sure that the signature is valid on every request, using the Ed25519 key pair that also signs tasks.

### Network requirements

Warden's HTTP proxy uses the Iggy HTTP API ports, 80 and 443. The [`iggy_http`](/networking/access-rules) toggle therefore controls browser access. With data access off, add the browser IP to an access rule with `iggy_http: true` on the deployment. LaserData does not proxy these requests then, so the rule needs no LaserData-owned IP. With data access on, LaserData's own [data access rule](/networking/access-rules#data-access-rule) admits its addresses, and Stream UI needs no browser rule.

### Data access through LaserData

[Turning data access on](/api/deployments#update-data-access) sends Stream UI's requests through the Supervisor that runs the deployment, which forwards each one to the deployment's Warden. API key clients, such as the [Laser CLI](/cli), use the same path. Stream contents, message bodies, and query results then pass through LaserData.

* The Console gives Stream UI a proxy token instead of the Warden session token. It is signed with Ed25519, lasts five minutes, and is bound to one deployment, one user, and that user's Console session. The Console session cookie never reaches Stream UI.
* The Supervisor checks every request against the live Console session. Signing out, leaving the organization, or losing a permission takes effect on the next request, before the token expires. A session bound to your IP address and browser binds the token the same way.
* An API key reaches the proxy without a token: send the request to `{supervisor_url}/deployments/{deployment_id}/data_access/sso/iggy/{path}`, where `{path}` is the Iggy HTTP path, with the `ld-api-key` header. The Supervisor checks the key on every request like any other API call, including its rate limit and allowed IP addresses. Revocation and permission changes can take a few seconds to reach all API servers because authentication is cached. Warden attributes the request to the user who created the key and never receives the key. A request that carries a bearer credential is decided by that credential alone, so an invalid or empty one is refused even beside a valid key.
* Reading needs `deployment:read`. Creating, changing, or deleting resources, sending messages, and polling with automatic offset commits need `deployment:manage`. Requests outside the operations of Stream UI and the Laser SDK HTTP client are refused.
* Each request is forwarded once and never retried. When Warden does not answer within 65 seconds, the request fails with `504` and may still complete on the deployment. Request bodies are limited to 10 MiB.
* Warden's own checks still apply, including the refusal of sends to protected session topics.
* The organization's [audit log](/api/audit) records each request that changes something as `deployment_data_accessed`, and the first read of each member or API key per deployment and UTC day. Bodies and query strings are never stored.
* An unknown API key returns `401` with `authentication_required`, a key used from an address it does not allow returns `403` with `access_forbidden`, and a key above its rate limit returns `429` with `rate_limit_exceeded`. A missing permission returns `403` with `insufficient_permissions`, an operation outside the proxy's list returns `403` with `data_access_route_not_allowed`, and data access off returns `401` with `data_access_disabled`. A deployment of another organization returns `404` with `resource_not_found`, whether its data access is on or off.
* Turning data access off refuses the next proxied request. Stream UI receives a new token for the direct path, and the Laser CLI connects to the deployment directly, which needs an [access rule](/networking/access-rules) for its address.

### Managed data

The data plane is the service that turns records into queryable views and state. A backend is a storage or query engine that this service uses. The Backends page configures those engines and requires `deployment:config:manage`. You do not need to open this page to send and read ordinary Iggy messages.

If the deployment runs the managed data plane, Stream UI also exposes its supported tools. Their requests take the same path as the rest of Stream UI.

The tools include these operations:

* Create, replace, and drop projections, definitions that extract events into queryable tables. Bind them to source topics and targets.
* Query a projection with the structured JSON query language. It supports filters, sorting, paging, aggregation, and vector search. Operational SQL backends also accept authorized read-only SQL through the managed query API.
* Register and drop writer schemas in the schema registry.
* Read and edit namespaced keys and values. An optional TTL sets each key's lifetime.
* Create forks, copy-on-write branches of a read model. Promote a fork onto the trunk or discard it.

The Managed data area appears only when the deployment advertises support. Individual tools appear according to its capabilities.

## Metadata sync

The Console lists streams, topics, and agentic streams from metadata that the Supervisor stores, so a list never connects to a deployment. Warden keeps that copy current from Iggy's metadata change log, the session index and agent registry of every agentic stream, and each node's live agent connections. No Console view lists agents or agent sessions. The [agent session API](/api/deployments#list-workspace-agent-sessions-supervisor-api) and the [agent API](/api/deployments#list-workspace-agents-supervisor-api) serve them from the metadata the Supervisor stores, and session contents stay in [Stream UI](#stream-ui).

Every node of a cluster applies the same metadata changes in the same order, so one node per deployment is enough. The Supervisor chooses that node and keeps it while it stays healthy. Warden on the chosen node reads the change log of its own Iggy every 5 seconds and pushes new changes in order. The other nodes only report whether their Iggy serves the change log. When the chosen node stops reporting, the Supervisor chooses another one, which continues from the last position the Supervisor accepted. Each push names that position, so a lost reply or a retry never applies a change twice, and a replaced node cannot write.

Warden saves the position the Supervisor accepted on the node's disk, so a restarted Warden continues where it stopped. While the Supervisor is unreachable, Warden retries and keeps nothing else, because Iggy keeps the change log on every node. If the outage outlasts the change log, Warden rebuilds the Supervisor's copy from the current state. The previous copy stays visible until the rebuild completes, and the lists mark its rows with `resyncing` meanwhile. When Iggy's change log itself was replaced, the previous copy leaves the lists at once and the Supervisor deletes it in the background before the rebuild is published. When the chosen node reports for 15 minutes, with no progress in between, that its Iggy has not reached the position the Supervisor accepted, for example after the deployment's data was restored from an older backup, the Supervisor deletes its copy the same way and Warden rebuilds it. While nothing changes, the chosen node confirms its position every minute, so the Supervisor knows the copy is current.

The chosen node also samples sizes and message counts from its own replica and sends the values that changed. Counter sweeps start every minute by default. A sweep whose own work, waits for the change sync and lookup pacing aside, takes longer than a tenth of the interval moves the next one out to ten times that work, so large deployments update less often. Lists show each value with the time it was read. Values from different nodes are never added up. A stream or topic whose name the Supervisor cannot store, such as one holding a NUL character, stays out of the lists.

On the same schedule, on deployments that run the AGDX plane, the chosen node reads the AGDX catalog of its Iggy: saved consumer filters with the consumer group that owns each group's own filter, the consumer groups bound to them, row and graph projections with the topics they read, forks of the managed tables, and the numbers of destinations, schemas in the deployment-wide registry, and key-value namespaces with their entries. When the catalog changed, Warden sends its names, ids, states, revisions, digests, and counts, never a filter policy, a projection definition, a schema, a key, a value, or a row. The stream and topic lists then show which ones have filtered consumer groups or projections, and the deployment table shows the catalog counts. A catalog sent in several parts shows only once its last part arrives. Warden binds its Iggy user a read-only role for these lists.

Every change the Supervisor applies from the change log also becomes a [deployment activity](/api/audit#query-deployment-activity) entry: a stream, topic, partition count, consumer group, Iggy user, or personal access token that was created, updated, or deleted, with the Iggy user that made the change when Iggy records one. The chosen node also sends the deployment's Iggy user ids and usernames: all of them after it starts and once a day, and in between only the users that were added or renamed, so every entry can name its Iggy user. It never sends passwords or permissions. A rebuild from the current state adds no entries for the resources it finds. When changes were lost because the rebuild was needed, one gap entry marks the interval, and a change the Supervisor could not store appears as a skipped entry.

On the same schedule, on deployments that run the AGDX plane, the chosen node reads the managed backend's audit log of applied AGDX changes from the deployment's ops stream, `_agdx`, and pushes the records after the position the Supervisor accepted. The managed backend writes this audit log as a new log in the ops stream, and Warden reads it from its start, so the changes made through the managed backend appear with their actor and time. Changes made before the deployment ran a managed backend that writes this log are not in it. A record of a filter, filter binding, projection, schema, or fork change names the resource with the ids, names, revisions, versions, and kinds of the change, the Iggy user whose request the managed backend applied, and the time it applied the change. The Supervisor turns each such record into a deployment activity entry with that user and time. No record carries a filter policy, a projection definition, a schema body, a value, or a row. A restart or a newly chosen node continues from the accepted position, and a record Warden cannot decode becomes a skipped entry instead of stopping the log. The managed backend seals every record with the deployment's catalog signing key over the record's id and content. When that key is set, Warden drops a record whose seal is missing or does not verify, so a record written or replayed by anyone else never becomes activity.

For every agentic stream, the chosen node also follows the stream's session index and pushes session summaries on the same schedule. Warden pages through the deployment's agentic streams with no limit on their number, and when one run has more to push than it sends at once, the next run continues where it stopped. The first read of a stream lists all its sessions. Later reads follow the index's change feed and push only the sessions that changed and the ids of the sessions the index removed. When the feed cannot report every change, for example after the index was rebuilt, Warden lists that stream again, and a listing sent in several parts shows only once its last part arrives. A summary carries the session id, label, namespace, agent, parent and root, status, idle and over-budget flags, start, end, event, and heartbeat times, the counts of events, model calls, tool calls, tokens, errors, and held records, the cost, the budget, and the SDK language and version. Labels and namespaces are cut to 255 characters. Tags, events, state, memory, and graph data never leave the deployment. The read-only role Warden binds for the catalog also grants these session reads.

On the same schedule, the chosen node folds each agentic stream's `agent.registry` topic into the latest card of each agent and the quarantine facts, as a Laser SDK registry does, drops cards past their ttl, and pushes a stream's cards when they changed. A card carries the agent id, display name, version, the skills with their health, load, and maximum concurrency, the quarantine flag, and the card time and expiry. Names and versions are cut to 255 characters. Skill schemas and card metadata never leave the deployment. Every node, not only the chosen one, reads its own Iggy's live connections every 5 seconds and pushes the agent id, inbox, Iggy user, transport, and address of each connection whose presence names an agent, when they change and otherwise once a minute by default. The Supervisor drops a node's connections when three of those pushes are missed. Connections of other clients never leave the deployment.

Metadata sync can be turned off per deployment with [Update Metadata Sync](/api/deployments#update-metadata-sync). The Supervisor then accepts none of the deployment's metadata, and every node's Warden stops reading and sending it after it reloads its configuration. The stored records, agents and agent sessions included, can be deleted at the same time. Deployment activity is never deleted, not by this purge and not when the deployment is deleted. The managed backend's audit log keeps its records while sync is off, so the AGDX changes it still holds appear once sync is on again. Other changes made while sync is off produce no activity, and the rebuild after it is turned on again marks that interval as a gap.

Every current Warden and [Laser Streams](/deployments/server) release supports metadata sync and shows Iggy usernames in deployment activity. A deployment created before metadata sync starts syncing once its Warden credentials are renewed. Until its first sync completes, its streams and topics do not appear in the Console lists.

## Task System

Configuration changes, upgrades, and certificate rotation follow the same signed task sequence:

1. A user or automated process requests an action through the Console or API.
2. The control plane creates and signs a task.
3. Warden retrieves the task on its next polling cycle.
4. Warden makes sure that the signature is valid before it runs the task.
5. Warden reports success or failure to the control plane.

Warden rejects invalid signatures. This protects tasks against changes in transit and authenticates their origin at the control plane.

## Cluster Readiness and Shared Hosts

Warden follows local Iggy leadership and node readiness. Provisioning waits for a healthy cluster roster before it creates credentials. Source connectors run only on the healthy local leader. If leadership is unknown, they remain disabled with their configuration intact.

Managed data also depends on plane replay, ownership, and backend health. Read [readiness and metrics](/api/observability) during recovery. A recent heartbeat alone does not prove that a node can serve requests.

Free deployments can share a host with separate runtime processes and CPU quotas. Host metrics and runtime metrics remain separate. The fleet upgrade workflow updates shared-host binaries in place.

## Upgrades and Rollbacks

Runtime upgrades can interrupt client connections. Clients must reconnect after an interruption. Warden follows this sequence:

1. A published version becomes available, and Warden receives an upgrade task.
2. Warden downloads the binary and makes sure that its signature is valid.
3. Warden replaces the binary atomically, as one indivisible operation.
4. If the new binary fails to start, Warden restores the previous version.

Warden can upgrade itself without stopping Iggy or Connectors. Those services continue to run during its upgrade. Warden rejects any binary with an invalid signature. Signatures apply to Warden, Iggy, and Connectors binaries.

## Credential Scope

Each Warden token grants access to one node. A compromised token remains limited to that node's permissions:

| Can Do | Cannot Do |
| --- | --- |
| Pull tasks for that single node | Access the VM shell |
| Push telemetry for that node | Read message payloads |
| Push stream and topic metadata for its deployment | |
| | Execute arbitrary commands |
| | Access other nodes |
| | Modify Iggy configuration directly |

## Network Requirements

| Direction | What | Port |
| --- | --- | --- |
| Outbound | HTTPS to the LaserData control plane | 443 |
| Inbound | Nothing | None |

Management requires outbound HTTPS. It does not require inbound firewall rules, SSH, SSM, or bastion hosts. Application listeners still require the client access described above.

- [Security Architecture](/security)
- [Deployment Models](/deployments)
- [Configuration](/deployments/configuration)

### AGDX activity

Warden collects retained authorization history and the managed backend's audit log through metadata sync. The [activity API](/api/audit#deployment-activity) shows role and binding events from the authorization history, and filter, filter binding, projection, schema, and fork changes from the audit log, each with the Iggy user that made it when one is recorded and the time of the change. It does not invent an actor or report a rebuilt baseline as resource creation.

Authorization history is bounded. If a collection is incomplete or unavailable, the activity pipeline reports a coverage gap and does not advance accepted event progress. The audit log does not expire by default. If its retention is changed and records are removed before Warden reads them, activity shows an `events_evicted` gap. Recreating the ops stream or its audit topic loses the records the log held, reported as a `history_changed` gap. Turning metadata sync off also stops this collection. Purging stored metadata keeps the deployment's activity.

### Outgoing data access

Warden 0.72.0 supports an outgoing HTTPS connection for on-premise deployments and BYOC deployments without a public IP. The deployment must already reach its Supervisor over outbound HTTPS. This connection does not create a private network route or require a new inbound firewall rule.

The deployment data-access toggle controls forwarding. Warden also enforces the received state on each request. Turning the toggle off closes the connection within about five seconds, and a request still in progress at that moment ends with an unknown outcome. Direct Warden access retains its normal behavior when the toggle is off. A dropped connection never causes a message send or another write to be replayed automatically. If the result is unknown, inspect the deployment before sending again.

The connection is ordinary HTTPS to the same Supervisor address Warden already uses for its configuration and tasks. Warden checks the Supervisor certificate and hostname against the node's system trust store, the same check as every other Warden request, so certificate renewals need no change on the node.

A node can refuse the outgoing connection whatever the Supervisor sends. Set `LD_WARDEN_DATA_ACCESS_OUTGOING_ALLOWED=false` in the Warden service environment, for example with `systemctl edit warden` and the line `Environment=LD_WARDEN_DATA_ACCESS_OUTGOING_ALLOWED=false`, then restart Warden. The Supervisor never writes this setting, and Warden configuration reloads keep it. It defaults to true, and the deployment toggle still decides whether a connection opens.

`LD_WARDEN_DATA_ACCESS_MAX_REQUESTS` defaults to 16. `LD_WARDEN_DATA_ACCESS_MAX_BODY_BYTES` defaults to 10485760 bytes. Warden applies the lower of its local limits and those advertised by the Supervisor, and reports the result back, so the Supervisor never sends more requests than the node accepts. Requests and responses use bounded transport chunks. Slow readers cannot create an unbounded relay queue.

A request the node refuses before running it, because it is full, the toggle changed or the body is too large, ends with 503, 401 or 413 and is safe to retry. Only a request that may have run ends with 502 and an unknown outcome.

When a request arrives at another Supervisor instance than the one holding the node connection, the instances relay it through encrypted temporary buffers. Queued frames expire after 65 seconds. This transport expiry does not define the retention of infrastructure backups. Message bodies and query results pass through LaserData while this access is enabled.

Source: https://docs.laserdata.com/deployments/warden
