Monitoring
Read deployment metrics, health reports, logs, and alerts
Warden collects metrics, heartbeats, and logs from each deployment node. It sends telemetry, the records of system activity, to the control plane. You can read it in the Console or through the API.
Telemetry goes to LaserData by default. You can send logs and traces to your own OpenTelemetry-compatible endpoint instead.
Understand a New Deployment
Platform services create resources before you send application data. The prober, a service that tests writes to the streaming server, sends a message to _ld/prober every five seconds by default. Warden and managed services can also appear as connected clients. Stream, topic, client, and message counts therefore do not start at zero or have one fixed initial value.
Use Include prober in Metrics where available to separate health traffic from application counts. For your first test, inspect the stream and topic named by the producer. Do not delete platform resources to make a dashboard count zero.
Which Health Status to Use
The cards measure different things. initialized is a provisioning state. A recent heartbeat shows that a service reported its condition. All nodes serving summarizes node readiness. The Health Probe card measures the age of the last observed probe write.
The current Health Probe card uses these thresholds:
| Label | Meaning |
|---|---|
| Healthy | The last probe is less than 15 seconds old |
| Degraded | The last probe is at least 15 seconds old and less than 60 seconds old |
| Unhealthy | The last probe is at least 60 seconds old |
| Probe data unavailable | Required probe timestamps are missing |
A node can report Healthy while the latest probe record is delayed. Read the probe time, node readiness, and logs together. The card calculates age from the browser clock, so make sure that your computer time is correct. If probe age continues increasing, test a send and read through your client and report the result with the deployment ID.
Shared-Host Capacity and Network Counters
On Free deployments, memory and disk values describe your slot limits. They do not describe the entire physical host. Compare them with Free allocations, not a dedicated VM size.
Slot telemetry currently reports network RX and TX as zero instead of exposing the shared host counters. A zero Network I/O card therefore does not prove that no messages moved. Network throughput has no points for these deployments for the same reason. Streaming server message counts and process disk writes measure different activity and can increase at the same time.
Times and Reports
Some Console dates use month-first formatting and local time. Do not infer date order from a value such as 9/9/2026. When reporting a problem, use YYYY-MM-DD, include the time and timezone, and include the detailed record timestamp when available.
Metrics
The metrics dashboard reports each node and runtime continuously. A runtime is a managed process, such as Iggy or Connectors.
What Is Collected
| Category | Metrics |
|---|---|
| System | CPU usage, total CPU usage, memory usage, available memory |
| Process | Process ID, run time, start time |
| Disk I/O | Bytes read, bytes written |
| Iggy | Messages count, messages size, streams count, topics count, partitions count, segments count |
| Clients | Connected clients count, consumer groups count |
| Iggy runtime | Open file count and limit, thread count, free and total data disk space, message cache hits and misses |
Runtime names are iggy, connectors, and connector:{id} for individual connector instances. On dedicated nodes, host reports the virtual machine. Shared-host deployments expose slot-scoped values instead. Runtime-specific fields appear directly in each sample.
Cluster and Managed Data Health
| Area | Signals |
|---|---|
| Iggy replication | role, view, expected and reachable voters, quorum, metadata and partition gaps, repairing and transferring groups |
| Partition replication | partitions_under_replicated, partitions_below_write_quorum, partitions_owing_retirement |
| Managed backend | Projection bindings and routes, decode failures, KV entries, open forks, projector lag, replay barrier and lag |
| Ownership and recovery | Owned, expected, and standby partitions, fence expiry and rejections, destination watermark, restore and synchronization state |
| Host | VM CPU and memory, load, swap, disk capacity and I/O, network receive and transmit counters |
| Shared-host quota | cpu_quota_cores, cpu_usage_pct_of_quota, cpu_throttled_periods, cpu_throttled_usec |
| Connector pipeline | Produced, sent, consumed, processed, filtered messages, and errors |
On a cluster, Warden reports three partition replication gauges for each node, each summed over the node's shards. partitions_under_replicated counts partitions the node leads with a member it cannot reach or a member that trails the commit point for too long. partitions_below_write_quorum counts partitions the node leads with too few members in sync to commit a write. partitions_owing_retirement counts partitions that refuse client sessions until they clear sessions that ended while the partition lacked a quorum of its members. Zero is healthy for all three. A node counts the first two only for the partitions it leads, so their deployment value is the sum of the nodes' latest samples, and a node at zero says nothing about partitions other nodes lead. The gauges are null on a single-node deployment and when Warden could not read them. See Replication factor.
A heartbeat shows recent liveness. It does not prove that a node caught up or can serve managed requests. Read observed_at, iggy_ready, last_probe_at, and serving from the node readiness endpoint. serving is true only when every readiness check on the node passes.
GET /deployments/{deployment_id}/health shows the whole deployment in one reply: how many nodes are expected and serving, whether a quorum is serving, the readiness of each node, the state of its Iggy and connector runtimes, and the operation that owns the deployment with its phase and pause reason. For each source and sink it shows the state, the error count and the last error with its time, so a failing connector says why it fails. It needs deployment:telemetry:read.
Network Throughput
Network throughput is the received and transmitted bytes per second of all deployment nodes together. It counts every interface of each node, so it includes replication between nodes, LaserData traffic, and traffic between processes on a node, not only client messages. Each node's rate comes from its own counters. A gap of more than 60 seconds between samples or a counter reset leaves a bucket partial or empty instead of producing a spike or a negative value. A bucket that misses some nodes reports fewer nodes than the deployment has. See Get Deployment Network Throughput.
If a runtime omits an optional metric, treat its value as unknown. For the managed backend's disk_free_bytes, the maximum unsigned 64-bit value means that no disk sample exists. It does not mean unlimited capacity.
Read host and process utilization separately. On shared hosts, increasing throttling counters and high quota utilization can explain latency even with low usage across the whole machine.
Viewing Metrics
- Open your deployment in the Console.
- Open the Metrics tab.
- Select the node and runtime.
- Select the time range.
Heartbeats
Warden sends periodic health reports for managed runtimes such as Iggy and Connectors. You can read them by node, runtime, and historical time range. Missing reports indicate a possible runtime problem.
Logs
Iggy logs include internal replication and connection events. Messages about groups, views, or disconnected consensus clients do not by themselves prove that application data failed. Select the affected node, runtime, level, and time range. Compare repeated warnings or errors with probe age and a client send-and-read test.
Warden collects logs from all runtimes and nodes. The platform stores them for searches. Large result sets use pages.
Searching Logs
Filter logs by these fields:
| Filter | Description |
|---|---|
| Node | Filter to a specific node |
| Runtime | Filter by runtime (Iggy, Connectors, Warden) |
| Level | Filter by log level (any, debug, info, warn, error) |
| Message | Pattern match on log message content |
| Scope | Filter by component |
| Time range | Start and end timestamps |
Viewing Logs
- Open your deployment in the Console.
- Open the Logs tab.
- Select filters for the records that you need.
- Read further pages for additional results.
Log Redirection (OpenTelemetry)
LaserData stores logs by default. To keep them in your own system, configure an OpenTelemetry-compatible destination through the Console or support. This can support your privacy, compliance, or monitoring requirements.
Warden sends logs with ExportLogsServiceRequest. Compatible collectors and storage systems include Grafana Loki, Datadog, Elastic, and custom pipelines. Traces use ExportTracesServiceRequest.
Monitoring Alerts
The platform watches deployment health and sends alerts through notification channels. An alert starts when a threshold is crossed. A resolution event follows when the condition clears.
| Alert | Threshold | Description |
|---|---|---|
high_cpu_usage | CPU ≥ 80% (5-min window) | Sustained high CPU usage on a node |
high_memory_usage | Memory ≥ 90% | Memory usage approaching capacity |
high_disk_usage | Disk ≥ 80% | Disk usage approaching capacity |
node_unreachable | No heartbeat for 120s | Node stopped reporting heartbeats |
runtime_degraded | Only failing heartbeats for 5 min | A runtime still reports but keeps restarting, failing its health check, or refusing its config |
plane_condition | See the table below | The managed backend on a node needs an operator |
replication_condition | A partition replication gauge above zero for 5 min | Partitions on a node are short of a copy or a quorum of their members |
Resolution events include cpu_usage_resolved, memory_usage_resolved, disk_usage_resolved, node_reachable, runtime_recovered, plane_condition_resolved, and replication_condition_resolved. The platform avoids repeated alerts for a condition that remains active. To receive alerts, create a channel and subscribe to its event types.
A plane_condition alert names its condition, the measured reason, and the recovery step. Most conditions must hold for the whole 5-minute window, so a short retry never raises one.
| Condition | Raised when |
|---|---|
fold_lag | A fold stays at least 50,000 records behind its source |
backend_realization_pending | A binding table waits more than 10 minutes for its destination |
source_history_truncated | A source partition stopped because retention removed records it had not folded |
sync_stalled | The Turso Cloud sync writer has not pushed for 5 minutes |
rebuild_blocked | The node cannot rebuild its local database from the retained log |
fold_stalled | A fold holds its cursor because its store or a batch keeps failing |
records_held | A binding holds on a rejected source record and no skip was decided |
stream_purge_pending | A deleted stream still has derived data to purge |
dead_letters_growing | New records reached a dead-letter topic |
disk_floor_near | Free space on the managed backend volume is below twice the write floor |
database_growth | The embedded database reaches the write floor within 12 hours at its current growth |
lease_flapping | The writer lease of a shared destination changed hands at least 3 times |
lease_missing | A shared destination partition had no lease holder on any node |
snapshot_overdue | The deployment has no managed backend snapshot from the last 24 hours |
A replication_condition alert is raised per node and per gauge when the gauge stays above zero in every sample of the 5-minute window. Its subject names the condition, and its body starts with the severity, then names the node, the partition count, and the recovery step. replication_condition_resolved follows once the gauge is reported again and is zero in at least one sample of the window. A node that stops reporting the gauge keeps its alert open.
| Subject | Gauge | Severity | Effect |
|---|---|---|---|
| Partitions under-replicated | partitions_under_replicated | Warning | Writes still commit, but one more failure can stop them |
| Partitions below write quorum | partitions_below_write_quorum | Critical | Writes to these partitions fail |
| Partitions refusing sessions | partitions_owing_retirement | Critical | Sessions on these partitions are refused |
Telemetry Retention
Metrics, heartbeats, and logs share a retention period. Managed deployments include these periods:
| Deployment | Included retention |
|---|---|
| Free | 7 days |
| Standard | 14 days |
| Performance | 30 days |
| Enterprise | 90 days |
New deployments use the tier's included period. The platform rejects requests above that allowance and does not charge for extra telemetry days.
For BYOC and deployments without managed tiers, the allowance follows Compute. Free and Small include 7 days, Medium and Large 30 days, and XLarge or larger 90 days.
Set retention.telemetry_days through the deployment API. Audit, snapshot, and backup retention are separate. Reading telemetry requires deployment:telemetry:read. Changing retention requires deployment:telemetry:manage.
API Reference
Get Deployment Metrics
curl {supervisor_url}/deployments/{deployment_id}/metrics \
-H "ld-api-key: YOUR_API_KEY"Get Node Metrics (by runtime)
curl "{supervisor_url}/deployments/{deployment_id}/nodes/{node_id}/metrics/iggy?page=1&results=10&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \
-H "ld-api-key: YOUR_API_KEY"Get Deployment Heartbeats
curl {supervisor_url}/deployments/{deployment_id}/heartbeats \
-H "ld-api-key: YOUR_API_KEY"Get Node Heartbeats (by runtime)
curl "{supervisor_url}/deployments/{deployment_id}/nodes/{node_id}/heartbeats/iggy?page=1&results=10&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \
-H "ld-api-key: YOUR_API_KEY"Get Deployment Logs
curl "{supervisor_url}/deployments/{deployment_id}/logs/iggy?page=1&results=10&level=any&message=*&from=2026-01-01T00:00:00Z&to=2026-02-01T00:00:00Z" \
-H "ld-api-key: YOUR_API_KEY"AGDX activity coverage
Deployment activity includes recorded managed-role and role-binding events and the changes to filters, filter bindings, projections, schemas, and forks that the managed backend records in its audit log. The same activity permissions apply, reads go back as far as the plan's audit retention allows, and activity is never deleted, not even by a metadata purge.
Recorded authorization events carry their actor and time when known. Entries from the managed backend's audit log carry the Iggy user whose request it applied and the time it applied the change. Authorization history is bounded, and recreating the ops stream or its audit topic loses the records the audit log held.
A gap entry reports known incomplete coverage. Activity contains resource identity and safe change details, without filter policies, projection definitions, schema bodies, credentials, or customer payloads.