Connectors
Activate source and sink plugins to move data between Iggy and external systems
Connectors move data between Iggy and external systems without application code. The Apache Iggy connectors runtime loads compiled Rust plugins for each integration. The Connector Catalog lists 16 sinks and 5 sources with their configuration.
The plugins run as native code. Their data path uses no JVM or garbage collector.
How It Works
Each plugin implements the Source or Sink trait from the Apache Iggy Connectors SDK. The runtime loads plugins at startup and manages configuration, execution, monitoring, and shutdown.
Warden manages the runtime as a separate process on the same node. The runtime connects to Iggy through TCP with TLS. Cluster routing can direct it to another deployment node. External sources and destinations use their own network endpoints.
Connector Types
| Type | Direction | What It Does |
|---|---|---|
| Source | External system → Iggy | Produces messages into Iggy streams from an external system |
| Sink | Iggy → External system | Consumes messages from Iggy streams and pushes them to an external destination |
Multiple instances can run together. For example, one source can read from an external system while several sinks write to different destinations.
Activating a Connector
From the Console
- Open the deployment's Connectors tab.
- Browse the catalog or filter it by source or sink.
- Click Activate for the plugin that you need.
- If needed, set a custom instance name and key.
The platform provisions the instance on every node with the Connectors runtime. Its status moves from Pending to Active after the nodes process the activation task.
Instance Naming
Each instance has a unique key within the deployment. Without a custom key, the platform generates {connector}-{type}-{operation_id}. Save the returned key as the name for the instance's configuration.
You can activate a plugin more than once. For example, two PostgreSQL sink instances can write to different databases.
Connector Lifecycle
| Status | Meaning |
|---|---|
| Pending | Instance created, waiting for nodes to process activation |
| Active | Running and processing messages |
| Inactive | Disabled but configuration preserved |
| Failed | Encountered errors - check logs for details |
Activation installs the plugin with its initial configuration disabled. To start processing, create an enabled configuration under the instance key.
In a cluster, sources run only on the healthy local Iggy leader. Followers and nodes with unknown leadership keep sources disabled while retaining their desired configuration. Sinks coordinate consumption through consumer groups.
Monitoring
The runtime reports these metrics for each instance:
| Metric | Description |
|---|---|
| messages_produced | Total messages produced (source connectors) |
| messages_consumed | Total messages consumed (sink connectors) |
| messages_sent | Messages sent to Iggy by sources |
| messages_processed | Messages processed by sink plugins |
| messages_filtered | Messages intentionally removed by transforms |
| errors | Error count |
| status | Runtime status (Starting, Running, Stopping, Stopped, Error) |
Runtime metrics also include CPU, memory, and the total sources and sinks running. Read them in the Console's Metrics tab or through the Monitoring API.
Deleting a Connector Instance
On the Connectors tab, click Delete for the instance. This removes it from all nodes and deletes its configuration.
Activation, configuration, and deletion require deployment:connector:manage. Reading instances requires deployment:connector:read.
API Reference
Use the Main API to browse the catalog and activate plugins. Use the Supervisor API to manage running instances.
List Available Connectors
curl "https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/{deployment_id}/connectors?type=sink&page=1&results=10" \
-H "ld-api-key: YOUR_API_KEY"The response includes each connector's availability and permission status for the deployment.
Activate a Connector
curl -X POST https://api.laserdata.cloud/tenants/{tenant_id}/divisions/{division_id}/environments/{environment_id}/deployments/{deployment_id}/connectors/activate \
-H "ld-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"connector_key": "postgres",
"connector_type": "sink",
"instance_name": "Orders to Postgres",
"instance_key": "orders-pg-sink"
}'The endpoint returns 202 Accepted. The instance moves from Pending to Active after the nodes process its task.
List Connector Instances
curl {supervisor_url}/deployments/{deployment_id}/connectors/instances \
-H "ld-api-key: YOUR_API_KEY"[
{
"id": 1,
"deployment_id": 611298765432109056,
"connector_type": "sink",
"connector_key": "postgres",
"name": "Orders to Postgres",
"key": "orders-pg-sink",
"status": "active"
}
]Delete a Connector Instance
curl -X DELETE {supervisor_url}/deployments/{deployment_id}/connectors/instances/{instance_id} \
-H "ld-api-key: YOUR_API_KEY"A successful request returns 204 No Content.