Connector Configuration

After activation, configure each instance through the deployment's Configuration system. Each instance has independent versions that you can save, activate, or restore. The Connector Catalog lists plugin-specific fields.

Sink Configuration

Source Configuration

Runtime and Plugin Settings

Before upgrading the runtime, make sure that source destination topics use persisted durability. The runtime creates missing topics with that policy. It rejects existing destinations that use replicated.

Instance configuration separates pipeline fields, such as streams, enabled, and transforms, from plugin_config. Shared connectors configuration controls the runtime's Iggy connection, HTTP API, telemetry, logging, and checkpoint storage.

Instance fields also include plugin_config_format, verbose, and benchmark. Formats are json, yaml, toml, and text. Cloud configuration accepts plugin values according to the schema. The platform owns type, key, version, name, and the library path.

For Avro, set schema: "avro". Supply avro_schema_json or a schema file available on the runtime node. An inline schema avoids a file dependency on every node.

Source Checkpoints and Failover

A checkpoint records completed source progress. File checkpoints remain on the node's disk. HTTP checkpoints can survive node replacement if the service that stores them is shared and durable.

The runtime saves a checkpoint only after Iggy acknowledges its batch. If sending or checkpoint storage fails, the connector receives the batch again on its next poll. This repeats work instead of skipping it.

The HTTP source first queues webhook requests in memory. Its success response proves queue admission, not a durable Iggy write. Restart can lose queued requests, and sender retries can duplicate them. Use durable retries at the sender and include a delivery ID so downstream systems can remove duplicates.

HTTP checkpoint storage uses conditional writes with ETags, identifiers for stored versions, and an idempotency key for each logical save. A missing checkpoint returns 404. Other load failures must not restart ingestion from the beginning. Version conflicts or revoked authorization stop checkpoint writes until the connector restarts.

Checkpoint configuration requires administrative access. Use IGGY_CONNECTORS_STATE_STORAGE and the schema's IGGY_CONNECTORS_STATE_HTTP_* fields. Retrieving configuration over HTTP and storing checkpoints over HTTP are separate features.

Warden enables cluster sources only on the healthy local Iggy leader. Other nodes keep sources disabled while preserving the saved desired configuration. Leadership changes can still repeat work. Use durable checkpoints and downstream operations that are safe to repeat.

Data Transforms

Apply transforms in order to change message fields:

To build a custom transform in Rust, implement Transform.

Examples

PostgreSQL Sink

{
  "name": "orders-pg-sink",
  "values": {
    "enabled": true,
    "streams": [
      {
        "stream": "orders",
        "topics": ["completed", "refunded"],
        "schema": "json",
        "batch_length": 100,
        "poll_interval": "1s",
        "consumer_group": "pg-sink-orders"
      }
    ],
    "plugin_config": {
      "connection_string": "postgres://user:pass@host:5432/orders",
      "target_table": "order_events"
    }
  },
  "activate": true
}

Random Source (Development)

{
  "name": "random-source",
  "values": {
    "enabled": true,
    "streams": [
      {
        "stream": "test_stream",
        "topic": "test_topic",
        "schema": "json",
        "batch_length": 1000,
        "linger_time": "5ms"
      }
    ],
    "plugin_config": {
      "interval": "3000ms",
      "max_count": 1000000,
      "messages_range": [1, 5],
      "payload_size": 200
    }
  },
  "activate": true
}

Plugin Schema Validation

Each plugin schema defines types, defaults, and secrets. Creation and updates follow these rules:

• Unknown plugin_config keys are rejected.
• Values must match their field types, such as string, integer, boolean, duration, or array.
• Fields marked secret, such as passwords and connection strings, appear as *** in API responses.

Before creating configuration, retrieve Get Config Schema for the available fields.

Configuration Flow

To configure an instance:

1. Retrieve its schema for fields, defaults, and rules.
2. Save values under the instance key to create a version.
3. Activate that version, or use "activate": true during creation.

Versions belong to individual instances. Changing one instance does not select a version for another.

Secret Masking

API responses show *** for secrets such as passwords, connection strings, and credentials. An update preserves masked values. Send only the fields that you want to change.

API Reference

A connector kind follows connector:{type}:{plugin_key}, for example connector:sink:postgres. Its configuration name is the activated instance key, such as orders-pg-sink. Use this name for activation and history. An arbitrary name does not create an instance.

Get Config Schema

Retrieve fields, types, defaults, and rules in the section format used by Iggy configuration:

curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/schema \
  -H "ld-api-key: YOUR_API_KEY"

{
  "sink": {
    "name": "Sink",
    "description": "Connector sink pipeline settings.",
    "schema": []
  },
  "plugin_config": {
    "name": "Plugin Config",
    "description": "Connector plugin-specific settings.",
    "schema": [
      {
        "key": "connection_string",
        "name": "Connection String",
        "description": "PostgreSQL connection string",
        "default_value": "",
        "kind": "string",
        "editable": true,
        "secret": true,
        "requirements": [],
        "rules": []
      }
    ]
  }
}

The sink or source section contains pipeline fields, such as enabled state, streams, and batching. plugin_config contains plugin fields, such as connection strings and table names. Fields with "secret": true appear as *** in responses.

Create a Config

curl -X POST {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres \
  -H "ld-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders-pg-sink",
    "values": {
      "enabled": true,
      "streams": [
        {
          "stream": "orders",
          "topics": ["completed", "refunded"],
          "schema": "json",
          "batch_length": 100,
          "poll_interval": "1s",
          "consumer_group": "pg-sink-orders"
        }
      ],
      "plugin_config": {
        "connection_string": "postgres://user:pass@host:5432/orders",
        "target_table": "order_events"
      },
      "transforms": {}
    },
    "activate": true
  }'

Supply these fields:

• Set name to the activated instance key.
• Put schema-compatible configuration in values.
• Set activate to true to select the new version as primary immediately.

A successful request returns 201 Created with the new configuration ID in ld-config. An existing name receives another version automatically.

Get Active Config

curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/primary \
  -H "ld-api-key: YOUR_API_KEY"

{
  "id": 1,
  "kind": "connector:sink:postgres",
  "name": "orders-pg-sink",
  "primary": true,
  "initialized": true,
  "version": 2,
  "created_at": "2026-03-16T12:00:00Z",
  "updated_at": "2026-03-16T12:05:00Z",
  "values": {
    "enabled": true,
    "streams": [],
    "plugin_config": {
      "connection_string": "***",
      "target_table": "order_events"
    },
    "transforms": {}
  }
}

List Config Versions

curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/orders-pg-sink/versions \
  -H "ld-api-key: YOUR_API_KEY"

Get a Specific Version

curl {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/orders-pg-sink/versions/2 \
  -H "ld-api-key: YOUR_API_KEY"

Activate a Specific Version

Activation selects the primary version and starts reconfiguration on every node:

curl -X PUT {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/orders-pg-sink/activate/2 \
  -H "ld-api-key: YOUR_API_KEY"

A successful request returns 204 No Content.

Delete a Config

curl -X DELETE {supervisor_url}/deployments/{deployment_id}/configs/connector:sink:postgres/{config_id} \
  -H "ld-api-key: YOUR_API_KEY"

Creation, activation, and deletion require deployment:config:manage. Reading configuration or schemas requires deployment:config:read.