# Kafka gateway

The Kafka gateway lets an application that uses a Kafka client library connect to a LaserData deployment. It speaks the Kafka wire protocol on TCP port 9092 and stores every Kafka topic as a topic of the deployment's streaming server, in a stream named `kafka`. The application keeps its Kafka client and changes only its connection settings. The gateway is optional and off by default. You turn it on per deployment.

| Item | Value |
| --- | --- |
| Address | The entries of `kafka_bootstrap_servers` in the deployment details, each on port 9092 |
| Security protocol | `SASL_SSL`, TLS with SASL sign-in. There is no plaintext listener |
| Sign-in | SASL `PLAIN` with the name and password of a user of the deployment, such as the `ld-admin` deployment login |
| Trust | The LaserData root certificate |
| Topics | The `kafka` stream of the deployment |
| Network | An [access rule](/networking/access-rules) with the `kafka` transport. Closed by default |

## Pick the interface

| You want to | Use |
| --- | --- |
| Run an existing Kafka producer, consumer, or tool against a deployment | The Kafka gateway |
| Build a new application, or use workflows, memory, sessions, and the other Laser SDK features | [Laser SDK](/laser-sdk) with a [connection string](/laser-sdk/connect) |
| Let an agent read or write a deployment's data with an API key | The [deployment data MCP endpoint](/mcp#deployment-data) |

## Where it runs

| Deployment | Kafka gateway |
| --- | --- |
| Managed cluster | One gateway beside the streaming server on every node. Clients connect to every node |
| Managed standalone, one dedicated node | One gateway on the node |
| Free | One gateway inside the deployment's slot on the shared host |
| BYOC | Not available yet |
| On-Premise | Not available yet |

A deployment created before the Kafka gateway became available cannot run it, and `enable` answers `400` with `deployment_runtime_missing`. On BYOC and On-Premise deployments, `enable` answers `400` with `unsupported_runtime_action`.

## Turn it on

`POST {supervisor_url}/deployments/{deployment_id}/activities/kafka_gateway/enable`

Turn on the deployment's Kafka gateway. Requires deployment:manage.

A successful request returns `202 Accepted` with no body. LaserData then creates the `kafka` stream for your topics, a `kafka.meta` stream for the gateway's own state, and a service user named `kafka-gateway`, and starts the gateway on every node. The `kafka_gateway_enabled` field of [Get Deployment](/api/deployments#get-deployment-supervisor-api) on the supervisor shows the setting, and the deployment's [activities](/api/activities) show the rollout. While LaserData prepares the streams and the user, `kafka_gateway_provisioning` is `true`. An accepted request does not mean that the gateway runs. Each node reports its own [heartbeat](#monitoring) once its gateway is running.

If the deployment already holds a stream named `kafka` or `kafka.meta`, or a user named `kafka-gateway`, that LaserData did not create, the enable activity fails. The gateway stays off, `kafka_gateway_enabled` returns to `false`, and `kafka_gateway_failure` in the deployment details gives the `reason`, with a `conflict` that names the stream or user and whether it is `foreign`, `missing` or `replaced`. A later successful `enable` clears it. LaserData never deletes, renames, or changes that resource. Rename or delete it yourself and send `enable` again. If you did not create it, contact LaserData support.

LaserData records the two streams and the user when it creates them. Do not delete, rename, or replace them. A gateway does not start when one of them is missing or was replaced.

The same route takes these actions for the runtime `kafka_gateway`. Each needs `deployment:manage` and answers `202 Accepted`.

| Action | Effect |
| --- | --- |
| `enable` | Creates the streams and the user the first time, then starts the gateway on every node |
| `disable` | Stops the gateway on every node and keeps it stopped. Kafka clients disconnect. Topics, their data, the `kafka-gateway` user, and the settings stay, and a later `enable` uses them again |
| `start` | Starts a stopped gateway |
| `stop` | Stops the gateway. It stays stopped across node restarts until `start` |
| `restart` | Restarts the gateway. On a cluster the nodes restart one at a time |
| `upgrade` | Moves the gateway to a newer version, one node at a time. Clusters only |

On standalone and Free deployments LaserData manages the gateway version, and `upgrade` answers `400` with `unsupported_runtime_action`. LaserData checks that the gateway version works with the deployment's streaming server version. It refuses an upgrade of either one that would pair versions that do not work together, also partway through a rolling upgrade.

## Open port 9092

The gateway listens on TCP port 9092, and an [access rule](/networking/access-rules) opens it for the source ranges the rule names. Set the `kafka` transport on the rule:

```bash
curl -X POST {supervisor_url}/deployments/{deployment_id}/access_rules \
  -H "ld-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kafka-clients",
    "cidr_blocks": ["203.0.113.0/24"],
    "rules": {
      "kafka": true
    }
  }'
```

The port stays closed until a rule sets `kafka`. The default rule of a Free deployment, `0.0.0.0/0`, does not set it either, so a Free deployment also needs a rule of its own for Kafka clients. [Allow My IP](/networking/access-rules#allow-my-ip) accepts `{"kafka": true}` to admit the address you call from for 24 hours.

Opening the port and running the gateway are separate. A rule with `kafka` on a deployment whose gateway is off opens a port that nothing listens on. A gateway without such a rule runs, and no client can reach it.

A rule that sets `kafka` for `0.0.0.0/0` makes Kafka sign-in reachable from the whole internet, and every sign-in runs a password check on the streaming server. Admit only the ranges your Kafka clients connect from.

## Connect a client

| Setting | Value |
| --- | --- |
| `bootstrap.servers` | The entries of `kafka_bootstrap_servers` from [Get Deployment](/api/deployments#get-deployment-supervisor-api) on the supervisor, joined with commas |
| `security.protocol` | `SASL_SSL` |
| `sasl.mechanism` | `PLAIN`. librdkafka names the property `sasl.mechanisms` |
| User name and password | A user of the deployment. The `ld-admin` login from [Get Deployment Credentials](/api/deployments#get-deployment-credentials) works as it is |
| Trust | The LaserData root certificate, which the Console offers for download with the deployment's connection details |

**Java**

```properties
bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.location=<path to laserdata.crt>
```

With `ssl.truststore.type=PEM`, the Java client reads the downloaded certificate file as its trust store.

**librdkafka**

```properties
bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username=<user>
sasl.password=<password>
ssl.ca.location=<path to laserdata.crt>
```

Clients built on librdkafka, such as confluent-kafka for Python, Go, and .NET, take the same properties.

Keep the password in an environment variable or a secret store, never in a file you commit.

On a cluster, `kafka_bootstrap_servers` lists one address per node, not the deployment domain. The gateway on each node points clients back at its own node, and newer clients ask for a node by its address. Your client must reach every address in the list, so put all of them in `bootstrap.servers` and place no load balancer or shared proxy in front of them. On a public cluster the entries are the nodes' public IP addresses, and on a private deployment their private addresses. Read the list from the deployment details instead of building it from the domain or a port pattern. PrivateLink and Private Service Connect do not carry Kafka traffic.

On a standalone deployment and on a Free deployment, the list holds one entry: the deployment domain on port 9092. A Free deployment shares its host with other deployments, and the host routes each Kafka connection by the TLS server name the client sends. Connect by the host name from the list, never by IP address. The Java client, librdkafka, franz-go, and Sarama send the server name when they connect by host name. The host drops a connection that does not send it.

Every new Kafka connection signs in again, and each sign-in runs a password check on the streaming server. Keep connections open and reuse them.

## Who can read and write

Every user of the deployment that can sign in can read and write every Kafka topic through the gateway, including a user that holds no permission on the streaming server. The gateway checks the user name and password when a client connects, then serves produce, fetch, and the other data requests as its own `kafka-gateway` service user. Kafka topics live in the `kafka` stream, and no other stream is reachable through the gateway.

Some administrative requests are also checked against the signed-in user's own permissions on the streaming server. Deleting topics, adding partitions, changing topic configs, deleting consumer groups or their offsets, and creating a topic automatically when a client names it need `manage_topics` or `manage_streams`. Committing offsets needs read permission on the topic. `ld-admin` passes all of these checks.

## Topics and partitions

Kafka topics are ordinary topics of the streaming server in the `kafka` stream, so they appear there in the Console and to Iggy clients.

A `CreateTopics` request is not checked against the signed-in user's permissions. Any client that signs in can create topics of up to 1000 partitions each. On a Free deployment the slot's cap of 100 partitions across all streams bounds this. A deployment on dedicated nodes has no partition cap by default, so set `metadata.partitions_max` in its [server configuration](/deployments/server#server-configuration) before you turn the gateway on.

A topic created through Kafka without its own retention keeps messages for seven days, the default of `IGGY_KAFKA_TOPIC_RETENTION_MS`. The retention becomes the topic's message expiry on the streaming server, which removes whole sealed segments, so a message can outlive the retention until its segment is sealed. See [Topic Storage Options](/deployments/server#topic-storage-options).

On a cluster the gateways share consumer groups. When a node stops, its groups move to the gateways on the other nodes. A planned stop or restart hands them over at once. After a crash the move takes up to about 45 seconds. On standalone and Free deployments the one gateway holds every group, and committed offsets of a group do not expire.

## Settings

Change the gateway's settings with the `kafka_gateway` kind on the [configuration API](/api/configuration#kafka-gateway-configuration), for example `{supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway/schema` and `POST {supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway`. Activating a version applies it to every node, one node at a time on a cluster, which can take minutes. Saving a version never starts a stopped gateway.

| Key | Default | Accepted values | What it sets |
| --- | --- | --- | --- |
| `RUST_LOG` | `info` | `off`, `error`, `warn`, `info` | Log level of the gateway |
| `IGGY_KAFKA_AUTO_CREATE_TOPICS_ENABLED` | `false` | `true`, `false` | Creates a topic that a client names and nobody created, like Kafka's `auto.create.topics.enable` |
| `IGGY_KAFKA_AUTO_CREATE_TOPICS_PARTITIONS` | `1` | 1 to 1000 | Partitions of a topic created automatically. Used only while automatic creation is on |
| `IGGY_KAFKA_TOPIC_RETENTION_MS` | `604800000`, seven days | `-1` or a positive number | Retention in milliseconds of a topic created without one. `-1` keeps messages forever |
| `IGGY_KAFKA_TOPIC_RETENTION_BYTES` | `-1` | `-1` or at least `1073741824`, 1 GiB | Size limit in bytes of a topic created without one. `-1` sets no limit |
| `IGGY_KAFKA_TOPIC_DURABILITY` | `replicated` | `replicated`, `persisted` | How a topic created through Kafka stores its messages and offsets. See [Topic Durability](/deployments/server#topic-durability) |
| `IGGY_KAFKA_IDEMPOTENT_PRODUCERS` | `true` | `true`, `false` | Drops duplicate retried batches of idempotent producers. Turn it off only when older Iggy consumers read the same topics |
| `IGGY_KAFKA_OFFSETS_RETENTION_MINUTES` | `10080`, seven days | 1 or more | Minutes the committed offsets of an empty consumer group are kept. Cluster deployments only |
| `IGGY_KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS` | `3000` | 0 or more | Milliseconds a new consumer group waits for more members before its first rebalance |

The schema lists only the keys that the deployment's gateway version and placement have. It also lists keys that LaserData fixes, such as the listener and the stream name. Only the keys in the table can change.

## What is not supported

The gateway serves these Kafka requests: ApiVersions, Metadata, Produce, Fetch, ListOffsets, CreateTopics, DeleteTopics, CreatePartitions, OffsetCommit, OffsetFetch, FindCoordinator, JoinGroup, Heartbeat, LeaveGroup, SyncGroup, DescribeGroups, ListGroups, InitProducerId, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DeleteGroups, OffsetDelete, DescribeCluster, and DescribeAcls. Any other request closes the connection.

| Feature | What happens |
| --- | --- |
| Transactions | A transactional producer fails at start. Exactly-once stream processing does not work |
| Log compaction | A topic created with `cleanup.policy=compact` keeps every record. Kafka Connect and Schema Registry create such topics. Schema Registry needs `kafkastore.topic.skip.validation=true`, and Apicurio Registry 3 does not start |
| SCRAM and OAuth sign-in | Only SASL `PLAIN` is accepted. `SCRAM-SHA-256`, `SCRAM-SHA-512`, and `OAUTHBEARER` are refused |
| ACL changes | `DescribeAcls` shows the signed-in user's own permissions. `CreateAcls` and `DeleteAcls` are not served |
| `DeleteRecords` | Not served |
| Deleting a topic by ID | `DeleteTopics` by topic ID is not served. Delete topics by name |

## Free deployment limits

A Free deployment runs its gateway inside the deployment's slot, with the slot's memory and CPU. These limits apply to Kafka clients:

| Limit | Value |
| --- | --- |
| Kafka connections | 60 open connections |
| Largest Kafka request | 1 MiB. Keep producer settings such as `max.request.size` at or below it |
| Throughput | The slot's network rate limit of 100 KB/s applies to Kafka traffic too |
| Partitions | 100 across all streams and topics, Kafka topics included |

See [Free](/deployments/tiers-storage#free) for the rest of the slot's allowance.

## Monitoring

The gateway reports as the runtime `kafka_gateway`. Each node sends heartbeats that show whether its gateway runs and reaches the streaming server. Metrics are process metrics, such as CPU and memory. Kafka request metrics are not reported yet. Logs follow the level that `RUST_LOG` sets. Read all three in [Monitoring](/observability) and through the [observability API](/api/observability).

The gateway is not part of deployment health. A stopped, failed, or disabled gateway never marks the deployment unhealthy, and a disabled gateway is not a failure.

## Errors

| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| `400` | `deployment_runtime_missing` | The deployment cannot run the gateway, for example because it was created before the gateway became available | Use a deployment created since then |
| `400` | `unsupported_runtime_action` | `enable` on a BYOC or On-Premise deployment, or `upgrade` on a standalone or Free deployment | Use a supported placement. LaserData upgrades standalone and Free gateways |
| `400` | `kafka_gateway_server_unsupported` | At `enable`, or at a gateway or server upgrade, a gateway and server version pair that can meet during the rollout or its rollback does not work together | Upgrade the gateway first to a version that supports both server versions, or contact LaserData support |

A name conflict appears after the `202`, in `kafka_gateway_failure` of the deployment details. See [Turn it on](#turn-it-on).

Source: https://docs.laserdata.com/kafka-gateway
