Kafka gateway
Connect existing Kafka clients to a deployment through the Kafka gateway: turn it on, open port 9092 with an access rule, connect Java and librdkafka clients over SASL_SSL, change topic defaults, and see what is not supported
The Kafka gateway lets an application that uses a Kafka client library connect to a LaserData deployment. It speaks the Kafka wire protocol on TCP port 9092 and stores every Kafka topic as a topic of the deployment's streaming server, in a stream named kafka. The application keeps its Kafka client and changes only its connection settings. The gateway is optional and off by default. You turn it on per deployment.
| Item | Value |
|---|---|
| Address | The entries of kafka_bootstrap_servers in the deployment details, each on port 9092 |
| Security protocol | SASL_SSL, TLS with SASL sign-in. There is no plaintext listener |
| Sign-in | SASL PLAIN with the name and password of a user of the deployment, such as the ld-admin deployment login |
| Trust | The LaserData root certificate |
| Topics | The kafka stream of the deployment |
| Network | An access rule with the kafka transport. Closed by default |
Pick the interface
| You want to | Use |
|---|---|
| Run an existing Kafka producer, consumer, or tool against a deployment | The Kafka gateway |
| Build a new application, or use workflows, memory, sessions, and the other Laser SDK features | Laser SDK with a connection string |
| Let an agent read or write a deployment's data with an API key | The deployment data MCP endpoint |
Where it runs
| Deployment | Kafka gateway |
|---|---|
| Managed cluster | One gateway beside the streaming server on every node. Clients connect to every node |
| Managed standalone, one dedicated node | One gateway on the node |
| Free | One gateway inside the deployment's slot on the shared host |
| BYOC | Not available yet |
| On-Premise | Not available yet |
A deployment created before the Kafka gateway became available cannot run it, and enable answers 400 with deployment_runtime_missing. On BYOC and On-Premise deployments, enable answers 400 with unsupported_runtime_action.
Turn it on
POST{supervisor_url}/deployments/{deployment_id}/activities/kafka_gateway/enableTurn on the deployment's Kafka gateway. Requires deployment:manage.
A successful request returns 202 Accepted with no body. LaserData then creates the kafka stream for your topics, a kafka.meta stream for the gateway's own state, and a service user named kafka-gateway, and starts the gateway on every node. The kafka_gateway_enabled field of Get Deployment on the supervisor shows the setting, and the deployment's activities show the rollout. While LaserData prepares the streams and the user, kafka_gateway_provisioning is true. An accepted request does not mean that the gateway runs. Each node reports its own heartbeat once its gateway is running.
If the deployment already holds a stream named kafka or kafka.meta, or a user named kafka-gateway, that LaserData did not create, the enable activity fails. The gateway stays off, kafka_gateway_enabled returns to false, and kafka_gateway_failure in the deployment details gives the reason, with a conflict that names the stream or user and whether it is foreign, missing or replaced. A later successful enable clears it. LaserData never deletes, renames, or changes that resource. Rename or delete it yourself and send enable again. If you did not create it, contact LaserData support.
LaserData records the two streams and the user when it creates them. Do not delete, rename, or replace them. A gateway does not start when one of them is missing or was replaced.
The same route takes these actions for the runtime kafka_gateway. Each needs deployment:manage and answers 202 Accepted.
| Action | Effect |
|---|---|
enable | Creates the streams and the user the first time, then starts the gateway on every node |
disable | Stops the gateway on every node and keeps it stopped. Kafka clients disconnect. Topics, their data, the kafka-gateway user, and the settings stay, and a later enable uses them again |
start | Starts a stopped gateway |
stop | Stops the gateway. It stays stopped across node restarts until start |
restart | Restarts the gateway. On a cluster the nodes restart one at a time |
upgrade | Moves the gateway to a newer version, one node at a time. Clusters only |
On standalone and Free deployments LaserData manages the gateway version, and upgrade answers 400 with unsupported_runtime_action. LaserData checks that the gateway version works with the deployment's streaming server version. It refuses an upgrade of either one that would pair versions that do not work together, also partway through a rolling upgrade.
Open port 9092
The gateway listens on TCP port 9092, and an access rule opens it for the source ranges the rule names. Set the kafka transport on the rule:
curl -X POST {supervisor_url}/deployments/{deployment_id}/access_rules \
-H "ld-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "kafka-clients",
"cidr_blocks": ["203.0.113.0/24"],
"rules": {
"kafka": true
}
}'The port stays closed until a rule sets kafka. The default rule of a Free deployment, 0.0.0.0/0, does not set it either, so a Free deployment also needs a rule of its own for Kafka clients. Allow My IP accepts {"kafka": true} to admit the address you call from for 24 hours.
Opening the port and running the gateway are separate. A rule with kafka on a deployment whose gateway is off opens a port that nothing listens on. A gateway without such a rule runs, and no client can reach it.
A rule that sets kafka for 0.0.0.0/0 makes Kafka sign-in reachable from the whole internet, and every sign-in runs a password check on the streaming server. Admit only the ranges your Kafka clients connect from.
Connect a client
| Setting | Value |
|---|---|
bootstrap.servers | The entries of kafka_bootstrap_servers from Get Deployment on the supervisor, joined with commas |
security.protocol | SASL_SSL |
sasl.mechanism | PLAIN. librdkafka names the property sasl.mechanisms |
| User name and password | A user of the deployment. The ld-admin login from Get Deployment Credentials works as it is |
| Trust | The LaserData root certificate, which the Console offers for download with the deployment's connection details |
bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.location=<path to laserdata.crt>With ssl.truststore.type=PEM, the Java client reads the downloaded certificate file as its trust store.
bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username=<user>
sasl.password=<password>
ssl.ca.location=<path to laserdata.crt>Clients built on librdkafka, such as confluent-kafka for Python, Go, and .NET, take the same properties.
Keep the password in an environment variable or a secret store, never in a file you commit.
On a cluster, kafka_bootstrap_servers lists one address per node, not the deployment domain. The gateway on each node points clients back at its own node, and newer clients ask for a node by its address. Your client must reach every address in the list, so put all of them in bootstrap.servers and place no load balancer or shared proxy in front of them. On a public cluster the entries are the nodes' public IP addresses, and on a private deployment their private addresses. Read the list from the deployment details instead of building it from the domain or a port pattern. PrivateLink and Private Service Connect do not carry Kafka traffic.
On a standalone deployment and on a Free deployment, the list holds one entry: the deployment domain on port 9092. A Free deployment shares its host with other deployments, and the host routes each Kafka connection by the TLS server name the client sends. Connect by the host name from the list, never by IP address. The Java client, librdkafka, franz-go, and Sarama send the server name when they connect by host name. The host drops a connection that does not send it.
Every new Kafka connection signs in again, and each sign-in runs a password check on the streaming server. Keep connections open and reuse them.
Who can read and write
Every user of the deployment that can sign in can read and write every Kafka topic through the gateway, including a user that holds no permission on the streaming server. The gateway checks the user name and password when a client connects, then serves produce, fetch, and the other data requests as its own kafka-gateway service user. Kafka topics live in the kafka stream, and no other stream is reachable through the gateway.
Some administrative requests are also checked against the signed-in user's own permissions on the streaming server. Deleting topics, adding partitions, changing topic configs, deleting consumer groups or their offsets, and creating a topic automatically when a client names it need manage_topics or manage_streams. Committing offsets needs read permission on the topic. ld-admin passes all of these checks.
Topics and partitions
Kafka topics are ordinary topics of the streaming server in the kafka stream, so they appear there in the Console and to Iggy clients.
A CreateTopics request is not checked against the signed-in user's permissions. Any client that signs in can create topics of up to 1000 partitions each. On a Free deployment the slot's cap of 100 partitions across all streams bounds this. A deployment on dedicated nodes has no partition cap by default, so set metadata.partitions_max in its server configuration before you turn the gateway on.
A topic created through Kafka without its own retention keeps messages for seven days, the default of IGGY_KAFKA_TOPIC_RETENTION_MS. The retention becomes the topic's message expiry on the streaming server, which removes whole sealed segments, so a message can outlive the retention until its segment is sealed. See Topic Storage Options.
On a cluster the gateways share consumer groups. When a node stops, its groups move to the gateways on the other nodes. A planned stop or restart hands them over at once. After a crash the move takes up to about 45 seconds. On standalone and Free deployments the one gateway holds every group, and committed offsets of a group do not expire.
Settings
Change the gateway's settings with the kafka_gateway kind on the configuration API, for example {supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway/schema and POST {supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway. Activating a version applies it to every node, one node at a time on a cluster, which can take minutes. Saving a version never starts a stopped gateway.
| Key | Default | Accepted values | What it sets |
|---|---|---|---|
RUST_LOG | info | off, error, warn, info | Log level of the gateway |
IGGY_KAFKA_AUTO_CREATE_TOPICS_ENABLED | false | true, false | Creates a topic that a client names and nobody created, like Kafka's auto.create.topics.enable |
IGGY_KAFKA_AUTO_CREATE_TOPICS_PARTITIONS | 1 | 1 to 1000 | Partitions of a topic created automatically. Used only while automatic creation is on |
IGGY_KAFKA_TOPIC_RETENTION_MS | 604800000, seven days | -1 or a positive number | Retention in milliseconds of a topic created without one. -1 keeps messages forever |
IGGY_KAFKA_TOPIC_RETENTION_BYTES | -1 | -1 or at least 1073741824, 1 GiB | Size limit in bytes of a topic created without one. -1 sets no limit |
IGGY_KAFKA_TOPIC_DURABILITY | replicated | replicated, persisted | How a topic created through Kafka stores its messages and offsets. See Topic Durability |
IGGY_KAFKA_IDEMPOTENT_PRODUCERS | true | true, false | Drops duplicate retried batches of idempotent producers. Turn it off only when older Iggy consumers read the same topics |
IGGY_KAFKA_OFFSETS_RETENTION_MINUTES | 10080, seven days | 1 or more | Minutes the committed offsets of an empty consumer group are kept. Cluster deployments only |
IGGY_KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS | 3000 | 0 or more | Milliseconds a new consumer group waits for more members before its first rebalance |
The schema lists only the keys that the deployment's gateway version and placement have. It also lists keys that LaserData fixes, such as the listener and the stream name. Only the keys in the table can change.
What is not supported
The gateway serves these Kafka requests: ApiVersions, Metadata, Produce, Fetch, ListOffsets, CreateTopics, DeleteTopics, CreatePartitions, OffsetCommit, OffsetFetch, FindCoordinator, JoinGroup, Heartbeat, LeaveGroup, SyncGroup, DescribeGroups, ListGroups, InitProducerId, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DeleteGroups, OffsetDelete, DescribeCluster, and DescribeAcls. Any other request closes the connection.
| Feature | What happens |
|---|---|
| Transactions | A transactional producer fails at start. Exactly-once stream processing does not work |
| Log compaction | A topic created with cleanup.policy=compact keeps every record. Kafka Connect and Schema Registry create such topics. Schema Registry needs kafkastore.topic.skip.validation=true, and Apicurio Registry 3 does not start |
| SCRAM and OAuth sign-in | Only SASL PLAIN is accepted. SCRAM-SHA-256, SCRAM-SHA-512, and OAUTHBEARER are refused |
| ACL changes | DescribeAcls shows the signed-in user's own permissions. CreateAcls and DeleteAcls are not served |
DeleteRecords | Not served |
| Deleting a topic by ID | DeleteTopics by topic ID is not served. Delete topics by name |
Free deployment limits
A Free deployment runs its gateway inside the deployment's slot, with the slot's memory and CPU. These limits apply to Kafka clients:
| Limit | Value |
|---|---|
| Kafka connections | 60 open connections |
| Largest Kafka request | 1 MiB. Keep producer settings such as max.request.size at or below it |
| Throughput | The slot's network rate limit of 100 KB/s applies to Kafka traffic too |
| Partitions | 100 across all streams and topics, Kafka topics included |
See Free for the rest of the slot's allowance.
Monitoring
The gateway reports as the runtime kafka_gateway. Each node sends heartbeats that show whether its gateway runs and reaches the streaming server. Metrics are process metrics, such as CPU and memory. Kafka request metrics are not reported yet. Logs follow the level that RUST_LOG sets. Read all three in Monitoring and through the observability API.
The gateway is not part of deployment health. A stopped, failed, or disabled gateway never marks the deployment unhealthy, and a disabled gateway is not a failure.
Errors
| Status | Code | Meaning | What to do |
|---|---|---|---|
400 | deployment_runtime_missing | The deployment cannot run the gateway, for example because it was created before the gateway became available | Use a deployment created since then |
400 | unsupported_runtime_action | enable on a BYOC or On-Premise deployment, or upgrade on a standalone or Free deployment | Use a supported placement. LaserData upgrades standalone and Free gateways |
400 | kafka_gateway_server_unsupported | At enable, or at a gateway or server upgrade, a gateway and server version pair that can meet during the rollout or its rollback does not work together | Upgrade the gateway first to a version that supports both server versions, or contact LaserData support |
A name conflict appears after the 202, in kafka_gateway_failure of the deployment details. See Turn it on.
MCP servers
Connect Claude Code, Cursor, VS Code, Codex, or any MCP client to LaserData with an API key: manage your organization and deployments, and read or write the data inside a deployment
Organization Hierarchy
Organize resources and control access through tenants, workspaces, environments, and deployments