LaserData Cloud

Kafka gateway

Connect existing Kafka clients to a deployment through the Kafka gateway: turn it on, open port 9092 with an access rule, connect Java and librdkafka clients over SASL_SSL, change topic defaults, and see what is not supported

View as Markdownllms.txt

The Kafka gateway lets an application that uses a Kafka client library connect to a LaserData deployment. It speaks the Kafka wire protocol on TCP port 9092 and stores every Kafka topic as a topic of the deployment's streaming server, in a stream named kafka. The application keeps its Kafka client and changes only its connection settings. The gateway is optional and off by default. You turn it on per deployment.

ItemValue
AddressThe entries of kafka_bootstrap_servers in the deployment details, each on port 9092
Security protocolSASL_SSL, TLS with SASL sign-in. There is no plaintext listener
Sign-inSASL PLAIN with the name and password of a user of the deployment, such as the ld-admin deployment login
TrustThe LaserData root certificate
TopicsThe kafka stream of the deployment
NetworkAn access rule with the kafka transport. Closed by default

Pick the interface

You want toUse
Run an existing Kafka producer, consumer, or tool against a deploymentThe Kafka gateway
Build a new application, or use workflows, memory, sessions, and the other Laser SDK featuresLaser SDK with a connection string
Let an agent read or write a deployment's data with an API keyThe deployment data MCP endpoint

Where it runs

DeploymentKafka gateway
Managed clusterOne gateway beside the streaming server on every node. Clients connect to every node
Managed standalone, one dedicated nodeOne gateway on the node
FreeOne gateway inside the deployment's slot on the shared host
BYOCNot available yet
On-PremiseNot available yet

A deployment created before the Kafka gateway became available cannot run it, and enable answers 400 with deployment_runtime_missing. On BYOC and On-Premise deployments, enable answers 400 with unsupported_runtime_action.

Turn it on

POST
{supervisor_url}/deployments/{deployment_id}/activities/kafka_gateway/enable

Turn on the deployment's Kafka gateway. Requires deployment:manage.

A successful request returns 202 Accepted with no body. LaserData then creates the kafka stream for your topics, a kafka.meta stream for the gateway's own state, and a service user named kafka-gateway, and starts the gateway on every node. The kafka_gateway_enabled field of Get Deployment on the supervisor shows the setting, and the deployment's activities show the rollout. While LaserData prepares the streams and the user, kafka_gateway_provisioning is true. An accepted request does not mean that the gateway runs. Each node reports its own heartbeat once its gateway is running.

If the deployment already holds a stream named kafka or kafka.meta, or a user named kafka-gateway, that LaserData did not create, the enable activity fails. The gateway stays off, kafka_gateway_enabled returns to false, and kafka_gateway_failure in the deployment details gives the reason, with a conflict that names the stream or user and whether it is foreign, missing or replaced. A later successful enable clears it. LaserData never deletes, renames, or changes that resource. Rename or delete it yourself and send enable again. If you did not create it, contact LaserData support.

LaserData records the two streams and the user when it creates them. Do not delete, rename, or replace them. A gateway does not start when one of them is missing or was replaced.

The same route takes these actions for the runtime kafka_gateway. Each needs deployment:manage and answers 202 Accepted.

ActionEffect
enableCreates the streams and the user the first time, then starts the gateway on every node
disableStops the gateway on every node and keeps it stopped. Kafka clients disconnect. Topics, their data, the kafka-gateway user, and the settings stay, and a later enable uses them again
startStarts a stopped gateway
stopStops the gateway. It stays stopped across node restarts until start
restartRestarts the gateway. On a cluster the nodes restart one at a time
upgradeMoves the gateway to a newer version, one node at a time. Clusters only

On standalone and Free deployments LaserData manages the gateway version, and upgrade answers 400 with unsupported_runtime_action. LaserData checks that the gateway version works with the deployment's streaming server version. It refuses an upgrade of either one that would pair versions that do not work together, also partway through a rolling upgrade.

Open port 9092

The gateway listens on TCP port 9092, and an access rule opens it for the source ranges the rule names. Set the kafka transport on the rule:

curl -X POST {supervisor_url}/deployments/{deployment_id}/access_rules \
  -H "ld-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kafka-clients",
    "cidr_blocks": ["203.0.113.0/24"],
    "rules": {
      "kafka": true
    }
  }'

The port stays closed until a rule sets kafka. The default rule of a Free deployment, 0.0.0.0/0, does not set it either, so a Free deployment also needs a rule of its own for Kafka clients. Allow My IP accepts {"kafka": true} to admit the address you call from for 24 hours.

Opening the port and running the gateway are separate. A rule with kafka on a deployment whose gateway is off opens a port that nothing listens on. A gateway without such a rule runs, and no client can reach it.

A rule that sets kafka for 0.0.0.0/0 makes Kafka sign-in reachable from the whole internet, and every sign-in runs a password check on the streaming server. Admit only the ranges your Kafka clients connect from.

Connect a client

SettingValue
bootstrap.serversThe entries of kafka_bootstrap_servers from Get Deployment on the supervisor, joined with commas
security.protocolSASL_SSL
sasl.mechanismPLAIN. librdkafka names the property sasl.mechanisms
User name and passwordA user of the deployment. The ld-admin login from Get Deployment Credentials works as it is
TrustThe LaserData root certificate, which the Console offers for download with the deployment's connection details
bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.location=<path to laserdata.crt>

With ssl.truststore.type=PEM, the Java client reads the downloaded certificate file as its trust store.

bootstrap.servers=<bootstrap servers>
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username=<user>
sasl.password=<password>
ssl.ca.location=<path to laserdata.crt>

Clients built on librdkafka, such as confluent-kafka for Python, Go, and .NET, take the same properties.

Keep the password in an environment variable or a secret store, never in a file you commit.

On a cluster, kafka_bootstrap_servers lists one address per node, not the deployment domain. The gateway on each node points clients back at its own node, and newer clients ask for a node by its address. Your client must reach every address in the list, so put all of them in bootstrap.servers and place no load balancer or shared proxy in front of them. On a public cluster the entries are the nodes' public IP addresses, and on a private deployment their private addresses. Read the list from the deployment details instead of building it from the domain or a port pattern. PrivateLink and Private Service Connect do not carry Kafka traffic.

On a standalone deployment and on a Free deployment, the list holds one entry: the deployment domain on port 9092. A Free deployment shares its host with other deployments, and the host routes each Kafka connection by the TLS server name the client sends. Connect by the host name from the list, never by IP address. The Java client, librdkafka, franz-go, and Sarama send the server name when they connect by host name. The host drops a connection that does not send it.

Every new Kafka connection signs in again, and each sign-in runs a password check on the streaming server. Keep connections open and reuse them.

Who can read and write

Every user of the deployment that can sign in can read and write every Kafka topic through the gateway, including a user that holds no permission on the streaming server. The gateway checks the user name and password when a client connects, then serves produce, fetch, and the other data requests as its own kafka-gateway service user. Kafka topics live in the kafka stream, and no other stream is reachable through the gateway.

Some administrative requests are also checked against the signed-in user's own permissions on the streaming server. Deleting topics, adding partitions, changing topic configs, deleting consumer groups or their offsets, and creating a topic automatically when a client names it need manage_topics or manage_streams. Committing offsets needs read permission on the topic. ld-admin passes all of these checks.

Topics and partitions

Kafka topics are ordinary topics of the streaming server in the kafka stream, so they appear there in the Console and to Iggy clients.

A CreateTopics request is not checked against the signed-in user's permissions. Any client that signs in can create topics of up to 1000 partitions each. On a Free deployment the slot's cap of 100 partitions across all streams bounds this. A deployment on dedicated nodes has no partition cap by default, so set metadata.partitions_max in its server configuration before you turn the gateway on.

A topic created through Kafka without its own retention keeps messages for seven days, the default of IGGY_KAFKA_TOPIC_RETENTION_MS. The retention becomes the topic's message expiry on the streaming server, which removes whole sealed segments, so a message can outlive the retention until its segment is sealed. See Topic Storage Options.

On a cluster the gateways share consumer groups. When a node stops, its groups move to the gateways on the other nodes. A planned stop or restart hands them over at once. After a crash the move takes up to about 45 seconds. On standalone and Free deployments the one gateway holds every group, and committed offsets of a group do not expire.

Settings

Change the gateway's settings with the kafka_gateway kind on the configuration API, for example {supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway/schema and POST {supervisor_url}/deployments/{deployment_id}/configs/kafka_gateway. Activating a version applies it to every node, one node at a time on a cluster, which can take minutes. Saving a version never starts a stopped gateway.

KeyDefaultAccepted valuesWhat it sets
RUST_LOGinfooff, error, warn, infoLog level of the gateway
IGGY_KAFKA_AUTO_CREATE_TOPICS_ENABLEDfalsetrue, falseCreates a topic that a client names and nobody created, like Kafka's auto.create.topics.enable
IGGY_KAFKA_AUTO_CREATE_TOPICS_PARTITIONS11 to 1000Partitions of a topic created automatically. Used only while automatic creation is on
IGGY_KAFKA_TOPIC_RETENTION_MS604800000, seven days-1 or a positive numberRetention in milliseconds of a topic created without one. -1 keeps messages forever
IGGY_KAFKA_TOPIC_RETENTION_BYTES-1-1 or at least 1073741824, 1 GiBSize limit in bytes of a topic created without one. -1 sets no limit
IGGY_KAFKA_TOPIC_DURABILITYreplicatedreplicated, persistedHow a topic created through Kafka stores its messages and offsets. See Topic Durability
IGGY_KAFKA_IDEMPOTENT_PRODUCERStruetrue, falseDrops duplicate retried batches of idempotent producers. Turn it off only when older Iggy consumers read the same topics
IGGY_KAFKA_OFFSETS_RETENTION_MINUTES10080, seven days1 or moreMinutes the committed offsets of an empty consumer group are kept. Cluster deployments only
IGGY_KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS30000 or moreMilliseconds a new consumer group waits for more members before its first rebalance

The schema lists only the keys that the deployment's gateway version and placement have. It also lists keys that LaserData fixes, such as the listener and the stream name. Only the keys in the table can change.

What is not supported

The gateway serves these Kafka requests: ApiVersions, Metadata, Produce, Fetch, ListOffsets, CreateTopics, DeleteTopics, CreatePartitions, OffsetCommit, OffsetFetch, FindCoordinator, JoinGroup, Heartbeat, LeaveGroup, SyncGroup, DescribeGroups, ListGroups, InitProducerId, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DeleteGroups, OffsetDelete, DescribeCluster, and DescribeAcls. Any other request closes the connection.

FeatureWhat happens
TransactionsA transactional producer fails at start. Exactly-once stream processing does not work
Log compactionA topic created with cleanup.policy=compact keeps every record. Kafka Connect and Schema Registry create such topics. Schema Registry needs kafkastore.topic.skip.validation=true, and Apicurio Registry 3 does not start
SCRAM and OAuth sign-inOnly SASL PLAIN is accepted. SCRAM-SHA-256, SCRAM-SHA-512, and OAUTHBEARER are refused
ACL changesDescribeAcls shows the signed-in user's own permissions. CreateAcls and DeleteAcls are not served
DeleteRecordsNot served
Deleting a topic by IDDeleteTopics by topic ID is not served. Delete topics by name

Free deployment limits

A Free deployment runs its gateway inside the deployment's slot, with the slot's memory and CPU. These limits apply to Kafka clients:

LimitValue
Kafka connections60 open connections
Largest Kafka request1 MiB. Keep producer settings such as max.request.size at or below it
ThroughputThe slot's network rate limit of 100 KB/s applies to Kafka traffic too
Partitions100 across all streams and topics, Kafka topics included

See Free for the rest of the slot's allowance.

Monitoring

The gateway reports as the runtime kafka_gateway. Each node sends heartbeats that show whether its gateway runs and reaches the streaming server. Metrics are process metrics, such as CPU and memory. Kafka request metrics are not reported yet. Logs follow the level that RUST_LOG sets. Read all three in Monitoring and through the observability API.

The gateway is not part of deployment health. A stopped, failed, or disabled gateway never marks the deployment unhealthy, and a disabled gateway is not a failure.

Errors

StatusCodeMeaningWhat to do
400deployment_runtime_missingThe deployment cannot run the gateway, for example because it was created before the gateway became availableUse a deployment created since then
400unsupported_runtime_actionenable on a BYOC or On-Premise deployment, or upgrade on a standalone or Free deploymentUse a supported placement. LaserData upgrades standalone and Free gateways
400kafka_gateway_server_unsupportedAt enable, or at a gateway or server upgrade, a gateway and server version pair that can meet during the rollout or its rollback does not work togetherUpgrade the gateway first to a version that supports both server versions, or contact LaserData support

A name conflict appears after the 202, in kafka_gateway_failure of the deployment details. See Turn it on.

On this page