On-Premises Kafka

You can create an on-premises Kafka connection to connect to an on-premises Kafka deployment. This enables data sources implemented as Kafka clients to send messages to ScaleOut MSG modules running within a ScaleOut Active Caching deployment.

Connecting to Kafka

When creating an on-premises Kafka connection using the Active Caching UI, specify the connection properties required for a producer and for a consumer to connect to the local Kafka deployment. The single property “bootstrap.servers” and “group.id” is often sufficient for localhost deployments. If the Kafka deployment requires other authentication properties, include each of them on a separate line. For example, if the Kafka service was running locally on port 9092 alongside a ScaleOut Active Caching deployment, you would specify the following connect.properties:

bootstrap.servers=localhost:9092
group.id=<group_name>

Configuring Topics

Specify each module’s topics as comma-separated list of the form:

<module name>,<topic for sending to SOSS objects>

Each list is specified on a separate line.

Note

Topics specified in the configuration must already exist. You can create the topics using the Kafka APIs or via Kafka convenience scripts

Note

By default, Kafka consumers use the configuration property “auto.offset.reset=latest”. If the topic for messaging MSG module SOSS objects already exists and contains messages that the Kafka connection should read, please specify the configuration property “auto.offset.reset=earliest”. Please read more about this configuration property from Apache Kafka’s documentation.