Kafka Topics

Kafka uses topics to logically separate where messages are sent and received. The Kafka connection polls messages from the supplied topic.

Creating a Topic

For full documentation regarding creating topics, please consult the Kafka documentation. Topic creation is accomplished using Kafka APIs or via Kafka convenience scripts. For example, you can create the “MyMsgModuleTopic” topic used in the example below with the kafka-topics.sh script. The “MyMsgModuleTopic” is the topic where a Kafka client representing a data source would send messages:

$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4 --topic MyMsgModuleTopic

Kafka messages are sent as key/value pairs. Each data source sending a Kafka message to a MSG module SOSS object should specify the SOSS object’s name in the key and message in the value.

Note

The Kafka connection requires messages within Kafka to specify a key. The key will be used as the instance identifier for the corresponding SOSS object.

Note

In multi-host environments, Kafka topics should be created with at least the same number of partitions as hosts to scale the workload to all hosts.