Kafka Notification Settings

This page documents settings for configuring an Kafka service as a target for Bucket Notifications. See Publish Events to Kafka for a tutorial on using these settings.

You can establish or modify settings by defining:

• an environment variable on the host system prior to starting or restarting the MinIO Server. Refer to your operating system’s documentation for how to define an environment variable.
• a configuration setting using mc admin config set.

If you define both an environment variable and the similar configuration setting, MinIO uses the environment variable value.

Some settings have only an environment variable or a configuration setting, but not both.

Multiple Kafka Targets

You can specify multiple Kafka service endpoints by appending a unique identifier _ID for each set of related Kafka settings on to the top level key.

Examples

For example, the following commands set two distinct Kafka service endpoints as PRIMARY and SECONDARY respectively:

• Environment Variable
• Configuration Setting

export MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"

export MINIO_NOTIFY_KAFKA_ENABLE_SECONDARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_SECONDARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"

mc admin config set notify_kafka:primary \
   brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
   [ARGUMENT=VALUE ...]

mc admin config set notify_kafka:secondary \
   brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
   [ARGUMENT=VALUE ...]

Settings

Enable

Required

• Environment Variable
• Configuration Setting

mc admin config set notify_kafka \
  brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
  [ARGUMENT="VALUE"] ... \

Brokers

Required

• Environment Variable
• Configuration Setting

Specify a comma-separated list of Kafka broker addresses. For example:

"kafka1.example.com:2021,kafka2.example.com:2021"

Topic

Optional

• Environment Variable
• Configuration Setting

Specify the name of the Kafka topic to which MinIO publishes bucket events.

SASL

Optional

• Environment Variable
• Configuration Setting

Specify on to enable SASL authentication.

SASL Username

Optional

• Environment Variable
• Configuration Setting

Specify the username for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).

SASL Password

Optional

• Environment Variable
• Configuration Setting

Specify the password for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).

SASL Mechanism

Optional

• Environment Variable
• Configuration Setting

Specify the SASL mechanism to use for authenticating to the Kafka broker(s). MinIO supports the following mechanisms:

• PLAIN (Default)
• SHA256
• SHA512

TLS Client Auth

Optional

• Environment Variable
• Configuration Setting

Specify the client authentication type of the Kafka broker(s). The following table lists the supported values and their mappings

See ClientAuthType for more information on each client auth type.

TLS

Optional

• Environment Variable
• Configuration Setting

Specify on to enable TLS connectivity to the Kafka broker(s).

TLS Skip Verify

Optional

• Environment Variable
• Configuration Setting

Enables or disables TLS verification of the NATS service endpoint TLS certificates.

• Specify on to disable TLS verification (Default).
• Specify off to enable TLS verification.

Client TLS Cert

Optional

• Environment Variable
• Configuration Setting

Specify the path to the client certificate to use for performing mTLS authentication to the Kafka broker(s).

Client TLS Key

Optional

• Environment Variable
• Configuration Setting

Specify the path to the client private key to use for performing mTLS authentication to the Kafka broker(s).

Version

Optional

• Environment Variable
• Configuration Setting

Specify the version of the Kafka cluster to assume when performing operations against that cluster. See the sarama reference documentation for more information on this field’s behavior.

Batch Size

Optional

• Environment Variable
• Configuration Setting

Specify the integer value to use as the batch size for sending records to Kafka.

Queue Directory

Optional

• Environment Variable
• Configuration Setting

Specify the directory path to enable MinIO’s persistent event store for undelivered messages, such as /opt/minio/events.

MinIO stores undelivered events in the specified store while the Kafka server/broker is offline and replays the stored events when connectivity resumes.

Queue Limit

Optional

• Environment Variable
• Configuration Setting

Specify the maximum limit for undelivered messages. Defaults to 100000.

Comment

Optional

• Environment Variable
• Configuration Setting

Specify a comment to associate with the Kafka configuration.

Compression Codec

Optional

• Environment Variable
• Configuration Setting

Specify the compression codec to use when sending records to Kafka.

Supports the following values:

• none
• snappy
• gzip
• lz4
• zstd

Compression Level

Optional

• Environment Variable
• Configuration Setting

Controls the level of compression applied based on the configured compression codec.

Specify an integer value greater than or equal to 0. The effect of the value depends on the selected codec.