I’m trying to connect to the Kafka using a KafkaTool. I got an error:
Error connecting to the cluster. failed create new KafkaAdminClient
Kafka and Zookeeper is hosting in the Docker. I run next commands
docker network create kafka
docker run --network=kafka -d --name zookeeper -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest
docker run --network=kafka -d -p 9092:9092 --name kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka:latest
Settings for KafkaTool
Why does KafkaTool not connect to the Kafka that is hosting in the Docker?
OneCricketeer
169k18 gold badges124 silver badges232 bronze badges
asked Sep 23, 2019 at 14:47
I’m assuming this GUI is not coming from a Docker container. Therefore, your host machine doesn’t know what zookeeper or kafka are, only the Docker network does.
In the GUI, you will want to use localhost for both, then in your Kafka run command, leave all the other variables alone but change -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
Zookeeper run command is fine, but add -p 2181:2181 to expose the port out to the host so that the GUI can connect
answered Sep 23, 2019 at 15:50
OneCricketeerOneCricketeer
169k18 gold badges124 silver badges232 bronze badges
7
Failed to create new KafkaAdminClient #112
Comments
nmadmon commented Sep 22, 2020
Hi,
when running the backup command I’m getting an error and it looks like the script is stuck till I’m pressing Ctrl+C.
What can cause this issue?
[kafka@ip-172-31-5-236 kafka-backup]$ ./backup-standalone.sh —bootstrap-server localhost:9092 —target-dir /home/kafka/kafka-backup/bk —topics ‘__consumer_offsets,dataRequest,invalidation,subscription,topicName’
Sep 22, 2020 7:44:46 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
[2020-09-22 07:44:46,238] INFO BackupSinkConfig values:
max.segment.size.bytes = 1073741824
target.dir = /home/kafka/kafka-backup/bk
(de.azapps.kafkabackup.sink.BackupSinkConfig)
[2020-09-22 07:44:46,293] ERROR WorkerSinkTask Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:69)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at de.azapps.kafkabackup.sink.BackupSinkTask.start(BackupSinkTask.java:53)
at de.azapps.kafkabackup.sink.BackupSinkTask.start(BackupSinkTask.java:39)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:267)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:163)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(Lorg/apache/kafka/common/security/auth/SecurityProtocol;Lorg/apache/kafka/common/security/JaasContext$Type;Lorg/apache/kafka/common/config/AbstractConfig;Lorg/apache/kafka/common/network/ListenerName;Ljava/lang/String;Lorg/apache/kafka/common/utils/Time;Z)Lorg/apache/kafka/common/network/ChannelBuilder;
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426)
. 13 more
[2020-09-22 07:44:46,295] ERROR WorkerSinkTask Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2020-09-22 07:44:46,296] ERROR WorkerSinkTask Task threw an uncaught and unrecoverable exception during shutdown (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at de.azapps.kafkabackup.sink.BackupSinkTask.stop(BackupSinkTask.java:138)
at org.apache.kafka.connect.runtime.WorkerSinkTask.close(WorkerSinkTask.java:144)
at org.apache.kafka.connect.runtime.WorkerTask.doClose(WorkerTask.java:149)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:176)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The text was updated successfully, but these errors were encountered:
Источник
Connecting SASL_SSL Kafka Cluster #583
Comments
tommyJimmy87 commented Mar 6, 2019 •
I’m using Security Protocol set to «SASL_SSL» which means I need to specify all the JKS and JAAS in order to connect to the Cluster. I’m providing those information through the property file, but I found out that when the KafkaAdminClient runs it gets wrong SSL configuration because of this :
I’m not using SSL but SASL_SSL and I still need those properties to be set in the KafkaClientAdmin configurations.
The class where i found that code portion is KafkaCruiseControlUtils.java.
This is the error I’m facing :
Tag Cruise Control version is 2.0.36. Kafka Version is 1.1.x.
The text was updated successfully, but these errors were encountered:
efeg commented Mar 7, 2019
@tommyJimmy87 Thanks for reporting the issue!
Cruise Control does not support SASL_SSL , yet — i.e. it currently supports SSL and Plaintext .
Would you be interested in submitting a patch to add SASL_SSL support?
becketqin commented Mar 7, 2019
@tommyJimmy87 Are you trying to use Kerberos for authentication? If so, one of the following configs is needed:
It would be more helpful if you can share the configurations you are using.
tommyJimmy87 commented Mar 7, 2019
@becketqin We are not using kerberos. we are just using SASL_SSL with the username and password in the JAAS.. but apparently as @efeg said is not supported.
tommyJimmy87 commented Mar 7, 2019
@efeg I can take a look how complicate is to patch it.
becketqin commented Mar 7, 2019
@tommyJimmy87 Hmm, CC should support all the mechanisms support by Kafka itself. Have you set sasl.mechanism in your config? By default it is going to be GSSAPI and kerberos will be used. With SASL_SSL, change the that config to PLAIN and have a try.
becketqin commented Mar 7, 2019
Also, please make sure you are using the org.apache.kafka.common.security.plain.PlainLoginModule with your username and password in the jaas config file.
tommyJimmy87 commented Mar 7, 2019 •
@becketqin Thx, here my config file :
And also I’m providing my JAAS file path in the KAFKA_OPTS env variable that should be added when CC runs.
We are using the same JAAS file for several application.
tommyJimmy87 commented Mar 7, 2019 •
@becketqin The problem is just the KafkaAdminClient Configs, the ConsumerConf for example are created with the right values (jks.. password and jaas are present in the ConsumerConf but not in the KafkaAdminClient).
tommyJimmy87 commented Mar 7, 2019
@becketqin Uhm.. with the tag 0.1.39 is working actually.. earlier I was using 2.0.36 Tag. what’s the different ? Master branch should be at the 0.1.39 right ? I guess it was a little bit confusing.
becketqin commented Mar 7, 2019
@tommyJimmy87 Thanks for the update. This sounds like a bug in 2.0.36. But I could be wrong. @efeg would be more familiar with the diff between those two versions.
efeg commented Mar 7, 2019
@tommyJimmy87 The reason why the tag 0.1.39 is working, but 2.0.36 is not is because former versions (i.e. 0.1.* ) do not call describeLogDirs to show logDirs in the response of kafka_cluster_state (i.e. the endpoint that your request is sent to). The describeLogDirs creates and uses a KafkaAdminClient to return the corresponding response (supported for Kafka: 1.1.0+ with Cruise Control: 2.* ).
To support SASL_SSL in 2.* , I believe there are (at least) two places to update (and test) — i.e. you may search for the string if (securityProtocol.equals(SecurityProtocol.SSL.name)) to find the exact locations:
^ So I suspect that all we need would be:
tommyJimmy87 commented Mar 8, 2019
@efeg Ok as I thought. If you want I can contribute on this.
efeg commented Mar 8, 2019
@tommyJimmy87 Sure, we appreciate contributions!
Sugaroverdose commented Apr 5, 2019 •
I’ve encountered the same issue with SASL_PLAINTEXT and sasl.mechanism=SCRAM-SHA-256:
Cruise Control does connects to the cluster but right after that it tries to start AdminClient which does not inherit client.id, jaas.config and sasl.mechanism
UPD: managed to fix this by adding
import org.apache.kafka.common.config.SaslConfigs;
and
setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_MECHANISM);
into KafkaCruiseControlUtils.java
i guess the same should be added into CruiseControlMetricsUtils.java
efeg commented Apr 5, 2019
@Sugaroverdose Sounds great! Would you consider creating a PR with the patch?
Sugaroverdose commented Apr 5, 2019
@efeg i’ll try to
it seems like that CruiseControlMetricsUtils.java does not inherit securityProtocol, so idk how it works with something different than plaintext
Sugaroverdose commented Apr 9, 2019
@tommyJimmy87 you may try latest migrate_to_kafka_2_0 branch, it should work in your case now
poyyi commented May 2, 2019
I’m novice in adopting to cruise-contol using 2.029,trying with Sasl_ssl going by above discussion it looks I’m may not be able use it sasl_ssl
could you suggest which release shld I pick ..
Thank u
tommyJimmy87 commented May 3, 2019
@poyyi What is your Cruise Control Metric Jar configuration?
poyyi commented May 4, 2019
@tommy
it is cruise-control-metrics-reporter-0.1.0-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/linkedin/
com/linkedin/kafka/
com/linkedin/kafka/cruisecontrol/
com/linkedin/kafka/cruisecontrol/metricsreporter/
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor$Context.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric$MetricClassId.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType$MetricScope.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/BrokerMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricsUtils.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/PartitionMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/TopicMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporterConfig.class
com/linkedin/kafka/cruisecontrol/metricsreporter/exception/
on the kafka servers.. «2024» is sasl_ssl port
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2024
auto.create.topics.enable=true
CC property file
bootstrap.servers=BK1-IP:2024,BK2-IP:2024,BK3-IP:2024,BK4-IP:2024,BK5-IP:2024
zookeeper.connect=ZK1-IP:2181,ZK2-IP:2181,ZK3-IP:2181
sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
ssl.key.password=XXXXX@123
ssl.keystore.password=XXXXX@123
ssl.keystore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.truststore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.client.auth=true
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
ssl.truststore.password=XXXXX@123
ssl.endpoint.identification.algorithm=
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_SSL
tommyJimmy87 commented May 5, 2019
I guess you are missing all the Cruise Control configs for Sasl SSL.
Try with these also :
poyyi commented May 17, 2019 •
@jimmy sorry could not try immly on SASL_SSL (but tired with SSL which i run into some other issue described below ).. i have few queries on your suggestions for SASL_SSL
1)Above mentioned properties shld go into server.properties on brokers?
2) shld they be given exactly like how you gave above
KAFKA_METRIC_REPORTERS: ?, currently i have it as below in my server.properties
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.security.protocol=SSL
PL note version I’m using:
Cruise control version: Cruise-control-2.0.42
Kafka :kafka_2.12-0.11.0.0 (Kafka version : 0.11.0.1)
I ran with SSL pl find my setting and Exception I’m getting for
Failures
«GET /kafkacruisecontrol/kafka_cluster_state?json=true HTTP/1.1» 500 4462 79
Success
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 2637 8
«GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1» 200 61 2
«GET /kafkacruisecontrol/state?substates=MONITOR&verbose=true&json=true HTTP/1.1» 200 288 1
«GET /kafkacruisecontrol/state?substates=ANALYZER&verbose=true&json=true HTTP/1.1» 200 2866 2
«GET /kafkacruisecontrol/load?allow_capacity_estimation=true&json=true HTTP/1.1» 200 3074 2
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 3075 2
«GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1» 200 61 1
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 3275 2
Источник
Connecting SASL_SSL Kafka Cluster #583
Comments
tommyJimmy87 commented Mar 6, 2019 •
I’m using Security Protocol set to «SASL_SSL» which means I need to specify all the JKS and JAAS in order to connect to the Cluster. I’m providing those information through the property file, but I found out that when the KafkaAdminClient runs it gets wrong SSL configuration because of this :
I’m not using SSL but SASL_SSL and I still need those properties to be set in the KafkaClientAdmin configurations.
The class where i found that code portion is KafkaCruiseControlUtils.java.
This is the error I’m facing :
Tag Cruise Control version is 2.0.36. Kafka Version is 1.1.x.
The text was updated successfully, but these errors were encountered:
efeg commented Mar 7, 2019
@tommyJimmy87 Thanks for reporting the issue!
Cruise Control does not support SASL_SSL , yet — i.e. it currently supports SSL and Plaintext .
Would you be interested in submitting a patch to add SASL_SSL support?
becketqin commented Mar 7, 2019
@tommyJimmy87 Are you trying to use Kerberos for authentication? If so, one of the following configs is needed:
It would be more helpful if you can share the configurations you are using.
tommyJimmy87 commented Mar 7, 2019
@becketqin We are not using kerberos. we are just using SASL_SSL with the username and password in the JAAS.. but apparently as @efeg said is not supported.
tommyJimmy87 commented Mar 7, 2019
@efeg I can take a look how complicate is to patch it.
becketqin commented Mar 7, 2019
@tommyJimmy87 Hmm, CC should support all the mechanisms support by Kafka itself. Have you set sasl.mechanism in your config? By default it is going to be GSSAPI and kerberos will be used. With SASL_SSL, change the that config to PLAIN and have a try.
becketqin commented Mar 7, 2019
Also, please make sure you are using the org.apache.kafka.common.security.plain.PlainLoginModule with your username and password in the jaas config file.
tommyJimmy87 commented Mar 7, 2019 •
@becketqin Thx, here my config file :
And also I’m providing my JAAS file path in the KAFKA_OPTS env variable that should be added when CC runs.
We are using the same JAAS file for several application.
tommyJimmy87 commented Mar 7, 2019 •
@becketqin The problem is just the KafkaAdminClient Configs, the ConsumerConf for example are created with the right values (jks.. password and jaas are present in the ConsumerConf but not in the KafkaAdminClient).
tommyJimmy87 commented Mar 7, 2019
@becketqin Uhm.. with the tag 0.1.39 is working actually.. earlier I was using 2.0.36 Tag. what’s the different ? Master branch should be at the 0.1.39 right ? I guess it was a little bit confusing.
becketqin commented Mar 7, 2019
@tommyJimmy87 Thanks for the update. This sounds like a bug in 2.0.36. But I could be wrong. @efeg would be more familiar with the diff between those two versions.
efeg commented Mar 7, 2019
@tommyJimmy87 The reason why the tag 0.1.39 is working, but 2.0.36 is not is because former versions (i.e. 0.1.* ) do not call describeLogDirs to show logDirs in the response of kafka_cluster_state (i.e. the endpoint that your request is sent to). The describeLogDirs creates and uses a KafkaAdminClient to return the corresponding response (supported for Kafka: 1.1.0+ with Cruise Control: 2.* ).
To support SASL_SSL in 2.* , I believe there are (at least) two places to update (and test) — i.e. you may search for the string if (securityProtocol.equals(SecurityProtocol.SSL.name)) to find the exact locations:
^ So I suspect that all we need would be:
tommyJimmy87 commented Mar 8, 2019
@efeg Ok as I thought. If you want I can contribute on this.
efeg commented Mar 8, 2019
@tommyJimmy87 Sure, we appreciate contributions!
Sugaroverdose commented Apr 5, 2019 •
I’ve encountered the same issue with SASL_PLAINTEXT and sasl.mechanism=SCRAM-SHA-256:
Cruise Control does connects to the cluster but right after that it tries to start AdminClient which does not inherit client.id, jaas.config and sasl.mechanism
UPD: managed to fix this by adding
import org.apache.kafka.common.config.SaslConfigs;
and
setStringConfigIfExists(configs, adminClientConfigs, SaslConfigs.SASL_MECHANISM);
into KafkaCruiseControlUtils.java
i guess the same should be added into CruiseControlMetricsUtils.java
efeg commented Apr 5, 2019
@Sugaroverdose Sounds great! Would you consider creating a PR with the patch?
Sugaroverdose commented Apr 5, 2019
@efeg i’ll try to
it seems like that CruiseControlMetricsUtils.java does not inherit securityProtocol, so idk how it works with something different than plaintext
Sugaroverdose commented Apr 9, 2019
@tommyJimmy87 you may try latest migrate_to_kafka_2_0 branch, it should work in your case now
poyyi commented May 2, 2019
I’m novice in adopting to cruise-contol using 2.029,trying with Sasl_ssl going by above discussion it looks I’m may not be able use it sasl_ssl
could you suggest which release shld I pick ..
Thank u
tommyJimmy87 commented May 3, 2019
@poyyi What is your Cruise Control Metric Jar configuration?
poyyi commented May 4, 2019
@tommy
it is cruise-control-metrics-reporter-0.1.0-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
com/
com/linkedin/
com/linkedin/kafka/
com/linkedin/kafka/cruisecontrol/
com/linkedin/kafka/cruisecontrol/metricsreporter/
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporter.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor$Context.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/YammerMetricProcessor.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric$MetricClassId.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/CruiseControlMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType$MetricScope.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/RawMetricType.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/BrokerMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde$1.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricSerde.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/MetricsUtils.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/PartitionMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/metric/TopicMetric.class
com/linkedin/kafka/cruisecontrol/metricsreporter/CruiseControlMetricsReporterConfig.class
com/linkedin/kafka/cruisecontrol/metricsreporter/exception/
on the kafka servers.. «2024» is sasl_ssl port
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.bootstrap.servers=0.0.0.0:2024
auto.create.topics.enable=true
CC property file
bootstrap.servers=BK1-IP:2024,BK2-IP:2024,BK3-IP:2024,BK4-IP:2024,BK5-IP:2024
zookeeper.connect=ZK1-IP:2181,ZK2-IP:2181,ZK3-IP:2181
sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
ssl.key.password=XXXXX@123
ssl.keystore.password=XXXXX@123
ssl.keystore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.truststore.location=/opt/app/work/kafka-0.10.01/certs/kafka.server.truststore.jks
ssl.client.auth=true
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
ssl.truststore.password=XXXXX@123
ssl.endpoint.identification.algorithm=
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_SSL
tommyJimmy87 commented May 5, 2019
I guess you are missing all the Cruise Control configs for Sasl SSL.
Try with these also :
poyyi commented May 17, 2019 •
@jimmy sorry could not try immly on SASL_SSL (but tired with SSL which i run into some other issue described below ).. i have few queries on your suggestions for SASL_SSL
1)Above mentioned properties shld go into server.properties on brokers?
2) shld they be given exactly like how you gave above
KAFKA_METRIC_REPORTERS: ?, currently i have it as below in my server.properties
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
cruise.control.metrics.reporter.security.protocol=SSL
PL note version I’m using:
Cruise control version: Cruise-control-2.0.42
Kafka :kafka_2.12-0.11.0.0 (Kafka version : 0.11.0.1)
I ran with SSL pl find my setting and Exception I’m getting for
Failures
«GET /kafkacruisecontrol/kafka_cluster_state?json=true HTTP/1.1» 500 4462 79
Success
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 2637 8
«GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1» 200 61 2
«GET /kafkacruisecontrol/state?substates=MONITOR&verbose=true&json=true HTTP/1.1» 200 288 1
«GET /kafkacruisecontrol/state?substates=ANALYZER&verbose=true&json=true HTTP/1.1» 200 2866 2
«GET /kafkacruisecontrol/load?allow_capacity_estimation=true&json=true HTTP/1.1» 200 3074 2
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 3075 2
«GET /kafkacruisecontrol/state?substates=EXECUTOR&verbose=true&json=true HTTP/1.1» 200 61 1
«GET /kafkacruisecontrol/user_tasks?json=true HTTP/1.1» 200 3275 2
Источник
Problem
In certain situations, for example after the upgrade from Information Server 11.7.0.x to 11.7.1, Kafka fails to properly register in Zookeeper. This causes malfunction of many capabilities, including synchronization of metadata to Enterprise Search database, metadata discovery, and others.
Diagnosing The Problem
When you run the Kafka health check tool, you can find the error «No resolvable bootstrap urls given in bootstrap.servers»:
# /opt/IBM/InformationServer/Clients/istools/cli/istool.sh kafka healthCheck -username isadmin -password Win4UG_u
Kafka health check started...
ERROR: processing command healtCheck.
Kafka service health check failed! ERROR: Failed create new KafkaAdminClient
Exception details:
org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:338)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:52)
...
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66)
at org.apache.kafka.clients.admin.KafkaAdminClient.<init>(KafkaAdminClient.java:368)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:331)
... 47 moreAlso, when you look at Kafka metadata in Zookeeper, no Kafka brokers are registered:
[zk: host:2181/kafka(CONNECTED) 11] ls /brokers/ids
[]Resolving The Problem
To resolve the issue, re-create the Kafka pod. Kafka broker is registered again in Zookeeper when Kafka is started.
Run the following command on the Microservices Tier control plane node:
# kubectl delete pod kafka-0
Document Location
Worldwide
[{«Business Unit»:{«code»:»BU059″,»label»:»IBM Software w/o TPS»},»Product»:{«code»:»SSZJPZ»,»label»:»IBM InfoSphere Information Server»},»Component»:»Microservices Tier»,»Platform»:[{«code»:»PF016″,»label»:»Linux»}],»Version»:»11.7.0 and above»,»Edition»:»»,»Line of Business»:{«code»:»LOB10″,»label»:»Data and AI»}}]
I have created a kerberized Kafka cluster. It is working fine and I can produce and consume messages. Now I want to use the kafkatool2 software, and I am running the program through cmd by this command:
kafkatool.exe -J-Djava.security.auth.login.config=d:client_jaas.conf
But after filling up the blank spaces I am getting this error:
ERROR Kafkatool.exe — SASL_PLAINTEXT
This is my JAAS file:
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="D:kafkaadmin.keytab"
storeKey=true
useTicketCache=false
principal="admin/kafkatool@TEST.COM";
};
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
storeKey=true
keyTab="D:kafkaadmin.keytab"
principal="admin/kafkatool@TEST.COM";
};
// For Zookeeper Client
Client {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
storeKey=true
keyTab="D:kafkaadmin.keytab"
principal="admin/kafkatool@TEST.COM";
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
storeKey=true
keyTab="D:kafkaadmin.keytab"
principal="admin/kafkatool@TEST.COM";
};
What should I have missing?
Kafka-cluster OS: Ubuntu 14.04
User OS: Windows 10 Enterprise
asked May 26, 2019 at 8:17
A.KazemiA.Kazemi
111 silver badge3 bronze badges
I faced similar issue in my virtual machine when I tried to look at messages in the kerberized kafka environment. Try setting serviceName=»kafka» in your KafkaClient configuration.
Also make sure you run kafkatool by passing -J-Djava.security.auth.login.config as an argument.
Example client JAAS configuration:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
serviceName="kafka"
useKeyTab=true
keyTab="<path_to>/user.keytab"
principal="user@domain";
};
I used the tool in linux environment but I think the above will solve the issue for windows as well.
answered Aug 14, 2019 at 20:46
Log inSkip to main contentSkip to sidebar
- XMLWordPrintableJSON
Details
-
Type:
Bug
-
Status:
Resolved -
Priority:
Major
-
Resolution:
Fixed
-
Affects Version/s:
None
-
Component/s:
None
Major Description
- AdminClient should only call Metadata#requestUpdate when needed.
- AdminClient should retry requests for which the controller has changed.
- Fix an issue where AdminClient requests might not get a security exception, even when a metadata fetch fails with an authorization exception.
- Fix a possible issue where AdminClient might leak a socket after the timeout expires on a hard close, if a very narrow race condition is hit