• Sl chevron_right

      How to inspect ZooKeeper logs and snapshots

      pubsub.slavino.sk / sleeplessbestie · Friday, 18 March, 2022 - 12:00 edit

    Inspect ZooKeeper logs and snapshots. Inspect ZooKeeper data directory. $ ls /opt/kafka/zookeeper_data/version-2/ acceptedEpoch log.2000001e0 log.2000001f6 log.20000020c log.200000222 log.200000238 log.20000024e log.200000264 log.20000027a log.200000290 log.2000002a6 log.2000002bc log.2000002d2 log.2000002e8 snapshot.200000236 currentEpoch log.2000001e2 log.2000001f8 log.20000020e log.200000224 log.20000023a log.200000250 log.200000266 log.20000027c log.200000292 log.2000002a8 log.2000002be log.2000002d4 log.2000002ea snapshot.2000002b3 log.100000001 log.2000001e4 log.2000001fa log.200000210 log.200000226 log.20000023c log.200000252 log.200000268 log.20000027e log.200000294 log.2000002aa log.2000002c0 log.2000002d6 […]

    Značky: #Linux, #DevOps, #ZooKeeper

    • Sl chevron_right

      How to debug ZooKeeper SSL connection issues

      pubsub.slavino.sk / sleeplessbestie · Wednesday, 16 March, 2022 - 12:00 edit

    Debug ZooKeeper SSL connection issues. Ensure that -Djavax.net.debug=ssl option is defined. $ sudo -u zookeeper -i cat zookeeper/conf/zookeeper-env.sh JVMFLAGS="-Djavax.net.debug=ssl" Restart service. $ sudo systemctl restart zookeeper.service Inspect log file to identify an issue. $ tail -f /opt/zookeeper/zookeeper/logs/zookeeper-zookeeper-server-zookeeper1.example.org.out Expired certificate. [...] 2040-01-01 00:00:32,390 [myid:1] - ERROR [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CertificateVerifier@457] - Unsuccessful handshake with session 0x0 javax.net.ssl|WARNING|2D|nioEventLoopGroup-4-1|2040-01-01 24:00:32.391 UTC|SSLEngineOutputRecord.java:168|outbound […]

    Značky: #DevOps, #Linux, #ZooKeeper

    • Sl chevron_right

      How to remove old ZooKeeper snapshots and transaction logs

      pubsub.slavino.sk / sleeplessbestie · Monday, 7 March, 2022 - 13:00 edit

    How to remove old ZooKeeper snapshots and transaction logs. Inspect ZooKeeper snapshots and transaction logs. $ ls data/version-2/ acceptedEpoch log.00000001 log.00000003 log.100000001 log.80000000c snapshot.0 snapshot.500000000 snapshot.700000000 snapshot.900000000 currentEpoch log.00000002 log.00000004 log.800000001 log.c00000001 snapshot.100000000 snapshot.600000000 snapshot.800000000 version-2 Use built-in shell script to keep last 3 snapshots/transaction logs. $ bin/zkCleanup.sh -n 3 /usr/bin/java 2021-07-30 22:26:54,732 [myid:] - […]

    Značky: #Linux, #ZooKeeper, #DevOps

    • Sl chevron_right

      How to secure the ZooKeeper client communication

      pubsub.slavino.sk / sleeplessbestie · Wednesday, 2 March, 2022 - 12:00 edit

    Secure the ZooKeeper client communication. Certification authority Generate certification authority for testing purposes. Create a simplified CA configuration. $ cat <<EOF | tee ca.config # OpenSSL configuration file for certification authority. [ req ] prompt = no distinguished_name = req_distinguished_name x509_extensions = v3_ca [ req_distinguished_name ] O = ZooKeeper OU = Backend emailAddress = admin@example.org […]

    Značky: #ZooKeeper, #DevOps, #Linux

    • Sl chevron_right

      How to secure the ZooKeeper quorum communication

      pubsub.slavino.sk / sleeplessbestie · Monday, 28 February, 2022 - 12:00 edit

    Secure the ZooKeeper quorum communication using SSL certificates. Certification authority Generate certification authority for testing purposes. Create simplified CA configuration. $ cat <<EOF | tee ca.config # OpenSSL configuration file for certification authority. [ req ] prompt = no distinguished_name = req_distinguished_name x509_extensions = v3_ca [ req_distinguished_name ] O = ZooKeeper OU = Backend emailAddress […]

    Značky: #Linux, #ZooKeeper, #DevOps

    • Sl chevron_right

      How to use PKCS #12 as ZooKeeper truststore

      pubsub.slavino.sk / sleeplessbestie · Monday, 7 February, 2022 - 12:00 edit

    PKCS #12 format works fine when used to provide ZooKeeper keystore, but additional steps are needed to use it for a dedicated truststore. You will get the following error when you try to use PKCS #12 truststore. 2021-08-08 14:06:48,947 [myid:3] - WARN [QuorumConnectionThread-[myid=3]-86:QuorumCnxManager@397] - Cannot open secure channel to 2 at election address zookeeper2.example.org/172.16.0.112:3888 org.apache.zookeeper.common.X509Exception$SSLContextException: […]

    Značky: #Linux, #ZooKeeper, #DevOps

    • Sl chevron_right

      How to provide additional JVM options to ZooKeeper instance

      pubsub.slavino.sk / sleeplessbestie · Monday, 11 October, 2021 - 06:00 edit

    Provide additional JVM options to the ZooKeeper instance. Assuming that configuration is located in /opt/zookeeper/zookeeper/conf directory. $ /opt/zookeeper/zookeeper/bin/zkServer.sh --config /opt/zookeeper/zookeeper/conf start Create zookeeper-env.sh shell script. $ sudo -u zookeeper cat /opt/zookeeper/zookeeper/conf/zookeeper-env.sh # additional JVM FLAGS JVMFLAGS="-Dzookeeper.tcpKeepAlive=true" # HEAP SIZE (M) ZK_SERVER_HEAP="512" Display ZooKeeper command. $ sudo -u zookeeper /opt/zookeeper/zookeeper/bin/zkServer.sh --config /opt/zookeeper/zookeeper/conf print-cmd /usr/bin/java ZooKeeper JMX […]

    Značky: #Linux, #DevOps, #ZooKeeper

    • Sl chevron_right

      How to display ZooKeeper startup options

      pubsub.slavino.sk / sleeplessbestie · Friday, 8 October, 2021 - 11:00 edit · 2 minutes

    Display ZooKeeper startup options. Use print-cmd action on zkServer.sh control script to display startup parameters. $ sudo -u zookeeper /opt/zookeeper/zookeeper/bin/zkServer.sh print-cmd /usr/bin/java ZooKeeper JMX enabled by default Using config: /opt/zookeeper/zookeeper/bin/../conf/zoo.cfg "java" -Dzookeeper.log.dir="/opt/zookeeper/zookeeper/bin/../logs" -Dzookeeper.log.file="zookeeper-zookeeper-server-zookeeper2.log" -Dzookeeper.root.logger="INFO,CONSOLE" -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p' -cp "/opt/zookeeper/zookeeper/bin/../zookeeper-server/target/classes:/opt/zookeeper/zookeeper/bin/../build/classes:/opt/zookeeper/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.7.0.jar:/opt/zookeeper/zookeeper/bin/../lib/zookeeper-jute-3.7.0.jar:/opt/zookeeper/zookeeper/bin/../lib/zookeeper-3.7.0.jar:/opt/zookeeper/zookeeper/bin/../lib/snappy-java-1.1.7.7.jar:/opt/zookeeper/zookeeper/bin/../lib/slf4j-log4j12-1.7.30.jar:/opt/zookeeper/zookeeper/bin/../lib/slf4j-api-1.7.30.jar:/opt/zookeeper/zookeeper/bin/../lib/simpleclient_servlet-0.9.0.jar:/opt/zookeeper/zookeeper/bin/../lib/simpleclient_hotspot-0.9.0.jar:/opt/zookeeper/zookeeper/bin/../lib/simpleclient_common-0.9.0.jar:/opt/zookeeper/zookeeper/bin/../lib/simpleclient-0.9.0.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-transport-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-resolver-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-handler-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-common-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-codec-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/netty-buffer-4.1.59.Final.jar:/opt/zookeeper/zookeeper/bin/../lib/metrics-core-4.1.12.1.jar:/opt/zookeeper/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-util-ajax-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-util-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-servlet-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-server-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-security-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-io-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/jetty-http-9.4.38.v20210224.jar:/opt/zookeeper/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper/zookeeper/bin/../lib/jackson-databind-2.10.5.1.jar:/opt/zookeeper/zookeeper/bin/../lib/jackson-core-2.10.5.jar:/opt/zookeeper/zookeeper/bin/../lib/jackson-annotations-2.10.5.jar:/opt/zookeeper/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/zookeeper/zookeeper/bin/../lib/audience-annotations-0.12.0.jar:/opt/zookeeper/zookeeper/bin/../zookeeper-*.jar:/opt/zookeeper/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper/zookeeper/bin/../conf:" -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain "/opt/zookeeper/zookeeper/bin/../conf/zoo.cfg" > "/opt/zookeeper/zookeeper/bin/../logs/zookeeper-zookeeper-server-zookeeper2.out" 2>&1 < /dev/null

    Značky: #Linux, #ZooKeeper, #DevOps

    • Sl chevron_right

      How to execute ZooKeeper commands using command-line interface

      pubsub.slavino.sk / sleeplessbestie · Wednesday, 6 October, 2021 - 06:10 edit

    Execute ZooKeeper commands using command-line interface. By default zkCli.sh is very verbose. $ sudo -u zookeeper /opt/zookeeper/zookeeper/bin/zkCli.sh [16/1996] /usr/bin/java Connecting to localhost:2181 2021-06-11 21:37:39,885 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC 2021-06-11 21:37:39,887 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zookeeper3 2021-06-11 21:37:39,887 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.11 2021-06-11 […]

    Značky: #Linux, #ZooKeeper, #DevOps