Tech ramblings by Marcin

Operational problems with Zookeeper

2013-03-21 00:00

This post is a summary of what has been presented by Kathleen Ting on StrangeLoop conference. You can watch the original here:

I’ve decided to put this selection here for quick reference.

Connection mismanagement

  • too many connections

      WARN [NIOServerCxn.Factory:$Factory@247] - Too many connections from / - max is 60
  • running out of ZK connections?
    • set maxClientCnxns=200 in zoo.cfg
  • HBase client leaking connections?
    • fixed in HBASE-3777, HBASE-4773, HBASE-5466
    • manually close connections
  • connection closes prematurely

      ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately.
  • in hbase-site.xml set hbase.zookeeper.recoverable.waittime=30000ms
  • pig hangs connecting to HBase

      WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect Connection refused!

    CAUSE: location of ZK quorum is not known to Pig

    • use Pig 10, which includes PIG-2115
    • if there is an overlap between TaskTrackers and ZK quorum nodes
      • set hbase.zookeeper.quorum to final in hbase-site.xml
      • otherwise add hbaze.zoopeeker.quorum=hadoophbasemaster.lan:2181 in

Time mismanagement

  • client session timed out

      INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session <id>, timeout of 40000ms exceeded
    • ZK and HBase need the same session timeout values
      • zoo.cfg: maxSession=Timeout=180000
      • hbase-site.xml: zookeeper.session.timeout=180000
    • don’t co-locate ZK with IO-intense DataNode or RegionServer
    • specify right amount of heap and tune GC flags
      • turn on parallel/CMS/incremental GC
  • clients lose connections

      WARN org.apache.zookeeper.ClientCnxn - Session <id> for server <name>, unexpected error, closing socket connection and attempting reconnect Broken pipe
    • don’t use SSD drive for ZK transaction log

Disk management

  • unable to load database - unable to run quorum server

      FATAL Unable to load database on disk ! Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for <file> at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(!
    • archive and wipe /var/zookeeper/version-2 if other two ZK servers are running
  • unable to load database - unreasonable length exception

      FATAL Unable to load database on disk Unreasonable length = 1048583 at org.apache.jute.BinaryInputArchive.readBuffer(
    • server allows a client to set data larger than the server can read from disk
    • if a znode is not readable, increase jute.maxbuffer
      • look for "Packet len <xx> is out of range" in the client log
      • increase it by 20%
      • set in JVMFLAGS="-Djute.maxbuffer=yy" bin/
      • fixed in ZOOKEEPER-1513
  • failure to follow leader

      WARN org.apache.zookeeper.server.quorum.Learner: Exception when following the leader Read timed out 


    • disk IO contention, network issues
    • ZK snapshot is too large (lots of ZK nodes)


    • reduce IO contention by putting dataDir on dedicated spindle
    • increase initLimit on all ZK servers and restart, see ZOOKEEPER-1521
    • monitor network

Best Practices


  • separate spindles for dataDir & dataLogDir
  • allocate 3 or 5 ZK servers
  • tune garbage collection
  • run script via cron


  • dont’ co-locate ZK with I/O intense DataNode or RegionServer
  • don’t use SSD drive for ZK transaction log

You may use Zookeeper as an observer - a non-voting member:

  • in zoo.cfg