2012-06-07 17:09:24,457 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode events.namenode80 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-cdh4.0.0--1, built on 06/04/2012 23:16 GMT 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop-106 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-sun-1.6.0.32.x86_64/jre 2012-06-07 17:09:24,661 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/var/run/cloudera-scm-agent/process/98-hdfs-FAILOVERCONTROLLER:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/json-simple-1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0.jar:/usr/share/cmf/lib/plugins/event-publish-4.0.1-shaded.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/lib/jline-0.9.94.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/avro-1.5.4.jar:/usr/lib/hadoop-yarn/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/junit-4.8.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.15.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.5.4.jar:/usr/lib/hadoop-mapreduce/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.15.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.0.0-cdh4.0.0.jar 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler= 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-220.el6.x86_64 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfs 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/usr/lib/hadoop 2012-06-07 17:09:24,662 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/98-hdfs-FAILOVERCONTROLLER 2012-06-07 17:09:24,664 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-07 17:09:24,704 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-07 17:09:24,717 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-07 17:09:24,717 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-07 17:09:24,730 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-07 17:09:24,748 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730000, negotiated timeout = 5000 2012-06-07 17:09:24,786 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-07 17:09:24,801 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019 2012-06-07 17:09:24,851 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-06-07 17:09:24,853 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting 2012-06-07 17:09:24,887 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[FAILOVERCONTROLLER], CATEGORY=[LOG_MESSAGE], ROLE=[hdfs1-FAILOVERCONTROLLER-10], SEVERITY=[IMPORTANT], SERVICE=[hdfs1], SERVICE_TYPE=[HDFS], LOG_LEVEL=[WARN], HOSTS=[hadoop-106], EVENTCODE=[EV_LOG_EVENT]}, content=SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration., timestamp=1339081764717} java.io.IOException: Error connecting to hadoop-108/10.196.68.150:7184 at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:198) at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:133) at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:122) at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:182) at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:221) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) at com.cloudera.cmf.event.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at com.cloudera.cmf.event.shaded.org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) ... 3 more 2012-06-07 17:09:24,983 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-07 17:09:24,983 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-07 17:09:25,008 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-07 17:09:25,017 INFO org.apache.hadoop.ha.ActiveStandbyElector: No old node to fence 2012-06-07 17:09:25,017 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-07 17:09:25,023 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-07 17:09:25,797 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 12:36:20,488 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:36:20,492 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:36:20,492 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:36:20,492 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:36:20,501 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:36:20,501 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:36:20,646 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730000 closed 2012-06-13 12:36:20,647 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730000 2012-06-13 12:36:20,647 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:36:22,657 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:22,660 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:24,661 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:24,663 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:26,664 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:26,666 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:28,667 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:28,669 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:30,670 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:30,672 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:32,675 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:32,677 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:34,678 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:34,680 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:36,681 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:36,683 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:38,685 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:38,687 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:40,688 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:40,690 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:42,691 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:42,693 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:44,694 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:44,696 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:46,698 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:46,700 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:48,702 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:48,704 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:50,705 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:50,707 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:52,709 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:52,711 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:54,712 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:54,714 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:56,715 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:56,717 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:36:58,719 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:36:58,720 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:00,722 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:00,724 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:02,725 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:02,727 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:04,729 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:04,731 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:06,732 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:06,734 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:08,736 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:08,738 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:10,739 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:10,741 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:12,742 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:12,744 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:14,745 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:14,747 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:16,748 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:16,750 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:18,755 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:18,757 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:20,759 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:20,760 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:22,761 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:22,763 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:24,764 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:24,766 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:26,767 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:26,769 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:28,770 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:28,771 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:30,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:30,774 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:32,775 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:32,777 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:34,779 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:34,780 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:36,782 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:36,784 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:38,785 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:38,786 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:40,788 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:40,789 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:42,790 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:42,792 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:44,793 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:44,795 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:46,797 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:46,799 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:48,800 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:48,802 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:50,803 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:50,805 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:52,807 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:52,809 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:54,811 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:54,813 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:56,814 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:56,816 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:37:58,817 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:37:58,819 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:00,821 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:00,823 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:02,824 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:02,827 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:04,829 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:04,831 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:06,832 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:06,834 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:08,835 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:08,837 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:10,838 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:10,840 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:12,841 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:12,843 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:14,843 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:14,845 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:16,846 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:16,847 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:18,848 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:18,850 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:20,851 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:20,853 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:22,854 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:22,856 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:24,857 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:24,859 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:26,860 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:26,862 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:28,863 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:28,865 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:30,866 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:30,867 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:32,868 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:32,870 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:34,871 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:34,873 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:36,874 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:36,876 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:38,877 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:38,879 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:40,880 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:40,882 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:42,883 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:42,884 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:44,885 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:44,887 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:46,888 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:46,890 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:48,891 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:48,893 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:50,894 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:50,896 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:52,897 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:52,899 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:54,900 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:54,902 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:56,903 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:56,904 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:38:58,905 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:38:58,907 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:00,908 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:00,910 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:02,911 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:02,913 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:04,914 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:04,916 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:06,917 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:06,919 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:08,921 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:08,923 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:10,924 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:10,926 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:12,927 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:12,929 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:14,930 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:14,931 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:16,933 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:16,935 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:18,936 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:18,938 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:20,939 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:20,940 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:22,941 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:22,943 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:24,944 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:24,946 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:26,947 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:26,949 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:28,950 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:28,951 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:30,952 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:30,954 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:32,955 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:32,957 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:34,958 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:34,960 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:36,961 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:36,963 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:38,964 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:38,965 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:40,967 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:40,968 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:42,969 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:42,971 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:44,972 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:44,974 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:46,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:46,978 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:48,979 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:48,981 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:50,982 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:50,984 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:52,985 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:52,987 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:54,989 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:54,997 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:56,998 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:57,000 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:39:59,001 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:39:59,002 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:01,004 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:01,006 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:03,007 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:03,009 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:05,010 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:05,012 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:07,013 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:07,015 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:09,016 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:09,018 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:11,019 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:11,020 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:13,022 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:13,024 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:15,025 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:15,027 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:17,028 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:17,029 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:19,031 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:19,032 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:21,033 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:21,035 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:23,036 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:23,038 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:25,039 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:25,041 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:27,042 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:27,043 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:29,044 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:29,046 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:31,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:31,048 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:33,049 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:33,051 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:35,052 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:35,054 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:37,055 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:37,057 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:39,058 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:39,060 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:41,061 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:41,062 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:43,063 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:43,065 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:45,066 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:45,068 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:47,069 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:47,070 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:49,071 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:49,073 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:51,074 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:51,077 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:53,077 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:53,079 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:55,080 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:55,082 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:57,083 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:57,084 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:40:59,085 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:40:59,087 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:01,088 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:01,090 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:03,091 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:03,093 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:05,094 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:05,096 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:07,097 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:07,098 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:09,099 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:09,101 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:11,102 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:11,104 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:13,105 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:13,106 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:15,107 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:15,109 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:17,110 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:17,112 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:19,113 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:19,115 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:21,116 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:21,118 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:23,119 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:23,120 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:25,121 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:25,123 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:27,124 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:27,126 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:29,127 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:29,129 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:31,130 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:31,132 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:33,133 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:33,135 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:35,136 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:35,138 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:37,139 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:37,141 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:39,142 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:39,144 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:41,145 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:41,146 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:43,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:43,149 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:45,152 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:45,153 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:47,154 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:47,156 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:49,157 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:49,159 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:51,160 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:51,162 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:53,162 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:53,164 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:55,165 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:55,166 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:57,167 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:57,169 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:41:59,170 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:41:59,171 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:01,174 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:03,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:03,177 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:05,178 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:05,180 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:07,182 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:07,183 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:09,185 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:09,188 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:11,189 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:11,191 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:13,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:13,194 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:15,195 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:15,196 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:17,197 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:17,198 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:19,199 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:19,201 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:21,203 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:21,205 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:23,206 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:23,207 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:25,208 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:25,210 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:27,211 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:27,213 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:29,214 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:29,215 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:31,216 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:31,218 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:33,219 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:33,220 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:35,222 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:35,223 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:37,225 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:37,226 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:39,227 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:39,229 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:41,230 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:41,231 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:43,233 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:43,234 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:45,236 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:45,237 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:47,238 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:47,240 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:49,241 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:49,243 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:51,244 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:51,246 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:53,247 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:53,249 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:55,250 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:55,251 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:57,252 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:57,254 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:42:59,255 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:42:59,257 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:01,258 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:01,259 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:03,261 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:03,262 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:05,263 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:05,265 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:07,266 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:07,268 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:09,269 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:09,271 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:11,274 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:11,888 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:43:11,888 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:43:11,893 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:43:11,895 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:43:11,896 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:43:11,896 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:43:11,896 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:43:11,899 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730002, negotiated timeout = 5000 2012-06-13 12:43:11,902 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:43:11,919 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:43:11,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:43:11,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:43:11,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:43:11,967 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:43:13,262 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:43:13,262 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:43:13,265 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:43:13,265 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:43:13,265 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:43:13,265 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:43:13,266 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:43:13,267 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:43:13,272 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730002 closed 2012-06-13 12:43:14,274 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:43:14,275 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:43:14,276 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:43:14,277 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:43:14,277 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:43:14,279 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:43:14,279 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:43:14,279 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:43:14,280 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730003, negotiated timeout = 5000 2012-06-13 12:43:14,284 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730003 closed 2012-06-13 12:43:14,284 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730003 2012-06-13 12:43:14,284 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730003 2012-06-13 12:43:14,284 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:43:16,285 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:16,287 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:18,289 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:18,291 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:20,292 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:20,295 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:22,298 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:22,303 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:24,305 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:24,307 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:26,308 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:26,310 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:28,312 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:28,317 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:30,318 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:30,320 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:32,321 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:32,323 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:34,324 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:34,326 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:36,327 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:36,329 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:38,330 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:38,332 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:40,333 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:40,334 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:42,335 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:42,337 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:44,338 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:44,340 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:46,341 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:46,343 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:48,344 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:48,346 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:50,347 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:50,348 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:52,349 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:52,351 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:54,352 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:54,354 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:56,355 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:56,357 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:43:58,358 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:43:58,360 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:00,361 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:00,362 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:02,363 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:02,365 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:04,366 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:04,368 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:06,369 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:06,371 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:08,372 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:08,374 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:10,375 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:10,377 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:12,377 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:12,379 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:14,380 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:14,382 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:16,383 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:16,386 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:18,387 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:18,389 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:20,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:20,392 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:22,392 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:22,394 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:24,395 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:24,397 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:26,398 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:26,399 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:28,400 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:28,402 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:30,404 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:30,405 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:32,407 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:32,408 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:34,409 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:34,411 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:36,413 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:36,415 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:38,416 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:38,417 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:40,419 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:40,420 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:42,421 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:42,423 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:44,424 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:44,425 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:46,426 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:46,428 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:48,429 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:48,431 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:50,432 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:50,434 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:52,435 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:52,437 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:54,438 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:54,439 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:56,440 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:56,442 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:44:58,443 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:44:58,445 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:00,446 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:00,448 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:02,449 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:02,451 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:04,452 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:04,454 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:06,455 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:06,457 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:08,457 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:08,459 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:10,461 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:10,462 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:12,463 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:12,466 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:14,467 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:14,469 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:16,470 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:16,471 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:18,472 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:18,474 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:20,475 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:20,477 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:22,478 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:22,480 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:24,481 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:24,483 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:26,484 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:26,486 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:28,487 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:28,489 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:30,490 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:30,492 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:32,493 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:32,495 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:34,496 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:34,498 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:36,499 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:36,501 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:38,502 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:38,504 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:40,505 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:40,507 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:42,508 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:42,510 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:44,511 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:44,513 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:46,514 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:46,516 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:47,696 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:45:47,697 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:45:47,703 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:45:47,705 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:45:47,709 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:45:47,709 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:45:47,714 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:45:47,736 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560001, negotiated timeout = 5000 2012-06-13 12:45:47,740 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:45:47,744 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:45:47,781 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:45:47,782 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:45:47,782 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:45:47,798 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:45:48,609 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:48,612 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:48,613 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:48,614 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:45:48,618 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560001 closed 2012-06-13 12:45:48,708 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:48,711 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:45:48,711 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:45:48,711 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:45:49,620 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:45:49,621 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:45:49,622 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:45:49,623 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:45:49,623 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:45:49,623 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:45:49,623 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:45:49,624 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:45:49,627 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730004, negotiated timeout = 5000 2012-06-13 12:45:49,628 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730004 closed 2012-06-13 12:45:49,628 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730004 2012-06-13 12:45:49,628 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730004 2012-06-13 12:45:49,629 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:45:51,629 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:51,632 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:53,633 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:53,979 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:45:53,979 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:45:53,982 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:45:53,982 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:45:53,983 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:45:53,983 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:45:53,985 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:45:53,987 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730005, negotiated timeout = 5000 2012-06-13 12:45:53,989 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:45:53,990 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:45:53,992 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:45:53,992 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:45:53,992 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:45:54,000 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:45:54,745 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:54,747 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:54,749 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:54,749 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:45:54,752 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730005 closed 2012-06-13 12:45:54,984 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:45:54,986 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:45:54,986 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:45:54,986 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:45:55,754 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:45:55,755 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:45:55,756 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:45:55,756 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:45:55,756 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:45:55,757 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:45:55,757 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:45:55,757 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:45:55,759 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560002, negotiated timeout = 5000 2012-06-13 12:45:55,761 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560002 closed 2012-06-13 12:45:55,762 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560002 2012-06-13 12:45:55,762 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560002 2012-06-13 12:45:55,762 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:45:57,762 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:57,764 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:45:59,769 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:45:59,772 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:01,199 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:46:01,199 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:46:01,202 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:46:01,203 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-06-13 12:46:01,205 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:46:01,205 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:46:01,210 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-06-13 12:46:01,254 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x237c76a78a70001, negotiated timeout = 5000 2012-06-13 12:46:01,255 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:46:01,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:46:01,302 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:46:01,302 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:46:01,302 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:46:01,306 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:46:02,029 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:02,031 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:02,033 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:02,033 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:46:02,037 INFO org.apache.zookeeper.ZooKeeper: Session: 0x237c76a78a70001 closed 2012-06-13 12:46:02,205 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:02,207 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:46:02,207 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:46:02,207 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:46:03,039 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:46:03,040 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:46:03,041 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:46:03,041 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:46:03,041 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:46:03,041 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:46:03,041 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:46:03,041 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:46:03,045 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560003, negotiated timeout = 5000 2012-06-13 12:46:03,046 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560003 closed 2012-06-13 12:46:03,047 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560003 2012-06-13 12:46:03,047 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560003 2012-06-13 12:46:03,047 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:46:05,050 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:05,052 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:07,053 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:07,058 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:08,377 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:46:08,377 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:46:08,379 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:46:08,380 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:46:08,381 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:46:08,381 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:46:08,382 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:46:08,385 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730006, negotiated timeout = 5000 2012-06-13 12:46:08,386 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:46:08,388 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:46:08,390 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:46:08,390 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:46:08,390 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:46:08,393 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:46:09,142 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:09,145 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:09,146 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:09,146 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:46:09,150 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730006 closed 2012-06-13 12:46:09,381 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:46:09,383 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:46:09,383 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:46:09,383 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:46:10,152 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:46:10,152 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-06-13 12:46:10,153 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:46:10,153 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:46:10,154 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:46:10,154 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:46:10,154 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-06-13 12:46:10,154 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:46:10,158 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x237c76a78a70002, negotiated timeout = 5000 2012-06-13 12:46:10,165 INFO org.apache.zookeeper.ZooKeeper: Session: 0x237c76a78a70002 closed 2012-06-13 12:46:10,165 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x237c76a78a70002 2012-06-13 12:46:10,165 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x237c76a78a70002 2012-06-13 12:46:10,165 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:46:12,166 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:12,168 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:14,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:14,171 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:16,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:16,173 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:18,174 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:18,176 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:20,177 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:20,179 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:22,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:22,182 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:24,183 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:24,185 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:26,186 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:26,187 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:28,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:28,190 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:30,191 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:30,193 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:32,194 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:32,196 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:34,197 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:34,198 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:36,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:36,202 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:38,203 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:38,205 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:40,206 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:40,208 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:42,209 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:42,215 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:44,216 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:44,218 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:46,219 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:46,221 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:48,222 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:48,224 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:50,225 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:50,227 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:52,228 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:52,230 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:54,231 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:54,232 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:56,233 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:56,235 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:46:58,236 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:46:58,238 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:00,239 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:00,240 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:02,241 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:02,243 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:04,244 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:04,246 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:06,247 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:06,249 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:08,250 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:08,251 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:10,252 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:10,254 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:12,255 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:12,257 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:14,258 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:14,260 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:16,261 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:16,263 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:18,264 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:18,265 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:20,266 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:20,268 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:22,269 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:22,271 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:24,272 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:24,273 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:26,274 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:26,276 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:28,277 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:28,279 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:30,280 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:30,282 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:32,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:32,284 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:34,285 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:34,287 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:36,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:36,290 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:38,291 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:38,293 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:40,294 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:40,296 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:42,298 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:42,299 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:44,301 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:44,302 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:46,303 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:46,305 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:48,306 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:48,308 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:50,309 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:50,311 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:52,312 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:52,313 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:54,314 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:54,317 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:56,318 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:47:56,322 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:47:57,435 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:47:57,435 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:47:57,437 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:47:57,439 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-06-13 12:47:57,440 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:47:57,440 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:47:57,441 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-06-13 12:47:57,447 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x237c76a78a70003, negotiated timeout = 5000 2012-06-13 12:47:57,448 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:47:57,449 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:47:57,451 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:47:57,451 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:47:57,452 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:47:57,454 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:47:58,189 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:47:58,192 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:47:58,193 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:47:58,193 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:47:58,197 INFO org.apache.zookeeper.ZooKeeper: Session: 0x237c76a78a70003 closed 2012-06-13 12:47:58,440 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:47:58,442 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:47:58,442 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:47:58,442 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:47:59,198 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:47:59,199 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:47:59,200 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:47:59,200 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:47:59,200 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:47:59,200 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:47:59,201 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:47:59,201 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:47:59,204 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560004, negotiated timeout = 5000 2012-06-13 12:47:59,206 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560004 closed 2012-06-13 12:47:59,206 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560004 2012-06-13 12:47:59,206 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560004 2012-06-13 12:47:59,207 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:01,207 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:01,209 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:03,211 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:03,576 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:48:03,576 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:48:03,578 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:03,579 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:48:03,580 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:03,580 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:03,581 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:48:03,583 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730007, negotiated timeout = 5000 2012-06-13 12:48:03,584 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:48:03,586 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:48:03,588 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:48:03,588 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:48:03,588 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:48:03,590 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:48:04,300 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:04,303 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:04,304 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:04,306 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:48:04,309 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730007 closed 2012-06-13 12:48:04,580 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:04,582 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:48:04,582 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:48:04,582 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:48:05,311 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:05,311 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:48:05,312 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:05,313 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:05,313 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:05,313 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:48:05,313 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:48:05,314 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:48:05,316 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a75730008, negotiated timeout = 5000 2012-06-13 12:48:05,318 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a75730008 closed 2012-06-13 12:48:05,318 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730008 2012-06-13 12:48:05,318 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a75730008 2012-06-13 12:48:05,318 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:07,319 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:07,321 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:09,324 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:09,330 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:10,763 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:48:10,763 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:48:10,765 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:10,766 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:48:10,768 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:10,768 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:10,770 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:48:10,781 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560005, negotiated timeout = 5000 2012-06-13 12:48:10,782 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:48:10,784 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:48:10,786 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:48:10,786 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:48:10,786 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:48:10,789 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:48:11,531 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:11,533 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:11,534 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:11,535 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:48:11,538 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560005 closed 2012-06-13 12:48:11,767 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:11,769 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:48:11,770 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:48:11,770 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:48:12,540 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:12,540 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-06-13 12:48:12,541 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:12,541 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:48:12,543 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:12,543 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:12,543 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:48:12,543 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-06-13 12:48:12,547 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x237c76a78a70004, negotiated timeout = 5000 2012-06-13 12:48:12,558 INFO org.apache.zookeeper.ZooKeeper: Session: 0x237c76a78a70004 closed 2012-06-13 12:48:12,559 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x237c76a78a70004 2012-06-13 12:48:12,559 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x237c76a78a70004 2012-06-13 12:48:12,559 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:14,559 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:14,561 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:16,563 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:16,565 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:18,569 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:18,924 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:48:18,924 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:48:18,926 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:18,927 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:48:18,928 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:18,928 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:18,929 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:48:18,932 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560006, negotiated timeout = 5000 2012-06-13 12:48:18,934 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:48:18,936 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:48:18,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:48:18,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:48:18,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:48:18,941 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:48:19,683 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:19,686 FATAL org.apache.hadoop.ha.ZKFailoverController: Couldn't make NameNode at hadoop-106/10.196.68.149:8020 active java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:19,687 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election org.apache.hadoop.ha.ServiceFailedException: Couldn't transition to active at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:398) at org.apache.hadoop.ha.ZKFailoverController.access$900(ZKFailoverController.java:59) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.becomeActive(ZKFailoverController.java:819) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:764) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1165) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy11.transitionToActive(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToActive(HAServiceProtocolClientSideTranslatorPB.java:100) at org.apache.hadoop.ha.HAServiceProtocolHelper.transitionToActive(HAServiceProtocolHelper.java:48) at org.apache.hadoop.ha.ZKFailoverController.becomeActive(ZKFailoverController.java:380) ... 6 more Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:788) Caused by: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:19,687 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-06-13 12:48:19,691 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560006 closed 2012-06-13 12:48:19,928 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:48:19,930 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:48:19,930 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:48:19,930 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:48:20,692 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:48:20,693 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 12:48:20,693 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:20,693 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:48:20,694 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:48:20,694 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:48:20,694 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:48:20,695 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 12:48:20,700 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560007, negotiated timeout = 5000 2012-06-13 12:48:20,702 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560007 closed 2012-06-13 12:48:20,702 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560007 2012-06-13 12:48:20,703 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a75560007 2012-06-13 12:48:20,703 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:48:22,703 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:22,705 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:24,706 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:24,708 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:26,709 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:26,711 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:28,712 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:28,713 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:30,714 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:30,716 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:32,717 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:32,719 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:34,720 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:34,721 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:36,723 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:36,724 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:38,725 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:38,727 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:40,728 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:40,730 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:42,731 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:42,741 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:44,742 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:44,744 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:46,745 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:46,747 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:48,748 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:48,750 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:50,751 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:50,753 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:52,754 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:52,756 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:54,757 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:54,758 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:56,759 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:56,761 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:48:58,762 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:48:58,763 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:00,764 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:00,766 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:02,768 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:02,769 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:04,771 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:04,772 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:06,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:06,775 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:08,776 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:08,778 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:10,779 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:10,780 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:12,782 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:12,783 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:14,785 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:14,786 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:16,787 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:16,789 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:18,791 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:18,792 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:20,793 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:20,795 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:22,796 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:22,797 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:24,799 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:24,800 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:26,802 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:26,803 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:28,804 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:28,806 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:30,807 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:30,809 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:32,810 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:32,811 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:34,812 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:34,814 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:36,815 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:36,817 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:38,820 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:38,822 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:40,823 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:40,825 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:42,826 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:42,836 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:44,837 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:44,839 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:46,840 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:46,842 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:48,844 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:48,845 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:50,847 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:50,849 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:52,850 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:52,852 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:54,853 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:54,854 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:56,855 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:56,857 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:49:58,858 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:49:58,859 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:50:00,860 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:50:00,861 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:50:02,862 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:50:02,864 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:50:04,866 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:50:04,872 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:50:06,874 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:50:07,206 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:50:07,206 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:50:07,208 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:50:07,209 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-06-13 12:50:07,210 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:50:07,210 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:50:07,211 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-06-13 12:50:07,214 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x237c76a78a70005, negotiated timeout = 5000 2012-06-13 12:50:07,215 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:50:07,216 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:50:07,218 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:50:07,218 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:50:07,218 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:50:07,220 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:50:08,121 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 12:58:55,007 WARN org.apache.hadoop.ipc.Client: Unexpected error reading responses on connection Thread[IPC Client (884096274) connection to hadoop-106/10.196.68.149:8020 from hdfs,5,main] java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781) 2012-06-13 12:58:55,010 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "hadoop-106/10.196.68.149"; destination host is: "hadoop-106":8020; 2012-06-13 12:58:55,010 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 12:58:55,010 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 12:58:55,010 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 12:58:55,010 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 12:58:55,029 INFO org.apache.zookeeper.ZooKeeper: Session: 0x237c76a78a70005 closed 2012-06-13 12:58:55,029 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x237c76a78a70005 2012-06-13 12:58:55,029 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 12:58:57,030 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:58:57,032 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:58:59,033 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:58:59,034 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:01,035 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:01,037 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:03,039 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:03,040 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:05,041 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:05,043 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:07,044 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:07,046 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:09,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:09,048 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:11,049 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:11,051 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:13,052 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:13,054 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:15,055 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:15,057 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:17,058 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:17,060 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:19,060 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:19,062 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:21,063 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:21,065 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:23,066 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:23,067 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:25,068 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:25,070 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:27,071 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:27,073 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:29,074 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:29,076 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:31,077 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:31,078 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:33,079 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:33,081 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:35,083 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:35,084 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:37,086 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:37,089 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 12:59:39,091 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 12:59:39,403 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 12:59:39,403 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 12:59:39,405 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 12:59:39,406 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 12:59:39,407 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 12:59:39,407 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 12:59:39,408 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 12:59:39,411 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a7573000a, negotiated timeout = 5000 2012-06-13 12:59:39,411 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 12:59:39,413 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 12:59:39,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 12:59:39,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 12:59:39,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 12:59:39,423 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 12:59:40,405 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 13:04:13,825 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Allowed RPC access from hdfs (auth:SIMPLE) at 10.196.68.150 2012-06-13 13:04:13,827 INFO org.apache.hadoop.ha.ZKFailoverController: Requested by hdfs (auth:SIMPLE) at 10.196.68.150 to cede active role. 2012-06-13 13:04:13,849 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully ensured local node is in standby mode 2012-06-13 13:04:13,849 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 13:04:13,850 INFO org.apache.hadoop.ha.ActiveStandbyElector: Deleting bread-crumb of active node... 2012-06-13 13:04:13,880 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337c76a7573000a closed 2012-06-13 13:04:13,881 INFO org.apache.hadoop.ha.ZKFailoverController: Would have joined master election, but this node is prohibited from doing so for 9968 more ms 2012-06-13 13:04:13,883 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a7573000a 2012-06-13 13:04:13,883 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 13:04:15,237 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Allowed RPC access from hdfs (auth:SIMPLE) at 10.196.68.150 2012-06-13 13:04:15,239 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 13:04:15,242 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-06-13 13:04:15,243 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 13:04:15,244 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 13:04:15,244 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-06-13 13:04:15,246 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a7573000c, negotiated timeout = 5000 2012-06-13 13:04:15,248 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 13:04:15,250 INFO org.apache.hadoop.ha.ZKFailoverController: ZK Election indicated that NameNode at hadoop-106/10.196.68.149:8020 should become standby 2012-06-13 13:04:15,252 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to standby state 2012-06-13 13:06:17,299 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Allowed RPC access from hdfs (auth:SIMPLE) at 10.196.68.149 2012-06-13 13:06:17,303 INFO org.apache.hadoop.ha.ZKFailoverController: Asking NameNode at hadoop-108/10.196.68.150:8020 to cede its active state for 10000ms 2012-06-13 13:06:17,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 13:06:17,346 INFO org.apache.hadoop.ha.ActiveStandbyElector: No old node to fence 2012-06-13 13:06:17,346 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 13:06:17,348 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 13:06:18,289 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 13:06:18,292 INFO org.apache.hadoop.ha.ZKFailoverController: ZK Election indicated that NameNode at hadoop-106/10.196.68.149:8020 should become standby 2012-06-13 13:06:18,303 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully became active. Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 13:06:18,305 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to standby state 2012-06-13 13:06:18,307 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 13:06:18,308 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-06-13 13:06:18,308 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-06-13 13:06:18,308 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 13:06:18,310 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 13:06:18,894 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 13:08:11,130 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode events.namenode80 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-cdh4.0.0--1, built on 06/04/2012 23:16 GMT 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop-106 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-sun-1.6.0.32.x86_64/jre 2012-06-13 13:08:11,560 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/json-simple-1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0.jar:/usr/share/cmf/lib/plugins/event-publish-4.0.1-shaded.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/lib/jline-0.9.94.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/avro-1.5.4.jar:/usr/lib/hadoop-yarn/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/junit-4.8.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.15.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.5.4.jar:/usr/lib/hadoop-mapreduce/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.15.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.0.0-cdh4.0.0.jar 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler= 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-220.el6.x86_64 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfs 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/usr/lib/hadoop 2012-06-13 13:08:11,561 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER 2012-06-13 13:08:11,562 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 13:08:11,622 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 13:08:11,656 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 13:08:11,656 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 13:08:11,661 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 13:08:11,682 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a75560008, negotiated timeout = 5000 2012-06-13 13:08:11,721 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 13:08:11,812 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019 2012-06-13 13:08:11,864 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-06-13 13:08:11,865 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting 2012-06-13 13:08:12,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:12,950 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:12,950 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_NOT_RESPONDING 2012-06-13 13:08:12,950 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_NOT_RESPONDING 2012-06-13 13:08:12,951 INFO org.apache.hadoop.ha.ZKFailoverController: Quitting master election for NameNode at hadoop-106/10.196.68.149:8020 and marking that fencing is necessary 2012-06-13 13:08:12,951 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-06-13 13:08:12,954 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137c76a75560008 closed 2012-06-13 13:08:12,954 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-06-13 13:08:14,957 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:14,959 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:16,961 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:16,962 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:18,964 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:18,966 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:20,967 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:20,969 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:22,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:22,972 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:24,974 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:24,975 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:26,977 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:26,979 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:28,980 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:28,982 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:30,984 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:30,986 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:32,987 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:32,989 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:34,991 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:34,993 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:36,995 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:36,997 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:38,998 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:39,000 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:41,002 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:41,003 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:43,007 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:43,008 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:45,011 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:45,012 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:47,014 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:47,016 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:49,017 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:49,019 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:51,021 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:51,022 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:53,024 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:53,026 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:55,028 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:55,030 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:57,031 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:57,033 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:08:59,034 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:08:59,036 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:01,037 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:01,039 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:03,041 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:03,042 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:05,044 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:05,046 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:07,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:07,049 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:09,050 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:09,052 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:11,054 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:11,055 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:13,056 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:13,058 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:15,060 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:15,062 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:17,063 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:17,065 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:19,066 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:19,068 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:21,069 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:21,071 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:23,072 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:23,074 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:25,075 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:25,077 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:27,078 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:27,080 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:29,081 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:29,084 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:31,085 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:31,086 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:33,088 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:33,090 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:35,091 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:35,093 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:37,094 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:37,096 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:39,097 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:39,099 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:41,101 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:41,103 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:43,104 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:43,106 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:45,107 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:45,109 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:47,111 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:47,112 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:49,114 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:49,116 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:51,117 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:51,119 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:53,121 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:53,123 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:55,124 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:55,126 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:57,127 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:57,129 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:09:59,130 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:09:59,132 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:01,133 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:01,135 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:03,136 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:03,138 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:05,139 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:05,141 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:07,142 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:07,144 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:09,145 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:09,147 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:11,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:11,151 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:13,152 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:13,154 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:15,155 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:15,157 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:17,158 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:17,169 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:19,171 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:19,173 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:21,174 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:21,176 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:23,177 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:23,179 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:25,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:25,182 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:27,183 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:27,184 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:29,186 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:29,187 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:31,189 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:31,190 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:33,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:33,193 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:35,195 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:35,197 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:37,198 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:37,199 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:39,201 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:39,203 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:41,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:41,206 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:43,207 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:43,209 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:45,210 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:45,212 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:47,213 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:47,215 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:49,216 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:49,218 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:51,220 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:51,221 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:53,222 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:53,224 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:55,225 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:55,227 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:57,228 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:57,230 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:10:59,231 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:10:59,233 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:01,234 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:01,236 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:03,238 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:03,239 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:05,240 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:05,242 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:07,243 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:07,245 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:09,246 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:09,248 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:11,249 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:11,251 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:13,252 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:13,254 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:15,256 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:15,258 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:17,259 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-106/10.196.68.149:8020. Already tried 0 time(s). 2012-06-13 13:11:17,262 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at hadoop-106/10.196.68.149:8020: Call From hadoop-106/10.196.68.149 to hadoop-106:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2012-06-13 13:11:18,349 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-06-13 13:11:18,350 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-06-13 13:11:18,358 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-06-13 13:11:18,379 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-06-13 13:11:18,383 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-06-13 13:11:18,383 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-06-13 13:11:18,383 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-06-13 13:11:18,396 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x137c76a7556000a, negotiated timeout = 5000 2012-06-13 13:11:18,398 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-06-13 13:11:18,415 INFO org.apache.hadoop.ha.ZKFailoverController: ZK Election indicated that NameNode at hadoop-106/10.196.68.149:8020 should become standby 2012-06-13 13:11:18,422 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to standby state 2012-06-13 14:48:33,157 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Allowed RPC access from hdfs (auth:SIMPLE) at 10.196.68.149 2012-06-13 14:48:33,181 INFO org.apache.hadoop.ha.ZKFailoverController: Asking NameNode at hadoop-108/10.196.68.150:8020 to cede its active state for 10000ms 2012-06-13 14:48:33,322 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-06-13 14:48:33,340 INFO org.apache.hadoop.ha.ActiveStandbyElector: No old node to fence 2012-06-13 14:48:33,340 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-06-13 14:48:33,345 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-06-13 14:48:34,929 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-06-13 14:48:34,943 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully became active. Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:00:12,025 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x137c76a7556000a, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:00:12,654 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:00:14,758 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:00:14,770 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:00:14,770 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:00:14,771 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2281ms for sessionid 0x137c76a7556000a, closing socket connection and attempting reconnect 2012-07-01 02:00:16,456 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:00:16,473 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:00:16,473 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:00:16,496 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:00:16,573 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x137c76a7556000a has expired, closing socket connection 2012-07-01 02:00:16,616 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:00:16,616 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:00:16,955 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:00:17,009 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:00:17,030 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:00:17,030 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:00:17,040 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:00:17,074 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x337c76a7573000f, negotiated timeout = 5000 2012-07-01 02:00:17,158 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x137c76a7556000a 2012-07-01 02:00:17,159 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:00:17,315 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:00:17,407 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:00:17,465 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:00:17,465 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:00:17,465 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:00:17,759 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:00:18,024 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:12:56,733 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x337c76a7573000f, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:12:57,434 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:12:57,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:12:58,480 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:12:58,536 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:12:58,536 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:12:58,579 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:12:58,688 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x337c76a7573000f, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:13:00,963 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:13:00,973 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:00,973 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:00,973 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:13:01,034 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x337c76a7573000f, negotiated timeout = 5000 2012-07-01 02:13:01,038 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:13:01,149 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:13:01,186 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:13:01,202 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:13:01,203 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:13:01,203 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:13:01,276 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:13:01,420 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:13:34,022 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x337c76a7573000f, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:13:34,742 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:13:34,986 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-109/10.196.68.136:2181 2012-07-01 02:13:34,986 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:34,986 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:35,002 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:13:35,057 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x337c76a7573000f has expired, closing socket connection 2012-07-01 02:13:35,066 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:13:35,066 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:13:35,229 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:13:35,347 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:13:35,394 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:35,394 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:35,490 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a7573000f 2012-07-01 02:13:35,491 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x337c76a7573000f 2012-07-01 02:13:35,491 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:13:35,495 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:13:35,517 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3383fe2130b0000, negotiated timeout = 5000 2012-07-01 02:13:35,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:13:35,624 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:13:35,760 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:13:35,760 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:13:35,760 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:13:35,891 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:13:36,074 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:13:46,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3383fe2130b0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:13:46,683 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:13:49,614 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:13:49,635 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:49,635 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:49,635 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2955ms for sessionid 0x3383fe2130b0000, closing socket connection and attempting reconnect 2012-07-01 02:13:52,958 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:13:52,959 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:52,959 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:52,960 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:13:53,039 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3383fe2130b0000 has expired, closing socket connection 2012-07-01 02:13:53,040 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:13:53,040 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:13:53,169 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:13:53,230 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:13:53,270 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:13:53,270 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:13:53,286 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:13:53,316 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3383fe2130b0000 2012-07-01 02:13:53,316 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:13:53,370 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3383fe2130b0001, negotiated timeout = 5000 2012-07-01 02:13:53,381 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:13:53,402 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:13:53,410 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:13:53,450 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:13:53,450 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:13:53,488 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:13:53,596 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:14:10,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3383fe2130b0001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:14:10,443 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:14:12,612 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:14:12,624 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:14:12,624 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:14:12,624 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:14:12,675 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3383fe2130b0001 has expired, closing socket connection 2012-07-01 02:14:12,676 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:14:12,676 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:14:12,721 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:14:12,833 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:14:12,865 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:14:12,979 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:14:12,979 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:14:12,980 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:14:13,014 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3383fe2130b0001 2012-07-01 02:14:13,110 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2383fe20ed20000, negotiated timeout = 5000 2012-07-01 02:14:13,111 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:14:13,154 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:14:13,199 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:14:13,199 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:14:13,211 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:14:13,244 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:14:13,380 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:09,899 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fe20ed20000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:10,410 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:10,441 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:10,917 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:26:10,948 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:10,948 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:10,979 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:26:10,980 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fe20ed20000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:13,171 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:26:13,208 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:13,208 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:13,208 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:26:13,269 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x2383fe20ed20000, negotiated timeout = 5000 2012-07-01 02:26:13,283 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:13,311 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:13,368 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:13,412 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:13,412 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:13,412 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:13,505 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:13,583 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:20,029 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fe20ed20000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:20,455 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:21,004 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-108/10.196.68.150:2181 2012-07-01 02:26:21,028 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:21,028 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:21,062 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:26:21,132 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2383fe20ed20000 has expired, closing socket connection 2012-07-01 02:26:21,133 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:26:21,133 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:26:21,153 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:26:21,171 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:26:21,185 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:21,185 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:21,188 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:26:21,216 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:26:21,243 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2383fee2d3d0000, negotiated timeout = 5000 2012-07-01 02:26:21,244 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:21,263 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:21,298 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:21,331 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:21,331 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:21,348 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:21,448 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:21,449 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fe20ed20000 2012-07-01 02:26:21,449 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fe20ed20000 2012-07-01 02:26:30,003 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fee2d3d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:30,201 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:30,682 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:26:30,697 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:30,698 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:30,718 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:26:30,719 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2383fee2d3d0000 has expired, closing socket connection 2012-07-01 02:26:30,720 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:26:30,720 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:26:30,740 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:26:30,796 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:26:30,814 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:30,814 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:30,837 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:26:30,985 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2383fee2d3d0001, negotiated timeout = 5000 2012-07-01 02:26:30,987 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:30,988 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fee2d3d0000 2012-07-01 02:26:30,988 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:26:31,012 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:31,043 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:31,043 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:31,044 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:31,075 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:31,161 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:36,036 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fee2d3d0001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:36,424 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:36,505 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:26:36,569 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:36,569 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:36,608 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:26:36,609 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2383fee2d3d0001 has expired, closing socket connection 2012-07-01 02:26:36,610 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:26:36,610 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:26:36,651 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:26:36,719 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:26:36,734 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:36,734 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:36,734 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:26:36,809 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1383fee2b8d0000, negotiated timeout = 5000 2012-07-01 02:26:36,828 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fee2d3d0001 2012-07-01 02:26:36,828 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:26:36,829 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:36,970 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:36,972 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:36,972 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:36,972 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:37,011 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:37,051 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:42,015 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1383fee2b8d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:42,156 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:43,590 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:26:43,622 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:43,622 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:43,638 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:26:43,638 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1383fee2b8d0000 has expired, closing socket connection 2012-07-01 02:26:43,639 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:26:43,639 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:26:43,662 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:26:43,745 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:26:43,760 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:43,760 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:43,761 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:26:43,940 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1383fee2b8d0001, negotiated timeout = 5000 2012-07-01 02:26:43,942 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1383fee2b8d0000 2012-07-01 02:26:43,942 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:26:43,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:43,946 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:43,949 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:43,949 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:43,949 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:43,997 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:44,071 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:26:56,009 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1383fee2b8d0001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:26:56,187 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:26:56,586 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:26:56,593 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:56,593 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:56,613 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:26:56,614 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1383fee2b8d0001 has expired, closing socket connection 2012-07-01 02:26:56,615 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:26:56,615 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:26:56,637 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:26:56,716 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:26:56,727 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:26:56,727 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:26:56,727 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:26:56,801 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1383fee2b8d0001 2012-07-01 02:26:56,801 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:26:56,802 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1383fee2b8d0002, negotiated timeout = 5000 2012-07-01 02:26:56,803 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:26:56,818 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:26:56,852 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:26:56,852 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:26:56,852 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:26:56,902 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:26:56,994 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:27:02,003 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1383fee2b8d0002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:27:02,534 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:27:04,958 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:27:04,959 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:04,959 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:04,959 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2427ms for sessionid 0x1383fee2b8d0002, closing socket connection and attempting reconnect 2012-07-01 02:27:05,389 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:27:05,440 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:05,440 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:05,453 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:27:05,454 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1383fee2b8d0002 has expired, closing socket connection 2012-07-01 02:27:05,455 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:27:05,455 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:27:05,458 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:27:05,541 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:27:05,543 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:05,543 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:05,543 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:27:05,563 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:27:05,596 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1383fee2b8d0002 2012-07-01 02:27:05,680 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1383fee2b8d0003, negotiated timeout = 5000 2012-07-01 02:27:05,680 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:27:05,681 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:27:05,701 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:27:05,702 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:27:05,702 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:27:05,712 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:27:05,763 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:27:16,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1383fee2b8d0003, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:27:16,781 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:27:17,040 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:27:17,064 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:17,064 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:17,103 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:27:17,104 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1383fee2b8d0003 has expired, closing socket connection 2012-07-01 02:27:17,104 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:27:17,105 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:27:17,116 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:27:17,235 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:27:17,288 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1383fee2b8d0003 2012-07-01 02:27:17,288 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:27:17,289 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:17,289 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:17,290 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:27:17,353 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1383fee2b8d0004, negotiated timeout = 5000 2012-07-01 02:27:17,354 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:27:17,354 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:27:17,453 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:27:17,453 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:27:17,453 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:27:17,490 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:27:17,556 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:27:24,010 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1383fee2b8d0004, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:27:24,186 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:27:26,028 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:27:26,043 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:26,043 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:26,044 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 1859ms for sessionid 0x1383fee2b8d0004, closing socket connection and attempting reconnect 2012-07-01 02:27:27,857 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:27:27,891 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:27,892 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:27,934 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:27:27,978 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1383fee2b8d0004 has expired, closing socket connection 2012-07-01 02:27:27,978 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:27:27,978 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:27:27,979 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:27:28,012 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:27:28,024 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:28,024 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:28,038 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:27:28,214 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1383fee2b8d0004 2012-07-01 02:27:28,214 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:27:28,314 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2383fee2d3d0002, negotiated timeout = 5000 2012-07-01 02:27:28,315 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:27:28,315 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:27:28,363 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:27:28,363 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:27:28,363 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:27:28,367 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:27:28,467 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:27:36,027 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fee2d3d0002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:27:36,537 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:27:37,817 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 02:27:37,818 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:37,818 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:37,818 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 02:27:37,858 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2383fee2d3d0002 has expired, closing socket connection 2012-07-01 02:27:37,859 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:27:37,859 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:27:37,860 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:27:37,908 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 02:27:37,936 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:37,936 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:37,955 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 02:27:38,022 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2383fee2d3d0003, negotiated timeout = 5000 2012-07-01 02:27:38,023 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fee2d3d0002 2012-07-01 02:27:38,023 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:27:38,023 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:27:38,024 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:27:38,048 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:27:38,048 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:27:38,048 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:27:38,071 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:27:38,106 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 02:27:44,003 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2383fee2d3d0003, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 02:27:44,521 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 02:27:45,194 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:27:45,201 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:45,201 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:45,207 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:27:45,208 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2383fee2d3d0003 has expired, closing socket connection 2012-07-01 02:27:45,208 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 02:27:45,208 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 02:27:45,209 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 02:27:45,255 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 02:27:45,267 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 02:27:45,268 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 02:27:45,274 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 02:27:45,348 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3383fee2ba70000, negotiated timeout = 5000 2012-07-01 02:27:45,354 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2383fee2d3d0003 2012-07-01 02:27:45,354 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 02:27:45,354 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 02:27:45,357 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 02:27:45,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 02:27:45,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 02:27:45,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 02:27:45,374 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 02:27:45,428 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:06:14,040 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3383fee2ba70000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:06:14,476 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:06:14,476 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:06:14,920 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 03:06:14,922 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:06:14,922 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:06:14,922 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 03:06:15,043 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3383fee2ba70000 has expired, closing socket connection 2012-07-01 03:06:15,044 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 03:06:15,044 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 03:06:15,045 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 03:06:15,105 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 03:06:15,111 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:06:15,111 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:06:15,129 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3383fee2ba70000 2012-07-01 03:06:15,129 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 03:06:15,224 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 03:06:15,231 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384012dc250000, negotiated timeout = 5000 2012-07-01 03:06:15,231 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:06:15,231 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:06:15,233 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:06:15,233 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:06:15,234 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:06:15,235 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:06:15,323 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:06:46,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384012dc250000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:06:46,339 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:06:47,604 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 03:06:47,605 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:06:47,605 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:06:47,605 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 03:06:47,624 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384012dc250000 has expired, closing socket connection 2012-07-01 03:06:47,624 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 03:06:47,624 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 03:06:47,626 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 03:06:47,661 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 03:06:47,682 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:06:47,682 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:06:47,683 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 03:06:47,816 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1384012c4fb0000, negotiated timeout = 5000 2012-07-01 03:06:47,817 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384012dc250000 2012-07-01 03:06:47,817 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 03:06:47,817 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:06:47,820 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:06:47,836 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:06:47,836 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:06:47,836 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:06:47,854 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:06:48,084 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:07:04,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1384012c4fb0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:07:04,335 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:07:04,342 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 03:07:04,397 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:07:04,397 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:07:04,421 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 03:07:04,422 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1384012c4fb0000 has expired, closing socket connection 2012-07-01 03:07:04,423 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 03:07:04,423 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 03:07:04,498 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 03:07:04,528 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 03:07:04,537 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:07:04,537 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:07:04,555 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 03:07:04,598 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 03:07:04,721 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384012c52e0000, negotiated timeout = 5000 2012-07-01 03:07:04,722 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:07:04,732 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:07:04,750 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:07:04,750 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:07:04,750 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:07:04,774 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:07:04,903 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:07:04,904 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1384012c4fb0000 2012-07-01 03:27:12,438 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384012c52e0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:27:12,900 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:27:12,900 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:27:13,381 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 03:27:13,403 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:27:13,403 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:27:13,403 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 03:27:13,444 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x2384012c52e0000, negotiated timeout = 5000 2012-07-01 03:27:13,445 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:27:13,445 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:27:13,514 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:27:13,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:27:13,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:27:13,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:27:13,535 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:27:13,603 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:27:20,016 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384012c52e0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:27:20,593 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:27:21,388 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 03:27:21,398 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:27:21,398 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:27:21,410 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 03:27:21,411 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384012c52e0000 has expired, closing socket connection 2012-07-01 03:27:21,411 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 03:27:21,411 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 03:27:21,412 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 03:27:21,474 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 03:27:21,483 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:27:21,483 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:27:21,494 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 03:27:21,590 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 03:27:21,593 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384026111d0000, negotiated timeout = 5000 2012-07-01 03:27:21,593 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384012c52e0000 2012-07-01 03:27:21,593 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384012c52e0000 2012-07-01 03:27:21,594 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:27:21,597 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:27:21,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:27:21,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:27:21,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:27:21,601 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:27:21,641 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:41:56,895 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384026111d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:41:57,083 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:41:57,083 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:41:59,226 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 03:41:59,228 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:41:59,229 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:41:59,243 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 03:41:59,244 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384026111d0000 has expired, closing socket connection 2012-07-01 03:41:59,244 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 03:41:59,244 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 03:41:59,257 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 03:41:59,297 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 03:41:59,304 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:41:59,304 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:41:59,329 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 03:41:59,561 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840336c760000, negotiated timeout = 5000 2012-07-01 03:41:59,562 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384026111d0000 2012-07-01 03:41:59,562 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 03:41:59,563 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:41:59,652 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:41:59,654 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:41:59,654 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:41:59,654 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:41:59,657 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:41:59,737 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 03:59:14,834 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840336c760000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 03:59:15,559 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:59:15,559 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 03:59:16,140 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 03:59:16,149 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 03:59:16,149 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 03:59:16,159 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 03:59:16,160 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x23840336c760000, negotiated timeout = 5000 2012-07-01 03:59:16,160 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:59:16,161 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 03:59:16,182 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 03:59:16,183 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 03:59:16,183 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 03:59:16,184 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 03:59:16,225 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 03:59:16,269 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:02:40,832 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840336c760000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:02:41,304 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:02:41,305 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:02:41,317 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:02:41,476 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:02:41,476 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:02:41,476 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:02:41,477 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:02:41,595 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840336c760000 has expired, closing socket connection 2012-07-01 04:02:41,596 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:02:41,596 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:02:41,597 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:02:41,669 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:02:41,735 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:02:41,735 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:02:41,750 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840336c760000 2012-07-01 04:02:41,750 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840336c760000 2012-07-01 04:02:41,750 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:02:41,902 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:02:41,930 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x338404689f60000, negotiated timeout = 5000 2012-07-01 04:02:41,931 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:02:41,931 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:02:41,945 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:02:41,945 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:02:41,945 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:02:41,947 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:02:42,115 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:04:46,007 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x338404689f60000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:04:46,535 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:04:47,231 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:04:47,250 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:04:47,250 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:04:47,276 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:04:47,277 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x338404689f60000 has expired, closing socket connection 2012-07-01 04:04:47,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:04:47,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:04:47,278 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:04:47,384 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:04:47,409 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:04:47,409 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:04:47,410 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:04:47,594 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:04:47,595 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x338404689f60000 2012-07-01 04:04:47,596 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840466b330000, negotiated timeout = 5000 2012-07-01 04:04:47,596 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:04:47,597 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:04:47,598 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:04:47,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:04:47,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:04:47,600 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:04:47,699 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:05:26,005 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840466b330000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:05:26,130 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:05:28,394 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:05:28,440 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:05:28,440 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:05:28,440 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2310ms for sessionid 0x13840466b330000, closing socket connection and attempting reconnect 2012-07-01 04:05:29,803 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:05:29,828 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:05:29,828 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:05:29,857 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:05:29,869 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840466b330000 has expired, closing socket connection 2012-07-01 04:05:29,870 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:05:29,870 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:05:29,894 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:05:29,939 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:05:29,942 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:05:29,942 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:05:29,950 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:05:30,024 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840466b6d0000, negotiated timeout = 5000 2012-07-01 04:05:30,033 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:05:30,033 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:05:30,038 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:05:30,041 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:05:30,041 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:05:30,041 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:05:30,093 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:05:30,233 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:05:30,233 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840466b330000 2012-07-01 04:36:24,039 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840466b6d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:36:24,204 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:36:24,205 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:36:27,161 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:36:27,162 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:27,163 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:27,165 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:36:27,165 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840466b6d0000 has expired, closing socket connection 2012-07-01 04:36:27,166 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:36:27,166 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:36:27,183 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:36:27,220 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:36:27,230 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:27,230 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:27,231 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:36:27,323 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138406554340000, negotiated timeout = 5000 2012-07-01 04:36:27,324 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840466b6d0000 2012-07-01 04:36:27,324 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:36:27,324 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:36:27,333 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:36:27,338 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:36:27,338 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:36:27,338 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:36:27,339 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:36:27,487 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:36:34,005 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138406554340000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:36:34,260 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:36:34,733 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:36:34,743 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:34,743 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:34,760 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:36:34,806 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138406554340000 has expired, closing socket connection 2012-07-01 04:36:34,806 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:36:34,806 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:36:34,807 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:36:34,888 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:36:34,905 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:34,906 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:34,906 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:36:35,142 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138406554340001, negotiated timeout = 5000 2012-07-01 04:36:35,143 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138406554340000 2012-07-01 04:36:35,143 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:36:35,144 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:36:35,180 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:36:35,196 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:36:35,196 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:36:35,196 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:36:35,198 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:36:35,236 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:36:40,005 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138406554340001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:36:40,498 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:36:42,099 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:36:42,111 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:42,111 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:42,148 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:36:42,172 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138406554340001 has expired, closing socket connection 2012-07-01 04:36:42,172 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:36:42,172 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:36:42,195 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:36:42,215 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:36:42,229 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:42,229 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:42,242 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:36:42,292 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:36:42,336 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x238406569350000, negotiated timeout = 5000 2012-07-01 04:36:42,336 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:36:42,337 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:36:42,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:36:42,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:36:42,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:36:42,346 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:36:42,365 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:36:42,377 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138406554340001 2012-07-01 04:36:48,015 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x238406569350000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:36:48,163 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:36:51,215 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:36:51,233 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:51,233 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:51,233 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:36:51,262 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238406569350000 has expired, closing socket connection 2012-07-01 04:36:51,262 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:36:51,262 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:36:51,291 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:36:51,320 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:36:51,332 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:36:51,332 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:36:51,349 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:36:51,369 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:36:51,395 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x238406569350001, negotiated timeout = 5000 2012-07-01 04:36:51,395 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:36:51,395 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:36:51,402 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:36:51,402 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:36:51,402 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:36:51,406 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:36:51,565 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:36:51,565 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x238406569350000 2012-07-01 04:37:30,010 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x238406569350001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:37:30,135 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:37:31,270 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:37:31,279 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:31,279 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:31,280 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:37:31,336 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238406569350001 has expired, closing socket connection 2012-07-01 04:37:31,336 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:37:31,336 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:37:31,354 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:37:31,387 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:37:31,398 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:31,434 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:31,435 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:37:31,471 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:37:31,497 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x238406569350002, negotiated timeout = 5000 2012-07-01 04:37:31,497 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:37:31,519 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:37:31,632 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:37:31,632 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:37:31,632 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:37:31,634 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:37:31,702 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:37:31,707 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x238406569350001 2012-07-01 04:37:38,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x238406569350002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:37:38,487 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:37:39,079 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:37:39,093 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:39,093 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:39,115 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:37:39,115 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238406569350002 has expired, closing socket connection 2012-07-01 04:37:39,116 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:37:39,116 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:37:39,161 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:37:39,204 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:37:39,226 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:39,226 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:39,249 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:37:39,423 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x238406569350003, negotiated timeout = 5000 2012-07-01 04:37:39,426 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x238406569350002 2012-07-01 04:37:39,459 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:37:39,459 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:37:39,462 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:37:39,464 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:37:39,464 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:37:39,464 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:37:39,467 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:37:39,498 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:37:46,005 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x238406569350003, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:37:46,365 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:37:47,715 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:37:47,716 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:47,716 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:47,716 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:37:47,727 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238406569350003 has expired, closing socket connection 2012-07-01 04:37:47,728 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:37:47,728 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:37:47,730 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:37:47,765 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:37:47,783 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:37:47,783 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:37:47,783 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:37:47,970 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384065544a0000, negotiated timeout = 5000 2012-07-01 04:37:47,971 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x238406569350003 2012-07-01 04:37:47,971 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:37:47,971 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:37:47,976 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:37:47,990 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:37:47,990 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:37:47,990 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:37:47,992 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:37:48,037 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:56:36,717 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384065544a0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:56:37,027 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:56:37,027 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:56:38,247 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:56:38,254 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:56:38,254 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:56:38,254 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:56:38,405 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x3384065544a0000, negotiated timeout = 5000 2012-07-01 04:56:38,405 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:56:38,406 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:56:38,407 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:56:38,436 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:56:38,436 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:56:38,436 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:56:38,448 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:56:38,536 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:56:44,021 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384065544a0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:56:44,443 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:56:47,251 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:56:47,269 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:56:47,269 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:56:47,269 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2826ms for sessionid 0x3384065544a0000, closing socket connection and attempting reconnect 2012-07-01 04:56:47,877 FATAL org.apache.hadoop.ha.ActiveStandbyElector: Received create error from Zookeeper. code:CONNECTIONLOSS for path /hadoop-ha/events/ActiveStandbyElectorLock. Not retrying further znode create connection errors. 2012-07-01 04:56:48,819 INFO org.apache.zookeeper.ZooKeeper: Session: 0x3384065544a0000 closed 2012-07-01 04:56:48,819 FATAL org.apache.hadoop.ha.ZKFailoverController: Fatal error occurred:Received create error from Zookeeper. code:CONNECTIONLOSS for path /hadoop-ha/events/ActiveStandbyElectorLock. Not retrying further znode create connection errors. 2012-07-01 04:56:48,820 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384065544a0000 2012-07-01 04:56:48,820 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:56:48,820 INFO org.apache.hadoop.ipc.Server: Stopping server on 8019 2012-07-01 04:56:48,897 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-07-01 04:56:48,897 INFO org.apache.hadoop.ha.HealthMonitor: Stopping HealthMonitor thread 2012-07-01 04:56:48,897 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8019 2012-07-01 04:56:48,897 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2012-07-01 04:57:31,090 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode events.namenode80 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-cdh4.0.0--1, built on 06/04/2012 23:16 GMT 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop-106 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-sun-1.6.0.32.x86_64/jre 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/json-simple-1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0.jar:/usr/share/cmf/lib/plugins/event-publish-4.0.1-shaded.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/lib/jline-0.9.94.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/avro-1.5.4.jar:/usr/lib/hadoop-yarn/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/junit-4.8.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.15.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.5.4.jar:/usr/lib/hadoop-mapreduce/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.15.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.0.0-cdh4.0.0.jar 2012-07-01 04:57:38,320 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler= 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-220.el6.x86_64 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfs 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/usr/lib/hadoop 2012-07-01 04:57:38,321 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER 2012-07-01 04:57:38,428 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:57:39,914 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:57:40,161 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:57:40,161 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:57:40,252 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:57:40,311 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384077eb6c0000, negotiated timeout = 5000 2012-07-01 04:57:40,440 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:57:40,844 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019 2012-07-01 04:57:43,330 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-07-01 04:57:43,441 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting 2012-07-01 04:57:48,213 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-07-01 04:57:48,217 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-07-01 04:57:48,333 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:57:48,475 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:57:48,475 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:57:48,475 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:57:48,627 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:57:48,960 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:57:56,050 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384077eb6c0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:57:56,682 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:57:56,853 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:57:56,854 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:57:56,854 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:57:56,855 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:57:56,947 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384077eb6c0000 has expired, closing socket connection 2012-07-01 04:57:57,004 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:57:57,004 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:57:57,138 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:57:57,241 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:57:57,247 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:57:57,248 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:57:57,249 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384077eb6c0000 2012-07-01 04:57:57,249 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:57:57,255 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:57:57,273 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384077eb6c0001, negotiated timeout = 5000 2012-07-01 04:57:57,275 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:57:57,288 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:57:57,317 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:57:57,317 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:57:57,317 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:57:57,348 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:57:57,368 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:58:08,005 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384077eb6c0001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:58:08,190 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:58:08,800 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:58:08,803 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:08,803 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:08,803 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:58:08,829 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384077eb6c0001 has expired, closing socket connection 2012-07-01 04:58:08,871 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:58:08,871 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:58:08,989 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:58:09,057 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:58:09,228 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:09,228 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:09,228 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384077eb6c0001 2012-07-01 04:58:09,228 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:58:09,231 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:58:09,237 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384077eb6c0002, negotiated timeout = 5000 2012-07-01 04:58:09,276 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:58:09,279 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:58:09,428 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:58:09,428 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:58:09,428 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:58:09,495 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:58:09,680 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:58:16,007 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384077eb6c0002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:58:16,116 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:58:17,924 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:58:17,963 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:17,964 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:17,964 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:58:18,124 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384077eb6c0002 has expired, closing socket connection 2012-07-01 04:58:18,125 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:58:18,125 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:58:18,127 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:58:18,153 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:58:18,171 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:18,171 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:18,172 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:58:18,290 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:58:18,291 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384077eb6c0002 2012-07-01 04:58:18,300 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1384077ec160000, negotiated timeout = 5000 2012-07-01 04:58:18,301 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:58:18,303 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:58:18,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:58:18,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:58:18,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:58:18,385 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:58:18,464 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:58:24,016 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1384077ec160000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:58:24,312 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:58:25,392 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:58:25,398 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:25,399 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:25,407 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:58:25,409 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1384077ec160000 has expired, closing socket connection 2012-07-01 04:58:25,410 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:58:25,410 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:58:25,437 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:58:25,495 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 04:58:25,512 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:25,512 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:25,539 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 04:58:25,542 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384077eb6c0003, negotiated timeout = 5000 2012-07-01 04:58:25,568 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:58:25,569 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher java.lang.NullPointerException at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 04:58:25,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:58:25,620 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:58:25,620 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:58:25,621 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:58:25,651 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:58:25,653 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 04:58:25,730 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1384077ec160000 2012-07-01 04:58:32,058 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384077eb6c0003, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 04:58:32,233 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 04:58:34,127 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 04:58:34,135 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:34,136 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:34,136 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 04:58:34,213 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384077eb6c0003 has expired, closing socket connection 2012-07-01 04:58:34,214 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 04:58:34,214 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 04:58:34,245 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 04:58:34,322 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 04:58:34,331 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 04:58:34,331 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 04:58:34,349 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 04:58:34,532 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384077eb7b0000, negotiated timeout = 5000 2012-07-01 04:58:34,533 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384077eb6c0003 2012-07-01 04:58:34,533 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 04:58:34,534 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 04:58:34,552 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 04:58:34,595 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 04:58:34,595 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 04:58:34,595 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 04:58:34,643 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 04:58:34,753 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:00:42,997 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384077eb7b0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:00:43,330 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:00:43,356 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:00:44,927 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:00:44,944 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:00:44,944 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:00:44,945 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:00:45,077 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384077eb7b0000 has expired, closing socket connection 2012-07-01 05:00:45,078 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:00:45,078 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:00:45,142 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:00:45,220 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:00:45,221 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:00:45,221 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:00:45,240 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384077eb7b0000 2012-07-01 05:00:45,240 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:00:45,309 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:00:45,335 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138407b943d0000, negotiated timeout = 5000 2012-07-01 05:00:45,337 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:00:45,347 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:00:45,398 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:00:45,398 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:00:45,398 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:00:45,436 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:00:45,585 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:00:58,009 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138407b943d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:00:58,190 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:00:59,061 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:00:59,063 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:00:59,063 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:00:59,081 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 05:00:59,168 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138407b943d0000 has expired, closing socket connection 2012-07-01 05:00:59,169 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:00:59,169 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:00:59,192 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:00:59,266 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:00:59,288 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:00:59,288 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:00:59,288 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:00:59,379 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138407b943d0000 2012-07-01 05:00:59,379 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:00:59,379 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138407b943d0001, negotiated timeout = 5000 2012-07-01 05:00:59,380 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:00:59,396 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:00:59,409 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:00:59,409 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:00:59,446 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:00:59,474 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:00:59,624 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:01:18,009 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138407b943d0001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:01:18,255 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:01:20,043 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:01:20,057 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:01:20,058 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:01:20,058 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 1807ms for sessionid 0x138407b943d0001, closing socket connection and attempting reconnect 2012-07-01 05:01:22,667 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:01:22,677 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:01:22,677 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:01:22,677 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2351ms for sessionid 0x138407b943d0001, closing socket connection and attempting reconnect 2012-07-01 05:01:27,370 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-106/10.196.68.149:2181 2012-07-01 05:01:27,370 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:01:27,370 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:01:27,393 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:01:27,394 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138407b943d0001 has expired, closing socket connection 2012-07-01 05:01:27,395 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:01:27,395 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:01:27,425 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:01:27,497 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:01:27,510 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:01:27,510 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:01:27,510 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:01:27,680 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138407b943d0002, negotiated timeout = 5000 2012-07-01 05:01:27,680 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138407b943d0001 2012-07-01 05:01:27,680 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:01:27,681 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:01:27,735 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:01:27,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:01:27,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:01:27,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:01:27,766 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:01:27,883 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:02:18,004 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138407b943d0002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:02:18,297 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:02:19,217 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:02:19,245 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:02:19,245 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:02:19,274 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:02:19,318 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138407b943d0002 has expired, closing socket connection 2012-07-01 05:02:19,319 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:02:19,319 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:02:19,355 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:02:19,473 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:02:19,489 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:02:19,489 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:02:19,502 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:02:19,532 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:02:19,570 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x338407bac000000, negotiated timeout = 5000 2012-07-01 05:02:19,571 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:02:19,576 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:02:19,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:02:19,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:02:19,737 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:02:19,743 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:02:19,782 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:02:19,784 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138407b943d0002 2012-07-01 05:03:12,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x338407bac000000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:03:12,209 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:03:13,947 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:03:13,994 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:03:13,994 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:03:13,994 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 1852ms for sessionid 0x338407bac000000, closing socket connection and attempting reconnect 2012-07-01 05:03:15,642 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:03:15,679 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:03:15,679 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:03:15,679 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:03:15,723 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x338407bac000000 has expired, closing socket connection 2012-07-01 05:03:15,724 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:03:15,724 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:03:15,731 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:03:15,747 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:03:15,751 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:03:15,751 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:03:15,751 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:03:15,765 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x138407b943d0003, negotiated timeout = 5000 2012-07-01 05:03:15,854 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher java.lang.NullPointerException at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 05:03:15,855 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x338407bac000000 2012-07-01 05:03:15,855 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:03:15,858 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:03:15,860 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:03:15,860 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:03:15,860 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:03:15,879 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:03:15,933 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:03:28,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x138407b943d0003, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:03:28,484 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:03:29,140 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:03:29,164 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:03:29,164 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:03:29,194 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:03:29,195 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x138407b943d0003 has expired, closing socket connection 2012-07-01 05:03:29,196 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:03:29,196 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:03:29,246 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:03:29,341 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:03:29,391 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:03:29,391 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:03:29,436 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 05:03:29,506 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x238407b94810000, negotiated timeout = 5000 2012-07-01 05:03:29,558 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x138407b943d0003 2012-07-01 05:03:29,558 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:03:29,559 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:03:29,628 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:03:29,697 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:03:29,697 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:03:29,697 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:03:29,737 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:03:29,869 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:47:40,275 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x238407b94810000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:47:41,002 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:47:41,003 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:47:42,140 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:47:42,141 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:47:42,141 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:47:42,142 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:47:42,143 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238407b94810000 has expired, closing socket connection 2012-07-01 05:47:42,143 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:47:42,144 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:47:42,162 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:47:42,230 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:47:42,239 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:47:42,239 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:47:42,239 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:47:42,380 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840a67ae10000, negotiated timeout = 5000 2012-07-01 05:47:42,381 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x238407b94810000 2012-07-01 05:47:42,381 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:47:42,387 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:47:42,391 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:47:42,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:47:42,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:47:42,415 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:47:42,437 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:47:42,502 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:47:48,010 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840a67ae10000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:47:48,176 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:47:50,344 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:47:50,345 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:47:50,345 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:47:50,345 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2169ms for sessionid 0x13840a67ae10000, closing socket connection and attempting reconnect 2012-07-01 05:47:51,302 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:47:51,324 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:47:51,324 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:47:51,335 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:47:51,336 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840a67ae10000 has expired, closing socket connection 2012-07-01 05:47:51,336 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:47:51,336 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:47:51,337 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:47:51,367 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 05:47:51,373 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:47:51,373 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:47:51,373 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:47:51,422 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:47:51,429 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840a67ae10001, negotiated timeout = 5000 2012-07-01 05:47:51,429 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:47:51,430 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:47:51,442 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:47:51,442 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:47:51,442 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:47:51,455 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:47:51,499 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:47:51,753 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840a67ae10000 2012-07-01 05:48:02,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840a67ae10001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 05:48:02,314 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 05:48:05,312 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 05:48:05,331 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:48:05,331 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:48:05,331 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 3017ms for sessionid 0x13840a67ae10001, closing socket connection and attempting reconnect 2012-07-01 05:48:08,313 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:48:08,328 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:48:08,328 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:48:08,329 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2736ms for sessionid 0x13840a67ae10001, closing socket connection and attempting reconnect 2012-07-01 05:48:15,971 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-106/10.196.68.149:2181 2012-07-01 05:48:15,971 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:48:15,971 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:48:15,971 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 05:48:16,023 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840a67ae10001 has expired, closing socket connection 2012-07-01 05:48:16,024 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 05:48:16,024 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 05:48:16,036 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 05:48:16,090 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 05:48:16,096 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 05:48:16,096 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 05:48:16,107 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 05:48:16,173 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 05:48:16,176 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840a67af70000, negotiated timeout = 5000 2012-07-01 05:48:16,176 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 05:48:16,185 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 05:48:16,190 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 05:48:16,190 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 05:48:16,190 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 05:48:16,199 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 05:48:16,313 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 05:48:16,456 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840a67ae10001 2012-07-01 06:06:57,994 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840a67af70000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:06:58,598 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:06:58,599 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:06:59,832 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:06:59,839 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:06:59,839 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:06:59,839 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:06:59,890 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x33840a67af70000, negotiated timeout = 5000 2012-07-01 06:06:59,890 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:06:59,891 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:06:59,892 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:06:59,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:06:59,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:06:59,943 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:06:59,953 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:07:00,004 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:07:06,009 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840a67af70000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:07:06,206 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:07:07,152 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:07:07,154 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:07,154 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:07,155 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:07:07,164 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840a67af70000 has expired, closing socket connection 2012-07-01 06:07:07,186 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:07:07,186 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:07:07,225 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:07:07,264 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:07:07,274 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:07:07,329 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:07,329 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:07,364 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840a67af70000 2012-07-01 06:07:07,364 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840a67af70000 2012-07-01 06:07:07,364 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:07:07,370 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840b868220000, negotiated timeout = 5000 2012-07-01 06:07:07,370 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:07:07,370 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:07:07,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:07:07,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:07:07,372 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:07:07,457 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:07:07,464 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:07:26,004 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840b868220000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:07:26,537 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:07:29,831 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:07:29,874 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:29,874 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:29,874 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 3338ms for sessionid 0x23840b868220000, closing socket connection and attempting reconnect 2012-07-01 06:07:33,532 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:07:33,540 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:33,540 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:33,540 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:07:33,547 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:07:33,547 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:07:33,584 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:07:33,631 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:07:33,647 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:33,647 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:33,668 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:07:33,670 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840b868220000 has expired, closing socket connection 2012-07-01 06:07:33,682 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:07:33,760 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840b868220001, negotiated timeout = 5000 2012-07-01 06:07:33,760 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:07:33,760 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:07:33,780 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:07:33,780 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:07:33,780 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:07:33,787 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:07:33,843 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:07:34,078 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840b868220000 2012-07-01 06:07:40,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840b868220001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:07:40,576 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:07:40,700 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:07:40,718 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:40,722 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:40,731 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:07:40,741 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:07:40,741 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:07:40,746 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:07:40,746 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840b868220001 has expired, closing socket connection 2012-07-01 06:07:40,768 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:07:40,783 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:40,783 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:40,783 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:07:40,836 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:07:40,838 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840b854910000, negotiated timeout = 5000 2012-07-01 06:07:40,838 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:07:40,840 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:07:40,849 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:07:40,849 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:07:40,849 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:07:40,852 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:07:40,876 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:07:41,126 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840b868220001 2012-07-01 06:07:48,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840b854910000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:07:48,408 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:07:50,990 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:07:51,012 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:51,012 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:51,012 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2604ms for sessionid 0x13840b854910000, closing socket connection and attempting reconnect 2012-07-01 06:07:52,477 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:07:52,502 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:52,503 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:52,520 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:07:52,534 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840b854910000 has expired, closing socket connection 2012-07-01 06:07:52,535 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:07:52,535 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:07:52,535 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:07:52,569 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:07:52,598 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:07:52,706 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840b854910000 2012-07-01 06:07:52,706 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:07:52,707 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:07:52,780 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:07:52,786 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840b854b80000, negotiated timeout = 5000 2012-07-01 06:07:52,786 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:07:52,787 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:07:52,788 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:07:52,788 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:07:52,788 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:07:52,790 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:07:52,836 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:09:25,203 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840b854b80000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:09:25,616 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:09:25,617 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:09:27,139 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:09:27,171 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:09:27,171 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:09:27,196 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:09:27,197 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840b854b80000 has expired, closing socket connection 2012-07-01 06:09:27,197 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:09:27,197 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:09:27,198 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:09:27,267 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:09:27,283 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:09:27,283 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:09:27,303 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:09:27,393 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840ba92510000, negotiated timeout = 5000 2012-07-01 06:09:27,397 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840b854b80000 2012-07-01 06:09:27,397 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:09:27,398 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:09:27,399 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:09:27,400 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:09:27,400 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:09:27,400 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:09:27,402 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:09:27,440 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:09:36,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840ba92510000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:09:36,419 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:09:37,910 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:09:37,921 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:09:37,921 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:09:37,921 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:09:37,986 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840ba92510000 has expired, closing socket connection 2012-07-01 06:09:37,986 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:09:37,986 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:09:37,987 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:09:38,024 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:09:38,040 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:09:38,040 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:09:38,058 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:09:38,342 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840ba92510001, negotiated timeout = 5000 2012-07-01 06:09:38,344 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:09:38,346 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840ba92510000 2012-07-01 06:09:38,346 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:09:38,347 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:09:38,353 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:09:38,354 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:09:38,354 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:09:38,365 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:09:38,420 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:10:16,006 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840ba92510001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:10:16,496 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:10:18,058 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:10:18,059 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:10:18,059 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:10:18,102 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:10:18,102 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840ba92510001 has expired, closing socket connection 2012-07-01 06:10:18,103 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:10:18,103 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:10:18,105 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:10:18,176 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:10:18,190 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:10:18,205 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:10:18,205 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:10:18,206 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:10:18,223 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840ba92510001 2012-07-01 06:10:18,289 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840ba77010000, negotiated timeout = 5000 2012-07-01 06:10:18,289 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:10:18,290 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:10:18,299 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:10:18,299 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:10:18,299 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:10:18,302 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:10:18,349 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:10:32,026 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840ba77010000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:10:32,268 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:10:34,769 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:10:34,792 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:10:34,792 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:10:34,792 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2524ms for sessionid 0x13840ba77010000, closing socket connection and attempting reconnect 2012-07-01 06:10:35,673 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:10:35,692 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:10:35,692 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:10:35,692 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:10:35,693 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840ba77010000 has expired, closing socket connection 2012-07-01 06:10:35,693 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:10:35,693 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:10:35,694 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:10:35,735 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:10:35,756 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:10:35,756 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:10:35,793 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:10:35,984 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840ba77480000, negotiated timeout = 5000 2012-07-01 06:10:35,984 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840ba77010000 2012-07-01 06:10:35,984 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:10:35,985 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:10:35,987 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:10:35,988 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:10:35,988 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:10:35,988 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:10:35,989 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:10:36,051 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:28:27,662 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840ba77480000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:28:28,357 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:28:28,357 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:28:28,799 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:28:28,820 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:28:28,820 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:28:28,856 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:28:28,856 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840ba77480000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:28:29,403 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:28:29,408 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:28:29,408 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:28:29,408 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:28:29,470 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840ba77480000 has expired, closing socket connection 2012-07-01 06:28:29,471 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:28:29,471 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:28:29,475 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:28:29,527 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:28:29,543 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:28:29,543 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:28:29,574 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840ba77480000 2012-07-01 06:28:29,574 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:28:29,593 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:28:29,671 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840cbe9920000, negotiated timeout = 5000 2012-07-01 06:28:29,671 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:28:29,671 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:28:29,691 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:28:29,691 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:28:29,692 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:28:29,699 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:28:29,894 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:35:31,260 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840cbe9920000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:35:31,638 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:35:31,638 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:35:33,232 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:35:33,263 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:35:33,263 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:35:33,279 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:35:33,299 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840cbe9920000 has expired, closing socket connection 2012-07-01 06:35:33,299 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:35:33,299 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:35:33,340 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:35:33,368 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:35:33,379 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:35:33,379 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:35:33,394 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:35:33,407 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840cbe9920000 2012-07-01 06:35:33,408 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:35:33,456 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840d277d60000, negotiated timeout = 5000 2012-07-01 06:35:33,456 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:35:33,461 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:35:33,463 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:35:33,463 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:35:33,463 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:35:33,464 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:35:33,496 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:35:58,040 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840d277d60000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:35:58,545 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:35:59,335 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:35:59,336 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:35:59,336 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:35:59,336 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:35:59,360 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840d277d60000 has expired, closing socket connection 2012-07-01 06:35:59,360 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:35:59,360 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:35:59,361 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:35:59,429 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:35:59,443 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:35:59,443 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:35:59,464 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:35:59,497 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840d277d60001, negotiated timeout = 5000 2012-07-01 06:35:59,541 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher java.lang.NullPointerException at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 06:35:59,547 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:35:59,555 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:35:59,571 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:35:59,583 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:35:59,583 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:35:59,607 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:35:59,682 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:35:59,691 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840d277d60000 2012-07-01 06:36:36,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840d277d60001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:36:36,685 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:36:38,275 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:36:38,294 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:36:38,294 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:36:38,319 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:36:38,339 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840d277d60001 has expired, closing socket connection 2012-07-01 06:36:38,339 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:36:38,339 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:36:38,340 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:36:38,401 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:36:38,415 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:36:38,415 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:36:38,415 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:36:38,463 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:36:38,521 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840d258220000, negotiated timeout = 5000 2012-07-01 06:36:38,521 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:36:38,532 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:36:38,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:36:38,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:36:38,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:36:38,552 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:36:38,615 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:36:38,863 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840d277d60001 2012-07-01 06:36:54,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840d258220000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:36:54,382 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:36:56,431 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:36:56,445 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:36:56,445 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:36:56,446 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2064ms for sessionid 0x13840d258220000, closing socket connection and attempting reconnect 2012-07-01 06:36:57,154 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:36:57,172 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:36:57,172 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:36:57,198 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:36:57,276 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840d258220000 has expired, closing socket connection 2012-07-01 06:36:57,276 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:36:57,276 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:36:57,277 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:36:57,330 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:36:57,346 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:36:57,346 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:36:57,375 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:36:57,524 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840d258220001, negotiated timeout = 5000 2012-07-01 06:36:57,525 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840d258220000 2012-07-01 06:36:57,525 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:36:57,526 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:36:57,537 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:36:57,544 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:36:57,544 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:36:57,545 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:36:57,554 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:36:57,622 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:37:34,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840d258220001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:37:34,555 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:37:36,284 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:37:36,290 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:37:36,290 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:37:36,290 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 1735ms for sessionid 0x13840d258220001, closing socket connection and attempting reconnect 2012-07-01 06:37:38,462 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:37:38,479 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:37:38,479 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:37:38,479 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 1856ms for sessionid 0x13840d258220001, closing socket connection and attempting reconnect 2012-07-01 06:37:42,387 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-106/10.196.68.149:2181 2012-07-01 06:37:42,388 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:37:42,388 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:37:42,388 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:37:42,426 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840d258220001 has expired, closing socket connection 2012-07-01 06:37:42,426 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:37:42,426 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:37:42,427 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:37:42,505 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:37:42,513 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:37:42,513 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:37:42,532 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:37:42,652 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840d277d60002, negotiated timeout = 5000 2012-07-01 06:37:42,655 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840d258220001 2012-07-01 06:37:42,655 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:37:42,657 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:37:42,688 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:37:42,689 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:37:42,689 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:37:42,689 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:37:42,690 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:37:42,757 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:38:18,090 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840d277d60002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:38:18,235 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:38:20,920 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:38:20,939 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:38:20,939 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:38:20,966 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2731ms for sessionid 0x33840d277d60002, closing socket connection and attempting reconnect 2012-07-01 06:38:23,638 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:38:23,687 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:38:23,687 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:38:23,713 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:38:23,733 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840d277d60002 has expired, closing socket connection 2012-07-01 06:38:23,734 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:38:23,734 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:38:23,734 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:38:23,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:38:23,787 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:38:23,787 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:38:23,787 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:38:24,026 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840d277d60002 2012-07-01 06:38:24,026 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:38:24,027 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13840d258220002, negotiated timeout = 5000 2012-07-01 06:38:24,027 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:38:24,029 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:38:24,029 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:38:24,029 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:38:24,029 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:38:24,031 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:38:24,118 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:38:34,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13840d258220002, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:38:34,618 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:38:35,809 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:38:35,810 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:38:35,810 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:38:35,810 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:38:35,852 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13840d258220002 has expired, closing socket connection 2012-07-01 06:38:35,852 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:38:35,852 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:38:35,883 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:38:35,885 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:38:35,887 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:38:35,887 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:38:35,888 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:38:35,903 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:38:35,936 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840d258590000, negotiated timeout = 5000 2012-07-01 06:38:35,937 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:38:35,938 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:38:35,939 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:38:35,939 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:38:35,939 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:38:35,940 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:38:36,109 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:38:36,109 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13840d258220002 2012-07-01 06:46:17,424 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840d258590000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:46:17,855 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:46:17,855 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:46:18,710 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:46:18,731 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:46:18,735 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:46:18,741 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:46:18,747 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840d258590000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:46:20,182 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:46:20,203 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:46:20,203 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:46:20,204 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:46:20,264 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840d258590000 has expired, closing socket connection 2012-07-01 06:46:20,264 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:46:20,264 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:46:20,265 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:46:20,301 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 06:46:20,311 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:46:20,311 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:46:20,331 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 06:46:20,460 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840d258590000 2012-07-01 06:46:20,460 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:46:20,461 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23840dc546d0000, negotiated timeout = 5000 2012-07-01 06:46:20,461 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:46:20,463 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:46:20,466 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:46:20,466 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:46:20,466 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:46:20,473 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:46:20,531 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:46:32,001 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23840dc546d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 06:46:32,337 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 06:46:32,834 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 06:46:32,835 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:46:32,835 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:46:32,835 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 06:46:32,859 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 06:46:32,859 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 06:46:32,967 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 06:46:33,025 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 06:46:33,035 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 06:46:33,035 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23840dc546d0000 has expired, closing socket connection 2012-07-01 06:46:33,040 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 06:46:33,040 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 06:46:33,060 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 06:46:33,106 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x33840dc3dc80000, negotiated timeout = 5000 2012-07-01 06:46:33,106 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 06:46:33,107 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 06:46:33,108 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 06:46:33,108 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 06:46:33,108 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 06:46:33,113 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 06:46:33,202 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 06:46:33,203 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23840dc546d0000 2012-07-01 07:46:45,556 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x33840dc3dc80000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 07:46:45,866 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 07:46:45,866 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 07:46:47,872 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 07:46:47,900 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:46:47,900 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:46:47,900 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 07:46:47,901 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x33840dc3dc80000 has expired, closing socket connection 2012-07-01 07:46:47,902 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 07:46:47,902 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 07:46:47,903 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 07:46:47,966 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 07:46:48,001 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x33840dc3dc80000 2012-07-01 07:46:48,001 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 07:46:48,003 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:46:48,004 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:46:48,005 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 07:46:48,081 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13841139b280000, negotiated timeout = 5000 2012-07-01 07:46:48,081 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 07:46:48,097 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 07:46:48,110 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 07:46:48,110 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 07:46:48,110 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 07:46:48,126 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 07:46:48,309 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 07:47:08,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13841139b280000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 07:47:08,365 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 07:47:12,378 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 07:47:12,379 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:12,379 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:12,379 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4014ms for sessionid 0x13841139b280000, closing socket connection and attempting reconnect 2012-07-01 07:47:16,481 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 07:47:16,490 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:16,491 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:16,508 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 3524ms for sessionid 0x13841139b280000, closing socket connection and attempting reconnect 2012-07-01 07:47:20,490 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-106/10.196.68.149:2181 2012-07-01 07:47:20,491 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:20,491 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:20,491 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 07:47:20,492 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13841139b280000 has expired, closing socket connection 2012-07-01 07:47:20,493 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 07:47:20,493 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 07:47:20,503 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 07:47:20,558 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 07:47:20,581 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:20,581 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:20,581 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 07:47:20,599 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13841139b280000 2012-07-01 07:47:20,599 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 07:47:20,651 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x13841139b280001, negotiated timeout = 5000 2012-07-01 07:47:20,651 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 07:47:20,663 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 07:47:20,670 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 07:47:20,670 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 07:47:20,670 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 07:47:20,674 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 07:47:20,851 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 07:47:50,066 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x13841139b280001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 07:47:50,567 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 07:47:51,358 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 07:47:51,369 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:51,369 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:51,399 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 07:47:51,400 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13841139b280001 has expired, closing socket connection 2012-07-01 07:47:51,400 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 07:47:51,400 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 07:47:51,404 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 07:47:51,463 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 07:47:51,499 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:47:51,499 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:47:51,501 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 07:47:51,526 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 07:47:51,593 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384113b0770000, negotiated timeout = 5000 2012-07-01 07:47:51,593 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 07:47:51,593 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 07:47:51,594 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 07:47:51,595 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 07:47:51,595 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 07:47:51,610 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 07:47:51,654 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 07:47:51,838 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x13841139b280001 2012-07-01 07:48:00,002 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384113b0770000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 07:48:00,197 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 07:48:02,825 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 07:48:02,835 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:48:02,835 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:48:02,835 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 07:48:02,958 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384113b0770000 has expired, closing socket connection 2012-07-01 07:48:02,959 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 07:48:02,959 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 07:48:02,959 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 07:48:03,102 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 07:48:03,104 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 07:48:03,104 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 07:48:03,134 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 07:48:03,265 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x23841139b4a0000, negotiated timeout = 5000 2012-07-01 07:48:03,267 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384113b0770000 2012-07-01 07:48:03,267 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 07:48:03,267 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 07:48:03,271 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 07:48:03,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 07:48:03,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 07:48:03,277 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 07:48:03,278 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 07:48:03,369 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:01:15,715 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23841139b4a0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:01:16,285 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:01:16,285 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:01:17,015 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:01:17,038 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:17,038 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:17,038 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:01:17,039 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23841139b4a0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:01:18,883 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:01:18,903 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:18,903 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:18,903 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:01:18,944 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x23841139b4a0000, negotiated timeout = 5000 2012-07-01 08:01:18,944 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:01:18,944 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:01:18,962 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:01:18,968 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:01:18,968 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:01:18,968 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:01:18,976 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:01:19,058 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:01:24,024 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23841139b4a0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:01:24,428 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:01:25,173 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-108/10.196.68.150:2181 2012-07-01 08:01:25,173 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:25,173 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:25,182 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:01:25,225 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x23841139b4a0000 has expired, closing socket connection 2012-07-01 08:01:25,226 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:01:25,226 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:01:25,226 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:01:25,266 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:01:25,271 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:25,271 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:25,271 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:01:25,323 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:01:25,328 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1384120f9a20000, negotiated timeout = 5000 2012-07-01 08:01:25,328 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:01:25,361 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:01:25,361 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:01:25,362 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:01:25,362 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:01:25,370 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:01:25,380 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:01:25,588 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23841139b4a0000 2012-07-01 08:01:25,589 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x23841139b4a0000 2012-07-01 08:01:42,004 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1384120f9a20000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:01:42,246 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:01:43,869 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:01:43,878 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:43,878 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:43,902 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:01:43,984 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1384120f9a20000 has expired, closing socket connection 2012-07-01 08:01:43,994 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:01:43,994 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:01:43,995 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:01:44,062 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:01:44,088 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:01:44,214 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:44,214 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:44,215 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1384120f9a20000 2012-07-01 08:01:44,215 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:01:44,267 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384120f98d0000, negotiated timeout = 5000 2012-07-01 08:01:44,268 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:01:44,268 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:01:44,325 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:01:44,326 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:01:44,326 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:01:44,327 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:01:44,366 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:01:54,019 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384120f98d0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:01:54,181 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:01:54,877 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:01:54,888 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:54,888 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:54,900 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:01:54,900 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384120f98d0000 has expired, closing socket connection 2012-07-01 08:01:54,901 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:01:54,901 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:01:54,901 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:01:54,955 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:01:54,971 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:01:54,971 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:01:55,003 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:01:55,029 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384120f9b60000, negotiated timeout = 5000 2012-07-01 08:01:55,055 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher java.lang.NullPointerException at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 08:01:55,069 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:01:55,080 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:01:55,092 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:01:55,092 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:01:55,092 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:01:55,102 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:01:55,185 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:01:55,185 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384120f98d0000 2012-07-01 08:09:53,134 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384120f9b60000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:09:53,356 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:09:53,356 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:09:56,771 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:09:56,791 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:09:56,791 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:09:56,791 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:09:56,898 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384120f9b60000 has expired, closing socket connection 2012-07-01 08:09:56,898 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:09:56,898 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:09:56,899 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:09:57,004 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:09:57,004 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:09:57,005 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:09:57,046 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384120f9b60000 2012-07-01 08:09:57,104 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:09:57,123 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:09:57,180 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1384128c95e0000, negotiated timeout = 5000 2012-07-01 08:09:57,180 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:09:57,181 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:09:57,245 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:09:57,245 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:09:57,245 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:09:57,260 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:09:57,310 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:10:10,021 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x1384128c95e0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:10:10,409 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:10:10,769 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:10:10,792 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:10:10,792 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:10:10,807 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:10:10,807 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x1384128c95e0000 has expired, closing socket connection 2012-07-01 08:10:10,808 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:10:10,808 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:10:10,808 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:10:10,851 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:10:10,876 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:10:10,876 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:10:10,891 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:10:11,063 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:10:11,063 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-109/10.196.68.136:2181, sessionid = 0x3384128dc680000, negotiated timeout = 5000 2012-07-01 08:10:11,064 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1384128c95e0000 2012-07-01 08:10:11,064 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:10:11,068 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:10:11,069 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:10:11,069 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:10:11,069 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:10:11,071 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:10:11,112 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:10:26,051 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x3384128dc680000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:10:26,490 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:10:27,324 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:10:27,339 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:10:27,339 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:10:27,339 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:10:27,343 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3384128dc680000 has expired, closing socket connection 2012-07-01 08:10:27,343 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:10:27,344 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:10:27,383 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:10:27,470 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:10:27,470 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:10:27,470 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:10:27,471 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:10:27,533 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384128c98b0000, negotiated timeout = 5000 2012-07-01 08:10:27,534 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher java.lang.NullPointerException at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 08:10:27,535 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x3384128dc680000 2012-07-01 08:10:27,535 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:10:27,545 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:10:27,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:10:27,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:10:27,546 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:10:27,547 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:10:27,610 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:24:13,805 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384128c98b0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:24:14,305 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:24:14,305 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:24:15,158 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:24:15,168 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:24:15,168 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:24:15,179 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:24:15,180 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384128c98b0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:24:17,448 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:24:17,479 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:24:17,479 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:24:17,511 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:24:17,586 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x2384128c98b0000, negotiated timeout = 5000 2012-07-01 08:24:17,586 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:24:17,587 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:24:17,611 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:24:17,625 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:24:17,625 INFO org.apache.hadoop.ha.ActiveStandbyElector: But old node has our own data, so don't need to fence it. 2012-07-01 08:24:17,625 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:24:17,655 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:24:17,720 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:24:24,030 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384128c98b0000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:24:24,394 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:24:27,247 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop-108/10.196.68.150:2181 2012-07-01 08:24:27,247 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:24:27,247 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:24:27,248 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 2855ms for sessionid 0x2384128c98b0000, closing socket connection and attempting reconnect 2012-07-01 08:24:27,384 FATAL org.apache.hadoop.ha.ActiveStandbyElector: Received create error from Zookeeper. code:CONNECTIONLOSS for path /hadoop-ha/events/ActiveStandbyElectorLock. Not retrying further znode create connection errors. 2012-07-01 08:24:29,680 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2384128c98b0000 closed 2012-07-01 08:24:29,680 FATAL org.apache.hadoop.ha.ZKFailoverController: Fatal error occurred:Received create error from Zookeeper. code:CONNECTIONLOSS for path /hadoop-ha/events/ActiveStandbyElectorLock. Not retrying further znode create connection errors. 2012-07-01 08:24:29,680 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384128c98b0000 2012-07-01 08:24:29,680 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:24:29,680 INFO org.apache.hadoop.ipc.Server: Stopping server on 8019 2012-07-01 08:24:29,790 INFO org.apache.hadoop.ha.ActiveStandbyElector: Yielding from election 2012-07-01 08:24:29,790 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8019 2012-07-01 08:24:29,790 INFO org.apache.hadoop.ha.HealthMonitor: Stopping HealthMonitor thread 2012-07-01 08:24:29,804 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2012-07-01 08:25:12,603 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Failover controller configured for NameNode events.namenode80 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-cdh4.0.0--1, built on 06/04/2012 23:16 GMT 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop-106 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_32 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-sun-1.6.0.32.x86_64/jre 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/json-simple-1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.0.0.jar:/usr/share/cmf/lib/plugins/event-publish-4.0.1-shaded.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/lib/avro-1.5.4.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.15.jar:/usr/lib/hadoop-hdfs/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/lib/jline-0.9.94.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/avro-1.5.4.jar:/usr/lib/hadoop-yarn/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/junit-4.8.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.15.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.5.4.jar:/usr/lib/hadoop-mapreduce/lib/jdiff-1.0.9.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.2.3.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.15.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.3.2.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/zookeeper-3.4.3-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.0.0-cdh4.0.0.jar 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2012-07-01 08:25:18,696 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler= 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-220.el6.x86_64 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hdfs 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/usr/lib/hadoop 2012-07-01 08:25:18,697 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/339-hdfs-FAILOVERCONTROLLER 2012-07-01 08:25:18,698 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:25:18,911 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:25:19,034 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:25:19,034 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:25:19,173 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:25:19,622 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384135ffe30000, negotiated timeout = 5000 2012-07-01 08:25:19,921 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:25:21,356 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8019 2012-07-01 08:25:22,575 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-07-01 08:25:22,708 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8019: starting 2012-07-01 08:25:26,004 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384135ffe30000, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:25:26,344 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:25:26,618 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.136:2181 2012-07-01 08:25:26,673 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:25:26,673 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:25:26,677 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-109/10.196.68.136:2181, initiating session 2012-07-01 08:25:26,678 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:25:26,678 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384135ffe30000 has expired, closing socket connection 2012-07-01 08:25:26,678 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:25:26,759 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:25:26,863 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:25:27,019 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:25:27,019 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:25:27,020 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:25:27,134 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384135ffe30001, negotiated timeout = 5000 2012-07-01 08:25:27,248 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:25:27,294 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:25:27,371 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:25:27,602 INFO org.apache.hadoop.ha.HealthMonitor: Entering state SERVICE_HEALTHY 2012-07-01 08:25:27,607 INFO org.apache.hadoop.ha.ZKFailoverController: Local service NameNode at hadoop-106/10.196.68.149:8020 entered state: SERVICE_HEALTHY 2012-07-01 08:25:27,850 INFO org.apache.hadoop.ha.ActiveStandbyElector: Old node exists: 0a066576656e7473120a6e616d656e6f646538301a206861646f6f702d3130362e616d73342e636f72702e626f6f6b696e672e636f6d20d43e28d33e 2012-07-01 08:25:28,068 INFO org.apache.hadoop.ha.ZKFailoverController: Should fence: NameNode at hadoop-106/10.196.68.149:8020 2012-07-01 08:25:28,574 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to standby state without fencing 2012-07-01 08:25:28,575 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/events/ActiveBreadCrumb to indicate that the local node is the most recent active... 2012-07-01 08:25:28,668 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at hadoop-106/10.196.68.149:8020 active... 2012-07-01 08:25:34,003 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2384135ffe30001, likely server has closed socket, closing socket connection and attempting reconnect 2012-07-01 08:25:34,577 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:25:34,587 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:25:34,587 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:25:34,587 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:25:34,650 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x2384135ffe30001 has expired, closing socket connection 2012-07-01 08:25:56,292 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at hadoop-106/10.196.68.149:8020 to active state 2012-07-01 08:25:56,369 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode... 2012-07-01 08:25:56,369 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session expired. Entering neutral mode and rejoining... 2012-07-01 08:25:56,369 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:25:56,441 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:25:56,543 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.149:2181 2012-07-01 08:25:56,574 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x2384135ffe30001 2012-07-01 08:25:56,574 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:25:56,736 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:25:56,737 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:25:56,737 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-106/10.196.68.149:2181, initiating session 2012-07-01 08:25:56,797 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-106/10.196.68.149:2181, sessionid = 0x1384135ffcb0000, negotiated timeout = 5000 2012-07-01 08:25:56,800 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:25:57,109 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:25:57,111 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:855) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:760) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 08:25:57,112 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:25:57,324 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1384135ffcb0000 closed 2012-07-01 08:26:00,062 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:26:00,131 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:26:00,198 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:26:00,198 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:26:00,311 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x1384135ffcb0000 2012-07-01 08:26:00,311 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:26:00,312 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:26:00,336 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384135ffe30002, negotiated timeout = 5000 2012-07-01 08:26:00,376 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:26:00,521 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:26:00,557 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:855) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:760) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 08:26:00,557 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:26:00,687 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2384135ffe30002 closed 2012-07-01 08:26:03,295 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-106:2181,hadoop-109:2181,hadoop-108:2181 sessionTimeout=5000 watcher=null 2012-07-01 08:26:03,433 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.196.68.150:2181 2012-07-01 08:26:03,465 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 2012-07-01 08:26:03,465 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 2012-07-01 08:26:03,603 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop-108/10.196.68.150:2181, initiating session 2012-07-01 08:26:03,610 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop-108/10.196.68.150:2181, sessionid = 0x2384135ffe30003, negotiated timeout = 5000 2012-07-01 08:26:03,678 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2012-07-01 08:26:03,774 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2012-07-01 08:26:03,844 INFO org.apache.hadoop.ha.ActiveStandbyElector: Checking for any old active which needs to be fenced... 2012-07-01 08:26:03,877 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:855) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:760) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:407) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) 2012-07-01 08:26:03,877 INFO org.apache.hadoop.ha.ActiveStandbyElector: Trying to re-establish ZK session 2012-07-01 08:26:03,998 INFO org.apache.zookeeper.ZooKeeper: Session: 0x2384135ffe30003 closed