Details
Major Description
A new HDFS/Yarn node was added to an existing cluster that only runs HDFS/Yarn. The other services have been removed. HDFS on the new node starts fine. Yarn does not. The logs do not indicate why. Even TRACE logging does not offer additional info as to the failure state compared with INFO. The Yarn instance does register with RM, then immediately fails afterward.
DN Yarn:
[root@nyc-stats-hadoop-11 conf]$ tail -f /var/log/hadoop-yarn/hadoop-cmf-yarn1-NODEMANAGER-nyc-stats-hadoop-11.int.peer1.squarespace.net.log.out ... 8 more 2013-05-23 17:56:21,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Public cache exiting 2013-05-23 17:56:21,988 INFO org.apache.hadoop.ipc.Server: Stopping server on 8041 2013-05-23 17:56:21,988 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system... 2013-05-23 17:56:21,988 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped. 2013-05-23 17:56:21,988 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete. 2013-05-23 17:56:21,988 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at nyc-stats-hadoop-11.int.peer1.squarespace.net/192.168.1.197 ************************************************************/ 2013-05-23 17:58:52,962 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NodeManager STARTUP_MSG: host = nyc-stats-hadoop-11.int.peer1.squarespace.net/192.168.1.197 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.0.0-cdh4.2.1 STARTUP_MSG: classpath = /var/run/cloudera-scm-agent/process/742-yarn-NODEMANAGER:/var/run/cloudera-scm-agent/process/742-yarn-NODEMANAGER:/var/run/cloudera-scm-agent/process/742-yarn-NODEMANAGER:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/hue-plugins-2.2.0-cdh4.2.1.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh4.2.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/avro-1.7.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/.//hadoop-annotations-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.2.1-tests.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-auth-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.5-cdh4.2.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.2.1-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.3.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.2.1-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.1-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.0.0-cdh4.2.1-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.0.0-cdh4.2.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.2.4.Final.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.8.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.3.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.8.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/var/run/cloudera-scm-agent/process/742-yarn-NODEMANAGER/nm-config/log4j.properties STARTUP_MSG: build = file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.2.1/src/hadoop-common-project/hadoop-common -r 144bd548d481c2774fab2bec2ac2645d190f705b; compiled by 'jenkins' on Mon Apr 22 10:26:03 PDT 2013 STARTUP_MSG: java = 1.7.0_13 ************************************************************/ 2013-05-23 17:58:53,450 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: setsid exited with exit code 0 2013-05-23 17:58:53,557 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher 2013-05-23 17:58:53,558 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher 2013-05-23 17:58:53,558 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService 2013-05-23 17:58:53,558 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices 2013-05-23 17:58:53,559 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl 2013-05-23 17:58:53,559 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher 2013-05-23 17:58:53,569 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl 2013-05-23 17:58:53,596 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2013-05-23 17:58:53,653 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2013-05-23 17:58:53,654 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started 2013-05-23 17:58:53,659 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.DeletionService is inited. 2013-05-23 17:58:53,662 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService is inited. 2013-05-23 17:58:53,665 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /dfs/yarn/nm error directory is not writable: /dfs/yarn/nm, removing from the list of valid directories. 2013-05-23 17:58:53,665 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/yarn/nm; 2013-05-23 17:58:53,665 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/yarn/nm; 2013-05-23 17:58:53,665 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService is inited. 2013-05-23 17:58:53,665 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl is inited. 2013-05-23 17:58:53,666 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler 2013-05-23 17:58:53,748 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker 2013-05-23 17:58:53,748 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker is inited. 2013-05-23 17:58:53,748 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService is inited. 2013-05-23 17:58:53,749 INFO org.apache.hadoop.yarn.service.AbstractService: Service:containers-launcher is inited. 2013-05-23 17:58:53,884 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce.shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config. 2013-05-23 17:58:53,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding auxiliary service httpshuffle, "mapreduce.shuffle" 2013-05-23 17:58:53,903 INFO org.apache.hadoop.yarn.service.AbstractService: Service:httpshuffle is inited. 2013-05-23 17:58:53,903 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices is inited. 2013-05-23 17:58:53,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@72b02f 2013-05-23 17:58:53,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:containers-monitor is inited. 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited. 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler is inited. 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl is inited. 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer is inited. 2013-05-23 17:58:53,911 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited. 2013-05-23 17:58:53,914 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl is inited. 2013-05-23 17:58:53,914 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeManager is inited. 2013-05-23 17:58:53,915 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.DeletionService is started. 2013-05-23 17:58:53,915 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService is started. 2013-05-23 17:58:53,916 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService is started. 2013-05-23 17:58:53,916 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl is started. 2013-05-23 17:58:53,949 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8041 2013-05-23 17:58:53,963 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ContainerManagerPB to the server 2013-05-23 17:58:53,964 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2013-05-23 17:58:53,965 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8041: starting 2013-05-23 17:58:53,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager started at nyc-stats-hadoop-11.int.peer1.squarespace.net/192.168.1.197:8041 2013-05-23 17:58:53,982 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8040 2013-05-23 17:58:53,984 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server 2013-05-23 17:58:53,985 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2013-05-23 17:58:53,985 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8040: starting 2013-05-23 17:58:53,988 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer started on port 8040 2013-05-23 17:58:53,988 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker is started. 2013-05-23 17:58:53,988 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService is started. 2013-05-23 17:58:53,988 INFO org.apache.hadoop.yarn.service.AbstractService: Service:containers-launcher is started. 2013-05-23 17:58:53,998 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760 2013-05-23 17:58:54,014 INFO org.apache.hadoop.mapred.ShuffleHandler: httpshuffle listening on port 8080 2013-05-23 17:58:54,014 INFO org.apache.hadoop.yarn.service.AbstractService: Service:httpshuffle is started. 2013-05-23 17:58:54,014 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices is started. 2013-05-23 17:58:54,015 INFO org.apache.hadoop.yarn.service.AbstractService: Service:containers-monitor is started. 2013-05-23 17:58:54,015 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is started. 2013-05-23 17:58:54,015 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler is started. 2013-05-23 17:58:54,015 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl is started. 2013-05-23 17:58:54,015 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at nyc-stats-hadoop-11.int.peer1.squarespace.net:8042 2013-05-23 17:58:54,041 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2013-05-23 17:58:54,079 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2013-05-23 17:58:54,081 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context node 2013-05-23 17:58:54,081 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2013-05-23 17:58:54,081 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2013-05-23 17:58:54,083 INFO org.apache.hadoop.http.HttpServer: adding path spec: /node 2013-05-23 17:58:54,084 INFO org.apache.hadoop.http.HttpServer: adding path spec: /node/* 2013-05-23 17:58:54,084 INFO org.apache.hadoop.http.HttpServer: adding path spec: /ws 2013-05-23 17:58:54,084 INFO org.apache.hadoop.http.HttpServer: adding path spec: /ws/* 2013-05-23 17:58:54,084 INFO org.apache.hadoop.http.HttpServer: Added global filter 'guice' (class=com.google.inject.servlet.GuiceFilter) 2013-05-23 17:58:54,086 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 8042 2013-05-23 17:58:54,086 INFO org.mortbay.log: jetty-6.1.26.cloudera.2 2013-05-23 17:58:54,105 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.0.0-cdh4.2.1.jar!/webapps/node to /tmp/Jetty_nyc.stats.hadoop.11_int_peer1_squarespace_net_8042_node____.xt3owp/webapp 2013-05-23 17:58:54,310 INFO org.mortbay.log: Started SelectChannelConnector@nyc-stats-hadoop-11.int.peer1.squarespace.net:8042 2013-05-23 17:58:54,311 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 8042 2013-05-23 17:58:54,576 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2013-05-23 17:58:54,577 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer is started. 2013-05-23 17:58:54,577 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is started. 2013-05-23 17:58:54,597 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Connecting to ResourceManager at nyc-stats-hadoop-zookeeper-1.int.peer1.squarespace.net/192.168.1.242:8031 2013-05-23 17:58:54,638 ERROR org.apache.hadoop.yarn.service.CompositeService: Error starting services org.apache.hadoop.yarn.server.nodemanager.NodeManager org.apache.avro.AvroRuntimeException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:141) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:176) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:274) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:291) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:61) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:190) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:137) ... 4 more Caused by: com.google.protobuf.ServiceException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length. at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) at sun.proxy.$Proxy24.registerNodeManager(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59) ... 6 more Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length. at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49) at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:754) at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:294) at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484) at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438) at org.apache.hadoop.yarn.proto.YarnServerCommonProtos$MasterKeyProto$Builder.mergeFrom(YarnServerCommonProtos.java:1676) at org.apache.hadoop.yarn.proto.YarnServerCommonProtos$MasterKeyProto$Builder.mergeFrom(YarnServerCommonProtos.java:1549) at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:275) at org.apache.hadoop.yarn.proto.YarnServerCommonProtos$RegistrationResponseProto$Builder.mergeFrom(YarnServerCommonProtos.java:2142) at org.apache.hadoop.yarn.proto.YarnServerCommonProtos$RegistrationResponseProto$Builder.mergeFrom(YarnServerCommonProtos.java:1993) at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:275) at org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos$RegisterNodeManagerResponseProto$Builder.mergeFrom(YarnServerCommonServiceProtos.java:1032) at org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos$RegisterNodeManagerResponseProto$Builder.mergeFrom(YarnServerCommonServiceProtos.java:892) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:716) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:709) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ... 8 more 2013-05-23 17:58:54,640 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is stopped. 2013-05-23 17:58:54,642 INFO org.mortbay.log: Stopped SelectChannelConnector@nyc-stats-hadoop-11.int.peer1.squarespace.net:8042 2013-05-23 17:58:54,743 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer is stopped. 2013-05-23 17:58:54,743 INFO org.apache.hadoop.ipc.Server: Stopping server on 8041 2013-05-23 17:58:54,744 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8041 2013-05-23 17:58:54,744 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2013-05-23 17:58:54,744 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.
RM Yarn
2013-05-21 21:01:12,904 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved nyc-stats-hadoop-11.int.peer1.squarespace.net to /default-rack 2013-05-21 21:01:12,904 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node nyc-stats-hadoop-11.int.peer1.squarespace.net(cmPort: 8041 httpPort: 8042) registered with capability: 61440, assigned nodeId nyc-stats-hadoop-11.int.peer1.squarespace.net:8041 2013-05-21 21:01:12,904 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: nyc-stats-hadoop-11.int.peer1.squarespace.net:8041 Node Transitioned from NEW to RUNNING