Details
-
Type: Task
-
Status: Resolved
-
Priority: Trivial
-
Resolution: Not A Bug
-
Affects Version/s: CDH4.5.0
-
Fix Version/s: None
-
Component/s: HDFS
-
Labels:
-
Environment:Ubuntu 12.04 64bit
Description
[bigdata1 mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar pi 5 10
Number of Maps = 5
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/03/19 21:37:54 INFO service.AbstractService: Service:smileysurprised:rg.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/19 21:37:54 INFO service.AbstractService: Service:smileysurprised:rg.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/19 21:37:54 INFO input.FileInputFormat: Total input paths to process : 5
14/03/19 21:37:54 INFO mapreduce.JobSubmitter: number of splits:5
14/03/19 21:37:54 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/19 21:37:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1395234357046_0007
14/03/19 21:37:55 INFO client.YarnClientImpl: Submitted application application_1395234357046_0007 to ResourceManager at bigdata1/192.168.65.66:8032
14/03/19 21:37:55 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1395234357046_0007/
14/03/19 21:37:55 INFO mapreduce.Job: Running job: job_1395234357046_0007
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 running in uber mode : false
14/03/19 21:38:09 INFO mapreduce.Job: map 0% reduce 0%
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 failed with state FAILED due to: Application application_1395234357046_0007 failed 1 times due to AM Container for appattempt_1395234357046_0007_000001 exited with exitCode: 1 due to:
.Failing this attempt.. Failing the application.
14/03/19 21:38:09 INFO mapreduce.Job: Counters: 0
Job Finished in 15.584 seconds
java.io.FileNotFoundException: File does not exist: hdfs://bigdata1:8020/user/root/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:824)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
I found the resourcemanager never read configuration from yarn-site.xml and mapred-site.xml,here is my configturation:
mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata1:19888</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>8070</value>
</property>
</configuration>
yarn-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>bigdata1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>bigdata1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>bigdata1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>bigdata1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>bigdata1:8088</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$YARN_HOME/,$YARN_HOME/lib/,$YARN_HOME/share/hadoop/common/,$YARN_HOME/share/hadoop/yarn,$YARN_HOME/share/hadoop/yarn/lib/,$YARN_HOME/share/hadoop/mapreduce/,$YARN_HOME/share/hadoop/mapreduce/lib/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/,$YARN_HOME/share/hadoop/common/lib/*</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/${user.name}/nm-local-dir</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/app</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
</configuration>
But the impala webserver shows like the attachment.