Details

    • Type: Task
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Not A Bug
    • Affects Version/s: CDH4.5.0
    • Fix Version/s: None
    • Component/s: HDFS
    • Labels:
    • Environment:
      Ubuntu 12.04 64bit

      Description

      [bigdata1 mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar pi 5 10
      Number of Maps = 5
      Samples per Map = 10
      Wrote input for Map #0
      Wrote input for Map #1
      Wrote input for Map #2
      Wrote input for Map #3
      Wrote input for Map #4
      Starting Job
      14/03/19 21:37:54 INFO service.AbstractService: Service:smileysurprised:rg.apache.hadoop.yarn.client.YarnClientImpl is inited.
      14/03/19 21:37:54 INFO service.AbstractService: Service:smileysurprised:rg.apache.hadoop.yarn.client.YarnClientImpl is started.
      14/03/19 21:37:54 INFO input.FileInputFormat: Total input paths to process : 5
      14/03/19 21:37:54 INFO mapreduce.JobSubmitter: number of splits:5
      14/03/19 21:37:54 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
      14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
      14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      14/03/19 21:37:54 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
      14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
      14/03/19 21:37:54 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
      14/03/19 21:37:54 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
      14/03/19 21:37:54 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
      14/03/19 21:37:54 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
      14/03/19 21:37:54 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
      14/03/19 21:37:54 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
      14/03/19 21:37:54 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
      14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      14/03/19 21:37:54 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
      14/03/19 21:37:54 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
      14/03/19 21:37:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1395234357046_0007
      14/03/19 21:37:55 INFO client.YarnClientImpl: Submitted application application_1395234357046_0007 to ResourceManager at bigdata1/192.168.65.66:8032
      14/03/19 21:37:55 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1395234357046_0007/
      14/03/19 21:37:55 INFO mapreduce.Job: Running job: job_1395234357046_0007
      14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 running in uber mode : false
      14/03/19 21:38:09 INFO mapreduce.Job: map 0% reduce 0%
      14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 failed with state FAILED due to: Application application_1395234357046_0007 failed 1 times due to AM Container for appattempt_1395234357046_0007_000001 exited with exitCode: 1 due to:
      .Failing this attempt.. Failing the application.
      14/03/19 21:38:09 INFO mapreduce.Job: Counters: 0
      Job Finished in 15.584 seconds
      java.io.FileNotFoundException: File does not exist: hdfs://bigdata1:8020/user/root/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:824)
      at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
      at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
      at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
      at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

      I found the resourcemanager never read configuration from yarn-site.xml and mapred-site.xml,here is my configturation:
      mapred-site.xml:
      <?xml version="1.0"?>
      <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
      <!-- Put site-specific property overrides in this file. -->
      <configuration>
      <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
      </property>
      <property>
      <name>mapreduce.jobhistory.address</name>
      <value>bigdata1:10020</value>
      </property>
      <property>
      <name>mapreduce.jobhistory.webapp.address</name>
      <value>bigdata1:19888</value>
      </property>
      <property>
      <name>mapreduce.shuffle.port</name>
      <value>8070</value>
      </property>
      </configuration>

      yarn-site.xml:

      <?xml version="1.0"?>
      <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
      <configuration>
      <property>
      <name>yarn.resourcemanager.resource-tracker.address</name>
      <value>bigdata1:8031</value>
      </property>
      <property>
      <name>yarn.resourcemanager.address</name>
      <value>bigdata1:8032</value>
      </property>
      <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>bigdata1:8030</value>
      </property>
      <property>
      <name>yarn.resourcemanager.admin.address</name>
      <value>bigdata1:8033</value>
      </property>
      <property>
      <name>yarn.resourcemanager.webapp.address</name>
      <value>bigdata1:8088</value>
      </property>
      <property>
      <description>Classpath for typical applications.</description>
      <name>yarn.application.classpath</name>
      <value>$HADOOP_CONF_DIR,$YARN_HOME/,$YARN_HOME/lib/,$YARN_HOME/share/hadoop/common/,$YARN_HOME/share/hadoop/yarn,$YARN_HOME/share/hadoop/yarn/lib/,$YARN_HOME/share/hadoop/mapreduce/,$YARN_HOME/share/hadoop/mapreduce/lib/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/,$YARN_HOME/share/hadoop/common/lib/*</value>
      </property>
      <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce.shuffle</value>
      </property>
      <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
      <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
      </property>
      <property>
      <name>yarn.dispatcher.exit-on-error</name>
      <value>true</value>
      </property>
      <property>
      <name>yarn.nodemanager.local-dirs</name>

      <value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/${user.name}/nm-local-dir</value>
      </property>

      <property>
      <name>yarn.nodemanager.log-dirs</name>
      <value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
      </property>
      <property>
      <description>Where to aggregate logs</description>
      <name>yarn.nodemanager.remote-app-log-dir</name>
      <value>/var/log/hadoop-yarn/app</value>
      </property>
      <property>
      <name>yarn.app.mapreduce.am.staging-dir</name>
      <value>/user</value>
      </property>

      </configuration>

      But the impala webserver shows like the attachment.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              huangxing huangxing
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: