Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-1024

Kite tools tarball fails when loading LocalJobRunner

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: Command-line Interface
    • Labels:
      None

      Description

      When I run kite-dataset from the tarball bundle, it fails when importing local data using the LocalJobRunner with a VerifyError:

      blue@work:~/tmp$ kite-dataset json-import movies.json dataset:file:data/movies_avro/
      Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
      Exception in thread "Thread-6" java.lang.VerifyError: Bad return type
      Exception Details:
        Location:
          org/apache/hadoop/mapred/LocalJobRunner.submitJob(Lorg/apache/hadoop/mapreduce/JobID;Ljava/lang/String;Lorg/apache/hadoop/security/Credentials;)Lorg/apache/hadoop/mapreduce/JobStatus; @29: areturn
        Reason:
          Type 'org/apache/hadoop/mapred/JobStatus' (current frame, stack[0]) is not assignable to 'org/apache/hadoop/mapreduce/JobStatus' (from method signature)
        Current Frame:
          bci: @29
          flags: { }
          locals: { 'org/apache/hadoop/mapred/LocalJobRunner', 'org/apache/hadoop/mapreduce/JobID', 'java/lang/String', 'org/apache/hadoop/security/Credentials', 'org/apache/hadoop/mapred/LocalJobRunner$Job' }
          stack: { 'org/apache/hadoop/mapred/JobStatus' }
        Bytecode:
          0000000: bb00 2259 2a2b b800 232c b700 243a 0419
          0000010: 04b8 0025 2db6 0026 1904 b800 27b0     
      
              at org.apache.hadoop.mapred.JobClient.init(JobClient.java:548)
              at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:520)
              at org.apache.hadoop.mapreduce.Job$1.run(Job.java:595)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
              at org.apache.hadoop.mapreduce.Job.connect(Job.java:593)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:581)
              at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:329)
              at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:204)
              at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:238)
              at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:112)
              at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
              at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:83)
              at java.lang.Thread.run(Thread.java:745)
      Unknown error: null
      2015-06-25 15:13:33 PDT
      

      It looks like the problem is that both mr1 and yarn jars are present and the classes are getting mixed together. The work-around is to remove the mr1 jars from the lib directory.

      The mr1 jars are being pulled in transitively through HBase, so we need to add exclusions for hadoop-core. It should be provided by the kite-hadoop-dependencies packages anyway.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                blue Ryan Blue
                Reporter:
                blue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: