Details
-
Type: Bug
-
Status: Resolved
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: 1.1.0
-
Fix Version/s: 1.2.0
-
Component/s: Command-line Interface
-
Labels:None
Description
When I run kite-dataset from the tarball bundle, it fails when importing local data using the LocalJobRunner with a VerifyError:
blue@work:~/tmp$ kite-dataset json-import movies.json dataset:file:data/movies_avro/ Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar Exception in thread "Thread-6" java.lang.VerifyError: Bad return type Exception Details: Location: org/apache/hadoop/mapred/LocalJobRunner.submitJob(Lorg/apache/hadoop/mapreduce/JobID;Ljava/lang/String;Lorg/apache/hadoop/security/Credentials;)Lorg/apache/hadoop/mapreduce/JobStatus; @29: areturn Reason: Type 'org/apache/hadoop/mapred/JobStatus' (current frame, stack[0]) is not assignable to 'org/apache/hadoop/mapreduce/JobStatus' (from method signature) Current Frame: bci: @29 flags: { } locals: { 'org/apache/hadoop/mapred/LocalJobRunner', 'org/apache/hadoop/mapreduce/JobID', 'java/lang/String', 'org/apache/hadoop/security/Credentials', 'org/apache/hadoop/mapred/LocalJobRunner$Job' } stack: { 'org/apache/hadoop/mapred/JobStatus' } Bytecode: 0000000: bb00 2259 2a2b b800 232c b700 243a 0419 0000010: 04b8 0025 2db6 0026 1904 b800 27b0 at org.apache.hadoop.mapred.JobClient.init(JobClient.java:548) at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:520) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:595) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapreduce.Job.connect(Job.java:593) at org.apache.hadoop.mapreduce.Job.submit(Job.java:581) at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:329) at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:204) at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:238) at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:112) at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55) at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:83) at java.lang.Thread.run(Thread.java:745) Unknown error: null 2015-06-25 15:13:33 PDT
It looks like the problem is that both mr1 and yarn jars are present and the classes are getting mixed together. The work-around is to remove the mr1 jars from the lib directory.
The mr1 jars are being pulled in transitively through HBase, so we need to add exclusions for hadoop-core. It should be provided by the kite-hadoop-dependencies packages anyway.
Attachments
Issue Links
- links to