Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.17.1
-
Fix Version/s: 0.18.0
-
Component/s: None
-
Labels:None
Description
On a 5.1 VM, I downloaded the 0.17.1 CLI bundle (the executable jar) and tried to copy from an Avro dataset to a Parquet dataset. The job failed with:
2014-12-29 10:27:12,377 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47) at org.kitesdk.data.spi.filesystem.FileSystemViewKeyInputFormat.createUnfilteredRecordReader(FileSystemViewKeyInputFormat.java:148) at org.kitesdk.data.spi.filesystem.FileSystemViewKeyInputFormat.createRecordReader(FileSystemViewKeyInputFormat.java:132) at org.kitesdk.data.mapreduce.DatasetKeyInputFormat.createRecordReader(DatasetKeyInputFormat.java:241) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:492) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:735) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
The problem is a binary incompatibility between avro-mapred and the Hadoop version on the cluster. Avro mapred is included in the jar, while the Hadoop libraries are not. The dependency tree reports that the included version isn't the hadoop2 avro-mapred.
Attachments
Issue Links
- blocks
-
-
- Open
-