Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.17.0
-
Fix Version/s: 0.17.1
-
Component/s: Data Module
-
Labels:None
Description
The stack trace below occurs when a Crunch-based job writes to a View instance in a dataset that uses a "provided" partition value. The job appears to be writing data to a temporary, task-specific dataset, and that task-specific dataset does not include the required view parameters. The result is the attempt to write is rejected with a "Cannot construct key, missing provided value" exception.
14/10/21 09:48:29 INFO run.RTNode: Crunch exception in 'Kite(dataset:hdfs://localhost:58957/var/folders/1c/qm_z1zxx52g92ptj0vyqjtxr0000gp/T/1413902908055-0/ns/out)' for input:
{"username": "test-0", "email": "email-0"}org.apache.crunch.CrunchRuntimeException: java.lang.IllegalArgumentException: Cannot construct key, missing provided value: version
at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:45)
at org.apache.crunch.MapFn.process(MapFn.java:34)
at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
at org.apache.crunch.MapFn.process(MapFn.java:34)
at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.IllegalArgumentException: Cannot construct key, missing provided value: version
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
at org.kitesdk.data.spi.EntityAccessor.partitionValue(EntityAccessor.java:118)
at org.kitesdk.data.spi.EntityAccessor.keyFor(EntityAccessor.java:103)
at org.kitesdk.data.spi.filesystem.PartitionedDatasetWriter.write(PartitionedDatasetWriter.java:107)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$DatasetRecordWriter.write(DatasetKeyOutputFormat.java:318)
at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$DatasetRecordWriter.write(DatasetKeyOutputFormat.java:295)
at org.apache.crunch.io.CrunchOutputs.write(CrunchOutputs.java:133)
at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:41)
... 17 more
14/10/21 09:48:29 INFO mapred.LocalJobRunner: Map task executor complete.
Attachments
Issue Links
- is duplicated by
-
KITE-993 DatasetKeyOutputFormat causes YARN AM Retries to fail
-
- Closed
-