Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-735

Crunch-based MapReduce fails when writing to a View with provided values

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.1
    • Component/s: Data Module
    • Labels:
      None

      Description

      The stack trace below occurs when a Crunch-based job writes to a View instance in a dataset that uses a "provided" partition value. The job appears to be writing data to a temporary, task-specific dataset, and that task-specific dataset does not include the required view parameters. The result is the attempt to write is rejected with a "Cannot construct key, missing provided value" exception.

      14/10/21 09:48:29 INFO run.RTNode: Crunch exception in 'Kite(dataset:hdfs://localhost:58957/var/folders/1c/qm_z1zxx52g92ptj0vyqjtxr0000gp/T/1413902908055-0/ns/out)' for input:

      {"username": "test-0", "email": "email-0"}

      org.apache.crunch.CrunchRuntimeException: java.lang.IllegalArgumentException: Cannot construct key, missing provided value: version
      at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:45)
      at org.apache.crunch.MapFn.process(MapFn.java:34)
      at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
      at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
      at org.apache.crunch.MapFn.process(MapFn.java:34)
      at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
      at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
      at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
      at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      at java.lang.Thread.run(Thread.java:695)
      Caused by: java.lang.IllegalArgumentException: Cannot construct key, missing provided value: version
      at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
      at org.kitesdk.data.spi.EntityAccessor.partitionValue(EntityAccessor.java:118)
      at org.kitesdk.data.spi.EntityAccessor.keyFor(EntityAccessor.java:103)
      at org.kitesdk.data.spi.filesystem.PartitionedDatasetWriter.write(PartitionedDatasetWriter.java:107)
      at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$DatasetRecordWriter.write(DatasetKeyOutputFormat.java:318)
      at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$DatasetRecordWriter.write(DatasetKeyOutputFormat.java:295)
      at org.apache.crunch.io.CrunchOutputs.write(CrunchOutputs.java:133)
      at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:41)
      ... 17 more
      14/10/21 09:48:29 INFO mapred.LocalJobRunner: Map task executor complete.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                blue Ryan Blue
                Reporter:
                rbrush Ryan Brush
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: