Uploaded image for project: 'Livy (READ-ONLY)'
  1. Livy (READ-ONLY)
  2. LIVY-200

Kyro deserialization fails when trying to return java.util.List on javaRDD.collect()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 0.3
    • Fix Version/s: None
    • Component/s: RSC
    • Labels:
      None

      Description

      Kyro deserialization fails when trying to return java.util.List when the collect API is called on a javaRDD

      CODE
      public class TestJob implements Job<List<Integer>> {

      public List<Integer> call(JobContext jobContext) throws Exception {
      Function<Integer, Integer> multiplyFunc = new Function<Integer, Integer>() {
      public Integer call(Integer v1) throws Exception

      { return v1 * v1; }

      };
      List<Integer> list = jobContext.sc().parallelize(Arrays.asList(1, 2, 3)).map(multiplyFunc).collect();
      return list;
      }
      }

      EXCEPTION STACK TRACE
      java.lang.NullPointerException
      at scala.collection.convert.Wrappers$IterableWrapperTrait$class.size(Wrappers.scala:23)
      at scala.collection.convert.Wrappers$SeqWrapper.size(Wrappers.scala:64)
      at java.util.AbstractList.add(AbstractList.java:108)
      at com.cloudera.livy.shaded.kryo.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
      at com.cloudera.livy.shaded.kryo.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
      at com.cloudera.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:776)
      at com.cloudera.livy.client.common.Serializer.deserialize(Serializer.java:63)
      at com.cloudera.livy.client.http.JobHandleImpl$JobPollTask.run(JobHandleImpl.java:239)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:

      WORK AROUND
      Populate the results of the collect() function onto a java.util.ArrayList

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              manikandan Manikandan Nagarajan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: