Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Not A Bug
-
Affects Version/s: 0.3
-
Fix Version/s: None
-
Component/s: RSC
-
Labels:None
Description
Kyro deserialization fails when trying to return java.util.List when the collect API is called on a javaRDD
CODE
public class TestJob implements Job<List<Integer>> {
public List<Integer> call(JobContext jobContext) throws Exception {
Function<Integer, Integer> multiplyFunc = new Function<Integer, Integer>() {
public Integer call(Integer v1) throws Exception
};
List<Integer> list = jobContext.sc().parallelize(Arrays.asList(1, 2, 3)).map(multiplyFunc).collect();
return list;
}
}
EXCEPTION STACK TRACE
java.lang.NullPointerException
at scala.collection.convert.Wrappers$IterableWrapperTrait$class.size(Wrappers.scala:23)
at scala.collection.convert.Wrappers$SeqWrapper.size(Wrappers.scala:64)
at java.util.AbstractList.add(AbstractList.java:108)
at com.cloudera.livy.shaded.kryo.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at com.cloudera.livy.shaded.kryo.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.cloudera.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:776)
at com.cloudera.livy.client.common.Serializer.deserialize(Serializer.java:63)
at com.cloudera.livy.client.http.JobHandleImpl$JobPollTask.run(JobHandleImpl.java:239)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
WORK AROUND
Populate the results of the collect() function onto a java.util.ArrayList