Details
Description
I have Scala code:
val master = "spark://<server address>:7077"
val sc = new SparkContext(new SparkConf()
.setMaster(master)
.setAppName("SparkQueryDemo 01")
.set("spark.executor.memory", "512m"))
val count2 = sc .textFile("hdfs://<server address>:8020/tmp/data/risk/account.txt")
.filter(line => line.contains("Word"))
.count()
I've got such an error:
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:0 failed 4 times, most
recent failure: Exception failure in TID 6 on host <server address>: java.lang.ClassNotFoundExcept
ion: SimpleApp$$anonfun$1
My dependencies :
object Version
object Library
{ val sparkCore = "org.apache.spark" % "spark-assembly_2.10" % Version.spark }My OS is Win 7, sbt 13.5, Scala 2.10.4. I've tried sbt 13.5, 13.6, 12.4.
If I change code to:
val count2 = sc .textFile("hdfs://<server address>:8020/tmp/data/risk/account.txt")
//.filter(line => line.contains("Word")) it seems it can't work with anonymous function
.count()
Without filter function with lambda inside, it does work!
I have also reported to Spark Jira, but seems it's not related to them:
https://issues.apache.org/jira/browse/SPARK-3761