Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-647

Class anonfun$1 not found exception / sbt 13.x / sbt 12.4 / Scala 2.10.4

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: CDH 5.1.0
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:

      Description

      I have Scala code:
      val master = "spark://<server address>:7077"
      val sc = new SparkContext(new SparkConf()
      .setMaster(master)
      .setAppName("SparkQueryDemo 01")
      .set("spark.executor.memory", "512m"))
      val count2 = sc .textFile("hdfs://<server address>:8020/tmp/data/risk/account.txt")
      .filter(line => line.contains("Word"))
      .count()
      I've got such an error:
      [error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:0 failed 4 times, most
      recent failure: Exception failure in TID 6 on host <server address>: java.lang.ClassNotFoundExcept
      ion: SimpleApp$$anonfun$1
      My dependencies :
      object Version

      { val spark = "1.0.0-cdh5.1.0" }

      object Library

      { val sparkCore = "org.apache.spark" % "spark-assembly_2.10" % Version.spark }

      My OS is Win 7, sbt 13.5, Scala 2.10.4. I've tried sbt 13.5, 13.6, 12.4.

      If I change code to:
      val count2 = sc .textFile("hdfs://<server address>:8020/tmp/data/risk/account.txt")
      //.filter(line => line.contains("Word")) it seems it can't work with anonymous function
      .count()
      Without filter function with lambda inside, it does work!

      I have also reported to Spark Jira, but seems it's not related to them:
      https://issues.apache.org/jira/browse/SPARK-3761

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              legart Igor Tkachenko
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: