Uploaded image for project: 'Livy (READ-ONLY)'
  1. Livy (READ-ONLY)
  2. LIVY-321

LivyClient comes out NullPointException while return spark-sql result

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.3, 0.4
    • Fix Version/s: None
    • Component/s: API, RSC
    • Labels:
      None
    • Environment:

      spark version : 2.0.1
      livy: 0.4.0.SNAPSHOT
      hbase:1.2.1
      hadoop:2.7.3

      Description

      It seems the spark-sql job is successfully done in addition to the livy-server log and the spark-web-ui.
      but some problems just show up at the later procedure.
      the following is my spark-sql job implement:

      class SQLQueryJob(tableName: String, sql: String, schemas: JMap[String,String]) extends Job[JList[Row]] {
      
        override def call(jc: JobContext): JList[Row] = {
          val session:SparkSession = jc.sparkSession()
          implicit val hConfig = HbConfig()
          query(session, tableName, sql, schemasToStructType(schemas)).collectAsList()
        }
      
        private[this] def query(sparkSession: SparkSession,
                                tableName: String,
                                sql: String,
                                schemas: StructType)
                               (implicit config: HbConfig): DataFrame = {
          val rdd = sparkSession.readHbase(tableName, COLUMN_FAMILY, schemas)
          val df = sparkSession.createDataFrame(rdd, schemas)
          df.createOrReplaceTempView(tableName)
          sparkSession.sql(sql)
        }
      }
      

      and the following is my client call:

      List<Row> rows = livyClient.submit(job).get();
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              haiyun Alex
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: