Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 0.1
-
Fix Version/s: None
-
Component/s: Core
-
Labels:None
Description
https://github.com/cloudera/hue/issues/244
With a string or garbabe it boots then stay in running: curl -X POST --data '{"kind": "pyspark", "pyFiles": "hdfs://localhost:8020/user/romain/dwdwdwpyspark_csv.py"}' -H "Content-Type: application/json" localhost:8998/sessions { "id": 3, "kind": "pyspark", "log": [ "15/10/08 17:56:56 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:56:57 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:56:58 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:56:59 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:00 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:01 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:02 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:03 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:04 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)", "15/10/08 17:57:05 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)" ], curl localhost:8998/sessions/2/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' java.lang.IllegalStateException: Session is in state starting I give a list as expected I get: curl -X POST --data '{"kind": "pyspark", "pyFiles": ["hdfs://localhost:8020/user/romaindwdwd/pyspark_csv.py"]}' -H "Content-Type: application/json" localhost:8998/sessions { "id": 1, "kind": "pyspark", "log": [ "Error: --py-files given but primary resource is not a Python script", "Run with --help for usage help or --verbose for debug output" ], "state": "error" },
`spark-submit ... --py-files ...` is explicitly checking that the program it's executing ends with .py. Livy with pyspark actually is executing a jar file, hence this issue. We'll need to find another way to pass along the py-files.