Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.1
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      https://github.com/cloudera/hue/issues/244

      With a string or garbabe it boots then stay in running:
      
      curl -X POST --data '{"kind": "pyspark", "pyFiles": "hdfs://localhost:8020/user/romain/dwdwdwpyspark_csv.py"}' -H "Content-Type: application/json" localhost:8998/sessions
      
              {
                  "id": 3,
                  "kind": "pyspark",
                  "log": [
                      "15/10/08 17:56:56 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:56:57 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:56:58 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:56:59 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:00 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:01 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:02 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:03 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:04 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)",
                      "15/10/08 17:57:05 INFO yarn.Client: Application report for application_1444345227311_0020 (state: ACCEPTED)"
                  ],
                 
      curl localhost:8998/sessions/2/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}'
      java.lang.IllegalStateException: Session is in state starting
      
      
      
      I give a list as expected I get:
      
      curl -X POST --data '{"kind": "pyspark", "pyFiles": ["hdfs://localhost:8020/user/romaindwdwd/pyspark_csv.py"]}' -H "Content-Type: application/json" localhost:8998/sessions
      
      
              {
                  "id": 1,
                  "kind": "pyspark",
                  "log": [
                      "Error: --py-files given but primary resource is not a Python script",
                      "Run with --help for usage help or --verbose for debug output"
                  ],
                  "state": "error"
              },
      
      `spark-submit ... --py-files ...` is explicitly checking that the program it's executing ends with .py. Livy with pyspark actually is executing a jar file, hence this issue. We'll need to find another way to pass along the py-files.
      

        Attachments

          Activity

            People

            • Assignee:
              erickt Erick Tryzelaar
              Reporter:
              romain Romain Rigaux
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: