Uploaded image for project: 'Livy (READ-ONLY)'
  1. Livy (READ-ONLY)
  2. LIVY-102

Don't leave spark-submit launchers lying around

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.2
    • Component/s: Core, RSC
    • Labels:
      None

      Description

      When Livy launches Spark in cluster mode, the spark-submit process used to launch Spark hangs around in the background, consuming resources and printing lots of things to the log.

      Instead, we should let the process go away since it's not useful anymore after the app is started. That's the spark.yarn.submit.waitAppCompletion config option (default = true), but we also need to make sure the RSC (and batch sessions) won't detect that as an error.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                vanzin Marcelo Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: