Details
Description
When Livy launches Spark in cluster mode, the spark-submit process used to launch Spark hangs around in the background, consuming resources and printing lots of things to the log.
Instead, we should let the process go away since it's not useful anymore after the app is started. That's the spark.yarn.submit.waitAppCompletion config option (default = true), but we also need to make sure the RSC (and batch sessions) won't detect that as an error.
Attachments
Issue Links
- links to