Details
Description
To pass Livy-specific options to the Spark driver, the RSC stashes them in the Spark configuration with the spark._livy_ prefix. The issue is that those show up in the "Environment" tab of the Spark UI and get written down to Spark event logs; so things like the secret used for authentication are visible.
We need to remove those from the Spark config to be able to run Livy with security enabled.