Details
Description
Currently RSC driver URI is gotten when PingJob is finished, which means only when SparkContext and other Spark entry point are fully started and job are finished we could get the RSC driver URI. This unnecessarily prolong the intermediate state of recovery metadata, if LivyServer is failed at this time period, it will never talk the RSC application again.
So we should shorten the time to get RSC driver URI to mitigate the intermediate state of recovery metadata, that will make session recovery more robust.