Details
-
Type:
Improvement
-
Status: Open
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 0.3
-
Fix Version/s: None
-
Component/s: REPL
-
Labels:None
Description
Current implementation of statement cancelling (refer from Zeppelin SparkInterpreter) is not so elegant and threading contention prone:
1. Using statement to set JobGroup is slow and ugly. Actually this could be done by SparkListener (tracking the mapping between statement and spark jobs). We should consider this with LIVY-272 to offer a sophisticated StatementStateListener to track the progress of statement as well as statement cancelling.
2. Current statement cancelling can only be worked with Spark jobs. If the statement doesn't include any spark job, the statement cannot be cancelled. We should think of way to handle it.
3. Current statement cancelling manages complex state transition and threading content. We should refactor to simplify or make the code easy to understand.
4. If we are still relying on setJobGroupId, move it to Interpreter interface.