Livy pyspark session returns before the computation is completed.

Working with Livy on the Hortonworks Sandbox we found the problem that on some cases the Livy API provide a result for the statement before the actual work has ended as indicated by the Spark monitoring tool. In some cases Livy provide a result in just less than 5 seconds for a job with normal execution time of more than 10 minutes. If we repeat the same task with the same initial conditions more than once on some cases we have seen this problem happen.

If you need anything else from me to duplicate this bug don't hesitate into asking.