Uploaded image for project: 'Hue (READ ONLY)'
  1. Hue (READ ONLY)
  2. HUE-6017

Notebooks don't close actively creating spark sessions when moving away from the Notebook app

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 3.11.0, 3.12.0
    • Fix Version/s: None
    • Component/s: con.spark
    • Labels:
      None

      Description

      If you create a pySpark snippet in a Notebook, and it is not yet to the idle/running state, and you then navigate away from the Notebooks app to another app, Hue will not attempt to close the session that is still being created. That session will then be "leaked" from Hue and basically become a zombie session in Livy--Hue won't be tracking it anymore. Leaked sessions can confused the user in the Job Browser and also consume resources unnecessarily on the cluster.

      If you put a breakpoint in editor_components.mako at the window.onbeforeunload(), you can see that the onbeforeunload correctly fires in this case and it does actually call close() on the selected Notebook.

      However, if you put logging statements in the notebook's api.py in close_notebook(), you can see that while in this case there is a Notebook, there is an empty set of sessions associated with the Notebook at this point. Which is kind of incorrect--the Notebook doesn't have any idle/running sessions, but it does have a session "in flight", not yet in the running/idle state. This session is not yet found in this Notebook object, and so close() is never called on the pending session.

      Somehow, the pending session needs to be tracked/found in the notebook api.py close_notebook(), and have a close() requested of it.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rickbernotas Rick Bernotas
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: