Uploaded image for project: 'Livy (READ-ONLY)'
  1. Livy (READ-ONLY)
  2. LIVY-133

For interactive sessions, spark.master is hardcoded

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.2
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Centos Linux, Standalone Spark 1.5.1, Livy master branch as of 28-04-2016.

      Description

      In the InteractiveSession class at this line:
      https://github.com/cloudera/livy/blob/a5f3cfc9ebf1a5b3caf9ac23583c105fcf49aa6a/server/src/main/scala/com/cloudera/livy/server/interactive/InteractiveSession.scala#L74

      The master of the Client Builder is hard coded to be "yarn-cluster". This overrides any value set in config as described in a comment within the client builder:

        /**
         * Creates a new builder that will optionally load the default Livy and Spark configuration
         * from the classpath.
         *
         * Livy client configuration is stored in a file called "livy-client.conf", and Spark client
         * configuration is stored in a file called "spark-defaults.conf", both in the root of the
         * application's classpath. Livy configuration takes precedence over Spark's (in case
         * configuration entries are duplicated), and configuration set in this builder object will
         * override the values in those files.
         */
      

      Locally, with this line removed, I could start an interactive session (using the simple example from the project README) with master set to 'local' in my spark-defaults.conf.

      It looks like an InteractiveSession would only work with YARN in the current version.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                mattinbits Matthew Livesey
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: