Details
Major Description
In the InteractiveSession class at this line:
https://github.com/cloudera/livy/blob/a5f3cfc9ebf1a5b3caf9ac23583c105fcf49aa6a/server/src/main/scala/com/cloudera/livy/server/interactive/InteractiveSession.scala#L74
The master of the Client Builder is hard coded to be "yarn-cluster". This overrides any value set in config as described in a comment within the client builder:
/** * Creates a new builder that will optionally load the default Livy and Spark configuration * from the classpath. * * Livy client configuration is stored in a file called "livy-client.conf", and Spark client * configuration is stored in a file called "spark-defaults.conf", both in the root of the * application's classpath. Livy configuration takes precedence over Spark's (in case * configuration entries are duplicated), and configuration set in this builder object will * override the values in those files. */
Locally, with this line removed, I could start an interactive session (using the simple example from the project README) with master set to 'local' in my spark-defaults.conf.
It looks like an InteractiveSession would only work with YARN in the current version.
Attachments
Issue Links
- links to