[spark] Cannot submit archives/files/jars with pySpark session

From the Hue notebooks in Hue 3.11, it is not possible to submit archives, files, or jars with the pySpark session. We are using Livy 0.2 from https://github.com/cloudera/livy

If I choose one of the archive/file/jar properties, add a file from HDFS, and then recreate the session, Livy throws an error:

"com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of scala.collection.immutable.List out of VALUE_STRING token\n at [Source: HttpInputOverHTTP@7cb46d37; line: 1, column: 114] (through reference chain: com.cloudera.livy.server.interactive.CreateInteractiveRequest[\"archives\"])" (error 500)

Livy is expecting a List but that is not what is sent by Hue.

In March, there was a commit to Hue that changed Hue's internally shipped Livy source to accept Option[String] instead of List[String]: https://github.com/cloudera/hue/commit/61ff7871a50588d9fc7db317cde67a1af2d8e557#diff-593d94f080a4e28edbe83539aa1eee5bL16

However, that change was not carried over to the Livy at https://github.com/cloudera/livy and now as of Hue 3.11, the Hue source no longer includes Livy (it must be sourced from the separate github project). Furthermore, making the change in the commit listed above to try and make Livy 0.2 compatible with Hue 3.11 is not possible anymore as the underlying architecture in Livy has changed (for example the InteractiveSessionFactory no longer exists).

We must use Livy 0.2 as it is the version that ships with security.

This issue was also present in Hue 3.10 if you used it with Livy 0.2.

Screenshot and logs attached.