Uploaded image for project: 'Livy (READ-ONLY)'
  1. Livy (READ-ONLY)
  2. LIVY-65

Clean up server dependencies, deployed packages, package names

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.2
    • Component/s: API
    • Labels:
      None

      Description

      There are some problems with the current package structure in Livy that we should clean up. I'll pile all of them in the same task since it's all modifications to the POM files, so better to do them all at once.

      • Livy should not have Spark jars in the final package

      Livy server should not depend on any Spark classes to run (with the eventual exception of the "spark-launcher" library). It doesn't run Spark code, it just starts Spark applications, which doesn't require anything Spark in the classpath.

      The modules that actually depend on Spark (like the REPL or client-local) are Spark applications, so they are run with spark-submit (or the equivalent), and so don't need Spark jars either.

      • Only "public" artifacts should be deployed.

      Deploy here means "upload to maven central (or equivalent repository) when the time arrives". Only a handful of packages are user-facing: things like client-http, client-local (useful for testing) and, potentially, repl, although I'm not sold on that one.

      All others should not be uploaded anywhere. They're packaged in the final Livy server zip and that's all that's needed.

      • The Scala version for package names is not needed.

      The version of Scala the server is compiled against shouldn't matter, since no one is linking against the server. client-http is pure java. client-local depends on Spark, but it depends on Spark's Java bindings, so the version of Scala there hopefully won't matter. So we should be able to remove "_2.10" from all artifact names.

      I know this might be a little painful if you got used to typing it, but I'm getting kinda tired of typing "_2.10" all the time.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                vanzin Marcelo Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: