Uploaded image for project: 'Hue'
  1. Hue
  2. HUE-7126

[oozie] Improve Spark action to pick up the hive-site.xml easier

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0
    • Fix Version/s: None
    • Component/s: connector.oozie
    • Labels:
      None

      Description

      Please see how to run a SparkSQL application from Oozie/HUE:

      1) Upload your Spark jar to HDFS
      2) Upload your hive-site.xml file to the Workflow's workspace lib directory (/user/hue/oozie/workspaces/<workspace_dir>/lib)
      You can look up your workspace directory by clicking on the Settings (gear) icon on the workflow editor page
      3) Add your Spark Action to your workflow and set the following properties:
      Spark Master: yarn
      Mode: cluster
      App name: <application name>
      Jars/py files: <hdfs path to the jar file>
      Main class: <main class>
      4) Click on the little 3gear icon on the top-right corner of the Spark action and set the following properties:
      Options list: --files hive-site.xml

        Attachments

          Activity

            People

            • Assignee:
              yingc Ying Chen
              Reporter:
              yingc Ying Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: