Uploaded image for project: 'Hue (READ ONLY)'
  1. Hue (READ ONLY)
  2. HUE-662

[config] Each Hadoop hdfs/mr/yarn cluster to define its own env

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: hue2
    • Fix Version/s: 2.0.0-beta
    • Component/s: core.api
    • Labels:
      None
    • Target Version:

      Description

      Currently, there is a global hadoop_home, hadoop_bin and hadoop_conf_dir under the [hadoop] section. There isn't enough information here for MR1 and MR2. First, to list the places where Hue depends on hadoop_home and friends:

      • Starting beeswax with the right hadoop_bin
      • Starting shells (e.g. pig)
      • Running tests – We start NN/DN/JT/TT as child processes to have a real cluster to test against.

      The solution is to let each hdfs, mr and yarn cluster define its own environment.

      • HDFS will have hadoop_hdfs_home, hadoop_bin and hadoop_conf_dir.
      • MR1 and MR2 will have hadoop_mapred_home, hadoop_bin and hadoop_conf_dir.

      The mr/yarn clusters have an additional submit_to config, which should only be enabled on one cluster. When Hue launches beeswax or shells, it'll find out the cluster to submit to, and prepare the $HADOOP_MAPRED_HOME and other setups for the child.

        Attachments

          Activity

            People

            • Assignee:
              bcwalrus bc Wong
              Reporter:
              bcwalrus bc Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: