Details
-
Type:
Task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: hue2
-
Fix Version/s: 2.0.0-beta
-
Component/s: core.api
-
Labels:None
-
Target Version:
Description
Currently, there is a global hadoop_home, hadoop_bin and hadoop_conf_dir under the [hadoop] section. There isn't enough information here for MR1 and MR2. First, to list the places where Hue depends on hadoop_home and friends:
- Starting beeswax with the right hadoop_bin
- Starting shells (e.g. pig)
- Running tests – We start NN/DN/JT/TT as child processes to have a real cluster to test against.
The solution is to let each hdfs, mr and yarn cluster define its own environment.
- HDFS will have hadoop_hdfs_home, hadoop_bin and hadoop_conf_dir.
- MR1 and MR2 will have hadoop_mapred_home, hadoop_bin and hadoop_conf_dir.
The mr/yarn clusters have an additional submit_to config, which should only be enabled on one cluster. When Hue launches beeswax or shells, it'll find out the cluster to submit to, and prepare the $HADOOP_MAPRED_HOME and other setups for the child.