Job history unavailable due to HDFS permissions check failure

Running CDH4.3.0/MRv2 on a physical cluster (not VMs and not in pseudo-distributed mode). OS is Ubuntu 12.04 LTS.

On the namenodes I have explicitly installed the packages:

• hadoop-hdfs-namenode
• hadoop-hdfs-journalnode

which, of course, grabs a few other packages to resolve dependencies. Importantly though, note hadoop-yarn and hadoop-mapreduce are not installed on the namenode.

The end result of this is the only Hadoop "system" account on the namenode is "hdfs". The "yarn" and "mapred" accounts, which are created by the hadoop-yarn and hadoop-mapreduce packages, respectively, do not exist on the namenode.

The namenode, then, will fail HDFS permissions checks for these accounts. This can manifest itself in several ways; the one I experienced is the job history server failing to read the history files. From the history server log:

2013-07-24 16:02:29,272 ERROR org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to scan the directory hdfs://hdpdev:8020/mr-history/intermediate/mreibert
org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=READ_EXECUTE, inode="/mr-history/intermediate/mreibert":mreibert:mapred:drwxrwx---

Note the user "mapred" (the history server) is unable to access a directory that is rwx to group "mapred". All because that permissions check is done on the namenode where the mapred account does not exist.

The solution is to ensure all the Hadoop "system" accounts exist on the namenode. These accounts could all be created in the hadoop-hdfs package, or even the base hadoop package.