Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-524

mapred.local.dir overwritten automatically by Task Tracker

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: CDH4.3.0
    • Fix Version/s: None
    • Component/s: MapReduce
    • Labels:
      None
    • Environment:
      Deployment: Blades on SAN
      DN blades each have a 133 GB local disk on the blade with 2 TB SAN as storage attached.

      Description

      Version: Hadoop 2.0.0-cdh4.3.0

      First I mounted mapred.local.dir (/opt/var/mapred/local/1) on local disk. The disk soon got filled up.
      After I softlink /opt/var/mapred/local/1 to SAN, everything ran fine for a day. Later, somehow the softlink was removed and Task Tracker process created a directory again as /opt/var/mapred/local/1 on local disk again by itself.

      Quesiton:
      1. What is disk quota requirement for mapred.local.dir?
      2. How often mapred.local.dir get a cleanup?
      2. Does Task Tracker automatically switch mapred.local.dir to local disk?

      FYI - local disk is used now. I did create softlink 1 -> /opt/var/hadoop/dfs/dn/1/mapred.tmp, but the soft link was removed.
      $ pwd
      /opt/var/mapred/local/1
      $ ls -al
      total 20
      drwxrwxr-x. 5 mapred hadoop 4096 Sep 15 23:02 .
      drwxrwxr-x. 3 mapred hadoop 4096 Sep 15 05:48 ..
      drwxr-xr-x. 5 mapred mapred 4096 Sep 16 04:39 taskTracker
      drwxr-xr-x. 3 mapred mapred 4096 Sep 15 23:02 ttprivate
      drwxr-xr-x. 20 mapred mapred 4096 Sep 16 05:49 userlogs

      SAN location for mapred.local.dir (softlinked)
      $ pwd
      /opt/var/hadoop/dfs/dn/1/mapred.tmp
      $ ls -al
      total 28
      drwxr-xr-x. 7 mapred hadoop 4096 Sep 13 23:44 .
      drwxrwxr-x. 5 hdfs hdfs 4096 Sep 13 23:12 ..
      drwxr-xr-x. 3 mapred mapred 4096 Sep 14 22:53 taskTracker
      drwxr-xr-x. 2 mapred mapred 4096 Sep 13 23:44 toBeDeleted
      drwxr-xr-x. 2 mapred mapred 4096 Sep 13 23:44 tt_log_tmp
      drwx------. 3 mapred mapred 4096 Sep 14 22:53 ttprivate
      drwxr-xr-x. 9 mapred mapred 4096 Sep 14 23:04 userlogs

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wayne2chicago Wayne
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated: