Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-679

Imbalanced disk scheduling with CDH5 hdfs

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: CDH 5.3.0
    • Fix Version/s: None
    • Component/s: HBase
    • Labels:
    • Environment:
      CentOS7-64

      Description

      I am working on HBase perf now and suffering from an odd problem: huge perf degradation in hdfs write with CDH5 deployment compared to Apache 1.2.1. Despite the same configurations of Hbase and hdfs… obvious disk imbalance is found with CDH5 deployment as below. My dfs.data.dir uses 8 disks - /mnt/sd

      {g..n}

      1

      ///////////////////////////CDH5 - Hbase 0.98.6 + Hadoop 2.5///////////////////////

      [root@perf2-worker-5 z-cdh5]# du -hs /mnt/sd*
      0 /mnt/sdb1
      336K /mnt/sdc1
      32K /mnt/sdd1
      1.1M /mnt/sde1
      40K /mnt/sdf1
      104K /mnt/sdg1
      96K /mnt/sdh1
      96K /mnt/sdi1
      96K /mnt/sdj1
      96K /mnt/sdk1
      88K /mnt/sdl1
      88K /mnt/sdm1
      108K /mnt/sdn1

      After execute hbase org.apache.hadoop.hbase.PerformanceEvaluation --presplit=8 --nomapred sequentialWrite 10

      [root@perf2-worker-5 z-cdh5]# du -hs /mnt/sd*
      0 /mnt/sdb1
      712K /mnt/sdc1
      32K /mnt/sdd1
      1.1M /mnt/sde1
      40K /mnt/sdf1
      1.3G /mnt/sdg1
      2.1G /mnt/sdh1
      1.4G /mnt/sdi1
      2.0G /mnt/sdj1
      1.3G /mnt/sdk1
      12G /mnt/sdl1
      1.5G /mnt/sdm1
      1.2G /mnt/sdn1

      ///////////////////////////Hbase 0.98.4 + Apache Hadoop 1.2.1///////////////////////

      [root@hbase-node1 z]# du -hs /mnt/sd*
      232K /mnt/sdc1
      0 /mnt/sdcg
      36K /mnt/sdd1
      1.1M /mnt/sde1
      48K /mnt/sdf1
      80K /mnt/sdg1
      72K /mnt/sdh1
      72K /mnt/sdi1
      72K /mnt/sdj1
      72K /mnt/sdk1
      72K /mnt/sdl1
      72K /mnt/sdm1
      84K /mnt/sdn1

      After execute hbase org.apache.hadoop.hbase.PerformanceEvaluation --presplit=8 --nomapred sequentialWrite 10

      [root@hbase-node1 z]# du -hs /mnt/sd*
      1.5M /mnt/sdc1
      0 /mnt/sdcg
      44K /mnt/sdd1
      1.1M /mnt/sde1
      48K /mnt/sdf1
      3.6G /mnt/sdg1
      3.8G /mnt/sdh1
      3.6G /mnt/sdi1
      3.6G /mnt/sdj1
      3.6G /mnt/sdk1
      3.9G /mnt/sdl1
      3.3G /mnt/sdm1
      4.0G /mnt/sdn1

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xinhui xinhui li
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: