Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-601

very slow HDFS file read when read is resumed after delay.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: CDH 5.0.1
    • Fix Version/s: None
    • Component/s: HDFS
    • Labels:
      None
    • Environment:
      Centos 6.5.
      8 Worker nodes, 8 HDD per node, 128 GB RAM, dual six CPU.

      Description

      When we read HDFS file using java API, and do significant delays between data accesses, next read after delay sometimes takes dozens of seconds. Problem reproduced inside MR jobs as well as in standalone java application.
      So set of operations to reproduce the problem:
      1. Open big file
      2. read 20K lines
      3. sleep for 60 seconds
      4. Read next line. This read sometimes (about 50% of cases) takes a lot of times - dozens of seconds.

      Code snap which shows the problem
      while (line != null) {
                   long tlineStart = System.currentTimeMillis();
                  line = br.readLine();
                  long tlineEnd = System.currentTimeMillis();
                  long tlineDelta = tlineEnd - tlineStart;
       
                  if (tlineDelta > MAX_READ_TIME_20){ // THIS PRINT INDICATES VERY LONG DELAY IN READ
                      System.err.println("Read time is " + (tlineDelta / 1000));
                  }
       
                  if (countOfLines % 200000 == 0){
                      Thread.sleep(1000 * 60 * 2);
                  }
              }
       
      

      attached is minimal program reproducing the problem (above is snap from it).

      We did many tests and found that:
      The problem does not reproduced on CDH 4.6
      The problem does happens when we read file which sits on the co-located datanode.
      The problem is independent on compression codecs.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              davidgruzman David Gruzman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: