Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Cannot Reproduce
-
Affects Version/s: CDH 5.0.1
-
Fix Version/s: None
-
Component/s: HDFS
-
Labels:None
-
Environment:Centos 6.5.
8 Worker nodes, 8 HDD per node, 128 GB RAM, dual six CPU.
Description
When we read HDFS file using java API, and do significant delays between data accesses, next read after delay sometimes takes dozens of seconds. Problem reproduced inside MR jobs as well as in standalone java application.
So set of operations to reproduce the problem:
1. Open big file
2. read 20K lines
3. sleep for 60 seconds
4. Read next line. This read sometimes (about 50% of cases) takes a lot of times - dozens of seconds.
Code snap which shows the problem
while (line != null) { long tlineStart = System.currentTimeMillis(); line = br.readLine(); long tlineEnd = System.currentTimeMillis(); long tlineDelta = tlineEnd - tlineStart; if (tlineDelta > MAX_READ_TIME_20){ // THIS PRINT INDICATES VERY LONG DELAY IN READ System.err.println("Read time is " + (tlineDelta / 1000)); } if (countOfLines % 200000 == 0){ Thread.sleep(1000 * 60 * 2); } }
attached is minimal program reproducing the problem (above is snap from it).
We did many tests and found that:
The problem does not reproduced on CDH 4.6
The problem does happens when we read file which sits on the co-located datanode.
The problem is independent on compression codecs.