Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Feedback Received
-
Affects Version/s: CDH4.5.0
-
Fix Version/s: None
-
Component/s: HDFS
-
Labels:None
-
Environment:$ hadoop version
Hadoop 2.0.0-cdh4.5.0
Subversion git://localhost/data/1/jenkins/workspace/generic-package-debian64-6-0/CDH4.5.0-Packaging-Hadoop-2013-11-20_14-31-53/hadoop-2.0.0+1518-1.cdh4.5.0.p0.24~squeeze/src/hadoop-common-project/hadoop-common -r 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Wed Nov 20 15:09:45 PST 2013
From source with checksum 9848b0f85b461913ed63fa19c2b79ccc
This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.5.0.jar
$ pig -version
Apache Pig version 0.11.0-cdh4.5.0 (rexported)
compiled Nov 20 2013, 16:05:49
$ hadoop version Hadoop 2.0.0-cdh4.5.0 Subversion git://localhost/data/1/jenkins/workspace/generic-package-debian64-6-0/CDH4.5.0-Packaging-Hadoop-2013-11-20_14-31-53/hadoop-2.0.0+1518-1.cdh4.5.0.p0.24~squeeze/src/hadoop-common-project/hadoop-common -r 8e266e052e423af592871e2dfe09d54c03f6a0e8 Compiled by jenkins on Wed Nov 20 15:09:45 PST 2013 From source with checksum 9848b0f85b461913ed63fa19c2b79ccc This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.5.0.jar $ pig -version Apache Pig version 0.11.0-cdh4.5.0 (rexported) compiled Nov 20 2013, 16:05:49
Description
Hi!
I recently upgraded a cluster to CDH 4.5.0 and noticed that after the upgrade, our pig script that could pull data in from s3 in the past, now just hangs and prints a bunch of these error messages:
2013-12-15 02:14:09,197 [Finalizer] WARN com.cloudera.org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream - Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation.
2013-12-15 02:14:09,216 [Finalizer] WARN com.cloudera.org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream - Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation.
2013-12-15 02:14:09,233 [Finalizer] WARN com.cloudera.org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream - Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation.
There is a similar problem mentioned here that seems related: http://community.cloudera.com/t5/Storage-Random-Access-HDFS/from-CDH-4-4-0-1-cdh4-4-0-p0-39-to-CDH-4-5-0-1-cdh4-5-0-p0-30/td-p/3869
I am still using the fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey in core-site.xml that worked before the upgrade. Interestingly, things worked fine when I was still running pig 0.9.2 against the CDH 4.5.0 cluster, upgrading pig to 0.11 that comes with CDH 4.5.0, however, and things stopped working.
Any info would be great, thanks!