Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: CDH 5.3.0
-
Fix Version/s: None
-
Component/s: HDFS
-
Labels:None
-
Environment:mini dfs cluster with 3 data nodes.
Description
I've tried and failed to copy large files ( > 300mb ) to s3 using the hdfs client. I've tried -cp, -put, distcpy, and all return with the same error. Note that in the mini dfs environment, we use the LocalJobRunner.
I've successfully copied files of ~100mb. I'm uncertain whether the problem is in my setup, or an artifact of our s3 connectors. I have not yet found evidence of this being done successfully, but maybe I'm wrong.
[jenkins@ishaan-impala-ec2 tpcds.store_sales_unpartitioned]$ time aws s3 cp store_sales.dat s3://ishaan-impala/
upload: ./store_sales.dat to s3://ishaan-impala/store_sales.dat
real 0m9.860s
user 0m5.910s
sys 0m1.082s
[jenkins@ishaan-impala-ec2 tpcds.store_sales_unpartitioned]$ aws s3 rm s3://ishaan-impala/store_sales.dat
delete: s3://ishaan-impala/store_sales.dat
hdfs dfs -du -s -h /test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat
370.5 M 1.1 G /test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat
[jenkins@ishaan-impala-ec2 tpcds.store_sales_unpartitioned]$ hadoop distcp /test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat s3a://ishaan-impala/
2015-01-28 14:53:13,718 INFO [main] tools.DistCp (DistCp.java:run(112)) - Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat], targetPath=s3a://ishaan-impala/, targetPathExists=true, preserveRawXattrs=false}
2015-01-28 14:53:13,733 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1060)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2015-01-28 14:53:13,734 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2015-01-28 14:53:14,421 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1060)) - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
2015-01-28 14:53:14,421 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1060)) - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
2015-01-28 14:53:14,456 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-01-28 14:53:14,491 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-01-28 14:53:14,589 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1159523254_0001
2015-01-28 14:53:14,644 WARN [main] conf.Configuration (Configuration.java:loadProperty(2551)) - file:/tmp/hadoop-jenkins/mapred/staging/jenkins1159523254/.staging/job_local1159523254_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-01-28 14:53:14,650 WARN [main] conf.Configuration (Configuration.java:loadProperty(2551)) - file:/tmp/hadoop-jenkins/mapred/staging/jenkins1159523254/.staging/job_local1159523254_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-01-28 14:53:14,766 WARN [main] conf.Configuration (Configuration.java:loadProperty(2551)) - file:/tmp/hadoop-jenkins/mapred/local/localRunner/jenkins/job_local1159523254_0001/job_local1159523254_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-01-28 14:53:14,768 WARN [main] conf.Configuration (Configuration.java:loadProperty(2551)) - file:/tmp/hadoop-jenkins/mapred/local/localRunner/jenkins/job_local1159523254_0001/job_local1159523254_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-01-28 14:53:14,775 INFO [main] mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-01-28 14:53:14,775 INFO [main] tools.DistCp (DistCp.java:execute(173)) - DistCp job-id: job_local1159523254_0001
2015-01-28 14:53:14,776 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1344)) - Running job: job_local1159523254_0001
2015-01-28 14:53:14,777 INFO [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-01-28 14:53:14,785 INFO [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.tools.mapred.CopyCommitter
2015-01-28 14:53:14,820 INFO [Thread-5] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-01-28 14:53:14,820 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1159523254_0001_m_000000_0
2015-01-28 14:53:14,854 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2015-01-28 14:53:14,856 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(753)) - Processing split: file:/tmp/hadoop-jenkins/mapred/staging/jenkins688878082/.staging/_distcp259455119/fileList.seq:0+271
2015-01-28 14:53:14,915 INFO [LocalJobRunner Map Task Executor #0] mapred.CopyMapper (CopyMapper.java:map(209)) - Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat
2015-01-28 14:53:15,022 INFO [LocalJobRunner Map Task Executor #0] mapred.RetriableFileCopyCommand (RetriableFileCopyCommand.java:getTmpFile(233)) - Creating temp file: s3a://ishaan-impala/.distcp.tmp.attempt_local1159523254_0001_m_000000_0
2015-01-28 14:53:15,780 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1365)) - Job job_local1159523254_0001 running in uber mode : false
2015-01-28 14:53:15,785 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 0% reduce 0%
2015-01-28 14:53:20,849 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:21,789 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1372)) - map 100% reduce 0%
2015-01-28 14:53:23,850 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:26,851 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:29,852 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:32,853 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:34,434 INFO [LocalJobRunner Map Task Executor #0] s3a.S3AFileSystem (S3AFileSystem.java:rename(286)) - Rename path s3a://ishaan-impala/.distcp.tmp.attempt_local1159523254_0001_m_000000_0 to s3a://ishaan-impala/store_sales.dat
2015-01-28 14:53:35,854 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:38,855 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 100.0% Copying hdfs://localhost:20500/test-warehouse/tpcds.store_sales_unpartitioned/store_sales.dat to s3a://ishaan-impala/store_sales.dat [370.5M/370.5M] > map
2015-01-28 14:53:39,528 INFO [s3-transfer-manager-worker-1] http.AmazonHttpClient (AmazonHttpClient.java:executeHelper(448)) - Unable to execute HTTP request: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at com.cloudera.org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at com.cloudera.org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at com.cloudera.org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at com.cloudera.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at com.cloudera.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at com.cloudera.org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at com.cloudera.org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at com.cloudera.org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at com.cloudera.org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at com.cloudera.org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at com.cloudera.com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at com.cloudera.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at com.cloudera.org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at com.cloudera.org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at com.cloudera.org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at com.cloudera.org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
at com.cloudera.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507)
at com.cloudera.com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143)
at com.cloudera.com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131)
at com.cloudera.com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189)
at com.cloudera.com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134)
at com.cloudera.com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)