Uploaded image for project: 'Sqoop (READ-ONLY)'
  1. Sqoop (READ-ONLY)
  2. SQOOP-214

with default Sqoop connector, running into IO Error: Connection reset

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.4.0
    • Component/s: docs
    • Labels:

      Description

      When using the default Sqoop connector for Oracle, some data does get transferred, but during the map-reduce job, seeing lots of errors as below:

      attempt_201105261333_0002_m_000001_0: log4j:WARN Not allowed to write to a closed appender.
      11/05/26 16:23:47 INFO mapred.JobClient: Task Id : attempt_201105261333_0002_m_000002_0, Status : FAILED
      java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLRecoverableException: IO Error: Connection reset
      at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
      at org.apache.hadoop.mapred.Child.main(Child.java:262)
      Caused by: java.lang.RuntimeException: java.sql.SQLRecoverableException: IO Error: Connection reset
      at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190)
      at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159)
      ... 9 more
      Caused by: java.sql.SQLRecoverableException: IO Error: Connection reset
      at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:428)
      at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:536)
      at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:228)
      at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
      at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
      at java.sql.DriverManager.getConnection(DriverManager.java:582)
      at java.sql.DriverManager.getConnection(DriverManager.java:185)
      at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152)
      at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184)
      ... 10 more
      Caused by: java.net.SocketException: Connection reset
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at oracle.net.ns.DataPacket.send(DataPacket.java:199)
      at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:211)
      at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:227)
      at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
      at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
      at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
      at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
      at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
      at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
      at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
      at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:366)
      at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:752)
      at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:366)
      ... 18 more

      At the end:

      11/05/26 16:27:51 INFO mapred.JobClient: Job complete: job_201105261333_0002
      11/05/26 16:27:51 INFO mapred.JobClient: Counters: 13
      11/05/26 16:27:51 INFO mapred.JobClient: Job Counters
      11/05/26 16:27:51 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=790364
      11/05/26 16:27:51 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
      11/05/26 16:27:51 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
      11/05/26 16:27:51 INFO mapred.JobClient: Launched map tasks=10
      11/05/26 16:27:51 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
      11/05/26 16:27:51 INFO mapred.JobClient: Failed map tasks=1
      11/05/26 16:27:51 INFO mapred.JobClient: FileSystemCounters
      11/05/26 16:27:51 INFO mapred.JobClient: HDFS_BYTES_READ=222
      11/05/26 16:27:51 INFO mapred.JobClient: FILE_BYTES_WRITTEN=108922
      11/05/26 16:27:51 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=462
      11/05/26 16:27:51 INFO mapred.JobClient: Map-Reduce Framework
      11/05/26 16:27:51 INFO mapred.JobClient: Map input records=8
      11/05/26 16:27:51 INFO mapred.JobClient: Spilled Records=0
      11/05/26 16:27:51 INFO mapred.JobClient: Map output records=8
      11/05/26 16:27:51 INFO mapred.JobClient: SPLIT_RAW_BYTES=222
      11/05/26 16:27:51 INFO mapreduce.ImportJobBase: Transferred 462 bytes in 379.7796 seconds (1.2165 bytes/sec)
      11/05/26 16:27:51 INFO mapreduce.ImportJobBase: Retrieved 8 records.
      11/05/26 16:27:51 ERROR tool.ImportTool: Error during import: Import job failed!

      As Paul Socha pointed out, the workaround is to change the setup for your application, by adding this parameter to the java command:

      -Djava.security.egd=file:/dev/../dev/urandom

      This should be documented in the Sqoop User Guide.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kate Kathleen Ting
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: