Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-516

Error - Sqoop import works with free form query but not --table

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: CDH3u3
    • Fix Version/s: None
    • Component/s: Sqoop
    • Labels:
    • Environment:
      Linux

      Description

      When we try to run sqoop import with --table argument, we get java.io.IOException: SQLException in nextKeyValue error
      sqoop import \
      -connect $conn \
      --username $user \
      --password '*****' \
      --table $tablename \
      --split-by $primary_key \
      --target-dir $output_path

      But when it is run with --query argument, it works fine
      import_sql="select t.* from $tablename as t where \$CONDITIONS with ur"
      sqoop import \
      -connect $conn \
      --username $user \
      --password '*****' \
      --query "${import_sql}" \
      --split-by $primary_key \
      --target-dir $output_path

      We eventually want to run import with incremental argument and it is also failing with SQLException in nextKeyValue error
      Sqoop Incremental import doesn't work with the query argument and it needs the table argument. So how do we go about it?

      This is what we have observed:
      The SQL that gets generated is different in both cases
      When run with --table
      13/08/16 00:01:52 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(ID), MAX(ID) FROM schema.table

      When run with --query
      13/08/16 00:08:40 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(ID), MAX(ID) FROM (select t.* from schema.table as t where (1 = 1) with ur) AS t1

      Detailed log:
      13/08/16 00:02:17 INFO mapred.JobClient: Task Id : attempt_201308110330_17886_m_000001_2, Status : FAILED
      java.io.IOException: SQLException in nextKeyValue
      at com.cloudera.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:251)
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
      at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
      at com.cloudera.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:189)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
      at org.apache.hadoop.mapred.Child.main(Child.java:264)
      Caused by: com.ibm.db2.jcc.a.bn: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=schema;ROM schema.table AS;JOIN <joi

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ishita Ishita Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: