Details
Description
When we try to run sqoop import with --table argument, we get java.io.IOException: SQLException in nextKeyValue error
sqoop import \
-connect $conn \
--username $user \
--password '*****' \
--table $tablename \
--split-by $primary_key \
--target-dir $output_path
But when it is run with --query argument, it works fine
import_sql="select t.* from $tablename as t where \$CONDITIONS with ur"
sqoop import \
-connect $conn \
--username $user \
--password '*****' \
--query "${import_sql}" \
--split-by $primary_key \
--target-dir $output_path
We eventually want to run import with incremental argument and it is also failing with SQLException in nextKeyValue error
Sqoop Incremental import doesn't work with the query argument and it needs the table argument. So how do we go about it?
This is what we have observed:
The SQL that gets generated is different in both cases
When run with --table
13/08/16 00:01:52 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(ID), MAX(ID) FROM schema.table
When run with --query
13/08/16 00:08:40 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(ID), MAX(ID) FROM (select t.* from schema.table as t where (1 = 1) with ur) AS t1
Detailed log:
13/08/16 00:02:17 INFO mapred.JobClient: Task Id : attempt_201308110330_17886_m_000001_2, Status : FAILED
java.io.IOException: SQLException in nextKeyValue
at com.cloudera.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:251)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at com.cloudera.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:189)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: com.ibm.db2.jcc.a.bn: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=schema;ROM schema.table AS;JOIN <joi