Sqoop Saved Incremental Job running through oozie fails

Hi,

This is an ugent request.

Sqoop Saved incremental Job running through oozie succeeds 1st run (Base Line Data)
Sqoop Saved incremental Job running through oozie fails 2nd run with below error ( Incremental Data)
see the workflow.xml

This needs to be fixed even at the apache sqoop project, as well as cloudera sqoop

WE HAVE COMPLETELY BLOCKED ON THIS SCENARIO, NEED TO GIVE DEMO TO CLIENT ABOUT THE CLOUDERA PRODUCT, BASED ON THIS CLIENT WILL BUY CLOUDERA PRODUCT

Error

[cdhadmin@hadoop-cdh oozie]$ sqoop job --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop --show mergetest | grep "db.required.password"
15/06/08 20:49:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.4.0
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: It seems that you have launched a Sqoop metastore job via
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: Oozie with sqoop.metastore.client.record.password disabled.
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: But this configuration is not supported because Sqoop can't
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: prompt the user to enter the password while being executed
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: as Oozie tasks. Please enable sqoop.metastore.client.record
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: .password in sqoop-site.xml, or provide the password
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: explicitly using --password in the command tag of the Oozie
15/06/08 20:49:32 ERROR sqoop.SqoopOptions: workflow file.

DB scripts

mysql> create table mergetest (id int not null primary key, val int, cdate timestamp);

BaselineData (before oozie 1st run)
-----------------------------------
mysql> insert into mergetest values (1, 10, NOW());
mysql> insert into mergetest values (2, 20, NOW());
mysql> insert into mergetest values (3, 30, NOW());
select * from myapp.mergetest;

Incremental Data (before oozie 2nd run)
-----------------------------------

update myapp.mergetest set val=201 where id=2;
update myapp.mergetest set val=301 where id=3;
insert into mergetest values (5, 50, NOW());
select * from myapp.mergetest;

1) Followed link ==> https://issues.apache.org/jira/browse/SQOOP-453

/etc/sqoop/conf/sqoop-site.xml

<property>
<name>sqoop.metastore.client.record.password</name>
<value>true</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://localhost:16000/sqoop</value>
</property>
<property>
<name>sqoop.metastore.server.location</name> <value>/tmp/sqoop-metastore/shared.db</value> </property>
<property> <name>sqoop.metastore.server.port</name> <value>16000</value> </property>

start metastore in the background
sqoop metastore &

2) sqoop job --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop \
--create mergetest \
– import --connect jdbc:mysql://10.109.51.3/myapp \
--table mergetest \
--incremental lastmodified \
--check-column cdate \
--last-value 0 \
--username root \
--password root1pass \
--m 1

3) job.properties
...
...
baseLineData=${nameNode}/user/${user.name}/${examplesRoot}/BaseData
incLineData=${nameNode}/user/${user.name}/${examplesRoot}/IncrementalData
mergedData=${nameNode}/user/${user.name}/${examplesRoot}/MergedData

4) workflow.xml

<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.4" name="cs-wf-fork-join">
<start to="control-node" />

<decision name="control-node">
<switch>
<case to="base-sqoop-import">
${fs:exists(baseLineData) eq "false"}
</case>
<case to="inc-sqoop-import">
${fs:exists(incLineData) eq "false"}
</case>
<default to="merge-data-node"/>
</switch>
</decision>

<action name="base-sqoop-import">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${oozieImportPath}"/>
</prepare>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
</configuration>
<arg>job</arg>
<arg>--meta-connect</arg>
<arg>jdbc:hsqldb:hsql://localhost:16000/sqoop</arg>
<arg>--exec</arg>
<arg>mergetest</arg>
<arg>--</arg>
<arg>--target-dir</arg>
<arg>${baseLineData}</arg>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>

<action name="inc-sqoop-import">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${oozieImportPath}"/>
</prepare>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
</configuration>
<arg>job</arg>
<arg>--meta-connect</arg>
<arg>jdbc:hsqldb:hsql://localhost:16000/sqoop</arg>
<arg>--exec</arg>
<arg>mergetest</arg>
<arg>--</arg>
<arg>--target-dir</arg>
<arg>${incLineData}</arg>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>

<action name="merge-data-node">
<fs />
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

5) export OOZIE_URL=http://localhost:11000/oozie
$ oozie job -config job.properties -run