Details
Description
When incremental imports are run in "lastmodified" mode, a new dataset is created in HDFS which contains records that supercede records of the old dataset. This implements a job which merges the datasets preserving only the most recent row for a given primary key value.