Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-652

DatasetKeyOutputFormat: getLocation() will be override unexpectedly

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None

      Description

      After having replaced jars of 0.16.0 to 0.17 (from trunk), dataset is not able to create dataset at specified location.

      In my case, I want to dataset at "hdfs:/tmp/par1", but actually Kite tries to create a dataset at "hdfs:/.temp/job_1408673538327_0357/mr/job_1408673538327_0357". As the current user does not have permission to create directory at root, it is failed.

      Here is the stack trace:

      14/09/04 15:59:34 INFO mapreduce.Job: Job job_1408673538327_0357 failed with state FAILED due to: Job setup failed : org.kitesdk.data.DatasetIOException: Unable to create metadata directory: hdfs://bdpe14.sh.intel.com:8020/.temp/job_1408673538327_0357/mr/job_1408673538327_0357/.metadata for dataset: job_1408673538327_0357
      	at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.create(FileSystemMetadataProvider.java:199)
      	at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.create(FileSystemDatasetRepository.java:136)
      	at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.createJobDataset(DatasetKeyOutputFormat.java:535)
      	at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.access$200(DatasetKeyOutputFormat.java:62)
      	at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$MergeOutputCommitter.setupJob(DatasetKeyOutputFormat.java:354)

      After some digging, I noticed the location property will get lost after DatasetDescriptor is copied. (Details were described in comment below)

      I regard it as a bug. Please kindly review the problem.

        Attachments

          Activity

            People

            • Assignee:
              tom Tom White
              Reporter:
              stanleyxu2005 Qian Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: