Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-1012

ReadSequenceFile command only emits the first K/V record

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0
    • Component/s: Morphlines Module
    • Labels:
      None

      Description

      In ReadSequenceFileBuilder at line 90 that you guys do this

      reader = new SequenceFile.Reader(FileSystem.getLocal(conf), new Path("/"), conf) {
      protected FSDataInputStream openFile(FileSystem fs, Path f, int sz, long l) throws IOException

      { return new FSDataInputStream(new ForwardOnlySeekable(in)); }

      It seems that the Reader is grabbing the file length of Path("/") which grabs the length of the root directory. The test passes because the length of the test file is shorter than the length returned from Path("/"). Since you guys are streaming the data through the Morphline command, can't you make use of the SequenceFile.Reader.stream? replace the reader on line 90 with this.

      reader = new SequenceFile.Reader(conf, SequenceFile.Reader.stream(new FSDataInputStream(new ForwardOnlySeekable(in)));

        Attachments

          Activity

            People

            • Assignee:
              whoschek Wolfgang Hoschek
              Reporter:
              newton Newton Truong
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: