Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-565

Expose parquet writer options as properties

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      A partner trying to write to parquet on the CDH 5.0.0-1 VM was having jobs fail with OutOfMemoryExceptions. Even turning down the writer cache to just one writer still hit the problem because the parquet block size was too large for one writer. We should pass any parquet.* options to the ParquetInputFormat when configuring MR.

        Attachments

          Activity

            People

            • Assignee:
              blue Ryan Blue
              Reporter:
              blue Ryan Blue
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: