Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 0.15.0
    • Component/s: Data Module
    • Labels:
      None

      Description

      The easiest option to guarantee durability for parquet writes is to write a shadow Avro file at the same time, which is deleted after the parquet file is committed to a dataset. That way, if anything happens while streaming data in, the data is recoverable as Avro and can be rewritten during recovery.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                blue Ryan Blue
                Reporter:
                blue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: