Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.14.1
-
Fix Version/s: 0.15.0
-
Component/s: Data Module
-
Labels:None
Description
The easiest option to guarantee durability for parquet writes is to write a shadow Avro file at the same time, which is deleted after the parquet file is committed to a dataset. That way, if anything happens while streaming data in, the data is recoverable as Avro and can be rewritten during recovery.
Attachments
Issue Links
- relates to
-
KITE-494 Add data recovery CLI command
-
- Open
-