As a safeguard for data removal it'd be nice if Kite moved deleted partitions to trash instead of directly dropping them from HDFS completely. This would provide a safeguard in case a delete command was issued incorrectly and completed or some data was deleted in error based on incorrect configuration.
This could be implemented in a couple of ways:
- Dataset.deleteAll would move things to trash instead of calling FileSystem.delete(...)
- A new Dataset.moveToTrash() method gets added (or Dataset.deleteAll(boolean skipTrash)) that could be called for people wanting to be safer with deletes.