Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-1029

Add existing partitions in batches

          Details

          • Type: Improvement
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: 1.1.0
          • Fix Version/s: 1.3.0
          • Component/s: Data Module
          • Labels:
            None

            Description

            It can take a long time to create a dataset backed by Hive external tables for existing partitioned data if there are many partitions. (E.g. a few thousand partitions can take ~10 minutes.) It would be good if the partitions were added in batches so that the number of calls to the metastore was reduced.

              Attachments

                Activity

                  People

                  • Assignee:
                    Unassigned
                    Reporter:
                    tom Tom White
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    2 Start watching this issue

                    Dates

                    • Created:
                      Updated: