Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-696

Store Avro schema in HDFS by default for Hive datasets

          Details

          • Type: Improvement
          • Status: Resolved
          • Priority: Minor
          • Resolution: Fixed
          • Affects Version/s: 0.16.0
          • Fix Version/s: 1.0.0
          • Component/s: None
          • Labels:
            None

            Description

            Storing the Avro schema in HDFS for a Hive-based dataset may solve a couple problems. Specifically:

            • The original version of the schema can be preserved to ensure future changes are compatible with it., as discussed at [1]
            • It avoids limitations around the size of the schema Hive supports (as discussed at [2]). (This is an issue we run into frequently.)

            [1]
            https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/fKyK8RfYsRg
            [2]
            https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/K4YNFbivVBA

              Attachments

              1. cdk-696-1.patch
                34 kB
                Ryan Brush
              2. cdk-696-2.patch
                42 kB
                Ryan Brush

                Issue Links

                is duplicated by

                Bug - A problem which impairs or prevents the functions of the product. KITE-786 Long parquet schemas cause Metastore exception during Hive import

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. KITE-38 HCatalogMetadataProvider should store the Avro schema on the filesystem

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      rbrush Ryan Brush
                      Reporter:
                      rbrush Ryan Brush
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      6 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: