Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-800

Empty arrays are not properly handled for Parquet tables

          Details

          • Type: Bug
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: CDH5 5.6.0
          • Fix Version/s: None
          • Component/s: Hive, Parquet
          • Labels:
          • Environment:
            Ubuntu

            Description

            Hive table columns can contain empty arrays.
            However, on attempt to write an empty array into Parquet table, it causes exception:

            Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
            at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
            at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
            ...

            The problem in a nutshell can be reproduced and demonstrated with CDH5.7.0 with one-liner HQL request:
            CREATE TABLE array_test STORED AS PARQUET AS SELECT array();

            The problem did not appear in the previous CDH (5.6.0).
            The issue is very close to one reported in https://issues.apache.org/jira/browse/HIVE-11625

              Attachments

                Activity

                  People

                  • Assignee:
                    Unassigned
                    Reporter:
                    DTsarou Dzmitry Tsarou
                  • Votes:
                    1 Vote for this issue
                    Watchers:
                    3 Start watching this issue

                    Dates

                    • Created:
                      Updated: