Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-469

Dataset CLI should use avro schema URI property

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      The CLI create command opens a schema and adds it to the DatasetDescriptor using the InputStream method. This should detect when the schema is in HDFS and use the Hive URI property.

      The literal schema property value is limited to the size of the TABLE_PARAMS value, which is about 4k for some back-ends. This results in the following error when inserting:

      2014-06-10 10:17:17 INFO  :: Creating a managed Hive table named: test_table
      Unknown error
      org.kitesdk.data.DatasetRepositoryException: Hive MetaStore exception
              at org.kitesdk.data.hcatalog.impl.HCatalog.createTable(HCatalog.java:153)
              at org.kitesdk.data.hcatalog.HCatalogManagedMetadataProvider.create(HCatalogManagedMetadataProvider.java:71)
              at org.kitesdk.data.hcatalog.HCatalogDatasetRepository.create(HCatalogDatasetRepository.java:64)
              at org.kitesdk.cli.commands.CreateDatasetCommand.run(CreateDatasetCommand.java:75)
              at org.kitesdk.cli.Main.run(Main.java:133)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.kitesdk.cli.Main.main(Main.java:185)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
      Caused by: MetaException(message:javax.jdo.JDODataStoreException: Put request failed : INSERT INTO "TABLE_PARAMS" ("PARAM_VALUE","TBL_ID","PARAM_KEY") VALUES (?,?,?) 
      NestedThrowables:
      org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO "TABLE_PARAMS" ("PARAM_VALUE","TBL_ID","PARAM_KEY") VALUES (?,?,?) )
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:20149)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:20117)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result.read(ThriftHiveMetastore.java:20043)
              at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table(ThriftHiveMetastore.java:732)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table(ThriftHiveMetastore.java:719)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:432)
              at org.kitesdk.data.hcatalog.impl.HCatalog$3.call(HCatalog.java:139)
              at org.kitesdk.data.hcatalog.impl.HCatalog$3.call(HCatalog.java:136)
              at org.kitesdk.data.hcatalog.impl.HCatalog.doWithRetry(HCatalog.java:67)
              at org.kitesdk.data.hcatalog.impl.HCatalog.createTable(HCatalog.java:145)
              ... 11 more
      

        Attachments

          Activity

            People

            • Assignee:
              blue Ryan Blue
              Reporter:
              blue Ryan Blue
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: