Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.10.1
-
Fix Version/s: 0.12.1
-
Component/s: None
-
Labels:None
Description
Exceptions occur when writing to multiple partitioned writers for datasets in repo:hive repositories.
These exceptions are caused because HCatUtil.getHiveClient(...) (in kite's HCatalog class) can return the same instance in different threads.
Example stacktrace:
writer 1 writing record 0
writer 2 writing record 0
14/02/18 16:28:57 INFO metastore.HiveMetaStore: 1: source:/127.0.0.1 append_partition_by_name: db=default tbl=test2 part=username=48
14/02/18 16:28:57 INFO HiveMetaStore.audit: ugi=karel ip=/127.0.0.1 cmd=source:/127.0.0.1 append_partition_by_name: db=default tbl=test2 part=username=48
14/02/18 16:28:57 INFO metastore.HiveMetaStore: 1: source:/127.0.0.1 append_partition_by_name: db=default tbl=test3 part=username=48
14/02/18 16:28:57 INFO HiveMetaStore.audit: ugi=karel ip=/127.0.0.1 cmd=source:/127.0.0.1 append_partition_by_name: db=default tbl=test3 part=username=48
java.lang.IllegalArgumentException: Problem creating view for entity:
at org.kitesdk.data.filesystem.PartitionedDatasetWriter.write(PartitionedDatasetWriter.java:96)
at org.kitesdk.data.hcatalog.TestExternalHCatalogDatasetRepository$WriteRecordTask.call(TestExternalHCatalogDatasetRepository.java:253)
at org.kitesdk.data.hcatalog.TestExternalHCatalogDatasetRepository$WriteRecordTask.call(TestExternalHCatalogDatasetRepository.java:234)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.RuntimeException: Hive metastore exception
at org.kitesdk.data.hcatalog.HCatalog.addPartition(HCatalog.java:108)
at org.kitesdk.data.hcatalog.HCatalogMetadataProvider.partitionAdded(HCatalogMetadataProvider.java:94)
at org.kitesdk.data.filesystem.PartitionedDatasetWriter$DatasetWriterCacheLoader.load(PartitionedDatasetWriter.java:179)
at org.kitesdk.data.filesystem.PartitionedDatasetWriter$DatasetWriterCacheLoader.load(PartitionedDatasetWriter.java:154)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834)
at org.kitesdk.data.filesystem.PartitionedDatasetWriter.write(PartitionedDatasetWriter.java:94)
... 7 more
Caused by: org.apache.thrift.TApplicationException: append_partition_by_name failed: out of sequence response
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_append_partition_by_name(ThriftHiveMetastore.java:1165)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.append_partition_by_name(ThriftHiveMetastore.java:1150)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:399)
at org.kitesdk.data.hcatalog.HCatalog.addPartition(HCatalog.java:102)
... 19 more