Details
-
Type: Bug
-
Status: Resolved
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: 0.17.0
-
Fix Version/s: 1.1.0
-
Component/s: Command-line Interface
-
Labels:None
Description
When using the CopyTask, Alan ran into an issue where his Hive MetaStore URI is not present in the job config and available to task nodes. Tasks fail because Kite detects that it can't talk to the MetaStore. The copy task works when importing local data (without a Crunch job) so the MetaStore is correctly configured in the environment. I think what is happening is that the HiveConf doesn't add settings to existing Configuration objects like the HDFS configuration does (this is speculation). If that's the case, then the configuration submitted by the CopyTask wouldn't have the metastore URI set, even though the configuration used to load Hive datasets can correctly connect to Hive.
The work-around is to embed the MetaStore host and port in the destination dataset URI, which will override the job config.
Attachments
Issue Links
- depends on
-
KITE-976 DatasetKeyInputFormat/DatasetKeyOutputFormat not setting job configuration before loading dataset
- Resolved