Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-758

Hive MetaStore URI from HiveConf is not present for MR jobs

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 1.1.0
    • Component/s: Command-line Interface
    • Labels:
      None

      Description

      When using the CopyTask, Alan ran into an issue where his Hive MetaStore URI is not present in the job config and available to task nodes. Tasks fail because Kite detects that it can't talk to the MetaStore. The copy task works when importing local data (without a Crunch job) so the MetaStore is correctly configured in the environment. I think what is happening is that the HiveConf doesn't add settings to existing Configuration objects like the HDFS configuration does (this is speculation). If that's the case, then the configuration submitted by the CopyTask wouldn't have the metastore URI set, even though the configuration used to load Hive datasets can correctly connect to Hive.

      The work-around is to embed the MetaStore host and port in the destination dataset URI, which will override the job config.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                blue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: