Uploaded image for project: 'CDH (READ-ONLY)'
  1. CDH (READ-ONLY)
  2. DISTRO-570

Loading Metadata for a table fails with what looks like an hdfs client error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Not A Bug
    • Affects Version/s: CDH5.0.0
    • Fix Version/s: None
    • Component/s: HDFS
    • Labels:
      None

      Description

      This happened after the move to the latest hdfs client.

      Impalad Logs:

      I0305 19:10:51.463315  4933 jni-util.cc:177] com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: functional_seq.alltypessmall
              at com.cloudera.impala.analysis.Analyzer.getTable(Analyzer.java:1340)
              at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:299)
              at com.cloudera.impala.analysis.BaseTableRef.analyze(BaseTableRef.java:71)
              at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:118)
              at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:248)
              at com.cloudera.impala.service.Frontend.analyzeStmt(Frontend.java:539)
              at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:563)
              at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:116)
      Caused by: com.cloudera.impala.catalog.TableLoadingException: Failed to load metadata for table: alltypessmall
      CAUSED BY: CancellationException: null
              at com.cloudera.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:113)
              at com.cloudera.impala.catalog.Table.fromThrift(Table.java:204)
              at com.cloudera.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:561)
              at com.cloudera.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:489)
              at com.cloudera.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:227)
              at com.cloudera.impala.service.Frontend.updateCatalogCache(Frontend.java:140)
              at com.cloudera.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:133)
              at ========.<Remote stack trace on catalogd>: com.cloudera.impala.catalog.TableLoadingException: Failed to load metadata for table: alltypessmall
              at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:828)
              at com.cloudera.impala.catalog.TableLoader.load(TableLoader.java:84)
              at com.cloudera.impala.catalog.TableLoadingMgr$1.call(TableLoadingMgr.java:174)
              at com.cloudera.impala.catalog.TableLoadingMgr$1.call(TableLoadingMgr.java:171)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.util.concurrent.CancellationException
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220)
              at java.util.concurrent.FutureTask.get(FutureTask.java:83)
              at org.apache.hadoop.hdfs.BlockStorageLocationUtil.queryDatanodesForHdfsBlocksMetadata(BlockStorageLocationUtil.java:159)
              at org.apache.hadoop.hdfs.DFSClient.getBlockStorageLocations(DFSClient.java:1219)
              at org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockStorageLocations(DistributedFileSystem.java:255)
              at com.cloudera.impala.catalog.HdfsTable.loadBlockMd(HdfsTable.java:237)
              at com.cloudera.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:535)
              at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:798)
              ... 8 more
      

      Catlogd Logs:

      W0305 19:07:52.011731 11445 Client.java:1387] interrupted waiting to send rpc request to server
      Java exception follows:
      java.lang.InterruptedException
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:979)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
              at java.util.concurrent.FutureTask.get(FutureTask.java:83)
              at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1028)
              at org.apache.hadoop.ipc.Client.call(Client.java:1382)
              at org.apache.hadoop.ipc.Client.call(Client.java:1362)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
              at $Proxy11.getHdfsBlockLocations(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolTranslatorPB.getHdfsBlocksMetadata(ClientDatanodeProtocolTranslatorPB.java:246)
              at org.apache.hadoop.hdfs.BlockStorageLocationUtil$VolumeBlockLocationCallable.call(BlockStorageLocationUtil.java:343)
              at org.apache.hadoop.hdfs.BlockStorageLocationUtil$VolumeBlockLocationCallable.call(BlockStorageLocationUtil.java:307)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ishaan Ishaan Joshi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: