Details
-
Type:
Bug
-
Status: Open
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: CDH 5.3.0
-
Fix Version/s: None
-
Component/s: Hive
-
Labels:None
-
Environment:Java application with Gradle dependencies as below.
compile 'org.apache.hive:hive-exec:0.13.1-cdh5.3.0'
compile 'org.apache.hive:hive-jdbc:0.13.1-cdh5.3.0'
Description
When hive2 server is up but metastore server is down, if you try to execute JDBC query (i.e. "show tables") SQLException is thrown with null cause. Real cause is connection lost between hive2 and metastore, and full stack trace is visible in exception message clearly indicating what root cause is. But still, you can't traverse down the tree because getCause() is null.
It is very easy to replicate. Just create simple Java app that executes "show tables" hive query, then stop metastore server and run app.
I caught exception and logged its content as follows:
LOGGER.debug("\n| msg -> {}\n| cause -> {}\n| suppressed -> {}",
throwable.getMessage(), throwable.getCause(), throwable.getSuppressed());
Result is:
16:57:40.530 [main] DEBUG c.r.c.c.commons.HiveFailoverExecutor - | msg -> Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:347) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:253) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:88) at com.sun.proxy.$Proxy10.getDatabase(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1189) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1178) at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:4054) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:270) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:145) at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:69) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:200) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:213) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 28 more ) | cause -> null | suppressed -> []