Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: search-0.9.2
-
Fix Version/s: None
-
Component/s: Search
-
Labels:None
Description
I discovered this after creating this bad setting in hbase-indexer-site.xml:
<property> <name>hbase.zookeeper.quorum</name> <value>search-testing-3rdparty-1.ent.cloudera.com:2181</value> </property>
(the port number should not be specified)
which still resulted in a successfully created and active indexer:
[systest@search-testing-3rdparty-1 ~]$ hbase-indexer list-indexers -z search-testing-3rdparty-1.ent.cloudera.com:2181
Number of indexes: 1
hbaseIndexer
+ Lifecycle state: ACTIVE
+ Incremental indexing state: SUBSCRIBE_AND_CONSUME
+ Batch indexing state: INACTIVE
+ SEP subscription ID: Indexer_hbaseIndexer
+ SEP subscription timestamp: 2013-07-11T13:27:49.922-07:00
+ Connection type: solr
+ Connection params:
+ solr.collection = hbase-collection
+ solr.zk = search-testing-3rdparty-1.ent.cloudera.com:2181/solr
+ Indexer config:
110 bytes, use -dump to see content
+ Batch index config:
(none)
+ Default batch index config:
(none)
+ Processes
+ 16 running processes
+ 0 failed processes
But any data put into hbase didn't get indexed.
Further investigation revealed a bad cluster key exception in the HBase regionserver:
2013-07-11 13:31:58,223 ERROR org.apache.hadoop.hbase.replication.ReplicationZookeeper: Can't get peer because:
java.io.IOException: Cluster key invalid, the format should be:hbase.zookeeper.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
at org.apache.hadoop.hbase.zookeeper.ZKUtil.transformClusterKey(ZKUtil.java:396)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.applyClusterKeyToConf(ZKUtil.java:379)
at org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:337)
at org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:527)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:311)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
which can also be confirmed from hbase shell (note the double port numbers):
hbase(main):002:0> list_peers PEER_ID CLUSTER_KEY STATE Indexer_hbaseIndexer search-testing-3rdparty-1.ent.cloudera.com:2181:2181:/ngdata/sep/hbase-slave/Indexer_hbaseIndexer ENABLED 1 row(s) in 0.0190 seconds
The indexer tool should confirm that the peer is valid before returning success. If it's not valid, it should clean the bad indexer out of HBase and return an error message instead.