Issues while running Secure Lily HBase Indexer

@Gregory and myself found some problems while trying to validate Lily HBase indexer on Secure cluster.

Here are the steps followed:

• We validated the Secure HBase replication (master-slave) against two secure HBase clusters
• We setup the lily hbase indexer by creating a JAAS conf and setting java.security.auth.login.config via Java config options.
• We wrote new data to hbase and verified that it still showed up in the slave cluster, but nothing showed up in solr.
• Looking in the hbase logs, it appears that the issue is the hbase RS from the master cluster wasn't able to connect via kerberos to the fake Lily RS. So, we edited hbase-indexer-site.xml to add the properties necessary for secure RPC in hbase.
• We restarted the hbase indexer but the process failed with an NPE. It appears that the RPCServer in the hbase indexer doesn't try to read the Configuration - it just returns a null configuration.

  2013-10-08 18:42:25,378 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http client, config:
  2013-10-08 18:42:25,378 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: /etc/hbase-solr/conf/jaas.conf
  2013-10-08 18:42:25,391 INFO org.apache.hadoop.ipc.RpcServer: regionserver/hbase-master.ent.cloudera.com/10.20.193.95:0: started 10 reader(s).
  2013-10-08 18:42:25,392 ERROR com.ngdata.hbaseindexer.supervisor.IndexerSupervisor: Problem starting indexer myIndexer
  java.lang.NullPointerException
  	at org.apache.hadoop.hbase.ipc.RpcServer.createSecretManager(RpcServer.java:2113)
  	at org.apache.hadoop.hbase.ipc.RpcServer.startThreads(RpcServer.java:2076)
  	at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2058)
  	at com.ngdata.sep.impl.SepConsumer.start(SepConsumer.java:141)
  	at com.ngdata.hbaseindexer.supervisor.IndexerSupervisor$IndexerHandle.start(IndexerSupervisor.java:386)
  	at com.ngdata.hbaseindexer.supervisor.IndexerSupervisor.startIndexer(IndexerSupervisor.java:244)
  	at com.ngdata.hbaseindexer.supervisor.IndexerSupervisor.access$900(IndexerSupervisor.java:82)
  	at com.ngdata.hbaseindexer.supervisor.IndexerSupervisor$EventWorker.run(IndexerSupervisor.java:424)
  	at java.lang.Thread.run(Thread.java:724)
  

Here's the RPCServer trying to read the Configuration to setup secure rpc:
https://github.com/apache/hbase/blob/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java#L2112
Here's the SEPConsumer setting up the rpcServer:
https://github.com/NGDATA/hbase-indexer/blob/master/hbase-sep/hbase-sep-impl-0.95/src/main/java/com/ngdata/sep/impl/SepConsumer.java#L139
Here's the RPCServer always returning null:
https://github.com/NGDATA/hbase-indexer/blob/master/hbase-sep/hbase-sep-impl-0.95/src/main/java/com/ngdata/sep/impl/BaseHRegionServer.java#L109
So, this may be not too difficult to fix if we can just get the proper configuration returned, but it definitely requires more investigation.