only able to execute hadoop commands using principals which are in the default realm

This might not be a bug. Here is the description. Any workarounds are appreciated.

I am only able to execute hadoop commands using principals which are in the default realm.

hadoop.security.auth_to_local seems to be ignored.

Attached is a log of everything done.

Here is overview of the configuration and some troubleshooting tests:

1. created and tested a principal using the KDC instead of AD and confirmed all OK
   hadoop org.apache.hadoop.security.HadoopKerberosName george@EC2.INTERNAL
   Name: george@EC2.INTERNAL to george

1. fails to use with principal from AD, seems to ignore rules in hadoop.security.auth_to_local
   hadoop org.apache.hadoop.security.HadoopKerberosName george@CLOUDSECURE.LOCAL
   Exception in thread "main" org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to george@CLOUDSECURE.LOCAL
   at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:378)
   at org.apache.hadoop.security.HadoopKerberosName.main(HadoopKerberosName.java:74)

1. note: ip-10-151-51-135.ec2.internal has Win 2008 R2 + AD DS with 1 forest, and defines all user accounts used for authentication
   /etc/krb5.conf

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = EC2.INTERNAL
dns_lookup_realm = false
dns_lookup_kdc = false
max_life = 1d
max_renewable_life = 7d
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts arcfour-hmac des3-hmac-sha1 des-hmac-sha1 des-cbc-md5 des-cbc-crc
default_tkt_enctypes = aes256-cts aes128-cts arcfour-hmac des3-hmac-sha1 des-hmac-sha1 des-cbc-md5 des-cbc-crc

[realms]
EC2.INTERNAL =

{ kdc = ip-10-191-70-81.ec2.internal admin_server = ip-10-191-70-81.ec2.internal default_domain = EC2.INTERNAL }

CLOUDSECURE.LOCAL =

{ kdc = ip-10-151-51-135.ec2.internal:88 admin_server = ip-10-151-51-135.ec2.internal:749 default_domain = EC2.INTERNAL }

[domain_realm]
.ec2.internal = EC2.INTERNAL
ec2.internal = EC2.INTERNAL

cat /etc/hadoop/conf.cloudera.hdfs1/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>

<!-Autogenerated by Cloudera CM on 2013-10-06T10:16:50.792Z->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ip-10-191-70-81.ec2.internal:8020</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](.*@\QEC2.INTERNAL\E$)s/@\QEC2.INTERNAL\E$//
RULE:[2:$1@$0](.*@\QEC2.INTERNAL\E$)s/@\QEC2.INTERNAL\E$//
RULE:[1:$1@$0](.*@\QCLOUDSECURE.LOCAL\E$)s/@\QCLOUDSECURE.LOCAL\E$//
RULE:[2:$1@$0](.*@\QCLOUDSECURE.LOCAL\E$)s/@\QCLOUDSECURE.LOCAL\E$//
DEFAULT</value>
</property>
</configuration>