Cluster planner membership is not updated after removing a planner

This is what I tried to reproduce the issue:
1. On a cluster, deploy a few planners and workers.
2. In CM, stop & remove the planner that's being used (i.e., the first planner in the list), and redeploy the client configuration.
3. Try some random MR job, the job will still try to use the removed planner. If you look at the /etc/recordservice, you'll find that there're two config directories now, and recordservice-site.xml in both directories still use the old planner membership.