Add file based config testing

We ran into many config issues at the end of the beta release. We typically test these by programmatically setting them but this left some holes with on disk config files. Add some end to end tests that mimic this.