It doesn't always though, with chef/puppet et al, you can write the code to throw up X (aapche w/ ssl etc), but unless you use a testing facility, there is no way to know for sure (other than manually checking obviously).
[1] http://docs.ansible.com/get_url_module.html & http://docs.ansible.com/uri_module.html for ansible, I'm fairly certain you can build up something similar for puppet and chef, though I haven't done it myself.