An attempt to replace our awkward use of null_resource + remote provisioner combination.