I think you are not talking about the same thing that the rest of us are talking about. I see three locations or context involved:
1. The public area - meaning anyone has access.
2. The application context (server, etc) -- should be a "secured" location, meaning few people or processes have legitimate access, and measures should be taken to prevent unauthorized access.
The private key should never be present in location 1, public area. The public key is expected to be available here. However, the problem would not be that the keys are together, but that the private key is available here at all, paired with the public or not. To be clear: I am saying the private key should never be available in context 1.
The typical assumption in most scenarios is that context 2 is secure, and by that I mean secure enough to house the private key. After all, the actions taken in context 2 usually require the private key. So it is reasonable to say that we consider this area secure enough to house private keys, and then take adequate measures to protect it. In this case, if the public key is also stored here it is irrelevant, as the public key is already available in context 1. If you have made the decision that context 2 is safe enough for the private key then there is no reason not to also store the public.
However, we have seen that in a typical deployment scenario context 2, the application context, is often compromised. This leads us to create context 3:
3. Dedicated secure key storage. This often comes in the form of an HSM or similar device. See http://en.wikipedia.org/wiki/Hardware_security_module
The purpose of having an HSM is that instead of doing the cryptographic work (which requires the private key in clear) in context 2 , you send a work request to the HSM saying, "Here is the data I need you to work on, here is the cryptographic operation I need performed, and here is the private key identifier representing the private key I want you to use," the HSM performs the crypto operation, and returns the result. Importantly, the private key is never known outside the HSM. Even in this situation, there is no reason not to also store the public key in the HSM. In fact, it is very common to store both the public and private key parts in the HSM so that you can use the same API in your code (the key id) to reference both. Many crypto APIs have limited or no support for cryptographic operations using a key supplied from the outside (outside the HSM). This is on purpose. So often you have to keep the public key in the HSM too if you want to use the HSM for the crypto.
> It's effectively required to store the keys separately because if you stored the public key on the key bastion, it would be useless.
No. No. No. It is true that you must publish the public key for others to use it, but that does not mean you cannot also store it in the secure location. As I have explained above it is often required that it also be present in the secure location.
> This makes no sense at all. You're saying that since the public key is public, it's safe to distribute the private key with the public key.
No. You are not understanding what you are reading. I am saying that anyone who has access to the private key also has access to the public key, so there is no reason not to store them together in the secure location. I'll say it again: If you have the private key, you might as well have the public key. You would never distribute the private key. Period. And just because you store the public key in the same secure location as the private key does not mean you have to give everyone access to that secure location. You could, I don't know, make a copy of the public key to distribute by itself.