I'm not American, so I have no stake in defending the US border patrol, but this is what I'd expect border patrol anywhere in the world to do. Their entire job function is to prevent illegal entry, toll violations etc.
By your own account you were in possession of a valid passport clearly indicating that you're working in an official capacity for a foreign state. Why would border patrol anywhere in the world continue to detain you at that point?
Is there some illegal immigration problem in any country on Earth stemming from people in possession of official government passports indicating that they work for their respective states that I'm unaware of? How is this not as clear of a signal as anything short of the agent personally recognizing you as a foreign ambassador that you should be sent on your way?