Sometimes it comes up and he talks about getting no sleep like everyone should be doing it. It is not aggressive, but maybe passive aggressive? I'm not even really sure, but it definitely makes me uncomfortable. Maybe this is my real concern with it?
Yes, this seems like the most important, and most detrimental part of this behavior. I'd recommend getting a new job -- it'll only get worse as the company gets older and more stress is added.