You could even say "The Fourth" and almost all Americans would understand. I think that's why; [begin speculation] it's such an exceptional date it necessitates saying it differently.

The Fourth of July and Cinco de Mayo are exceptional days.

Tax day is the ides of April, but we just say April 15th.