It's an 8 hour day with about 10 minutes of racing. Lots of different experiences to be had. The big party is in the infield. The people in the stands might sit and drink mint julips, but theres still lots going on, the biggest is usually watching the horses get ready at the paddock.
It used to be getting in the gate would get you access to the entire grounds, except the private suites. So you could pay for the infield and go sit in seats in the stands no one was using, and people with paid seats could still join the fun in the infield (there's tunnels under the track). This was the first year they blocked access between the stands and the infield (access was limited to those with seats in a corporate tent). I didn't go this year, but I imagine it changed the atmosphere quite a bit. It's also getting more expensive, just for the infield, drawing a different crowd. Things really changed after 9/11 when Chirchill Downs experimented with how much they could limit people bringing in. That didn't really have an effect on what alcohol got past security, but made it a lot harder to bring in snacks, etc. So you had to buy their food, making it for a more expensive day and discuraging some people from coming. And they've been slowly making it more and more expensive every year. Today, you can't even bring a good camera in without a special permit. It's really changed the environment from a cheap, fun day to someplace you go to spend a lot of money.