Since I have your ear, I thought about it some more and here are some ideas.
Moonshot: Let users snap a picture and use OCR to suggest times that user can then edit.
Another idea to make it a bit easier for businesses with odd hours: A option for a wizard.
Something like:
1. Click here to start Wizard
2. Monday: Select open time (or say not opened)
3. Monday: Select closed time
4. Monday: Option to add another time block
5. Tuesday: Option to copy and modify times from Monday
6. Repeat
7. Option to copy time from Monday/Tuesday
8. Repeat, etc
I'm not sure how exactly this would play out, and it would only be good in the more complex cases but I personally think it would be easier.