Oh, it's definitely possible to go lower than 5x8 and include symbols, but the issue is aesthetics. A lot of your lowercase letters are going to look taller than they usually would, which can be a distraction when reading.
If you don't care about appearances and just whether or not every character is distinct enough to be understood, then I think 4x6 is probably the limit, but I'd be happy to be proven wrong.