Your instincts may push you into a small startup where your work has immediate impact and feedback, but there's a space between 10 employees at 100K employees. I used to be in consulting (at a software company) and I worked with everybody from small startups that wanted to partner to companies with 200K+ employees. I tended to enjoy working the most with companies with 1K-10K employees. Those had interesting problems, interesting people, a sensible pace of work, and a somewhat tolerable amount of processes and bureaucracy. 50K+ employees almost always felt like I was in a satire. With really large companies (100K+), I came to understand how they could layoff 10% of the employees and still go on like nothing happened.
I'm not telling you to stay at your current job(don't stay at a job you hate!), but try to think about what you want your spare time to be like before make a move. Going to a place with a constant stream of challenging and interesting work might be nice at first, but after a while you might find your energy for side projects is zero.
There must be teams at the big 4 where you get the opportunity to do some interesting work and then go home and work on things that you are really passionate about. JSeymourATL provided a great answer - Assuming it is possible, network within the company and find a group that is doing something you find interesting. Unfortunately some companies are too political to be able to do this. Or you may have a horrible manager who will retaliate if you try to do this.
All that being said, some people need to be in charge. They need to be entrepreneurs. I know a brilliant engineer who owns a small construction company now because he'll do anything to not report to somebody else. He simply isn't built to have a boss and not have control of his work, or be obligated to VCs for a tech startup.