Woz is not dumb. His point was that he chose to adopt this mindset on purpose. He'd rather be the "average person joking all the time than a powerful businessman stressing over work everyday".
By not calling out Jobs for screwing him out of $5,000, you can easily say Woz was just avoiding confrontation. But I really think his whole mindset transcends that. It just wasn't important to him. And that's how Woz lived his whole life, simply focusing on the things that really mattered to him.
I have always been so impressed with how true to himself Woz was.