You should have taken the director job, you probably would have handled it as well as anyone else. I was a "Director" of Test at one company. I didn't even have a budget. Even in interviews I'll tell folks that I was a test manager with a glorified title (though with no budget, was I even a "test manager"?).