Interviewing only with your eventual manager is massively inefficient. Imagine I'm a hiring manager with one headcount available. Let's say Google gets about 5000 applications per day and hires 50 people per day, so I need to interview around 100 people to fill that spot. If two of those people meet my criteria, the extra ones get rejected anyway. Even more time is wasted than with the algorithm question approach (and honestly, stuff like tree traversal comes up all the time, even in client application code, so you might still get asked about it).