His superiors might be very upset if he ends up prompting you to leave. They may have no clue this is going on. Talk to the original people and tell them it's not working out.
The program manager wasn't initially aware of the arrangement. When I mentioned it to him, he basically threw a professional fit along the lines of "To get work done I need you here when everyone else is here." He'll even readily admit that I got plenty of work done while contracting and setting my own hours and location - enough that he and the company wanted me salaried - "but it's not the same now". The execs that I negotiated with don't have much backbone and won't stand up to him or our owner. They are well aware of the situation, and when I go through them and leave the PM out of scheduling, they're always okay with whatever I say I'm going to do (I don't even bother asking, I just tell them); after the fact, he'll huff and puff for a while, but he never actually says anything explicitly. Unfortunately, he and I work very closely together, so I don't often have the opportunity to go through the execs.
The night before a particularly harsh snowstorm, I even had a conversation with him about working from home the next day. We talked about what I could do from home, I literally showed him the printouts of the background material (and had digital copies on our office server and on my laptop), and he said he was okay with it. When I wasn't at the office shortly after 9:00am the next morning he started calling me. Even though I had already been working for close to an hour and sent him updated files showing what I had already completed, he wanted me to drive in.
To add to the frustration, we frequently have to travel for work. Of course, no one, including the PM has a problem with work done remotely in those situations.
Caveat emptor of course. I have a sample size of 2 in my personal life and its worked for me once (but wonderfully so). The other time, it got me out of a bad situation that I should have been working harder to leave anyway (with good references even).