Glad I could be of assistance -- let me see if I can assist further:
I have to say I'm a little surprised you're running into issues on the negotiation front -- but that may be where you live and other circumstances. Where I'm at (the Detroit area), there are fewer of the well-known huge tech companies, and I'm unaware of any, in particular, that do things like catered lunches, but there are plenty of great jobs and the market is very hot. On that, though, the best talent wants to work at the top places but the places that are traditionally "unsexy" are the ones that are going to be more willing to negotiate. I'm not sure, however, how many of these places are looking for ML candidates -- you're probably going to find that limits you a bit and that might be part of the problem[0].
As far as salary negotiations, that's the hardest thing to get right. It's very difficult to pull off, for me, but the golden rule is to not be the one to give the first figure. This will not make your hiring manager happy. I've always been able to successfully deflect the conversation before the final interview with some choice words[1]. If they insist and are turned off by a candidate seeking the best possible outcome in a salary negotiation, I'd take that as a big red flag and be done with it. If you're working with a recruiter,... stop returning his/her calls. It takes creative finesse to handle this well and I strongly suggest doing a few HN searches for good advice (there was a post I laboured to find which helped me out a lot and got me what I wanted in this area). The thing of it is, it feels like you're negatively impacting your chances of getting the job when you play a bit of hardball on these things, but if handled correctly and professionally, it demonstrates characteristics that a hiring manager expects out of someone in a senior position. It also indicates that you carry ambition along with your technical skills.
Here's the single best piece of advice I have gotten about job hunting -- and bear in mind, I'm from Detroit, so we do a lot of car analogies: "What color do you want? The cheapest color" and "Don't sit in the car". Both of these things are not common-sense advice -- we test drive the cars we want to buy because we think we're going to learn something about the car by doing so and we pick a color because we think that's going to be particularly important to me. The reason that car color is the first thing you're asked about is because it puts you a little closer toward the illusion that you own the car, already. The test drive creates a very powerful connection between you and this potential purchase. The salesman knows if he puts you in the seat and you drive away, the probability of you buying that car goes up substantially. It's not enough to just be aware of these things, it's important to guard against things that will unreasonably bias you toward becoming attached to the car. This advice applies to all negotiation. If you avoid things that will trigger loss aversion, you will have an unemotional focus on the target attributes and can evaluate the decision you're making based on those things[2].
You're coming in with a requirement that might very well be a non-starter because of some "corporate philosophy" (somehow, open floor plans have a religious fervor about them). I have different requirements than yours, but when I encounter a company that has a "policy" against things that I feel make me more effective, I graciously turn the job down, early. I do so before they have the opportunity to even offer to see if there's a way they can work around this policy for me[3]. Aside from the footnote, the biggest red flag I find is inflexible policy -- it's the kind of thing that causes a company to lack nimbleness. Companies that I want to work for treat policy as guidance and are willing to break policy[4] when the circumstances fit.
I can see the collaborative thing coming up a lot because of the fallacy that open office environments foster collaboration. I'd find a way to work collaboration in directly with your initial argument -- something along the lines of "As a Senior Developer, I understand that part of my responsibilities will be to share my skills with my coworkers with the goal of making more senior developers. I'll make sure my door is open except for the rare occasion that a deadline necessitates me to be "nose down in code".[5]" There's a small mind hack you can play if you're good at that sort of thing, but be careful, because if you're not it can backfire pretty badly. Follow-up with "At this point in my career, I've learned that communication has to be deliberate and that's the way I approach things. I'm confident that my approach to communication will make any concerns around collaboration a non-issue." Deliver that as casually as you can -- as if it's a simple, obvious, fact and you end up putting the hiring manager in a tough spot for a rebuttal[6].
I completely understand not relishing wanting to put up with months of job hunting/"tech hazing". Job hunting isn't terribly fun. Might not hurt to ask questions in forums where employees of the companies you're looking at frequent asking about offices if you want to improve the hit ratio. Another option that could help to eliminate places earlier in the process is to request a quick tour after the first on-site interview[7]
Unfortunately, though, when you have a lot of requirements, or when your requirements are not the norm, it's normal for this sort of thing to take a while and require a lot of work on your part to sort out. The tech hazing will wear you down and increase loss aversion, making it harder to walk away from a good offer that doesn't meet your requirements. And maybe you shouldn't walk away -- that's your call. At the same time, if this is really important to you, this is the market to be picky in. The market is hot. If you can keep yourself from being discouraged, you'll probably end up where you want with enough persistence.
As an aside - I got a kick out of the excuses you received "I'm the hiring manager and I don't even have an office." There are so many things I can think of to say to that which would get my resume binned: "Oh ... (shuffles feet) ... well, can I talk to your boss then?" or a sarcastic quip along the lines of "Work this out for me and you'll have one more argument to make for why you deserve one!"
[0] And I'm not suggesting you avoid ML jobs -- if that's where you fit best, stick with it.
[1] ala "I'm seeking Senior level position and with my experience, I expect to be paid above average for the market however, salary isn't my greatest concern and other things like benefits and such greatly affect any number I'd throw out right now. If after all of the interviewing is done you find that I'm going to be a good fit, I'm confident we can come to an agreement that works out for both of us."
[2] And it's entertaining, too.
[3] Generally, I don't want to work at a place where they had to make a major exception for something that I would have considered a reasonable accommodation, particularly if that accommodation is one that 90% of the people who currently work there would similarly want. If they are interested enough in hiring me that after I have turned down the job, they pop up with a way to provide that accommodation, I would consider it -- and this actual situation happened to me about 15 years ago, but I ended up not taking the position.
[4] And I'm not referring to ethics-based or legal-based policies.
[5] Here's the thing -- if offices are unusual where you're looking, the fact that you have a door serves as a psychological barrier -- you'll rarely need to close it. But it also doesn't matter -- that's something you work out with your boss after you've got a place to plant your hind-quarters. And you work through that concern by checking in with your boss every two weeks with a "Hey, just wanted to make sure I'm meeting your expectations as far as collaboration is concerned"... no mention of the door. Leave an opportunity for them to speak up if there are concerns, but you'll probably find that your boss has better things to be concerned with. Incidentally, I know that Microsoft at one point was big on offices. I'm not sure if that's still the case, though.
[6] And a rebuttal isn't a bad thing -- at least they want you there badly enough to try to persuade you to change your mind.
[7] "Wow - your office is far more impressive than the pictures online led me to believe -- would you be willing to give me a brief tour?" I've never had this fail at places that have fewer than 200 employees -- people like to show off where they work. If it's a high-security operation, though, don't expect to see much. :)