There are a few different questions here; I haven't worked with an OpenTable system yet, so I can only guess at some of it. But:
> ...it seems what restaurants want to do is increase profits.
Yes, but -- based on my experience and dealing with restaurant owners in my area, anyway -- they're not interested in things like metrics or VIP programs. VIP programs don't convert as well as you might expect -- pretty poorly, in fact, given the effort that it takes to manage them -- and restaurants are typically run by chefs or foodies, not by numbers geeks.
Restaurants are really hurting right now, so anything that doesn't immediately get butts in their seats is going to be a hard sell.
Daily P&Ls are already built into the reporting functions of the POSs I've seen so far, and they have some functions for reporting on popular dishes, although that's usually more clunky. But, the line cooks are usually the ones that tell the boss what was popular that night anyway.
> Is OpenTable too big to fail (e.g. restaurants will continue to pay them annual maintenance because it is a legacy system)?
Yes and no. Once a restaurant has committed to a particular POS, it is very, very, very hard to get them away from it. First, it cost them -- at a safe ballpark estimate -- $10,000 to install it in the first place, so they're going to milk it for as long as they can before paying for anything else, even if it costs them more money that way in the long run.
Second, change is really scary for these folks, because if they're trying a new POS, and it crashes on its first evening, they're going to have a lot of pissed off customers and that translates directly to empty seats. It doesn't help that every single system that they've tried -- or that their buddies in the industry have tried -- have been buggy, problematic, and prone to crankiness.
It doesn't matter if you know that you're selling the best system in the world at the cheapest price; the owners are still going to be really gun-shy about converting.
> Can OpenTable's database be cleanroom reverse engineered so that it can be migrated to a new competitor platform?
In theory, anything's possible, but I would feel pretty safe guessing that OpenTable isn't going to make that easy. They're probably using something pretty funky. On the plus side, they're probably not encrypting it, they're just relying on security-through-obscurity. So, you should be able to figure out the basics, like item description, pretty easily. Figuring out sales or quantities might be a lot harder, not to mention figuring out links between items (like, add-ons for a particular dish).
To be honest, you might be better off committing staff to manually re-entering the restaurant's menu in your POS, rather than trying to convert OpenTable's (or anyone else's), and then just roll that into your price. This is not as horrible as it sounds; most restaurants have only a couple of pages of items, with a variety of add-ons. I've done this by myself in under a day, so it's not too bad.