Do you want to keep doing the job if they get the CTO so you're not overloaded? Or do you want to get out of management?

There's nothing wrong with deciding you don't want to climb the management ladder, but getting "un-promoted" is something where you'll probably want to find a new job. Fortunately, almost everyone is trying to hire competent senior developers, so you should be able to find a new gig before too long - and probably get yet another pay raise to go with it.