First up, no, you don't need to maintain two completely different code bases. I'd recommend having a separate manifest.json for each browser or having a build step generate the per-browser manifest.json. A lot of the manifest changes are relatively small, like splitting "host_permissions" out from "permissions" and should be automatable. For example, a build step would merge these two arrays for the MV2 manifest.
As for your background scripts, if possible I'd recommend limiting the JS capabilities you use to the intersection of what all browsers support.
- Chrome supports background service workers - Safari supports event pages and I believe has service worker support in beta builds of their OSs - Firefox supports persistent background pages
In other words, target the capabilities available in both worker and page contexts. You'll also want to design your background logic to embrace ephemerality. This is required by Chrome, but it also benefits Firefox users in that it helps keep the browser responsive.
Best of luck!