I usually keep a common source folder for all browsers. This includes a common manifest (V2, still experimenting with V3). At build time a script generates a browser specific flavour of the manifest and copies all resources into a target folder (e.g /dist-chrome, /dist-safari, /dist-firefox) to create a bundle per browser and apply browser specific tweaks (like copying resources into an Xcode project for Safari etc).