DISCLAIMER: I don't use Unity because, at the time I started getting back into game dev, I think you still needed a plugin to run in browser whereas I wanted to target purely HTML5, JavaScript and CSS for maximum device compatibility (hah!). That may no longer be true.
The thing is, I could make the same point as this post but about developing cross-browser games with standardised web technologies. You can waste days at a time fixing behavioural quirks that appear in only one particular browser - I've done exactly that fixing bugs in iOS Safari, whereas I don't even bother supporting Edge in Windows 10 Phone (yet) because very few people use it.
I've had a full screen bug in iOS appear multiple times in both of the games I'm currently working on - a "nearly done" version of Star Castle (https://arcade.ly/games/starcastle/), and a very alpha-ish take on Asteroids (https://arcade.ly/games/asteroids/ - takes a while to load due to graphics pre-generation, which I need to sort out) - due to very minor changes in markup and CSS. They largely share both an engine, along with many front-end components, so issues like that tend to affect both games.
Actually, my "engine" really isn't an engine at all: more an assorted set of libraries of useful functionality that can be bolted together to help build games of a certain type.
Back to the point, some browsers are open source but it really doesn't help because I can't (easily) change what's deployed on millions of peoples' computers around the world. Having access to the source might make it quicker/easier for me to come up with a workaround. I doubt it though because, for example, Firefox has a lot of source code, which it would take weeks to understand before you could figure out why it was behaving oddly in a particular case. It would just be easier to create a workaround. (Of course, it's fragile - the next release of FF might break your workaround, and then you have to have a workaround for the workaround for that release and later.)
I could make the same points about building on top of any platform, open source or otherwise. I've spent a lot of years professionally doing .NET and Java development. With .NET I spent years building desktop tools for developers and DBAs, and we couldn't redistribute a custom runtime so, again, you have to put in workarounds sometimes.
I'd still say the leg up these platforms gave us was worth the hassle of having to deal with some bizarre pieces of behaviour (and we did occasionally encounter some really nasty issues, like race conditions involving database connections, and suchlike).
With web development on an OSS platform and framework I can see it might be different. You have complete control over your deployment environment, so you can put the fix directly in the Fx and submit a patch, if that's the best way to deal with it.
Likewise, if you're distributing the runtime yourself you have control and can apply the fix in its code rather than a workaround in yours. But whilst it's easy to say that the reality probably wouldn't be, and it might cause headaches in the future.
The costs of building your own engine are pretty clear:
- You start from nothing, so it's harder to make progress initially,
- If you want to hire people well, there's a pool of Unity developers, but nobody with experience of your engine,
- Say you start developing for PC but want to release a PS4 version of your game: adding support for more platforms is going to be a lot of work (I don't just mean dev: Unity is tested across all platforms they support, but now you have to do that for your engine),
- When your game engine gets large and complex it's going to start exhibiting hard to debug behavioural quirks, just like everybody else's; you might be able to figure out why more easily, because you have the source, but that doesn't mean the fixes will always be easy.
So, yeah, closed source game engines are a risk, but then so are open source engines, as well as engines you create yourself: they're just a different set of risks. Which you choose should be defined by the objectives of your project(s).