I think the biggest reason for the custom code in the small teams is that they make 2D games. A 2D engine barely deserves the name and it's usually more hassle adapt to a 3rd party engine and its limitations than just write it yourself.
If you say so. The vast majority of programmers would have no clue where to start if you put them up to the task.
A game engine is much more than a renderer, which is the only easier part in 2D versus 3D.
I have a toy game in play around in my free time and started with SDL like so many. You can create windows and draw bitmaps really easily. But sooner rather than later you discover limits (bugs, performance limitations, less than ideal APIs) and see that just writing OpenGL code yourself is actually easier and more flexible. Many people go down that road.
edit: Concerning your edit about "more than a renderer": Yeah that other stuff is annoying and many still use SDL, SFML or whatever for that. See Factorio and many many others who do the rendering themselves and offload the other crap. I do that myself too.
And bring physics, input, asset loading, networking, systems, scripting, etc.
Along with their own idiosyncrasies and assumptions. Which is something you may decide you do not want to deal with. E.g. hand-coding the little physics you need may be easier than learning the physics engine's API and adjusting your own architecture to mesh with it.
Roblox is the only really big engine company I can think of that actually develops their own physics tech (because they started off doing it).
So both Unreal and Unity now have their own physics tech.
A lot of engines are still using PhysX (albeit stuck on version 3.x). Upgrading a physics engine is often a pain point, either because it is so tightly integrated in the code base or because game assets have been tuned for the current physics implementation and even a small accuracy or behavior change will require assets to be re-tuned.
You’re right that it seems like Unreal and Unity are ditching PhysX for something else (Unreal for the new Chaos engine, Unity for their new DOTS engine in collaboration with Havok). I’m just surprised that PhysX‘s performance problems had something to do with the move.
NVIDIA's incompetence is what kicked both of them off, basically. Both engines will continue to support PhysX 3.6 for existing customers.
Unity has an ECS physics module they made in collaboration with Havok called Unity Physics which is in development.
And again the point I was trying to make was about graphics. Most if not all do not roll out their own code for input management, window creation, Networking and so on. But 2D graphics is relatively easy to DIY.
I've never seen a 2D game engine that doesn't give you at least graphics, physics, and input.
You can make simple games out if just hose libraries, but anything complex will tend to require a ton of support code (characters, animation, maps/levels, camera control, collision detection, event triggers and such for example) and other libraries, at which point you’re basically writing a small custom engine on top of those libraries.
Godot has a 2D renderer and physics option, Defold is 2D by default, Game Maker Studio, Cocos2d, Corona, Construct, Phaser.io, Buildbox, Gamesalad, Stencyl, Clickteam Fusion
Unity is very capable of 2d games too and provides a lot of game structure and so do many other engines (I want to mention Heaps.io because it’s awesome).
So while not at engine level, it already provides higher abstractions than those libraries, and a content pipeline.
You are missing asset management (loading, streaming, caching...), audio (big enough to split into libraries and actual audio engines), input (of different kinds), networking (big enough to split too), AI, scripting, physics...
Calling SDL/SFML for "other stuff" is, again, a tiny part and it is in no way offloading "the other crap" (as you call it).
I assume you are biased because you have only coded toy examples. When you start anything remotely complex, you have to do way more things than just "drawing bitmaps".
As for Factorio, it is actually a very complex and fast simulation. Downplaying engine devs as "rendering and other crap" is derogatory to say the least.
However, you can absolutely make 2D games that need lots of kinds of assets. RPGs happen to be one of them, in fact: The incidental details of adding inventory, NPCs, abilities, dialogue, etc. does add up. Every little message and description, every item's properties. You can ship a game by hardcoding much of it, but that's not going to scale with any substantial team. You need real data management, build processes, etc. Where assets interact with each other you get incidental complexity of the Dwarf Fortress bug kind, so where you can, you add static checks. The engine is built in tandem with the asset pipeline, in effect.
So - whether it's 2D or 3D, what actually matters is the assets. That's why even in the early 80's you had vector games with 3D effects in them; they just took approaches that simplified the assets and the resulting scenes.
This reeks of moving the goalposts.
A 2D sprite-based game might be able to get away with a much simpler animation system and still look great. E.g. sprite flip-books and really basic translation will get very far for a lot of game styles. Even if you want rotations, you probably don't even need to learn about quaternions since 2d rotations are so much simpler and you might not need to do interpolation/blending.
A lot of other parts can be more complicated as well. E.g. testing navigability is difficult in 3D since it's not always clear what you are able to jump on to/fall off of/squeeze through. Relatively common bugs in 3D games are falling out the bottom of the map, escaping out the side, or getting stuck, but they almost never happen in 2D games.
Controlling the camera requires a lot more thought in a 3D game. In a 2D game, a fixed camera or a camera that rigidly follows a character is often sufficient. In 3D, the field of view is awkwardly shaped in world-space, and obstructions are more likely and you may need a way to deal with them (e.g. manual camera controls, detecting obstructions and making them transparent, or moving the camera to avoid them).
I guess a lot of this comes down to being able to design a 2d game that requires a very simple engine while even seemingly basic 3d games seem to need tons of features to be palatable.
Quaternions are not hard, they are just a tool to deal with 3D rotations. On the other hand, 2D rotations have other set of challenges. For instance, sprites that are naively rotated look very bad.
Other rendering shenanigans happen in 2D, too: arbitrary scaling, for instance, is trivial in 3D games, while a nightmare in 2D. There is no general algorithm to do so properly, and some people suggest NNs to have a good-looking solution, similar to NVIDIA's DLSS.
Camera is not easy in 2D either. There are tons of ways to make cameras for different kinds of games, with different tweaks. There are entire articles written on this out there. Obstructions also happen in 2D for some kinds of perspective and different games provide different solutions.
Not everything 2D is an 80's platformer. For sure, you can make trivial games in 2D easily, but that does not mean they will be good, polished games. Same way making trivial 3D games is easy yet not good enough.