Don't get me wrong, boards like this (and ESP/Realtek boards with WiFi) are great for hacky prototypes. But if you're building an embedded device that will be deployed in any sort of volume, a small application processor running Linux will make your life dramatically easier. You can keep it asleep most of the time if your power budget is tight.
The hard part is not cramming in an Ethernet PHY or a 2.4 GHz radio, it's the mountain of software that you need to run on top of it to get the kind of reliable, secure communications channels that we've come to expect. Bare metal networking stacks like LwIP have a reputation for being buggy, and are nowhere near as battle-tested as the Linux/BSD networking stacks security-wise. Some of the more memory constrained devices can barely fit a complete certificate chain. Are you really prepared to roll out updates on a system like this when the next Heartbleed comes along?
Also, the layers tend to pile on, so while you might get away with a bare-bones wire protocol initially, you'll be starting your project close to the limits of what is practical. Somebody will eventually ask you if you can connect to a websocket. I know buildroot and yocto look a little scary at first, but they're better than designing yourself into a corner before you even get off the ground.