The problem with this approach is that you can make it as small as you want but it's still Linux. At a certain point are you going to start patching things out like support for users? Support for management of multiple processes? There's a non-trivial set of syscalls and data structures designed solely for these constructs. You can't just seccomp it and call it a day.
For us it's not about the size (that's nice of course) but it's more about the performance and security.