What happens with the "I'll build everything from scratch" apps is they are oftentimes not easy to understand, different patterns end up being used, 3rd party packages get slapped on ad-hoc and over time it just becomes a frankenstein. Much better to start with a hard set of rails that follow a set of conventions you just have to learn once.
The whole "I can move faster" without a framework is just an illusion that appears at the beginning of building an app. It will quickly disappear as more devs and complex code features are requested.