For single-file-no-project simplicity your best bet is dotnet-script: https://github.com/filipw/dotnet-script
For native code your best bet is NativeAOT: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT
I'm not aware of anything that combines the two.
As to why: it's not so much that they've made it hard as they haven't made it easy. The reason they haven't made it easy is that it's a fringe use case. The main benefits of AOT are faster startup time, smaller storage requirements, and compliance with the iOS interpreted language ban. Most people who worry about those things don't mind having a project file.
They _have_ made some changes that may partly address your concerns. Visual Studio is no longer required to build C#; you can do it purely from the command line. There is also a new project file format. It's still XML, but much simpler. The HelloWorld example[0] from NativeAOT is a good example of how simple it can get. And the command line tools include an easy way of creating basic project files[1], so you don't have to memorize what little boilerplate remains.
[0] https://github.com/dotnet/runtimelab/tree/feature/NativeAOT/...
[1] https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-ne...