LFS is a hack and a workaround for the fact that git ships the whole thing and requires you to have every copy of every blob from the whole of history.
LFS cheats this by storing the big blobs elsewhere and only fetching the ones needed for your checkout. It's a giant pain as everyone knows because you've got two sources of data. Instead lots of work is being done on git to remove the "every copy of every blob" requirement. This is in 2 dimensions - only some specified depth of history, and only some subset of the checkout (LFS only helps with the first). Effectively you still only end up fetching the blobs you need from the server for the checkout you're doing. That gets the same benefit of LFS but all in core git.
LFS makes a 'centralised' assumption about Git usage. Git is designed to be decentralised, even if that's not how we use it, whereas LFS requires a single central object store. An example of where this is a problem: we do a lot of development for clients. It's very convenient to be able to mirror the git repo to their Gitlab and git usually makes it very easy. If you have LFS in your project then you have to jump through a lot of hoops.
The second problem is that LFS copies its data exclusively via https. This creates some security complexity if you normally authenticate via SSH.
Regardless, I'm going to be sad if the UX we get to replace it is any of the newish/cutting-edge Git features, because they're damn ugly. In fact, despite how annoying LFS is for being a separate thing and requiring extra effort for server-side support if you're self-hosting, given Github and Gitlab (among others) support for LFS, I don't see things like sparse-checkout replacing it unless they get a lot nicer to use for that purpose specifically.
One thing that might help would be for Git to make distributing & sharing default repo config easy and built-in. Only thing that gets that treatment is the .gitignore. I feel like I need source control for my source control config, sometimes, and, indeed, some software's cropped up to do just that for e.g. hooks, but that ought to be built-in and cover more than that. One nice thing about LFS is that there's zero client-side config, aside from installing & enabling it (globally, if you like). You don't have to pass around lists of LFS-managed files or checkout-patterns or set-up scripts (hope you made them idempotent!) or anything out-of-band.
I don't quite get this point. The .gitattributes file is key to the operation of LFS and is modified and passed around in exactly the same way as .gitignore.
A good built-in implementation should do exactly what LFS does, yes. But they don't. LFS does, and that's part of why it's nicer to use for its specific purpose than doing something with shallow clones and sparse checkouts and all the rest. The best way to replace LFS in git itself would be to copy its UI as completely as possible, because it's pretty good.
Having to install & enable it is a pain, which making it part of git would fix. Having to maintain separate support for it server-side, and a separate transport mechanism, is a pain, which making it part of git at least could fix if the authors wanted to (in fact, I'd think it'd be more work not to solve that problem, in implementing it, though it'd leave vendors who want the two to be significantly separate to fend for themselves)