If you have a high performing team, great, do what the fuck you want.
If not, then chances are the team will need some guardrails, that’s what agile and scrum offers. It’s a place to start.
Just because you don’t believe in having a system to break down work doesn’t mean it isn’t useful.
Personally, I’ve seen the alternative in action and it always leads to burnout.