I used to use caldav but then stopped because there was once a bug on the server side where I couldn't delete events. The main thing I also don't like is there's no encryption or privacy really from the provider unless you go with a more a less proprietary solution like from an encrypted email provider. The closest I saw was EteSync but it requires special apps, and can include special bugs :)

I also don't need my immediate todo list on a calendar. I organize my to-do list simply as "Immediate", "Future", "Distant future", and then put things under heading. Sometimes I add a due date if there is one.

I just had a few markdown documents in a "todo" folder, eg <work>.md <project1>.md etc. Recently I changed it to org-mode because that's a syntax designed for the purpose. https://nvim-orgmode.github.io/ works excellently.

Never been an emacs user in my life. I spent about 5 minutes perusing https://orgmode.org/org.html and was able to do the same.

I use syncthing to sync it between my devices. https://www.orgzlyrevived.com/ works great on android.