Really cool to see this I hope you're able to continue this work. As others have said, CalDAV / CardDAV are complex standards and getting the nuances right without using some tried/true libraries (sabre-dav) might be a challenge.
Two points:
For just CalDAV / CardDAV there's Radicale, Davical and Baïkal all of which work pretty well and are relatively easy to set up, so this might be a bit of wheel re-invention.
Perhaps you might consider relying on those projects for calendar / contact sync and then add your other pieces (file, photos, etc).
Second point is I think the reason why Nextcloud / Owncloud end up becoming super-slow for file sync is unfortunately the use of WebDAV.
I can't find the thread at the moment but I remember the developers from one of those projects saying that WebDAV itself makes the file-sync super slow and resource heavy.
As a result I think the Owncloud++ or their golang based re-implementation prefers the use of S3-compatible for backend file store.
I'm not saying you should use S3, but if you continue down the path with WebDAV for file-sync you might find it's resource heavy and slow as Nextcloud/Owncloud might be.