For anyone looking in this direction, Obsidian[0] with the excalidraw-plugin[1] is an awesome solution: Free for personal use and cross-platform.

[0]: https://obsidian.md/ [1]: https://github.com/zsviczian/obsidian-excalidraw-plugin