Yes, P/Invoke should work, especially when you are targeting a single platform. However, for multiple platforms, there might be some unexpected obstacles I am not aware of.
I'm just putting a disclaimer that using WASM is a very wrong kind of suggestion, would likely not work the way you expect it to (you would have to use WASM for .NET too which is in many places experimental and is a huge performance killer) and no one does it - there are appropriate ways to target multiple platforms in a solution that splits logic between .NET and C/C++/Rust/Zig/Swift/etc., especially that Zig offers nice cross-compilation toolchain. Mind you, the use case for this is accessing language-specific libraries and for performance the solution really is writing faster C# instead.
I want the editor to be usable in other GUI stacks, a C-compatible library is the only approach that makes sense here