You probably have some fun disk caching turned on in the OS or your disk has a built in cache that it’s not properly flushing. I’ve seen this before with cheap SSDs.
Maybe, I don't know enough about this, but I've been using this veracrypt disk on 6 machines, now, and I have the problem with each of them.
Also, I don't have any problem with non encrypted data.
Looks like a veracrypt problem then.
Maybe a dumb question, but are you on windows? Check that Defender isn't scanning every single time data is written to your veracrypt drive. When I was on windows, the most painful slowdowns were almost invariably caused by Windows Defender.