My main problem is that I need to do operations on the data while it's in the DB. This means that I cannot leave it encrypted end-to-end there.
The data accessed by the app is not encrypted, you can still work on the data as you would usually do. It's mostly a compliance thing. Not sure what level of security it _actually_ brings to the data itself, but most companies are okay with "encryption at rest".