It might be to formal for your use-case, but there is a standard defined for error responses in RFC 7807:
I’d love to hear from anyone who has encountered APIs in the wild that actually implement this standard!
https://docs.microsoft.com/en-us/aspnet/core/web-api/handle-...
https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnet...
One thing to keep in mind re. "maybe-json-maybe-plaintext-maybe-empty" responses is that the more complex your errors, the more likely the error handling encounters an error. If you're trying to send back some JSON or XML but it fails it's usually better to at least shove out a line of plain text and hope it reaches a human than to mask the real error with a second fallback in the "right format" but with unhelpful fixed content.