Why reinvent the wheel? Just use a common standard... I prefer https://jsonapi.org/.
If you want jsonapi, you probably really want graphql...
> ou cannot represent an API backed by jsonapi with swagger/openapi
This doesn't seem right to me. What issues were you having trying to specify a JSON:API service using OpenAPI?
I personally often use jsonapi.net[1], a C# implementation of JSONAPI. This supports OpenAPI/Swagger with swashbuckle, has a very good filtering implementation and together with Orbit.js[2] it is pretty much without having to decide too many things...
[2]: https://orbitjs.com/