Stream Compression in NATS
qaze.app
qaze.app
If you look at [0] how does this explain anything? Seems like I need to write a service utilizing NATS which talks to the OIDC server.
NATS auth story is a complicated one, and now with auth callout it's even more complicated.
What I also quite didn't succeed in were JS client Jetstream consumers, async polling or w/e. All the examples, async or sync would block execution.
The regular request/response with microservices went well and straightforward. No issues at all. But writing middleware is also not very pretty, you have to have dedicated functions which you would call in the request handler (or microservice if you will) instead of being able to define a chain of middleware functions. Potejto potato but organization and structure and naming is half the code.
And more now you can't rely on http error codes, even if you can use them.
Then debugging is difficult, because messages are sent in binary format, and with no middleware to conditionally plug in, it's even more painful. Yeah you could write a dedicated service listening to >.* and printing deserialized JSON payloads but that's also extra work.
However dropping the HTTP overhead leads to a lot more throughput.
For now I'll stay with HTTP.
The reason I experimented with it was there is a company which required NATS knowledge.
[0] https://docs.nats.io/running-a-nats-service/configuration/se...
Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration.
> NATS auth story is a complicated one, and now with auth callout it's even more complicated.
There is a spectrum of auth options, starting with simple config-based, token or user/pass leading up to decentralized auth for use cases that need it. Auth callout is an opt-in thing, so it should only be adopted if it is truly necessary.
> you have to have dedicated functions which you would call in the request handler [...] instead of being able to define a chain of middleware functions
I don't quite understand this statement. Wrapping a NATS handler is the same approach as wrapping an HTTP handler (within the same client app). The function would take a handler and return a handler. There can be inspection of the message within that function and the choice of calling the next handler, responding early, doing some external call, or doing nothing.
If you want to distribute this middleware, then you would need to have separate, explicit subjects that each middleware would subscribe to and then publish to for the next element in the chain (for a choreography approach).
There is also the "message slip pattern" where the ingest component sets the path as metadata (e.g. headers) that each middleware component uses that to know (at request time) which subject to publish the result to next.
func loggingMiddleware(next micro.Handler) micro.Handler {
return micro.HandlerFunc(func(req micro.Request) {
spew.Dump(req.Data())
next.Handle(req)
})
}
Regarding auth, yes, spectrum of options, but for "production" use it's very complicated and you don't want a shared key, you need the whole operator, account, user thing, and external auth aka auth callout. The documentation is lacking. nkey, xkey what? It's confusing is complicated. Managing auth is a pain, more than once I've been locked out of my test server because of how complicated auth is and had to wipe it. It's so complicated because you have have isolated "namespaces" or operator spaces? Naming an organization operator, or a solution or service group doesn't help. user is synonymous with client. All that you don't get from the documentation, you have to watch Youtube videos to understand it. The documentation is bad.
How do I consume a stream from the deno/ws client without the whole client app blocking? No idea. The code snippets don't help. All there is, is a pay for company that offers to host NATS clusters, which you can pay to ask. Slack is not a good medium.
I'm guessing that the documentation is bad so one has to pay for support. And that's a no-go for me. I get that you spent time and effort into it, I'm assuming you're one of the NATS people, but what good is it when I don't know how to effectively use the software? The natsbyexample page is also hard to digest.
You're in your own little bubble and expect people to be mind readers.
If it's too complicated people won't adopt it. I know I won't because it's too complicated.
I would've loved to build upon it, because the core functionality is nice and works, easy to grasp.
I've not seen a single Jetstream stream and consumer tutorial with the new API and nats.ws and Go. Old API sure, STAN sure. New API no. Some real world stuff, not just code demo snippets. Why not? Because it's complicated.
Auth callout, that was the big thing everyone was waiting for? I would work on actually solving problems instead of building components so I can use this extra piece of software who's documentation is so bad.
You know how many Kafka tutorials there are? A LOT.
I want to like it, it seems very promising, but it suffers from the "coder no likey documentation" illness.
Today I saw a Twitch stream of some Netflix dev, talking about DX, developer experience.
NATS' DX has a lot of room for improvement, to put it friendly.
Out of curiosity, have you asked questions in Slack or Github? If so, and you had a bad experience with the interaction, I get the sentiment and would offer help. But this comment is not constructive without context.
> I'm guessing that the documentation is bad so one has to pay for support.
The NATS project has been open source for around 12 years, and part of the CNCF since 2018. This is an incorrect statement and very poor assumption to make because the documentation doesn't make sense for you.
I 100% agree it can be improved and we are working on a new docs site, but it is not quite ready.
In case its helpful, there is an increasing collection of examples on https://natsbyexample.com with new JetStream client examples among others. If you have specific requests, feel free to open issues in the corresponding repo: https://github.com/ConnectEverything/nats-by-example
I found some threads saying auth callout in 2.10 would solve this and decided to table the project until 2.10 but it's really really unclear how to work through the details of converting a "traditional" OAuth access token into the NATS-specified access token required by the auth callout contract.
- Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform client auth - If further authz/fine-grained control is needed the auth callout mechanism can be used. This would have access to the provided cookie/token so any claims needed for access control could be stapled on during step one and used at this point?
For GPs original question -- I'm running a fairly old Keycloak version (v8) but it does appear to set a JWT in KEYCLOAK_IDENTITY and KEYCLOAK_IDENTITY_LEGACY.
Am I right in understanding that IFF the token is signed with Ed25519 and both sub and iss are an NKEY value this is sufficient for NATS to accept that cookie as a credential?
As long as it can verify the chain of trust for the user JWT that is returned, it should work.
The three schema types are shown here: https://docs.nats.io/running-a-nats-service/configuration/se...
auth request comes in -> generate user jwt, sign + encode -> respond with auth response.
As long as the necessary bits of the response and user JWT conform, it will work.
Tried to click on the Docs link at the bottom of the page but that doesn't work.
Stopped looking.
And I also still have absolutely no fucking idea what any of this is.
I presume you are talking about the roadmap after the 2.10 release?