HTTP Feeds | Hacker News Reader

HTTP Feeds

http-feeds.org

214 pointsmcp_95 comments

Feels like a bit silly not to have made this compatible with SSE since all the mechanisms for acessing that are built into browsers these days.

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... https://html.spec.whatwg.org/multipage/server-sent-events.ht...

Yeah, we're also pushing for SSE for this kind of things.

Having a standard for what events look like is a decent idea as people generally just make some random thing up without thinking too hard and having a standard is generally helpful - but making your own polling system etc when there's already a battle tested one available seems like wasted effort and a bit counterproductive

Does SSE work reliably in mobile browsers nowadays? A few years back when I tried, it was working OK in Chrome for Android but mobile browser support didn't seem complete.

Sadly, there are very few mobile browsers that aren't Chrome or Safari. In fact, Safari/Webkit is the only mobile browser engine available on IOS at all -- all other browsers are just chrome (pun intended) around IOS webkit.

To your point, https://caniuse.com/eventsource

Also there are polyfills for very old browsers like IE.

I had previously used SSE in a project and had no idea it existed or that it was widely supported by most browsers. It's a beautiful and straightforward technology, but it doesn't seem to be as widely used as it should be, and more tooling is required. Also more tooling is required.

100% onboard with better tooling.

Starting with out of the box swagger support: https://github.com/OAI/OpenAPI-Specification/issues/396

Shame they are not interested in adding it.

The really huge issue with SSE, and the reason why it’s historically been dismissed, is that it us subject to (and counts towards) the HTTP 1.1 domain connection limit.

Sadly that makes it a bit of a pain in the ass to manage for sites served as (/ compatible with) http 1.1.

I know SSE works on scenarios with few or no state updates, like in dashboard UI's. What about other cases where they deal with input data? like in a chat/game

That’s the beauty, the client just do a normal HTTP request. It really simplify the system.

> "Feels like a bit silly not to have made this compatible with SSE ..."

The feedback is great, and valid. I wish it could have been expressed less dismissively though.

Depending on the culture you are living in, this can be perceived as either dismissive or respectful.

I was curious enough to browse through the comments of @lexicality to conclude that he is European, so there is a chance that you were considered capable of receiving a direct feedback.

That is, by giving direct feedback, @lexicality showed you respect.

The feedback isn't to me.

Also, I live in northern Europe. Generally feedback is direct, but respectful. Calling something silly is usually (but not always) out of bounds. European-style feedback might be: "I'm surprised you did not use X because it perfectly matches your use case."

Or "Great feedback but I wish it could have been expressed less dismissively"

You know, this might not apply here but lemme contribute this:

Recently at my place of work we were required to take some diversity training. In this training we learned about high context and low context regional and familial cultures. High context individuals convey and respond to indirect details and nuance, while low context cultures are more direct with what they receive and convey.

Learning this helped me a lot and I notice these differences often, as I too come from a different cultural background than my team mates.

I don't understand. Does the diversity training teach you to treat people differently based on where you think they came from?

It teaches you that if you are a high context individual receiving low context feedback to try and negate your frustration by understanding the possible factors influencing the response you received before letting it be a blocker in your interactions, also the inverse.

I think it's more similar to learning to speak somebody else's language and not get lost in translation.

Less this and more that one should be cognizant of their own inherent communication biases.

I've seen these skills described as CQ (Cultural Intelligence), distinct from EQ or IQ. IMHO, it's the 3rd leg of the stool for effective intelligence in any non-solo endeavor.

I hadn't heard those terms before, thanks. I'll check those out.

A couple things:

A) People have vastly different (even abusive) forms of communications. What works for one group does not work for another. "Sensitivity training" does not account for this, in fact it causes issues.

B) EQ and CQ are made up terms, but more importantly they aren't realistic. IQ, also is somewhat made up, its used as a sort of "general intelligence" marker and it is that in the context of broad studies, not so much on the individual level. IQ has been long used to browbeat certain ethnic groups, these new "made up" terms are similar, in that people are using them to browbeat various groups, in attempt to put themselves on a higher "moral ground". Its disgusting, and insensitive nonsense. They don't exist, and we don't have the bandwidth to all just "conform". And no, thanks I don't take kindly to being brainwashed.

This is interesting, thanks for mentioning. I wonder what the learning from that is though? Like, if you notice the difference, do you adapt to the other style? I feel like I can do that, but it also feels weird and "not me".

It's more about addressing ways out of social conflicts due to misunderstanding context. Both sides can mean well but a mismatch in speech patterns can cause undue conflict. Understanding these differences gives you an "out" from those negative feelings.

Part of learning to be an adult - growing thicker skin, a backbone, and ALSO understanding that everyone is different, learning to navigate that rather than just get frustrated or upset like you are two-year-old.

Just wanted to add that as a rule of thumb, Western cultures tend to be low-context, while Eastern cultures tend to be high-context.

This also ties in indirectly to punctuality of cultures I think. Lower context cultures tend to be punctual.

And I've felt the Eastern cultures are becoming lower context due to doing business with the West. This may or may not be a good thing.

PS: I'm from a high context culture, but I've lived in America for a while before returning back to my home country. Ever since, I've felt like something was off.

"silly" is used a little more casually stateside, it's actually more of a light touch descriptor

I believe the HN guidelines actually say something about this, but either way: I think there is tremendous value in interpreting other people's statements as charitably as possible.

Why take offense when none may have been intended?

At least where I come from (Ireland, but I'd say it's the same in the UK where Lexicality is from), "silly" and "stupid" are distinct. You can describe something like this as "silly" perfectly respectfully, but describing it as "stupid" would be out of bounds.

I don't agree that this instance is among the kind that can "be perceived as respectful" - I think we can make space to acknowledge that that is a thing, and that nevertheless this is not an instance of respectfully direct communication. So that's a red herring.

Moreover I don't see that "being European" has anything to do with anything, and classifying someone as either being "capable of" or not "capable of" receiving direct feedback, far from respectful, is an instance of insulting with plausible deniability, which is perhaps the most practiced art form of internet comment sections.

Calling something silly is not direct, nor respectful.

Perhaps your view of communication with Europeans is influenced by other factors that's led you to this conclusion? (or to put it respectfully, silly)

For me atleast as an Australian this would not be considered even remotely disrespectful.

First, it's addressing the idea rather than the person.

Secondly silly isn't used to convey an idea being bad, stupid or otherwise without merit.

The way I read this is that "I think this should address why it's not built on SSE or perhaps should be redefined to be built on SSE", silly just does that with much less words.

"Silly" also conveys the feeling that it's probably too late to redefine this as SSE.

I am reading all the replies and thinking the word silly seems to convey a different thing to different people. At least in the British context it is extremely mild.

But somehow others read it as like you said "being bad, stupid or otherwise without merit."

Yeah I'm honestly surprised too, but I'm Australian so effectively British for all intents and purposes when it comes to what language we find offensive (i.e very little lol).

Do you find it easy to find great, honest, and valid feedback on the internet?

How much of the supply of great, honest, and valid feedback would you sacrifice in order to make the supply that remains use "less dismissive"[0] language like "a bit silly"?

[0] Less dismissive in quotes because I do not find that language dismissive in the slightest

Let's chalk it up to cultural difference! Some commenters think "silly" is fine (Ireland, UK, US - although I'm from the US and I disagree, but let's grant it). Some think it's not.

Let's assume that "silly" was meant kindly and not dismissively!

This reminds me of CometD - https://github.com/cometd/cometd. Is it same in concept?

Had never heard of SSE before. Followed the link and found this warning:

> Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100).

This is the first time I’ve heard of per-domain connection limits. Seems… not great? Doesn’t this turn into a client side DoS? User opens 6+1 tabs and now the browser has exhausted all HTTP connections to your domain serving SSE connections?

I’ve used long polling before, I don’t understand how I’ve never observed the connection limit of 6…

Perhaps we should define a new HTTP header for servers to say: "It's ok for clients to initiate N connections with me" so that we can write web apps without this restriction.

...or we just keep moving to http/2 which avoids this limit.

That'd be great, but is a lot harder than adding a header. HTTP/2 changes the entire wire format into a binary protocol that multiplexes multiple streams over a single connection. It's a lot harder to implement than HTTP/1, which you can type out manually over telnet:

    telnet news.ycombinator.com 80

    GET / HTTP/1.1
    Host: news.ycombinator.com
    \n
    \n

Adding a header could be as simple as:

    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Multiple-Subscriptions: OK

The limit applied only to browsers which have all implemented HTTP/2 at this point.

Yes, the limit is implemented in browsers, but it's for the protection of servers. The idea is to prevent a web app from DDOSSing a server somewhere.

This is especially a problem for servers implemented with one thread or process per connection, rather than an async event queue like nodejs, because each thread takes up a few megabytes of memory, and if a browser opens 1000 connections, that's 2 gigs of memory right there.

But if a server is implemented properly to support long-lived connections (by using node, or greenlets, or other async system) then it should be able to opt out of that browser protection. A standardized header would allow a server to opt out of it.

> I don’t understand how I’ve never observed the connection limit of 6…

Practically speaking this limitation has been obsolete for 5 years or so. With HTTP/1.1 there used to be ways to achieve more parallel downloads through domain sharding [1] on the server side and browser tweaks [2] on the client side.

[1]: https://developer.mozilla.org/en-US/docs/Glossary/Domain_sha...

[2]: http://kb.mozillazine.org/Network.http.max-persistent-connec...

This limit was(/is) actually a common problem with HTTP/1.1, and was one of the motivations for HTTP/2. A common workaround is "domain sharding"[1] where you would split your site up between several domains, so that you could get more concurrent connections. Which is important for improving page load time, but doesn't really help with having SSE for multiple tabs (unless you do something crazy like using a randomly generated domain for each SSE connection).

[1]: https://developer.mozilla.org/en-US/docs/Glossary/Domain_sha...

Hi! I am the author of http-feeds.org. Thank you for your feedback.

For this spec I aimed to keep it as simple as possible. And plain polling-based JSON Endpoints are the most simple and robust endpoints IMHO.

If you need, you could implement an SSE representation on the server endpoint by prober content negotiation.

The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages. Plus, it is quite hard to debug SSE connections, e. g. no support by Postman and other dev tools. And long lasting HTTP connections are still a problem in todays infrastructure. E. g. there is currently no support for SSE endpoints in Digital Ocean App Platform, and I am not sure about them in Google Cloud Run.

Overall, plain GET endpoints felt much simpler.

Thank you for writing this great spec up for others to use!

I think totally right that back-pressure and plain GETs are an important use-case to support, and am really happy to see a beautiful spec written up to articulate concretely how to support them.

It is also great to be able to switch amongst these methods of subscription, for instance, if your server can keep a persistent connection open, it's nice to be able to get realtime updates over a single channel, but to still be able to fall back to polling or long-polling if you can't. And if you switch between a polling and a subscription, it's nice if you don't have to change the entire protocol — but can just change the subscription method.

There's a growing effort to integrate all our various subscription-over-http protocols into an IETF HTTP standard, that we can use for all these use-cases: https://lists.w3.org/Archives/Public/ietf-http-wg/2022JanMar...

Maybe you'd be interested in incorporating your experience, use-cases, and design decisions into that effort? We have been talking about starting at this November's IETF. [1]

For instance, you can do polling over the Braid protocol [2] with a sequence of GETs, where you specify the version you are coming from in the Parents: header:

    GET /foo
    Parents: "1"

    GET /foo
    Parents: "2"

    GET /foo
    Parents: "3"

Each response would include the next version.

And you can get back-pressure over Braid by disconnecting a subscription when your client gets overwhelmed, and then reconnecting again later on with a new Parents: header:

    GET /foo
    Subscribe: true
    Parents: "3"

    ..25 of updates flow back to the client..

    **client disconnects**

Now the client can reconnect whenever it's ready for new data:

    GET /foo
    Subscribe: true
    Parents: "28"

Or if it wants to re-fetch an old version, it can simply ask for it via the version ID it got:

    GET /foo
    Version: "14"

And if the source for these updates is a git repository, we could use a SHA hash for the version instead of an integer:

    GET /foo
    Version: "eac8eb8cb2f21c5e79c305c738aa8a8171391b36"
    Parents: "8f8bfc8ea356d929135d5c3f8cb891031d1539bd"

There's a magical universality to the basic concepts at play here!

[1] https://www.ietf.org/how/meetings/115/

[2] https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...

I'm not entirely sure what you mean by this. SSEs are just normal GET requests with a custom header and some formal logic around retries. I've even implemented them manually with PHP using the `retry` command to mean I don't need to have the connection open for longer than a normal pageload.

> The main reason, why I dropped SSE it the lack of proper back pressure, i.e. what happens when a consumes slower than the server produces messages.

Could you point me to where the spec handles this please? As far as I can tell it has the same problem of the server needing to buffer events until the client next connects

Very cool! This looks very similar to what we're doing with the braid spec[1], though I really like how clear and concise your examples are! This is a great little website.

Some differences between our approaches:

- I think its a good idea to support arbitrary patch formats via a content-type style field, just like we have different formats for images. This lets you use the same protocol for things like collaborative editors.

- For some data sets (like CRDTs), you want each change to be able to refer to multiple "parents".

- Your protocol is very JSON-y and not very HTTP-y. It looks like you're basically using JSON to express HTTP. Why not just use HTTP? One big downside of the JSON approach is that it makes it awkward to transmit binary patches. (Eg, 'patching' an image)

Feel free to reach out if you're up for a chat! Looks like we're working on the same problem.

[1] https://github.com/braid-org/braid-spec/blob/master/draft-to...

This made me all warm and fuzzy inside that nerds can still help other nerds even when we are on different teams. Hope y'all are able to share some ideas.

It'd be great to get a fairly standard way of doing this. :)

Having worked in this problem space a bit recently, I find this part a bit too optimistic:

> The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.

The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.

Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.

Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.

The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)

A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.

Perhaps the time problem can be handled with an eventually consistent Lamport clock system

Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes.

Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.

> Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think.

My point was that even if you have a single non-scaleout database with a single time source, a sequence or a timestamp or a combination of both isn't as reliable a sequencer as you might think, unless you have at most one writer.

Thus, I think a "standard" should encourage a cursor concept that can use something that may reliably provide _all_ changes. If you have a single writer, you have a pretty easy job implementing that, as a plain sequence would work. (A timestamp could still break on clock adjustments, though)

This pertains to the "data replication" part of the listed goals, where getting everything is more important than in e.g. a social media news feed style thing where chronological order may be tenable - or less consequential if an item is missed.

> Thus, I think a "standard" should encourage a cursor concept that can use something that may reliably provide _all_ changes.

I think the Braid standard might support what you want: https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...

We don't call it a "cursor", but the idea is to encode whatever clock or versioning information you want as an arbitrary string, with the only general requirement that the string is unique to that version.

You can use this with Lamport clocks, Vector Clocks, Version Vectors, Wall Clocks, (Agentid+Sequence) pairs, SHA hashes, or even blockchain transactions.

Hadn't come across Braid yet. Looks interesting. Thanks for the link :)

JSON inside server sent events gives you a browser native client, no need for streaming extensions to your JSON lib which is kinda rare, and various intermediates know not to buffer the content type already. Sorry to be the what about X guy, maybe there's something I missed?

Yeah this feels very much like someone retyped the Server Sent Events spec and bolted an arbitrary spec on for each message...

Yeah batched arrays isn't the way to go when when you want to mimic what you would get from a traditional message broker. This really should be a spec that builds on SSE but specifies the encoding (CloudEvents) and parameters to negotiate the serialisation (JSON/Protobuf/Avro) and lastEventId, then it would be much more useful.

Can someone ELI5 the significance of this?

* What are some scenarios where you would need a feed?

* What's being done now to solve the problem and how is this different?

Many thanks

This basically puts a REST/HTTP GET frontend on your messaging backend (Kafka, RabbitMQ, MQTT, etc).

So whatever events you want to expose publicly can be done over plain HTTP.

Having said that, IMO using SSE or websockets would be a better fit than raw HTTP.

I could see this being used as part of a distributed system for shopping. for example, say you have 20 stores, each with its own storage area for items, but you also have 2 large warehouses. Your online site might need to know which stores have stock now, or which warehouses have stock. The site might allow you to do "Click and collect" do it would need to know, in somewhat real time, which stores actually have the stock. each store would have its own endpoint with a data feed that the central server can get data from. Same with online orders for delivery. It needs to know what warehouses have the stock, and if none do, how to get the store to ship it to a customer.

Likewise, the stores might need to know what is in the warehouses, or even across town; someone walks in store to order something, but its not in that store. But they know, in real time, that its in the warehouse for delivery next day or that the store across the city has it.

What is done now is a more centralised approach; all sites would have a connection to a single DB in head office that stores everything. This makes things more distributed and, in theory, removes a single point of failure.

I should clarify, I am not working on any of this, this is just how I think it would work... if anyone wants to step in and tell me if I'm wrong, right, or just plan stupid, please shout.

Great answer and exactly what I was looking for! Thank you

I like this idea, however I think the event ID being encoded in the response body places constraints on what each element looks like. Perhaps it would make more sense to encode the last ID/feed position in a response header and have the client submit that in a subsequent request header? That would decouple the feed position from any one element or the response structure itself.

This seems similar, but not as fleshed out as RPDE feeds, which has already been accepted by the W3 [1], and provides many answers to lots of the 'eventually up to date' questions.

[1] https://www.w3.org/2017/08/realtime-paged-data-exchange/

The idea is nice and needed. However maybe the spec is a bit elaborate for me to adopt it immediately. I've been rolling my own for some time at some clients, for our kafkaesque/event sourcing patterns.

However what I used there was simple http stream/json stream like this:

- No start of [] but JSON newline entries a new line is an new entry

- Using Anything as an id (we've been using redis XSTREAMS as lightweight kafka concepts, just 64bit integers)

- have an type as an event, and versioning is just done by upgrading the type, ugly, but easy.

- We'er considering using SSE at this moment

Compaction is not something that I would do in the protocol I think I would just expose another version of it on a different url I think or put it in a different spec.

How do you recover when there is e.g. some connection error and a client misses some events? Can the client ask to replay the events? Then, how far back can they go?

It's an interesting idea, but I wonder how well the promise will hold up that this is less complex than using a message broker.

Yes, the network topology and protocol are certainly less complex, but there are now additional strong requirements how an endpoint has to store and manage existing events. (See the bits about strict time ordering, compaction, aggregation, etc).

A lot of this is effectively what a message broker is doing in a "traditional" system to guarantee consistency. Those tasks aren't gone, they are just pushed to the endpoints now.

My thoughts exactly. While this is more human readable on the wire, a message broker delivering the feed would provide many different other features that might be useful, such as transactions, load-balancing, guaranteed delivery and per-endpoint state to simplify the individual application instances.

For those not aware of what message brokers are, there are many to choose from such as: Mosquito, RabbitMQ, ActiveMQ, Solace... If delivery over HTTP is a requirement, many of these brokers support delivery over websockets or (in the case of Solace) also support long polling.

A few issues with message brokers, esp. in the system-to-system integration:

- Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... - Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, ... Who is on call when something goes wrong? - Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...

> Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ...

That's a valid point. I think it's a pity we don't have an equivalent standard for asynchronous messaging with the same support as HTTP. However, there are lots of options for presenting an asynchronous public API that would use your message broker behind the scenes, without fully exposing it: Websockets, SSE, web hooks, etc...

> Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user,

True, but don't you have that anyway? How is this different from requesting a new database, service route, service definition, etc?

> Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...

How simple or complex this is depends on the concrete broker at hand. There are some protocols, e.g. STOMP that are simple enough that you could write your own client. And as I wrote in the parent: HTTP feeds are a technology as well. You'll have to think about troubleshooting and resiliency there as well.

This seems like a really attractive way of publishing events to a third party. Platform independent and easy to understand.

  > Platform independent and easy to understand.

The platform independence is implicit in the "HTTP" part of the name.

With more attention going to long polling again, I wonder if it would be useful to introduce some kind of HTTP signaling (header or 1xx status) to indicate on the protocol level that long polling is going on. This might be useful information for intermediaries - e.g. proxies, firewalls, browser network tab, etc.

My first thought is if the client does not specify a start ID it could be sent billions of records depending on the elapsed time and frequency of events. What would be the best way to avoid overloading a client?

I think the server will give a truncated response.

They need to clearly document what is the behaviour for this case.

This could form the foundation of a distributed social media network.

How about "catching up" on events, if a client had an outage, can it indicate the last event ID it had and then get a natural replay of the older events?

That's the second example on the page.

Doh... thanks for pointing it out. I asked because it says elsewhere that persistence of objects is a non-goal, I guess it meant storing them clientside or something.

I wrote a blog post advocating for something like this recently, with an additional push to notify the client when it should poll: https://blog.separateconcerns.com/2022-03-05-push-to-poll.ht...

I wasn't aware of this but it fits the use case perfectly, I will update the post.

Indeed, these kinds of abbreviated notifications are a great way to implement realtime updates. The code is simpler and there's less chance of screwing something up.

I've also seen this described as hints: https://blog.andyet.com/2014/09/23/send-hints-not-data/

I think serving data from multiple Kafka partitions gets unnecessarily hard if your "continue from this point" token is tied to a singular event ID. For that, it'd be better to have the "cursor token" be an arbitrary blob of data you repeat back to the server. Then the server can e.g. encode a list of partition->offset values into it.

I always implement my own standards. I am faster that way instead of learning what others created. Obviously this is only true for less complex subjects.