matt_oriordan

Netlify/Vercel for static site hosting, intercom for chat, SendGrid for mail sending, Ably for pub/sub real-time messaging (yes I’m from ably.com!), paddle for subscriptions as opposed to stripe...

Check out https://guild.co. Quite different to Slack, more like a private professional version of WhatsApp. We use it to communicate with professional groups around topics.

How does QUIC or HTTP/3 change things? Polling is polling, regardless of whether it's over HTTP/1, HTTP/2 or HTTP/3 (QUIC).

He he, that is true, and not something we're proud of. We're continuing to optimize things where we can with the resources we have. Given our size, we are spending our engineering efforts on our product where we can bring our optimization work on streaming to our customers. Sadly, as a result, our blog has plenty of room for improvement. As we grow, I hope the existing optimization tasks in our backlog are prioritized.

I appreciate this may come across as hypocritical, and you're more than welcome to think that. But I don't think that changes the analysis of my article, being that Google are telling everyone else to optimize their sites because it makes a better web, or you'll be penalized (https://www.sitecenter.com/insights/141-google-introduces-pe...), and then on the other hand they have over 100k staff and haven't optimized their own results.

I agree complexity should be avoided. But I don't really follow the logic that if every feature can break security, reliability, etc. you should never change anything or progress. Google is doubling down on these features as you can see at https://www.seroundtable.com/google-pin-live-scores-28261.ht.... If they're continuing to invest in adding more realtime features, and penalising sites in search results that don't optimise their websites https://www.sitecenter.com/insights/141-google-introduces-pe..., then I think they should apply that same logic to their own site IMHO.

This feature is not for the Laver Cup, it's for all sports events and scores they have access to, and has been so since 2016 - http://www.thesempost.com/google-adds-live-sports-scores-sea...

Sure, but I'm never suggested front-end developers should build new tech every time they want a new feature. But there are solutions to do things better now, so it seems odd to ignore that. It's a bit like saying React/Angular/Vue/any other frontend framework won't make things tangibly better, so why bother. Or optimising page size won't make things better, so why bother. Google actively invests in telling everyone else to optimize their websites and even prioritizes fast sites over slow sites (https://developers.google.com/speed). Why shouldn't they care about their performance too?

falcolas I am not aware of any single firewall that blocks long polling, or XHR streaming (at least for a fixed period). Can you substantiate that? We regularly check our transports against legacy devices at Ably, so I don't think this is true. I stand to be corrected of course :)

Agreed, and what firewalls block long polling, which is in itself significantly better?

Yup, the article covers SSE, long polling and XHR streaming as all vastly superior solutions that would have improved things. So yes :+1:

fooblitzky it's interesting you think that handling millions of open websocket connections with state is needed. Firstly, all of these browsers already have connections to Google's servers. HTTP keeps a connection pool open, so keeping connections open is still needed regardless of the transport. Given that, why not use long polling? No state is needed in that situation. And if you're doing that, what's wrong with one additional step, and stream updates over an XHR connection, which too is effectively stateless given any request can fail, reconnect, and subscribe to the stream of updates.

I appreciate streaming is harder than polling. But IMHO, if that's Google's thinking, that feels a little defeatist that has designed and built complex systems like Millwheel (https://ai.google/research/pubs/pub41378) to solve these types of problems.

I appreciate when features are in test phase that it makes sense to keep it simple. But this is in production, and there are many dead-simple solutions that do this efficiently, long polling, SSE, etc. This is Google after all.

Sure, but long polling and XHR streaming work for all browsers too. The article was not saying you should use Websockets, it was meant to say HTTP polling is the least efficient way possible to do this :)

Sure, but that was not the point of the article. It works, but badly, and even long polling would have been a significant improvement and work for every user.

Ably realtime | Distributed Systems | Full Time | London, UK | www.ably.io

We're growing quickly and want more nice smart people to join our deeply technical engineering and devrel teams.

INFRASTRUCTURE

- Head of SRE & Senior SRE

ENGINEERING

- VP Engineering

- Distributed Systems Engineers

WEB ENGINEERING

- Head of Web Dev & Rails Engineers

- Front-End Developers

DEV RELATIONS

- Head of Developer Experience & DevRel advocates

- Technical Writer

Find our more on our careers page at https://www.ably.io/careers

Ably realtime | Distributed Systems | Full Time | London, UK | www.ably.io

We're growing quickly and want nice smart people to join our deeply technical engineering and devrel teams.

INFRASTRUCTURE

- Head of Infrastructure

- SRE

ENGINEERING

- VP Engineering

- Distributed Systems Engineers

WEB ENGINEERING

- Head of Web Dev & Ruby Engineers

- Front-End Developers

DEV RELATIONS

- Head of Developer Experience & DevRel advocates

- Technical Writer

Find our more on our careers page at https://www.ably.io/careers

Ably realtime | Distributed Systems | Full Time | London, UK | https://www.ably.io

We're growing quickly and want smart and nice people to join our deeply technical engineering and dev rel teams.

INFRASTRUCTURE

- Head of Infrastructure

- Head of Client Software

ENGINEERING

- VP Engineering

- Lead Distributed Systems Engineer

- Distributed Systems Engineer

- Site Reliability Engineer

WEB ENGINEERING

- Lead Ruby Engineer

- Senior Ruby Engineer

- Senior Front-End Developer

DEV RELATIONS

- Head of Developer Experience

- Developer advocate

- Technical Writer

Find our more on our careers page at https://www.ably.io/careers

Serious stuff indeed ;)

Thanks spking for posting! Although not entirely clear why.

Disclaimer: I am from Ably

Ok, thanks for the heads up. Will raise a status website issue.

Thanks :)

Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service.

At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647

We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.

Ah, used to be an Enterprise only feature. We have access to that, but didn't realise it's now available to Business. Perhaps time for them to consider offering it more widely!

Problem is not just about using Cloudflare as your DNS registrar really. Even if you have a different registrar, the Cloudflare model is to have the NS (nameserver) records set up to point to Cloudflare, and then they in turn resolve the DNS. You cannot really use Cloudflare without that set up. Changes to nameservers at a registrar level are rarely quick, at least quick enough to mitigate a disaster like this. It's why we've used two completely different domains at Ably (ably.io and ably-realtime.com) for all services we provide.

We wrote about a strategy to circumvent this sort of thing a little while back https://www.ably.io/blog/routing-around-single-point-of-fail.... Given two incidents in a matter of weeks, I think a revisit of that article in light of most businesses who operate on a single domain would be useful :)

Ably realtime | Site Reliability Engineer | Full Time | London, UK | On-site / Remote | https://www.ably.io

If you'd like to work somewhere where you're solving complex internet-scale problems for a distributed serverless messaging system, then do consider join us.

We are currently looking to grow our engineering team with the a Site Reliability Engineer (https://jobs.ably.io/o/site-reliability-engineer) in our London office or remote (commutable to our office when necessary).

Oh my god, this post is ridiculous. The entire substance behind this article is based on a satirical article written by the Basecamp founder Jason Fried himself, https://m.signalvnoise.com/press-release-basecamp-valuation-.... I applaud the thinking that exits, rapid scaling, and VC funding is not the only option (I know, I did well without any of that but did eventually exit), but if you’re going to base your entire article off a valuation that was a fabrication, I am afraid the rest of this article loses all credibility. Sorry to be the bearer of bad news!

Ably realtime | SRE and Distributed Systems Engineer | Full Time | London, UK | Onesite / Remote | https://www.ably.io

If you'd like to work somewhere where you're solving complex internet-scale problems for a distributed peer-to-peer serverless messaging system, then do consider join us. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-....

We are currently looking to grow our engineering team with the following on-site in our London or remote (commutable to our office when necessary) roles:

- Site Reliability Engineer: https://jobs.ably.io/o/site-reliability-engineer

- Distributed Systems Engineer: https://jobs.ably.io/o/distributed-systems-engineer-1

We at Ably Realtime have a largely remote engineering team. I’ve been searching for a good remote recruiter for some time now and not come across one. Unfortunately I think it’s just largely easier for recruiters to focus on local markets, and sadly it seems the demand is mostly for local workers in spite of what you read in the tech media.

Top talk was mentioned and are good, however I’ve only used them for contract roles.

We’re recruiting now too, perhaps see if it’s right for you: https://jobs.ably.io

Best of luck, remote work rocks (if you like it!).

Matt Ably https://www.ably.io

Being part of a team who is designing and running a globally distributed system has been the best 5 years of my 20 year tech career. Things go wrong sure, and it’s hard, but is crazy rewarding and intellectually challenging in all the right ways. I never quite got that when I was a full stack web dev.

I wrote a post on what a distributed system engineer is, at least from our perspective at Ably Realtime. See https://blog.ably.io/what-is-a-distributed-systems-engineer-... if you’re interested in what it’s like.

Matt, tech co-founder Ably, www.ably.io

Fascinating and brilliant article!

We have a similar approach to health in our distributed messaging platform in that during situations where there are capacity problems, we ensure existing connections and requests continue to get serviced, yet new requests are rejected triggering a fail over to other regions. We effectively put the affected nodes into an “under siege mode” until the load subsides.

Interestingly, in our recent distributed deep dive interview (https://blog.ably.io/hidden-scaling-issues-of-distributed-sy...) with Paul Nordstrom, a former systems architect at AWS & Google, he also talked about how they designed systems around how they would recover when there were failures. For example, if a huge backlog of work built up, trying to catch up is probably not going to work, so you need to think about that too in the design.

On a separate note, one of our engineers Simon wrote an article recently on distributed system rate limiting (https://blog.ably.io/how-adopting-a-distributed-rate-limitin...), and the challenges he faced. He also found that hard static limits would often cause significant yo-yo’ing of load which did not help solve the problem.

Matt, technical co-founder, Ably Realtime (https://www.ably.io)

For https://ably.io (Ably Realtime)

- Stripe (payments)

- PipeDrive & Intercom (CRM integration)

- Zapier (some automation between systems)

- Rollbar & Sentry (error tracking)

- AWS S3 (file storage)

- AWS Route53 & CloudFlare (DNS)

- AWS EC2 APIs (Infrastructure provisioning)

- AddSearch (search)

- Freshdesk (tickets)

- Sendgrid (email)

- Google Recaptha (captcha)

- APILayer (tax)

- Monitis & Uptrends (monitoring)

- Browserstack (browser automation)

We use 100s of services (https://blog.ably.io/94-tools-you-need-to-grow-your-startup-...), but I've only listed the key services where we integrate with their APIs.