matt_oriordan

We at https://www.ably.io host our website on Heroku for convenience as the site is less mission critical than our globally distributed realtime messaging platform.

15 minutes ago our site went down, and Heroku have scheduled maintenance which means every single API is unavailable. That means:

- I cannot tail the logs to see what's going on

- I cannot restart the apps

- I cannot roll back

- I cannot view my dashboard

- I cannot contact Heroku (their dashboards and login are down)

We are completely and utterly helpless, so have been forced to redirect all website traffic to our status site in the short term.

Whilst we have invested serious engineering into complete redundancy in every part of our realtime platform (https://goo.gl/ZyZ9zT for example), it's we should be putting some of that engineering into our website too! I now seriously question using Heroku as a reliable platform for hosting given a decision that they have made to cripple everything all at once.

Heroku, you have failed badly IMHO. How did you think it's OK to disable every single API and not notify me many times about the severity of this in advance.

This reminds me of the CPU issues there were previously with Slack: https://medium.com/@matt.at.ably/wheres-all-my-cpu-and-memor...

Sounds like Slack doesn't have any type of performance testing as part of their release cycles.

I'm excited by the possibilities. Interesting to see the different approaches different vendors are taking in this regard i.e. bundle in serverless functionality or integrate with serverless. APIs will of course always win out :)

Disclaimer: Co-founder of Ably

Ably realtime | SDK Developer | Full Time | Remote in Europe | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering teams. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest.

If you'd like to work somewhere where you're involved in solving complex internet-scale problems and enjoy being a polyglot programmer, then you'll fit right in.

Find out more and apply at http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

Ably realtime | Distributed Systems Engineer | Full Time | Remote in Europe | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering teams. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

If you enjoy solving hard distributed system problems that are mostly platform agnostic and theoretical in nature, then you'll love working at Ably.

Find out more and apply at http://jobs.ably.io/o/distributed-systems-engineer-remote

Ably realtime | Developer Advocate | Full Time | Onsite in London, UK | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering and marketing teams. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

Advocating for Ably, you’ll be joining a deeply technical and passionate team that is working with the developer community to provide reliable realtime data delivery at scale. For us, a developer advocate is both a skilled programmer with an interest in the latest technologies and trends, and also one that enjoys speaking, blogging, being part of the developer community and contributing to open source projects.

Find out more and apply at http://jobs.ably.io/o/developer-advocate-in-london-option-fo...

Ably realtime | Site Reliability Engineer | Full Time | Onsite in London or Remote in EU | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering team. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

As a Senior Engineer in our Site Reliability Engineering team, you’ll build solutions to enhance availability, performance and stability of the Ably platform as well as developing new network services whilst automating away repetitive work. Find out more and apply at http://jobs.ably.io/o/senior-site-reliability-engineer-remot...

Upgrading to iOS 11 has effectively bricked my phone, it’s painfully slow. The thing I find (sadly) amusing is I cannot use the torch feature now. Works roughly 20% of the time. It’s binary, on or off, and it doesn’t work. What the fuck is going on?

Ably realtime | Site Reliability Engineer | Full Time | Onsite in London or Remote in EU | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering team. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

As a Senior Engineer in our Site Reliability Engineering team, you’ll build solutions to enhance availability, performance and stability of the Ably platform as well as developing new network services whilst automating away repetitive work.

Find out more and apply at http://jobs.ably.io/o/senior-site-reliability-engineer-remot...

Ably realtime | London | Full Time | Onsite or Remote | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering team. Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest. Things like live chat, realtime location tracking, live document collaboration, gaming and elearning. One of our customers even uses Ably for their air traffic control system for drones.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

We're recruiting for the following roles at present:

- Developer Advocate (London mostly, part remote): http://jobs.ably.io/o/developer-advocate-in-london-part-remo...

- Infrastructure Engineer (Ruby & Go, remote in Europe OR onsite): http://jobs.ably.io/o/devops-engineer-for-ablys-distributed-...

- Distributed Systems Engineer Node.js & Go/Elixir (remote in Europe): http://jobs.ably.io/o/distributed-systems-engineer-remote

- Head of Infrastructure & DevOps (London): http://jobs.ably.io/o/head-of-infrastructure-and-devops

- Mobile & web client SDK dev (remote in Europe): http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

- Full stack rails dev (remote in Europe): http://jobs.ably.io/o/full-stack-rails-developer

Ably realtime | London | Full Time | Onsite or Remote | https://www.ably.io

I am the co-founder of Ably realtime, and we're accelerating the growth of our engineering team.

Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest. Things like live chat, realtime location tracking, live document collaboration, gaming and elearning. One of our customers even uses Ably for their air traffic control system for drones.

If you'd like to work somewhere where you're solving complex internet-scale problems in a truly distributed and peer-to-peer messaging system, then you'll fit right in. To get a taster for the types of things we talk about each day read https://blog.ably.io/what-is-a-distributed-systems-engineer-...

We're recruiting for the following roles at present:

- Developer Evangelist (London mostly, part remote): http://jobs.ably.io/o/developer-evangelist-in-london-part-re...

- DevOps Engineer (Ruby & Go, remote in Europe OR onsite): http://jobs.ably.io/o/devops-engineer-for-ablys-distributed-...

- Distributed Systems Engineer Node.js & Go/Elixir (remote in Europe): http://jobs.ably.io/o/distributed-systems-engineer-remote

- Head of Infrastructure & DevOps (London): http://jobs.ably.io/o/head-of-infrastructure-and-devops

- Mobile & web client SDK dev (remote in Europe): http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

- Full stack rails dev (remote in Europe): http://jobs.ably.io/o/full-stack-rails-developer

I ended up buying a second hand MacBook Pro instead of a new one because Apple have stupidly for some reason decided that if you want a more powerful configuration, you have to have a shitty touch bar. It makes no sense, a touch bar and a high spec are orthogonal.

Ably | SDK developer | London head office | Remote (in Europe) | Full time

Ably is a global realtime data delivery platform that powers low-latency communication between internet-enabled devices. We solve the hardest parts of delivering the service reliably at scale so developers don’t have to.

We have a number of SDKs in all popular languages (see http://www.ably.io/download), and a spec that all client libraries strictly adhere to (see https://docs.ably.io/client-lib-development-guide/features/). We are looking for polyglot engineer with a minimum of iOS, Android and Javascript experience to maintain and lead the development of our open source client libraries.

Find out more at http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

Ably | DevOps Engineer (remote) | London head office | Remote (in Europe) | Full time

If you enjoy the idea of programatically managing infrastructure for a large distributed system and developing network services and layers primarily in Go, then read more at http://jobs.ably.io/o/devops-engineer-remote

Ably | Distributed Systems Engineer (remote) | London head office | Remote (in Europe) | Full time

If you enjoy solving truly difficult distributed problems, have Go and Node experience and are looking to work with a deeply technical team to learn from, then take a lok at http://jobs.ably.io/o/distributed-systems-engineer-remote

Ably realtime | London | Full Time | Onsite and Remote | https://www.ably.io

I am the co-founder of Ably and we're accelerating the growth of our engineering team.

Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest. Things like live chat, realtime location tracking, live document collaboration, gaming and elearning. One of our customers even uses Ably for their air traffic control system for drones.

We're recruiting for five new roles:

- DevOps Engineer (Ruby & Go, remote in Europe OR onsite): http://jobs.ably.io/o/devops-engineer-remote--onsite

- Head of Infrastructure & DevOps (London): http://jobs.ably.io/o/head-of-infrastructure-and-devops

- Realtime dev using Node.js & Go/Elixir (remote in Europe): http://jobs.ably.io/o/realtime-developer--nodejs-and-goelixi...

- Mobile & web client SDK dev (remote in Europe): http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

- Full stack rails dev (remote in Europe): http://jobs.ably.io/o/full-stack-rails-developer

Just to confirm, we are not in any way a competitor to Slack, we are a realtime data delivery platform. The article confirms we use Slack ourselves. See www.ably.io/platform for more info.

Please don't troll and have a look at what Ably actually is.

It's data messaging, not user / VoIP / chat messaging. We are in no way a competitor to Slack or ever will be. We deliver a platform realtime data delivery. In fact, companies like Slack are often our customers.

See https://www.ably.io/platform if you want to see what we offer and how it is in no way related.

Hi Toby

Quick responses:

> I'm signed into two accounts while the author is signed into 11.

Yup. Also, see a subsequent update at https://medium.com/@matt.at.ably/wheres-all-my-cpu-and-memor.... One of the Slack developers got in touch and confirmed this issue is resolved in beta 2.7.0. I tried it and it is indeed.

> The author is CEO of a competing messaging platform

Ably is not a competitive platform in any way whatsoever. We are a realtime data distribution platform, we have no consumer apps or products you can use, only a low level platform-as-a-service for broadcasting realtime data.

Ably realtime; London; Full Time; Onsite and remote

I am the co-founder of Ably and we're accelerating the growth of our engineering team.

Ably helps power next generation digital experiences. Ones which are live rather than static, where data is in motion rather than at rest. Things like live chat, realtime location tracking, live document collaboration, gaming and elearning. One of our customers even uses Ably for their air traffic control system for drones.

We're recruiting for four new roles:

- Head of Infrastructure & DevOps: http://jobs.ably.io/o/head-of-infrastructure-and-devops

- Realtime dev using Node.js & Go/Elixir: http://jobs.ably.io/o/realtime-developer--nodejs-and-goelixi...

- Mobile & web client SDK dev: http://jobs.ably.io/o/mobile-and-web-client-library-sdk-deve...

- Full stack rails dev: http://jobs.ably.io/o/full-stack-rails-developer

Ably | Full time | Remote | Ideally Europe

Ably is a realtime data delivery platform solving distributed problems at scale for businesses like Yahoo, Offerup and Computer Associates.

We have our base in London, but mostly work remotely on flexible time. Every day we solve truly challenging distributed and scalability problems that go well beyond what common frameworks, languages and platforms address. If you enjoy taxing your brain, managing your time, having your code reviewed by like minded people, working with a range of technologies and getting shit done, then we think you might just fit in and enjoy yourself!

Find out more about Ably at https://www.ably.io

We have two roles currently we are trying to fill:

- Realtime systems engineer - https://www.wfh.io/jobs/3630-node-go-distributed-systems-eng...

- Web developer with Rails focus - https://www.wfh.io/jobs/3631-full-stack-ruby-rails-developer...

Joel, the point of my article was nothing to do with architecture and using multiple AZs. Of course we use a minimum of two AZs and often three in reach region. The problem is that AWS continued to report the entire region as completely healthy in spit of an entire AZ being down, and worse, continuing to route traffic from an ELB in that AZ to that unhealthy AZ data center.

This article was NOT about reliance on a single AZ. That was not the point as we use three AZs in most AWS region, and at least two in every region. The point was simply that an entire AZ was down and Amazon continued to report the entire region as completely healthy throughout.

We use three availability zones. Unfortunately that was not the point of the article. The point was one AZ was effectively completely down, yet AWS was reporting the entire region as healthy without any warnings.

That's am impressive bit of technology, and nice to see it's making extensive use of Erlang (we're using Elixir).

I'd be interested to know if the service works across multiple regions though. Some of the biggest challenges we've faced when engineering our realtime platform has been in having no single point of congestion and effectively peer-to-peer routing within the cluster. This is not that important if all your servers can be in a single region, however if you want your users in Australia for example to have a similar latency profile to those in USA, then clients in Oz need to connect to servers in Oz and all routing for customers in Oz does not need to traverse continents for other customers in Oz, but does for customers in US. I'd be interested to know if that was tackled in the design. Michal you following this thread?

Matt Ably realtime - https://www.ably.io

Read my previous comment. We use multiple AZs. That's not the point of my blog post. I am not complaining about availability of services, I am complaining that there is no accuracy of their service status.

Well it's not that's simple. We do run in multiple availability zones in every region. But if the connectivity between them is partially working, which it was, and shared service from Amazon itself aren't working fully from every instance, ou have a huge mess to contend with where the cluster consensus cannot be formed. So in cases like this we did what we should have done and routed traffic away form a network that was unreliable and partly partitioned. The point for us was not that this availability zone went down at all. It was that Amazon throughout claimed everything was operating normally for hours when this was very far from the truth.

Thanks Justin, I obviously concur!

Hey Sean, Matt here, co-founder of Ably. I have been meaning to write a blog post on this in fact, like I did some time back for ActionCable, see https://blog.ably.io/rails-5-actioncable-the-good-and-bad-pa....

I've not done an in-depth review yet, but from what I can tell SignalR has the following shortfalls:

- It relies on a single message bus for message passing between nodes. If your rate of messages exceeds what the bus can handle, you're in trouble.

- I don't believe it handles connection state recovery as a service, and certainly does not do that across moving connections i.e. a disconnected client may arrive an another server yet messages may have been published whilst disconnected.

- SignalR does not support multi-regions, once again due to the central bus, so if you have users in Asia and your datacenter in US, they'll all have to route everything through US.

To be quite honest though, we're now rarely having discussions about technology X versus Ably, because Ably is realtime platform-as-a-service, and not a software solution. To provide the scale, reach, resilience and feature set we have, it can only be achieved with a combination of infrastructure and software. So our customers are choosing us because we're investing a lot of time and effort in all aspects of the service, not just the realtime software itself. See https://blog.ably.io/routing-around-single-point-of-failure-... for details on how the Dyn attack did not affect us for example, or https://blog.ably.io/the-18-ghosts-in-your-infrastructure-st... for some details on we've built necessary resilience into every aspect of the system.

I plan on doing an in-depth SignalR review soon in fact, thanks for the prod.

For those interested, we wrote a blog post about single points of failure with DNS last Friday when the Dyn attack happened. Our strategy ensures our customers are mostly unaffected by issues like this. See https://blog.ably.io/routing-around-single-point-of-failure-...