Hacker News Reader

Comment by tazard | Hacker News Reader

Still seems down for me. Getting a 502

It went down again for a short time. We're continuing to monitor and adjust resources as needed. We weren't expecting this traffic to our monitoring dashboard, but it's great that so many people are interested in taking a look.

Is your monitoring page a recent build, is it scaling well?

I ask because it's not a particularly good luck that the viewers from HN are capable of bad gateway hug of deaths to the site?

It sounds like it's a separate, single instance. Definitely doesn't use the same infrastructure as gitlab.com itself (which is a good thing, since that's what it's monitoring), nor is it built to be scalable really. So, no great surprise that HN-level traffic overpowered the instance.

I do hope they're using Prometheus federation to expose this instance to the fickle internet and that they have one or more internal Prometheus instances that aren't directly queried by this instance. After all, that stuff is responsible for paging if something goes wrong in prod.

It is a separate instance from our internal one. We have a cron job that automatically copies the internal Grafana dashboards to the public one, so you still see exactly what we see.

We used to use Federation, but now we just have the public server scrape the same targets as our private one.

I'm adding a caching proxy (https://github.com/Comcast/trickster) to the public server now to improve performance. :-)

Reply on news.ycombinator.com