It happens more often than you think. Plus, odd issues, like a glitch with their servers causes the card reader to not work. Theres a ton of device verification and security checks on those readers that bounce back and forth between their servers. Square having a bad day? Everything works, but your reader wont sync, wont read cards = you're done for the day.
Days like this aren't the "norm", but constant aggravating issues are pretty much a weekly thing.
It got so bad, we just got an old-school merchant account with a reliable card reader. ZERO issues almost with that, ever. It's a little funkier, but the merchant account is 1/4 the cost to run than Square. Not as fancy of a webui from the merchant provider, but eh, all the same connections to Quickbooks for the accountants.
How slow are the rest of your tools? The Slack client probably performs worse today than it did a few years ago. It has the laggiest interface of any of my tools, you can watch your CPU spike to 60-80% just switching channels. Just do it right now, open up htop/top/atop/Activity Monitor - whatever you want, and just switch channels.
Laugh as the Slack client wastes a universe's worth of time just... rendering a DOM with plain text. It is genuinely pathetic how bad the client is.
Of course this isn't a typical single load balancer for cross AZ - but the general gist of their "new" architecture is first principles level of design. But sure, we can celebrate their minor achievement I guess
It even pains me to see they're suffering from so many own goals. And it's unfortunately reflected in the poor experience using the Slack client. Not to mention the multiple deprecated bot/integration APIs with such bad feature parity between all the different ways to integrate your own tooling into Slack.
XMMP was extensible to support all this in the early 2000s. Slack reinvented simple services in the most obtuse way. I have to use Slack and I sideline quarterback all the ways things could have been better every day.
Geo traffic distribution, multi regions/AZs with functionality to weight and drain traffic should be used in most SaaS services where a simple failure somewhere could cost users time and lose company money/goodwill. It's not terribly hard nor expensive.
I migrated some old BigIP load balancers over to Apache in 2004ish, and extended some of mod_proxy to do some "unholy" things at the time. We also did a lot of direct server return stuff when no load balancer you could buy could handle the amount of traffic statefully. Man, how times have changed, and lesson forgotten.
It may be shallow, but architecting for this is not really "advanced, FAANG-only accessible methodology". I'm surprised their services have been as "reliable" as they have been considering such trivial stuff is just now being employed in their architecture.
Just odd a company worth billions and billions of dollars is just now discovering HA models standard since the 90s. Can expand the Clos network architecture to these distributed service applications too. But judging by Slack's client quality, mature concepts such as those must be new to them.
So they used a feature built into a load balancer to gracefully drain traffic from specific availability zones? Odd that a feature found in load balancers from the last 25 years is a blog post worthy thing.
May not cover this scenario, but lots of emergency vehicles don't use sirens at night. The thought is, the bright flashing lights is plenty - and if the road isn't busy, why wake all the neighborhoods up on a routine service call.
AI is becoming more and more human every day!
Though, honestly, at least these driverless cars are backed by billion dollar companies, and can pay for the damages. I've seen plenty of less than savory people in junky cars nearly hit emergency vehicles, and they would've never had insurance to cover the damages.
Have not - looks nice though. Around here, you'll mostly only encounter the Dell/Supermicro/HP/Lenovo. I actually find Dell to have acheived the lowest "friction" for deployments.
You can get device manifests before the gear even ships, including MAC addresses, serials, out of band NIC MAC, etc. We pre-stage our configurations based on this, have everything ready to go (rack location/RU, switch ports, PDUs, DHCP/DNS).
We literally just plug it all up and power on, and our tools take care of the rest without any intervention. Just verify the serial number of the server and stick it in the right rack unit, done.
"it depends". Dell is fairly good overall, on-site techs are outsourced subcontractors a lot so that can be a mixed bag, pushy sales.
Supermicro is good on a budget, not quite mature full fault management or complete SNMP or redfish, they can EOL a new line of gear suddenly.
Not that nothing will fail - but some manufacturers have just really good fault management, monitoring, alerting, etc.
And even the simplest shit like SNMP with a few custom MIBs from the vendor (which theres some that do it better). Facilities and vendors that lend a good hand with remote hands is also nice, if you remote management infrastructure should fail. But out of band, full featured management cards with all the trimmings work so well. Some do good Redfish BMC/JSON/API stuff too on top of the usual SNMP and other nice builtin Easy Buttons.
And today's tooling with bare metal and KVM, working around faults to be quite seamless. Even good NVME raid options if you just absolutely must have your local box with mirrored data protection, 10/40/100Gbps cards with a good libvirt setup to migrates large VMs in mere minutes, resuming on the remote end with nigh 1ms blip.
What hardware are you buying? Across tens of thousands of physical nodes in my environment, only a few would have "fatal" enough problems that required manual intervention per year.
Yes we had hundreds of drives die a year, some ECC ram would exceed error thresholds, but downtime on any given node was rare (aside from patching, but we'd just live migrate KVM instances around as needed.
The blog headline is "dope shit only", but this blog entry is the lamest account of a lame "industry" with the lamest "point" (if there even is one?) I've ever seen.
Likewise, cool stuff from the EU or China that don't have it warehoused in the US makes it a drag waiting a long time for shipping and customs to get.
Minor gripe, but I want it now :)
My house was built in 1885, timber framed. There's no rot, it's been maintained, re-plumbed and re-wired a few times, and new owners only need to keep with basic maintenance. It's been insulated as well, utility bills are in the $90-150 range depending on how extreme the seasons get. The structure has another century left in it, easy.