Parler’s epic fail: A crash course on running your own servers
blog.alexgleason.me
blog.alexgleason.me
One of parler’s problems was they didn’t scrub headers which made it very obvious they were using AWS and which AWS services they were using.
AWS is just a bunch of APIs - they could have colo’d their front end or put a bunch of varnish servers in front if it at a colo and nobody would have been the wiser they were using AWS on the backend (assuming their AWS account didn’t have Parler Inc in the payment details). From there getting taken down from one colo would just be an issue of spinning up a front end at another colo. Front that setup with a couple CDNs and now you have layers where you have to coordinate between half a dozen companies to bring the site down completely, meanwhile your data is safe in AWS because nobody knows your using AWS or even if they do it’s hard to pick you out from the millions of other AWS users.
Their failure to make it cloud agnostic being 1 (even though they claimed before they went down that their app could run without AWS without issue).
Their failure to be incapable (still) to get it running again on non-cloud hardware..
And of course their inability to, as you suggest, move the frontend routing to a CDN/Cloudflare/DDosguard type service and have all the heavy lifting continue to be done via AWS/Azure/Google till you can run 100% on your own hardware.
AWS dropped WikiLeaks as a customer back in 2010 for political reasons, and apparently this was announced by a US senator even before AWS did so themselves:
http://edition.cnn.com/2010/US/12/01/wikileaks.amazon/index....
Parler's case has zero to do with incompetence, and it boggles the mind of how suddenly putting together web apps with bare metal servers is somehow depicted as normal or usual or basic competency skills. It isn't. At all.
The majority of the world's web apps are hosted by one of the four major cloud providers. AWS alone hosts about 1/3 of the world's websites. The only reason anyone ever had to move away from AWS ever was due to their high prices, and even then the decision was always to move to another major player.
Hell, whenever I point out that there are bare ones cloud service providers which are far cheaper.than AWS, such as Hetzner, I'm faced with at best an industrial dose of skepticism.
Let's face it: up until now there wasn't any reason at all to even consider hosting your startup app on a non-major cloud provider. At all. In fact, it's tremendously hard to argue using anything beyond them. You need scale on-demand, high-availability, and reliability. You do not get any of that by even picking most of the non-major cloud providers, let alone running bare metal servers.
Post-facto accusations of incompetence are not justifiable, specially when criticising the undisputed go-to rule that driver over half of the world's websites.
Just calling them servers is sufficient. I don't know when this "bare metal" crap started
Standing up a service is a basic competitancy
So it sounds easy, but is only usefull for companies with multiple networks. But even then i guess aws is publishing that they handle your networks.
This is a gross oversimplification of what AWS is. For starters, no one pays for AWS for the APIs. People do pay AWS for the on-demand infrastructure, and the global network of datacenters, which makes it trivial to setup a highly-availiable web application distributed globally. AWS allows you to put your highly-availablr global application together by click-click-clicking on dashboards, run AWS-specific infrastructure-as-code scripts, or run third-party scripts like Pulumi. In the process, APIs don't play any role in the decision process.
From a technical standpoint I think half the fun of working for Parler would have been evading the left’s efforts to take them down. I don’t know who they hired for technology but they lacked spirit in addition to fundamentals! ;)
Hosting servers isn't the hard part, and it isnt the same as running a social media platform. This blog is riddled with neivete and a misuse of the word "fear" to get clicks. Don't even get me started on trying to use the "they didn't come for me" quote when relating to online posts. Government != Private companies
If you want to compete you will require moderation, period. Illigal and disturbing content will have to be removed and sectioned off from your users. Or do really you want the average user seeing a trending video of gore and targeted harassment of people? Even 4chan moderates.
You will have to serve hundreds of petabytes of image and video data - you will have to use cloudflare or some other cache mechanism like everybody else. Cloudflare was the reason 4chan was kept alive btw.
You need lawyers on retainer that understand law pertaining to your business. If not then you're left open to civil liabilities anyone can exploit.
You need money. You will require algorithms for promotion of content which requires user tracking which requires databases that contains peoples habitual information.
You've now created facebook and twitter or you're running on investor money. Eventually you will have to sell out your users.
People forget that the "big social media" all came from the same small places when it was a lot harder to do ( do you know how much more expensive bandwidth used to be?). The difference is they got popular and smelled the money. It takes a very special community to not do the same.
It's been fascinating watching how dramatically that viewpoint has shifted over the years to the point where it is now a novel idea to do so.
The level of convenience that cloud providers give is just orders of magnitude more efficient and easier.
On the flip side, I wonder how much better the support from a cloud provider is if it's an isolated problem and not something that's setting twitter aflame.
If it takes a cloud provider in the order of hours to get me back online, I could probably get the same sort of service from one of the better colos/hosting providers, especially if they were local and I had the ability to make a call to get support.
There are other conveniences to cloud providers of course, but I think I if I could find highly skilled ops people and pay them well, I would run my own servers every time. For the kind of games I've worked on, the money/CPU cost of cloud is ludicrous.
The trick these days is even finding high-level ops people who aren't already working 3-400k jobs for AWS/Azure/GCP
I’ve had 4w to resolution for major service degradation on a big name provider and ~2d for full site outage (highest support tier)
You said it yourself - aws, goog, msft have very good sres but they’re not your sres. Meaning they dont care if you have a big event/demo/deal close coming up before they start doing network gear upgrade and such...
The other consideration is that things generally don't just 'go down' like they do with bare metal (because HA - replication and so on), but if they do, it's likely affecting a large portion of the internet too.
I don't even know if these salaries are truthful now.
Those are certainly in range for DevOps people at the higher end of the scale.
In the same way most people/companies don't really need to care what hardware their application runs on, only that it meets some bar of quality/cost that's appropriate for them. If someone else is delivering this then you've removed a small department's worth of overhead/planning from your corporate structure.
So, not a ideal analogy...
The sibling comment pointed out that vendor lock in can be a problem which I agree with, but I think for most of the industry that's a problem of protecting yourself from predatory price hikes/services being deprecated rather than the problem of actively pissing off people you need.
I've had zero downtime due to running on a Digital Ocean VPS the past few years and very, very brief outages due to my own decisions around various upgrade and backup decisions. I spend less, I get full control and there's zero risk of a surprise bill.
I think the key thing is that most customers won't blame you if your service goes down due to an AWS outage.
So much money is wasted with AWS. (Look at at their margins.)
It's useful in development/testing. It's irresponsible to use managed services that don't have an easy migration path to something open.
Deep-Learning in the cloud really is a scam and a complete waste of money.
Do you look at a car manufacturer, see that they're making a profit, and assume "I could clearly build a comparable car for less money?"
On both sides of this, the thing to do is to critically examine your needs and decide if one or another solution fits. AWS can be a game changer for some companies and an immensely complicated money sink that never gets good enough performance for others.
In any case, Parler’s problem (well, one problem of many) was that they had MASSIVE hardware requirements that dramatically cut down on the number of places that could practically host them.
It also seems that despite assurances and good sense, Parler had deeply tied itself to Amazon’s APIs, making migration off AWS slow even once a host was found.
Oh crap. There is zero chance I could do it quickly.
So a turn around from 20 years ago when I would setup LAMP applications on new new hardware in an afternoon.
Hurricane Electric will happily lease you an entire rack for $400/mo with a 1Gbps connection. I would argue that, in comparison to the prices the OP mentions, Hurricane Electric's price is quite good. Also, there is one fewer middleman between you and the Internet (and power, and rack space, etc, and the front door, etc).
(I believe that $400 number does not include vast amounts of power, so the actual price tag for filling that rack with conventional hardware may be rather higher. On the other hand, depending on your use case, fitting in a small power budget may be straightforward.)
Just about all ISP have terms and conditions that prohibit use that they find offensive.
Data Foundry, mentioned in the article, acceptable use policy is below and I’m sure could be used to kick out Parler
Then either a subpoena or publicity against the ISP will force them to do disclose who the customer is and/or remove them.
As for the subpoena, you can make it harder to find you by spreading your addresses across multiple jurisdictions. Put the company that owns the IP in one country and make it rent them to another company in a different country who owns the servers and the website and rents the domain name from a third company in yet another jurisdiction.
Eventually they'll find you (there aren't many ways to escape a government forever), but this game can buy you enough time to move someplace where extradition is unlikely.
Even if you host your own server on your own premises they could forbid you from using the location, shut off your electricity, etc.
Is it possible to host a website entirely on bittorrent? I suppose there's also IPFS but I'm not entirely familiar.
Maybe once solar is cheap enough you could launch some sort of array of powered drones into the sky that follow the sun that send files to people via P2P - solar mesh network if you will.
IPFS, I believe, has a solution. The one I am most familiar with is Dat Browser.
Does anyone have concrete evidence that contradicts the parent AND the grandparent statements? Or is it just the emotional reaction to two websites that one just felt the need to downvote a fact without reason or counter evidence?
Let's not get any form of irrationalities in the way of the actual facts and the truth.
Explanation? (With evidence of course)
That language is not neutral, so the response isn’t surprising.
I'm not sure how ISPs fit into this though, and to what extent they can say "we don't like the content you're serving" and cut your net. Obviously if the content is outright illegal that's one thing, but I wonder if they can "pull an Amazon".
You could at least get around DNS level stuff with a P2P application layer. But if you're cut off at the routing level by everyone then yeah, no go...
I once lived in an apartment tower that didn’t pay the water bill for 3 years (because of an incompetent transfer of management responsibilities). They didn’t shut off the water. They didn’t threaten to shut off the water. After 3 years, they threatened to charge interest on the unpaid amount if it wasn’t paid within 30 days!
Given anyone can do such nodes and it's relatively inexpensive to get gigabit lines nowadays - it's no doubt a good number of the nodes are owned by various three letter agencies.
Bandwidth isn't expensive on taxpayer dollars, but it is expensive for individuals.
Also because a lot of relays go outside the U.S. the metadata can be collected and stored at will.
CPU and RAM last a very long time so it shouldn't be a problem (I've had literally 1 RAM stick failure in over a decade of hosting and the datacenter swapped out the bad memory stick in less than 30 minutes -- it also didn't bring down the server either, the kernel log started showing a bunch of ECC issues and a quick memory check pointed to the stick that had the problem).
And of course you can just get double or triple the amount of servers with IP failover and such to be even more resilient to hardware failures.
This article is just showing how to get started. Throw more servers at it for more redundancy and scaling as needed.
There's always OpenNIC, DNS.watch, or Quad9 if you're after something that isn't operated by a creepy megacorp.
"The pirate bay, the most censored website in the world, started by kids, run by people with problems with alcohol, drugs and money, still is up after almost 2 decades. Parlor and gab etc have all the money around but no skills or mindset. Embarrassing.
The most ironic thing is that TPBs enemies include not just the US government but also many European and the Russian one. Compared to gab/parlor which is supported by the current president of the US and probably liked by the Russian one too.
First time tpb got shut down was because a Mexican gang wanted some cash. Took a day to move. Second time was when the USG forced Sweden to send 50 cops and even arrest the lawyer. Took 3 days to get back online, 1.5 days was spent to get drunk and party."
But the network itself prevents this. Bittorrent has to jump though multiple NAT busting hoops to allow two internet users to talk to each other directly. limited IPv4 addresses mean home internet connections have dynamic IP addresses which means no one can realistically host at home. Because no one hosts at home they consume far more than they upload. Because of this the physical network was built with a much higher download bandwidth than upload. It's a vicious cycle that wasn't intentional it just sort of evolved that way.
I shudder when tech folks themselves insist on NAT as a core security tenet - sure it might be a convenient extra layer but it’s so sad what’s been sacrificed in its name
It's been an obsession of mine for a long while, and i am overjoyed it has been written down
I have not been comfortable with the Cloud or nothing and the fact that it looks like it no longer possible.
This video also helps for people who want to go into self hosting
Contrary to what this article insinuates, the vast majority of companies do not have to worry about violating the terms of services of cloud providers and other external services, because they don't offer services designed to violate besaid terms of services.
As for Parler, of course they should have seen that coming. It's pathetic that they didn't. They should have looked at sites like the Piratebay for how to do it and prepared a bit more.
https://blog.checklyhq.com/why-the-recent-digital-ocean-kill...
That's why Parler was such an epic fail, not because they didn't host everything themselves. They were acting as if they'd care about free speech - unless it wasn't totally compatible with their personal views, in case of which they'd ban instantly - and did not prepare in the slightest for contingencies and were caught in the cold. All of that in the light of plenty of precedents, ranging from Napster over Piratebay to ISIS propaganda websites. It baffles my mind how a company could be so unprepared.
So many companies got 'hacked' simply by having open S3 buckets with customer data hanging around.
A cloud service doesn't shield against incompetence.
Had Parler just moderated their posts like they were asked to do, they would still be around. You can still have right wing, even extreme right wing views without calling for violence or organizing insurrections. The fact they essentially refused to moderate and let calls for violence fester on their platform is why they were shut off.
I always try to design a "degraded mode" read-only into my apps that covers maintenance windows and unscheduled outages. That is usually easier to lift and shift than the full app and can even be hosted someplace else if you don't mind it lagging behind the live website.
Would be interesting to hear how much it costs Twitter to have video features. If I was Parler right now I’d just switch off media and throw up a very lean MVP version with just text. Even with millions of users Parler has the luxury of almost no consistency problems at all. It’s all eventual consistency and simple replication. Shouldn’t need a lot of hardware at all.
Remember they are (or at least were) funded by the Cambridge Analytica folks.
The insurrectionists did not actually use Parler. They used Twitter and Facebook.
Something like... shoving someone so they fall and break their teeth, and then trying to blame them and saying, "it's your own fault for having such poor balance". A coping strategy for guilt? I just meant to harm you a little bit?
A better analogy would be: Parler likes to hang out with the school bully, even though everyone told them not to. One day, the bully shoved Parker and it broke it's teeth. Should the bully's conduct have been allowed? Almost certainly not, and that's a problem. But at the same time Parler was warned about the bully and yet they disregarded all warnings, believing that it wouldn't happen to them.
Parler can make a stand, saying "it is the school's job to keep bullies under control" (which is true) and live it's life as if there were no bullies because that's what life should be. Some people do precisely that for worthy causes, knowing full well what the consequences are (see: Rosa Parks). Or they can do what most of us did in school and stay away from the bully. Less principled? Sure. But our dentist bills are a lot cheaper as a result.
and then you wished you would have went to the cloud after all.
now you've got another problem.
> tech oligarchs
...here we go.
I really struggle to understand how some folks didn’t realize they were using other people’s things until recently. Maybe the fact that they weren’t physically in a space was hiding the proverbial threshold crossing activity that takes place a dozen or so times as you travel to twitter.com.
It is free speech to kick someone off your platform. Freedom minded individuals seem to think freedom only goes in one direction. I will defend DJTs right to tweet stupid shit all day and every day, just as I’ll defend Twitter’s right to kick him off the platform.
Can you live without 'using other people's stuff' except like a hermit in a cave?
Someone else's speech isn't a protected class, except for speech due to religion. Discrimination based on things that aren't a property of being in a protected class is legal for any non-utility.
Personally I'd say that internet service providers should be utilities and regulated as such, but currently they aren't so the ISPs are allowed to decide what speech to carry.
Most of the bullshit our western democracies have experienced in the last years online should be archived and studied for the crimes our Russian Chinese friends have committed against our free democracy.
We literally have registered political parties in the west that do the exact same thing and the matter was proven in a court of law (Leave EU). They have faced no material consequences.
For groceries, that’s a private business. If you run into a grocery store and cause a scene or do something the owner doesn’t like the owner can remove you. There are fairly narrow exceptions to this, such as how you can’t usually be discriminated against for being a member of a protected class.
Freedom of Speech in the US was never designed to protect you from other people, it was designed to protect you from the government.
If so, the government centric line of argument does not get you anywhere, because there are many essential services are not government owned or controlled monopolies..
My electric, water, internet and mobile suppliers are private Same goes for the bank. There are only a handful of companies in each category in the country, and they could wake up tomorrow and remove me from their network.
You can either have personal liberty, or oligopolies with contracts 'we can remove you at any time for any reason'. Not both.
In other parts of the world there would be limits and requirements set by the parties elected by the people, i.e. rules on what things you can and can't discriminate on (like ability to pay you can measure, but skin color is not allowed to be a factor for selling electricity). Generally there is a list of factors that you cannot use to allow/deny sale.
But living in a society means not upsetting people so badly that they won’t sell you groceries. There’s tons of ways to do that while retaining your ability to speak what’s on your mind, you just have to be thoughtful about how your words effect others, even if just to make sure you don’t get kicked out of society.
But it falls apart when 'people' are multinationals and "upsetting them" means their PR department gets a funny idea. In that case we need a contract that provides more protection than 'we can remove you at any time for any reason'
Imagine a corp optimizing click through. Any negative news could dampen that and require immediate action.
You have to be perfect or too big to remove.
Parler wasn’t removed from AWS because it was a conservative site, it was removed because it wasn’t willing to handle the legally troubling volume of potentially criminal activity. You can post your thoughts on any conservative idea you want on Twitter, no one is getting banned for that. The only reason anyone has been removed from any popular social media (or AWS) is because they became unwilling to recognize the fact they owe more than nothing to their fellow man, which includes not trying to silence others through intimidation.
Ideas have not been and are not still the problem, is the refusal to think carefully about how to express oneself.
I've never gotten COVID, so clearly there is something wrong with how you breathe!
Do some reading, google employees got banned from gmail and never found out why.
There is an important point that you are missing - I am not claiming that you can't kick people off your platform. I am asking that you have fair due process, and it's the same rules for me and for some loudmouth politician.
They are freedom of association, but in some cases—in US law for the first consideration—they are within the scope of such freedom that government can (under the strict scrutiny test) and has chosen to (e.g., either as part of regulation of monopolies or as part of public accommodation law) limit.
Under the US Constitution, Christians can also refuse to bake gay cakes, fire gay employees, or evict gay members from positions of leadership.
At the heart of cancellation is the freedom of association; perhaps we should redraw the boundaries on where that freedom begins and ends, but while keeping in mind the balance of affairs.
The same goes for getting deplatformed: if you are being nasty the problem isn't the non-governmental org removing you, that's a symptom of nobody wanting to deal with you. (or nobody 'big and easy')
Just because it looked good, or looked like a lot of people were 'with' you, doesn't mean it's true and doesn't mean it keeps looking that way if the barrier to entry gets higher. Hanging around in a browser and 'liking' or otherwise interacting/spreading things isn't actually a replacement for "a group of likeminded people sharing ideals", it's much more comparable to schoolkids forming groups.
At some point we might see a digital platform that is expensive enough and has a higher entry complexity to actually only grow and maintain people that are 'true' to the ideals fostered by that platform. Only then do we have an example that would be 'real' enough to converse about.
According to your logic, Twitter can do that.
Why allowing (socially as in bad PR) any censorship is bad.
Otherwise you get where we are at. What the mainstream deems bad is silenced. What transient outrage seems bad is silenced. What authorities convince the masses is scary gets silenced. Any voice that platform doesn't like and isnt popular enogh gets silenced.
Namely run on your own stuff and not on someone else's platform.
Some people's livelyhood depend on their google or facebook accounts every bit as much as they depend on telephone service.
There are two things everyone knows are market failures that need to be regulated: monopolies and third party negative externalities. Right and left wing economists alike. Trust bust and regulate monopolies, and return net neutrality.