Hacker News moved off AWS again at 10:17 PM PST
previously AWS IP 50.112.136.166, now M5 hosting IP 209.216.230.240
Glad to see it back on bare metal. Thanks HN team for all the hard work!
(Yes, I set up monitoring for it haha)
Glad to see it back on bare metal. Thanks HN team for all the hard work!
(Yes, I set up monitoring for it haha)
An IaaS VM from the same vendor (M5 Hosting) would provide equivalent resources with much higher reliability. What benefit is 2 bare metal servers providing over a single properly sized VM in a managed resource pool with SAN backed storage?
I don't think it saves any money, and when those bare metal servers suffered identical hardware failures from a firmware bug HN was down for most of a day and temporarily migrated to an IaaS provider anyway (AWS).
[citation needed]. Sure, it solves some reliability problems, but introduces others, and I’m not sure if it’s a net win.
Many people just don't consider failure scenarios. Offsite live database backups, for example, are a great idea. Say ... how does your site perform, in percent of normal QPS, when the database is now 150ms away instead of 1ns? 1% ... that's not redundancy, despite the site being up, let's just call that a failure.
And people forget one thing about hosting on AWS. Say ... when AWS is slow/has problems/blocked on firewall/down ... when your competitor is down, would you like your site to be up? How about vice versa?
“Off-site database backups” that mean you now have a 150ms round trip for user facing queries? … what on earth are you talking about.
Is your argument that “simple” things are better because “you can do stupid things and blame it on complexity”?
Because they had a really fast local database essentially all the time, every pageview started requiring more and more database queries, some 50 for the front page alone, as the developers added features.
Then the database needed to failover. And the complexity hadn't actually killed IT (yet): it actually worked ... But of course 50 * 2 * 150ms = 15000, or 15 seconds per page.
I'm saying simple things can be better even when they don't provide redundancy because there's a bunch of problems that increase complexity so much that you can literally fix a simple problem faster than redundancy can take care of it.
So for us this doesn't increase complexity, it greatly reduces it while increasing availability and general confidence, even though the underlying system (RDS/Aurora) is clearly very complex.
If you're running a tinpot site with a single developer on a couple of pet servers, then fair enough. But it's definitely not correct to say that a simple, direct system is the epitome of reliability. It's not.
Every quarter we test it, and without fail, it has worked. So I added maybe a couple hours of research and two minutes of additional configurations during set up for reliably fast failover. Seems pretty no brainer to me.
When we moved to a managed instance with our cloud provider it took even less time to set up failover, maybe 30 seconds.
With the numerous options for cloud offerings these days I see no reason to not have a failover set up whether you are a massive corporation or a small business.
[1] https://blog.rmotr.com/the-best-time-to-post-on-hacker-news-...
It won't catch a 1 in 10000 5xx error, but it'll catch the whole site being unavailable for several minutes.
Do you know why they fail to implement ipv6 for years? I mean it's shame when tech-oriented website/service can't keep up with new technologies.
> Until then you can derive and use the IPv6 address for their old Cloudflare endpoint.
Would be cool if you could share more information how to do that.
Little motivation to make changes and their anti-spam systems only handle IPv4.
> Would be cool if you could share more information how to do that.
Put the bytes of the IPv4 Cloudflare endpoint at the end of a Cloudflare IPv6 Anycast prefix and voila. Fastly is similar but you take the bytes containing the site ID in the IPv4 address.
Also seems they didn't forget to update Cloudflare this time around. :)
HN Cloudflare IPv4: 104.16.104.110
HN Cloudflare IPv6: 2606:4700::6810:686e
Being able to access the site from my IPV6 only devices?
Almost all of my computers, including my phone, are behind wireguard on a globally routed /64 IPV6 virtual network.
It's a bit of a pain for some sites who do not offer a V6 addie via DNS, but it's extremely flexible and offer tons of other advantages.
Specifically, NAT is basically a thing of the past and any of my devices can talk to all of my other devices by establishing a simple TCP connection or shooting a UDP packet at them.
I can also access all of my devices from wherever I am connected to the internet, as long as the device has a globally routed V6 addie.
I was waiting for the catch, and I was not disappointed.
Any internet device of mine would immediately go into a quarantine subnet. It is a feature not a bug.
Don’t you need a basic router/nat to protect your systems?
You are under the mistaken impression that your router / NAT protects you.
It doesn't. It may mitigate some of the most basic attacks, the ones what were cutting edge in the 90s.
No, not really. It's no longer the 90s so tcp/ip stacks aren't easily crashed. And it's no longer the 90s, so no services are listening by default or it's say openssh which isn't easily crashed either (you may want to consider if you want to accept passwords via ssh though).
Additionally, decent OSes will rate limit responses to pings and SYNs and what not, so you won't be a good reflector out of the box.
And the way you say, you need a "decent OS" to avoid flood attacks without tinkering, whichever OS that is.
Sure, but that's not by default. You've got to take affirmative steps to enable that; although it's certainly easier to listen without limiting the source than to do it right.
> And the way you say, you need a "decent OS" to avoid flood attacks without tinkering, whichever OS that is.
Yeah, I just don't know for sure what's decent. I have no problem putting FreeBSD out on the internet without a firewall, and I think Linux would be ok too; but I wouldn't put MacOS if it's got any tcp listening ports, because it can be easily SYN flooded, and I'm not sure off hand if it has ICMP limits. If you put Windows on the internet and tell it it's a 'public' network, it'll run a firewall and you should probably be pretty ok (again, as long as you don't misconfigure applications)
If you browse the net on your phone, it's likely you already do or sit behind some kludged up NAT situation which - among other things - severly curtails your freedom to interact with other devices on the internet.
And things aren't going to improve in that regard given the shortage of V4 addies.
My ISP (Aussie Broadband) follows that recommendation and provides me with a /48 that I can break into multiple /56s or /64s.
Although to be fair even with non-contiguous address space I might still want a VPN since ISPs in the US are allowed to sell your browsing history.
Also if I'm hosting a public facing service at home I'm going to proxy it via wireguard through a VPS I rent for obvious security reasons. I don't actually want public facing services directly exposed from my home network and I have to question the sanity of anyone who says they do.
And I've always disabled webrtc for obvious privacy (ie network fingerprinting) reasons. What's so great about getting rid of NAT again?
Some exec decided we must be modern and all cloud. So they closed our datacenters and moved everything to AWS as-is. Including the process to request new "servers" with 10 tabs of complex Excel forms. They just replaced the rack location tab with one about AWS regions lol. It's as static as it can be. We even have the old lead-time for hardware emulated in the cloud now because of the complex approval process taking up similar time.
When you take the agile and auto scaling out of the cloud you just end up paying more. But I'm done telling them they're holding it wrong... Nobody wants to know.
I hope being modern was worth it. /s
As a first swipe ... this is smart - stepping your way into change is often better than one swell foop
However, if you never revisit it / plan to improve and do things The New Way(tm), it's stupid
Of course, Microsoft, Google, Oracle, Amazon, etc aren't going to do you any favors and show you how to use cloud computing more effectively than dedicated hardware unless you ask them to: they're just providing a service; if you want to pay them $100,000 for something you could do for $1750...who are they to argue?
But it doesn't seem like we are stepping anywhere. For example, if there was a clear vision to a real cloud-based company, I would have at least made facilities to to do new things The New Way rather than forcing everything to be the old way. Which is what they're doing. I needed a beefy server do run a big datacrunch periodically, using only minimal software which I can easily autodeploy for every crunch, run it a few hours and kill it till the next time. Sounds like a great option to use cloud, right? Kubernetes would be a bit heavy for this but I could totally ansible this.
However there seems to be no way in our org to get a server I can simply spin up when I need it. It needs to be requested in advance in threefold, internal billing agreed etc, and even if I turn it off we'll still be billed for it because I can't destroy it. Destroying and spinning up a new one would mean going through the entire rigmarole again :P
The funny thing is, we always do this. By the time we actually go full-on cloud, the world will already have moved on and be on the next great thing, and our neat cloud setup will be all deprecated tech.
Sure
Work for a company with money than brains?
Ideally not :)
By the way, it's only in the US. In Europe, people are a lot less zombified and susceptible to advertisement/brand image.
And now this exec has this "success story" in his CV and either got a promotion or jumped ship to a better role at another company. It's a tried and true strategy for the ridiculous corporate rat race.
As a solo developer, you don't encounter problems of scaling your engineering team and hiring devops engineers because you are doing everything yourself. The moment you have to hire expensive developers/devops all this starts making sense.
But even for solo/smaller accounts, many people are using AWS as a jump host with a programmable whitelist to reach their VPS boxes. Why? Because AWS security is likely best in the world - they spend massive amounts of money on this and many banks and even government services run on them.
(And even if you did us it only to scale servers, you don't need to click a button :) AWS autoscaling is finely programmable on all kind of application metrics even custom ones. Or by time of day: many teams routinely scale down their UAT and DEV clusters at night and weekends to zero with an autoscaling rule, more than halving server costs.)
Reserved instances may or may not be competitive with other providers. But they’d check the CTO’s or CIO’s lists on “we’re on the cloud” and save some money.
If you use a fair amount of bandwidth, AWS is outright a scam.
Of course bare metal is much cheaper.
In my previous job (gaming), colocation was literally 20x cheaper than using AWS EC2.
The only place it's good for is for backups where traffic is almost one way.
Besides, there are plenty of VPS providers with excellent customer service.
And it's not specifically the "I didn't have a card" problem that was solved. It was the far more general "I had an issue that required human intervention and sacrifice on behalf of the provider", which was resolved. I can think of zero other tech giants with that level of service.
Even if you have 100,000 simultaneous users egressing 10k/minute (which seem implausibly high, but usable for a quick approximation), you're only looking at 40 TB/mo
And that has to be off by at least an order of magnitude (if not 2 or three)
There's no reason, from a bandwidth perspective, you can't host HN on $100 worth of DO droplets and some Object storage/hosted db
If HN cost even $500/mo in hosting, I'd be quite surprised
That post doesn't say how much memory they were running in 2018 (I'll guess 32G, which is probably about right for a dual-core CPU or that era[4]), but an as-close-to-similarly-specced phsyical box from Hetzner[5] (6 core, 64G RAM, mirrored 1TB NVMe, mirrored 2TB spinny) is currently € 64.26 per month (68+change in Germany vs Finland) with 20TB transfer per month (if you add-on the 10G uplink, unlimited (within fair use policies) on the 1gbit connection)
With 4M requests/day (let's even triple that to 12M/day now), at 10k per request (a ridiculously high guess, I'm sure), that's only 40-120GB/day tops in egressed data (or 1.2-36TB per month)
HN's a pretty cheap thing to run, as far as hardware and hosting is concerned :)
------------
[0] https://news.ycombinator.com/user?id=terramex
[1] https://news.ycombinator.com/item?id=32466789
[2] https://news.ycombinator.com/user?id=dang
[3] https://news.ycombinator.com/user?id=sctb
[4] https://ark.intel.com/content/www/us/en/ark/products/64598/i...
[5] https://www.hetzner.com/dedicated-rootserver/ax51-nvme/confi...
Customer service : if you need to use cusomter service something has gone wrong, and I literally move provider.
Enterprise public cloud like AWS really doesn't make financial sense for most large enterprises or small organisations. If you have enough scale there's no need to pay the premium. If you don't need the flexibility you don't need to pay the premium. Only those in are stuck in the middle or have use cases that need extreme 'elasticitiy' or 'agility' really benefit from using this model.
Would be cool to know how this thing is run.
Last month HN had issues with SSD's suddenly dying after 40000 hours and moved to AWS temporarily: https://news.ycombinator.com/item?id=32031639
but i would also like to know how much traffic in terms of traffic volume (TB and Mbit/s peak) they do.