Comment by krab | Hacker News Reader

Comment by krab | Hacker News Reader

A bit less terrible way in my opinion:

Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking.

Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware.

The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider.

Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.

This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes).

The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)

One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday.

I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone.

Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.

You're right you need to find a company you can trust.

And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).

I often recommend a "embrace, extend, extinguish" approach to AWS: Starting there for simplicity is fine, then "wrap" anything bandwidth intensive with caches elsewhere (every 1TB in egress from AWS will pay for a fleet of Hetzner instances with 5TB included, or one or more dedicated servers).

Gradually shift workloads, leaving anything requiring super-high durability last (optionally keeping S3, or competitors, as a backup storage option) as getting durability right is one of the more difficult things to get confidence in and most dangerous ones to get wrong.

Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.

Well, S3 is hard to beat for our use case. We make a heavy use of their various tires, we store a somewhat large amount of data but only a minor part ever goes out.

The compute and network heavy stuff we do is still out of AWS.

That's pretty much the one situation where they're competitive, so sounds very reasonable. Some of their competitors (Cloudflare, Backblaze) might compete, but the biggest cost issue with S3 by far is the egress so if not much goes out it might still be best for you to stay there.

Sounds like (unlike most people who use AWS) you've done your homework. It's great to see. I've used AWS a lot, and will again, because it's often convenient, but so often I see people doing it uncritically without modeling their costs even as it skyrockets with scale.

S3 is a decent product with zero competition. You should keep s3, it’s a fair price.

S3 has plenty of competition. It can be a fair price if you rarely read and need it's extreme durability, but that leaves plenty of uses it's totally wrong for.

> Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.

Is there an off the shelf implementation of this?

Minio used to support this, but removed the feature a while back. It was called “gateway mode”. Sadly, I know that doesn’t help much now…

https://blog.min.io/deprecation-of-the-minio-gateway/amp/

Every time I contemplate increasing my use of S3 or similar cloud services, I get annoyed at the extremely sorry state of anything on-prem that replicates to S3.

Heck, even confirming that one has uploaded one's files correctly is difficult. You can extract MD5 (!) signatures from most object storage systems, but only if you don't use whatever that particular system calls a multipart upload. You can often get CRC32 (gee thanks). With AWS, but not most competing systems, you can do a single-part upload and opt into "object integrity" and get a real hash in the inventory. You cannot ask for a hash after the fact.

I understand why computing a conventional cryptographically-secure has is challenging in a multipart upload (but that actually all that bad). But would it kill the providers to have first-class support for something like BLAKE3? BLAKE3 is a tree hash: one can separately hash multiple parts (with a priori known offsets, but that's fine for most APIs but maybe not Google's as is), assemble them into a whole in logarithmic time and memory, and end up with a hash that actually matches what b3sum would have output on the whole file. And one could even store some metadata and thereby allow downloading part of a large file and proving that one got the right data. (And AWS could even charge for that!)

But no, verifying the contents of one's cloud object storage bucket actually sucks, and it's very hard to be resistant to errors that occur at upload time.

I usually just use a suitable Nginx config unless there's a compelling reason. It means you "waste" the first read - you just let post/put etc. hit S3, and just case reads, but it's easier to get right. It's rare this matters (if your reads are rare enough relative to writes that avoiding the cost of the first read matters, odds are the savings from this are going to be questionable anyway - the big benefit here comes when reads dominate massively)

I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.

> how do you find a reliable, trustworthy datacenter?

drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself

That’s unfortunate. I consider SoC 2 compliance as a negative indicator of security (I’ve been on the vendor side of it, and have seen it have significant negative impacts on security and reliability in multiple organizations).

Ideally, there’d be locked doors, and the data center wouldn’t be subsidizing performative checkboxing.

I'm curious, can you tell us what the negative impacts you've seen are? Are there any audits that you can say are a positive indicator for security?

Spending all their security time/budget on box checking rather than actual security.

I'd rather see open ended red team pentest reports.

Yep. My experience as well. Once a place starts doing useless box checking stuff like SOC2 it’s time to find a new job or switch vendors.

Positive indicators would be talking to employees and getting an idea of organizational clue level. There are no shortcuts here I’ve ever found beyond doing this sort of old fashioned “know your vendor” style work.

This is my complaint with "cyber insurance". Companies spending money on insurance premiums and checklists for the insurance company rather than spending money on security.

I remember visiting a colo in the valley in the early aughts that had both of these: all the biometric-man trap-security drama in the front door, and in the back, garage doors & loading docks wide open to the Santa Clara breeze...

I had a similar mid 90's tour at a major provider in Washington DC (Ashburn) that had this exact problem, mantraps everywhere, great facility for the generation and then they took me out the back door to show me the basketball court in the back parking lot and no alarm went off.... Considering root dns servers, yahoo, hotmail and other majors of the day were running there, it was... sketchy at best.

As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)

The absolute trust killer is when your customer finds out about some issue on their own - even though you were supposed to monitor that. :-)

More important than raw uptime.

After years of dealing with colocation, I would never deploy a server without KVM in a data center. The cost of truck rolls is just too damned high! Even with KVM, I have had to make an emergency trip to a data center that is a 3.5 hour drive away due to a hardware errata on circa 2010 Intel Xeon boards where the network port shared between the KVM and the host would lock up under certain rare circumstances. The second time that happened I pulled the system from production.

If you do happen to have a system without on-board KVM, check out NanoKVM, which is a cheap (~$40) option for an add on KVM. It's rather more affordable than PiKVM. https://github.com/sipeed/NanoKVM

Reply on news.ycombinator.com