Hacker News Reader

You really should contact me in confidence. We’ve done almost exactly this and the data I have is compelling.

I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying.

The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...

Let me guess... you're using kit from the vendor at the bottom of that list? :)

No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.

The commentator your/this reply is to points out an interesting aspect:

> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.

This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.

Well... the landscape is indeed changing, as clarified by the 9-year old article above.

So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.

(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)

And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.

If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).

I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.

Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...

> I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally.

I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.

There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.

This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)

Cannot keep writing as I must go, may continue later

> Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards

I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.

Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?

I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.

What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).

To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.

> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard.

Hmmm. Good point. (Maybe I scared OP away, heh.)

> Is there an example of any large company, other than Google, who has mastered this?

Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015).

Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].

Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)

As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D

Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.

I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.

> I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta

I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).

Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.

My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.

> Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior

That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market.

> I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively

Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale.

Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.

> (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc

That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.

At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.

Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms.

One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).

[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.

> I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices

I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative.

I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly.

Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any).

The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway.

> you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs

I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap.

VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true).

> and autoscaling.

This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well.

Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings.

The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking.

Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill).

[1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.