HashiCorp’s multi-product strategy | Hacker News Reader

HashiCorp’s multi-product strategy

foundationinc.co

102 pointsgmays88 comments

Vault is great. But if I were to start learning Terraform now, I'd probably ditch it in favor of Pulumi.

Terraform was supposed to make writing DRY easy. It's often more reasonable to copy-paste parts of configuration than to try make it DRY. HashiCorp seems to have the approach of "if something can be hacked around, even in a very convoluted way, we can ignore any feature requests for that thing to be doable in a normal fashion".

Instead of fixing Terraform, Hashi is creating more and more products that likely will be increasingly unpleasant to use and won't get reasonable fixes, just as Terraform.

The way HashiCorp approaches creating an ecosystem, makes me want to stay away.

I found a way to make Terraform code completely dry by using Hiera as a data source and writing every stack and module to be entirely data-driven (ie: only ever use "for_each" to iterate over a data map, never hard code anything, use strict naming conventions). However there are some major obstacles to using Terraform this way, that emerge from it being a clunky DSL with hidden "special rules", instead of behaving in a consistent way like a proper language. Like these:

https://github.com/hashicorp/terraform/issues/31072

https://github.com/hashicorp/terraform/issues/24476

I find that HCL (Terraform's declarative resource specification format) is an excellent target for a compiler for a DRY description of my resources. :-D

HCL doesn’t interop with anything. In a moment of weakness I once generated HCL using M4 (because passing a DynamoDB key schema into a module used to be impossible), but now I think it’s better to generate the JSON equivalents they also parse.

Their gRPC API for plugins looks pretty tough, and I keep hoping for an SDK for custom resources that doesn’t require writing Go.

100% - HCL just isn’t expressive enough to handle complex environments and you end up needing to do very strange things that would have been simple in a general purpose language.

I’ve been using Pulumi over the last year and it’s the best time I’ve ever had provisioning infrastructure. I actually enjoy it now. The automation API is also incredibly powerful, I don’t think the ecosystem has wrapped their heads around it yet.

I agree that Hashi's approach to Terraform is “if there's a workaround, we won't fix it” I don't see Terraform as hard to use for DRY stuff, we have fully adopted Terraform, we now have 100+ modules that are used in more than 20+ AWS accounts with no major issues, these modules are opinionated and are mostly just plug and play for the teams that use them.

Pulumi is great, but you need to really trust your people to write stuff with Pulumi, in our case most teams can just grab a module, submit a pull-request with the values they want and it gets reviewed by a member of the infrastructure team, if everything looks ok then it's merged and the infra gets deployed/changed.

Even with that simple workflow we have issues with devs misunderstanding some concepts, even devs that have been using Terraform for months, it's not a simple tool to use, I would be worried about them having to use Pulumi for the same things.

It might just be an issue of scale though, we have a gigantic dev team and are riddled with regulations, so Terraform is the perfect fit for us, and again, we don't have DRYness issues, nothing obvious at least.

AWS provider seems to be covering for some of Terraform's shortcomings. Until I was using just AWS, Cloudflare and other widely-used providers the problems weren't that much annoying. After switching to some niche services with not so extensively developed (yet not buggy to my knowledge) provider modules, the whole Terraform experience went sideways.

Those "let's just use silly count or for_each to enable/disable the block" suddenly not always worked for randomish weird reasons requiring digging into TF_LOG=trace logs. Other simple things that you are used to with AWS were not so simple anymore.

DRY writing wasn't an issue with AWS provider but I find it to be an issue with Terraform itself when provider module isn't covering for tf's shortcomings.

Agree, smaller providers are more difficult to work with, hard to draw the line between an issue with the tool or a developer introduced issue though, I do agree that smaller providers tend to be weird, so I think Terraform/Hashi could do a better job at making it easier for developers to create better/more reliable providers.

It baffles me that there's still no way of just disabling a block without having to introduce a count parameter, that should have been fixed ages ago.

> disabling a block without having to introduce a count parameter

for_each is (weird and unpleasant but) an option, too.

    for_each = var.enabled ? ["thing"] : []

why is this absolute dreck on HN?

nothing in this piece gives any indication that the authors have any idea what Hashi even does. the extent of analysis is "hurr durr cloud computing is big" and then a long nondescript "Open Source is Key to Product-Led Growth" buzzword bingo that reads like a noob discovering how bottom up open source works for the first time

i need a downvote button at the post level, i want a refund on my time spent reading this shit

think a lot of saas/developer tools/open source in corporate clothing are manipulating opinions, flagging, censoring criticisms. you can buy HN votes/aged accounts like Reddit.

noticed this especially when a YC company is in the submission. its funny because all they are doing is selling to the same buyers (many who are also YC companies) creating a sorta ponzi scheme where you constantly have a supply of "IPO ready" to take advnatage of the bubble which is now bursting and you have increased astroturfing

HN used to be above this. I feel increasingly that many submissions are just purely marketing hidden under a FREE sign.

> nothing in this piece gives any indication that the authors have any idea what Hashi even does.

Yep, quite fully agree with you here. (to be mentioned that I worked at AWS, I am very familiar with the industry, I met Mitch and other people in the team several times early on, and have extensively used their products until 3-4 years ago).

Especially considering Hashi is down around 60% from IPO and has an earnings call this week.

Anything to show growth. While I appreciate and understand having a vision, I'd have loved some of these products to just be as it is. With solid feature and we'll maintained. Alas that probably motivates no one and definitely not wall street

I can't comment on all the marketing aspects, but the idea of creating products to handle all of the concerns of application development and deployment seems like a really good one: everything from provisioning infrastructure with Terraform, to running containers with a combination of Nomad and possibly Consul. In my eyes, that's enough to validate HCL as a viable language, versus it just being a weird DSL for a single product instead.

Also, the other day someone suggested that Nomad doesn't have a future when compared to Kubernetes, though it would seem that there are plenty of orgs also investing in HashiCorp's offering and quite possibly benefiting from the somewhat simpler architecture.

That said, I can't help but to wonder why the cloud vendors don't embrace Nomad as well - after all, if many of them can offer managed Kubernetes, then surely managed Nomad would be both cheaper and easier to implement. Most of those platforms already offer a variety of databases (e.g. MySQL/MariaDB and PostgreSQL), so it's not like the idea could be dismissed on the grounds of duplicated effort alone.

I use terraform everyday, and I’m not exactly Warren Buffet but it is kinda funny juxtaposing this article with the current stock price and trajectory since IPO.

I never use terraform. I use web console first and then if something really needs to be automated I use bash script to string together aws cli. If anything gets more complex than that I do not automate it, instead I use CDK to essentially write infrastructure just like any code.

There is a lot of time wasted in learning a DSL and than wrestling with the limits of what it can do. For most infrastructure scaffolding you can get away with just CLI and anything longer should require something as natural as CDK.

I also will never use Azure or Google! Why should I waste time learning DSL and it's syntaxes when I am using the best Infrastructure as Code provided by AWS? The UI is familiar and I can always reach into CLI to quickly experiment and web console for learning what each parameter does.

Sure there is value in using Terraform but I think all it does is just another Kubernetes type of busywork. 20% gain for 80% investment. Not really a good way to spend your hours, sort of like writing tests before you write code from scratch.

We will disagree but I trust my experience than what I'm told to do exactly because I realize these standards are largely just dog whistling junior developers eager to please their managers to get them on a "free" solution that ultimately bubbles up to the C-suite with some McKinsey or Forrester branded market analysis pdf attached with setting up next steps.

Not only has big corporate interests invaded consumer interest, they've hijacked the open source movement, into just series of raising money and hoping for a large IPO payout or acquisition by other corporate whales. This is why these days avoid popular opinions/standards, especially when large number of people push for them on social media.

TLDR; HashiCorp is a good example of how to use open source offering as a trojan horse to boost the personal wealth of it's investors and founders and I am increasingly wary of "standards" or generating voluntary champions within companies to parrot and becomes salesman for a bit of glory and recognition. THIS NEEDS TO STOP. Very few end up surviving the test of time. Relational database and Java is stronger than ever. Javascript took a good decade to become recognized but it's still treated as a different class than boring technology. Boring and straight forward is resilient, novelty and complex is not.

Enjoy your zealotry!

not a very useful comment. if you got any rebuttal please fell free to share but the second half of what i wrote probably effects your future if you are in YC or a VC.

enjoy the ever costly capital as rates go to the moon!

Perhaps it doesn't apply to your situation, but that happens if you need to build 5 more copies of your current setup? The CDK stuff would be fine as long as your code is DRY, but all that clicking on the console... Also, can other people understand and extend your infrastructure? (Honest questions)

you are describing different scenarios here with different scope and i did not say to use web console only

I literally said "The CDK stuff would be fine as long as your code is DRY, but all that clicking on the console..." So I was asking specifically about the console parts. If you had to redo it again or repeat it a number of times, wouldn't you prefer if it was code and not clicks?

Open source rules the world. A lot of people use HashiCorp's OSS products; not a lot pay for them. I certainly haven't and I don't think I've ever met a paying Hashicorp customer even though I know lots of people that use their products.

They are a consulting business ultimately. And like many such companies they thrive on complexity. They are doomed to become part of the problem they are trying to solve. If they succeed in their mission, the result is OSS software that is so simple to use that nobody needs their consulting services.

Like many OSS companies, they confuse control over the software with controlling the wider market. Amazon, Google, MS, etc. don't provide just software but their 630B dollar industry that is enabled by their massive data centers and infrastructure. That's what people pay for.

Hashicorp is a small fish in that pond as they don't really own or control any such infrastructure. I'd say they might end up being gobbled up by a bigger consulting company. Oracle and IBM come to mind.

Honest question/suggestion: don't you think many big companies are more than happy to pay for the consulting to give them a feeling of comfort and safety, someone to turn to if there's an outage in the middle of the night?

Not that I'd need much Hashicorp folks on my "bridge call" when my Websphere cluster runs out of memory, but since IBM and Microsoft are on speed dial (in their million(s) dollar support contracts), we may want the same for Hashicorp. This "we" is not "me", but I did work for this sort of company in the past. Open source made them uncomfortable, consulting may help with that.

And sure, it lasts only as long as it does.

Anybody can go and install Linux on almost anything that computes. How do Red Hat, Canonical, Suse even exist?

I'd say two parts are key: support (you can see it as a kind of technology insurance) and customized, tailor-made solutions in more demanding cases.

Those who use Hashicorp products for free mostly would never pay for them. But some of those people will work in larger companies which would pay, and it is important for Hashicorp (Microsoft, Google, etc) that those people, as technical experts, would suggest buying the tools they know and love.

Red Hat is the exception that proves the rule. Canonical and Suse haven't been huge successes from a financial perspective. They've survived mostly by investors hoping they'd turn into the next Red Hat but they haven't.

Certainly Amazon, Microsoft, and Google have generated much more revenue selling cloud services based on Linux than even Red Hat ever did selling Linux support.

Don't forget Gitlab.

People don't buy linux licenses but support licenses. If you want to use Red Hat, you can get the free version. Which is exactly what smaller companies do. Big companies tend to want to pay for ass coverage.

IBM, and Oracle are in that business. IBM owns Red Hat, Oracle maintains a derivative for their customers. The primary thing they sell is consulting, support, training, cerfification, etc.

Hashicorp is similar except much smaller. You don't buy their software but their support.

Hashicorp isn't just a small fish in the pond, it's a bunch of smaller fish in different ponds. I use Terraform to deploy EKS and a bunch of other AWS stuff. I have used Vault as part of a secret management strategy. I've only used Nomad as a toy. I ditched Vagrant after docker came along.

Hashicorp isn't one cohesive product, it's a bunch of parts that can work together. "But you may need an adult to help you with the scissors and glue".

> If they succeed in their mission, the result is OSS software that is so simple to use that nobody needs their consulting services.

If that comes to pass I think that's okay, even admirable.

Not every company needs to keep growing and growing, trying to take over the world. Not every company needs to continue in perpetuity.

> If that comes to pass I think that's okay, even admirable.

Just not very profitable. That's why devops is such a complex job.

I think you may be surprised.

There are very very large companies that surprisingly have hashicorp products deep in their central core - Vault is an obvious one.

At the heart of all huge companies are InfoSec and architects looking for a solution they can trust will scale - and what better than the OSS product they have hands on experience with.

It might not be profitable in Microsoft terms, but then again we won't see musicians earning like the Rolling Stones anymore. Open Source chnaged the software publishing game just like internet chnaged music,

But there is enough profit out there.

I worked for a shop that payed them for enterprise licenses, it was $$$ and imo the experience had been subpar. That was number of years ago so hopefully support has seen some improvement with all that extra IPO capital... That managed to capture a huge mindshare across all levels of infrastructure devs so that ought to be worth something.

We looked at paying for Terraform Cloud but their pricing was off the charts compared to all their competitors in the TACO space and they were being out innovated by 3x.

We signed Spacelift instead for 10% of what they were asking

Although there's a paywall towards the bottom (is it 25%, 60%, 90% through the article, who can tell?), the above-the-fold resonates and is a reasonable narrative about HashiCorp's success. Certainly HashiCorp shines in the UX department.

That said, I think there's a another angle: HashiCorp is provider agnostic. HashiCorp tooling naturally lends itself to a multi-cloud strategy because they aren't owned by a cloud provider. Current requirements aside, coupling your business to the whims of Amazon, Google or Microsoft is always a risk. Even Kubernetes, the closest thing to an orchestration standard at scale makes me nervous because Google's needs are not the needs of most businesses, and what happens when they decide to throw their weight around?

As long as HashiCorp remains independent, I think it will attract a lot engineering-focused businesses that want to control their own destiny. I hope they can resist the urge to chase the enterprise money dragon to its logical conclusion. We are all better off for having some smaller-scale competition to the cloud behemoths who will only care about developer experience as long as they have to.

Kubernetes is managed by a foundation so less of a risk than a tool owned by one company.

I think one way in which it plays out opposite to this is that Kubernetes, among companies I’ve talked to, is being deployed as EKS or GKE with cloud provider specific extensions/integrations while Nomad is much much easier to self manage on top of bare VMs. Even being owned by a single company, there is something about being to self host that feels more “free”.

Obviously this could make a 180 easily because of how much control HashiCorp has, but, thus far, they have been pretty good stewards.

Wanting to turn on just one more server to find months of lead time and a bunch of red tape to get stuff done, yeah I don't miss "owning the hardware" one bit. In times of economic hardship when finding skilled labor to manage infrastructure is easier to achieve, trading hardware costs for labor costs may make more sense, but I don't see that happening in today's economies for most western economies.

I think there is a quiet but noticeable push back to owning hardware for predictable workloads but that’s beside the point. You can run nomad on EC2, GCE, whatever the Azure equivalent is, in addition to bare metal you own.

Setting up your own, non-managed Kubernetes cluster is definitely possible but I’d argue it’s on a whole different level of difficulty.

Single point of failure!

Kubernetes absolutely eats the HashiStack lunch. I think they’re aware of this. That’s why I think HashiCorp invested so much VC funding into their SaaS offerings: they know consul, vault, and nomad are a huge pain in the ass to configure and manage, particularly in support of each other.

It’s all a little bit more agnostic than k8s but If you actually run consul + vault + nomad in production you know that managing it and repairing it can be a nightmare. GKE or EKS on the other hand feels like I almost shouldn’t even have a job.

In fact, two of the most high profile outages in the last two years(Robinhood March 2020 and whatever Roblox outage happened recently) we’re due to consul or vault or something going completely hare brained.

As a counterpoint I manage multiple HashiStack clusters in production with little overhead — it's not a nightmare. I am a Kubernetes refugee and Nomad is such a better experience with batteries included.

Nomad now has its own service discovery (along with secure variables coming in 1.4.0) so you won't actually need Consul and Vault anymore. This is perfect for smaller deployments like on a Raspberry Pi.

I came to say pretty much the same thing. Nomad is awesome software. I use Nomad at work for internal workloads and automating its deployment with TLS + Consul is wayyyyyyy easier than k8s. I am really looking forward to 1.4.0.

Also, I made some automation for setting up Nomad on a single Fedora CoreOS server (still kind of a work in progress) if anyone wants to give it a try.

https://github.com/quickvm/butane-nomad

Are your clusters at the same scale as Robinhood or Roblox? We’re talking many millions of inbound requests per second generating multiples of that on internal service calls.

Nomad, Consul and Vault work find at small to medium scales. Sometimes. If things are working really well and there aren’t any real problems on the network. It’s very fragile, though, and tends to shit itself under the slightest bit of abnormal pressure.

Exactly this; I would be knocking on all kinds of wood if you’ve had nothing but success with anything Consul/Vault/raft backed. Either that or beefing up your consul server ec2 instance types as soon as possible.

So far for me Nomad has been very pleasant to work with. We are a small startup with just three devs - and are running multi-region Nomad in production for a few weeks now.

We migrated around 20 services within a couple of days, coming from a mixture of all sorts of AWS services and Hetzner infra.

Documentation was fairly good and the community was helpful.

(This is without consul or vault, we use native service discovery so far)

> Google's needs are not the needs of most businesses, and what happens when they decide to throw their weight around

Nothing. Because AWS has invested heavily in Kubernetes and they would just fork the whole ecosystem.

They've done it in the past with Elasticsearch, Spark etc.

> This multi-product strategy of creating a consistent cloud foundation has launched them to a successful IPO in December 2022, is driving their $5.6-billion valuation, and helps them bring in some high-value customers like Lufthansa, AstraZeneca, Ubisoft, and thousands more. (Source: https://foundationinc.co/lab/hashicorp-product-led-growth)

I don't agree with this assessment. Yes, the IPO was successful for the first few weeks [0], reaching $90 at the end of 2021, but is now down to ~$36.

Seems to me that the market is not too excited about them, or any other high-tech company right now.

[0]: https://finance.yahoo.com/quote/hcp/

Article is locked to members but the part that isn't doesn't convince me much.

The narrative is hard to buy into when they've lost >50% of their value since IPO so far.

I can buy they are doing awesome in the abstract sense, but ...

Not $600B. Yeah.

> The narrative is hard to buy into when they've lost >50% of their value since IPO so far.

Many companies on NASDAQ have lost 50% of their value in that time window. It just happened to co-incide with a global downturn.

Sure, but that's also kind of the point - they aren't special in any meaningful way.

If you look at their reports - they are going to have to massively cut costs somehow.

If you look at their EBITDA and EPS and compare with their growth, future is not looking all to bright. They either are going to have to massively cut expenses (thereby potentially losing their R&D advantage), increase their conversions/sales or greatly raise their prices. Not saying that it's not possible, but it's a high risk bet to be fair. Nevertheless, a stock with a great upwards potential if they execute rightly. But that's a big if.

> There’s company-wide alignment on delivering the best user experience possible. That means sales, marketing, product engineering, product design, and customer success all channel their efforts into creating a product that provides an unparalleled UX. Product development puts all departments on the same wavelength in terms of the end goal.

Who the fuck are they talking about? Clearly not HashiCorp. The most popular tool in HashiCorp's ecosystem that wasn't written by them was written by a consulting firm who hated HashiCorp's UX. Hashi is like Amazon: shitty UX, but features that capture a market early, and so they remain a dominant player despite shitty UX and unnecessary overcomplicated bullshit.

There are only two things that you need to succeed with open source. 1) solve a problem nobody else has solved yet and drive massive adoption. 2) make it so that eventually people realize they can't get any farther without paying for enterprise features, and their lock-in and over-confidence from the free product will make them convince their bosses to pay you.

> The most popular tool in HashiCorp's ecosystem that wasn't written by them was written by a consulting firm who hated HashiCorp's UX

Which tool is that?

On a related note, Vault has a really excellent API. A joy.

I'm assuming this is Terragrunt, which itself perpetuates all kinds of horrific practices which are absolutely unnecessary if you apply basic software engineering principles to infrastructure as code/config.

Where does it say Terragrunt is part of Hashicorp?

> The most popular tool in HashiCorp's ecosystem that wasn't written by them was written by a consulting firm who hated HashiCorp's UX

This is the initial comment about Terragrunt, which doesn't say that

Indeed: it says the exact opposite.

Good, we both agree that no one in this thread suggested terragrunt is developed by hashicorp?

Do you know of any articles or other writeups about this?

Terragrunt is an antipattern. Made the mistake once with a very poorly structured project I inherited, never again.

Terragrunt is not an anti-pattern, but it is a tool that is easier to get wrong than Terraform and so people create great messes with it.

Used right it's the only way to remain sane on any non-trivial set of infra where you don't end up hand-rolling a half-baked version of Terragrunt.

Managing 10^5 servers right now with Terragrunt. Can confirm, it's the only way to stay sane.

Getting to this point was a learning process though. Probably six solid years of investment now.

From experience terragrunt + atlantis has been much better experience than hashicorp's own terraform enterprise service.

not using terraform enterprise.

"Code generation" is pure evil.

I second that Terraform needs to work at least on dynamically specifying providerblocks. This is where people usually resort to terragrunt.

Workspaces and state layering are in my experience hard on the novices and unfortunately they turn to terragrunt.

Could you express more? There's gotta be a good use case for Terragrunt; they have been useful so far for me

Not disagreeing, but you have to create a shit ton of scaffolding to work around TF's lack of useful functionality and painful UX. Terragrunt exists because somebody had to do the same, and then kept adding on "smart" features making it even more overcomplicated.

Vault is excellent.

> Vault has a really excellent API. A joy.

Strongly disagree: suppose we have secrets mounted at my/secrets, and we want to read a secret top/secret, which is represented as my/secrets/top/secret path in vault. However, the only way to access it via API is to read _all_ mount points, and match them with the path to split path to mount point and secret. vault cli itself follows the same logic: https://github.com/hashicorp/vault/blob/main/command/kv_help...

We’re using different Vaults it seems. HashiCorp’s tech is a mess of over engineering, poor UX, and insufficient documentation.

>HashiCorp’s tech is a mess of over engineering, poor UX, and insufficient documentation.

Second that. Very steep learning curve for some use cases that could be accomplished in a much easier way. Another kubernetes in disguise.

k8s in disguise? Vault? Have you used either of these tools?

They're talking about Nomad here, with the "tech" in the GP comment widening the discussion, I assume.

I'm assuming you are talking about terraform?

The UX of all the other hashi products I've used have been absolutely sublime compared to the industry equivalents. Nomad compared to kubernetes is a particularly stark contrast.

Yeah vault is great… unless you need to store 4096bit keys on a postgres backend… can't do that.

All of HashiCorp's products have splendid UX — what are you on about? Have you actually used their other products aside from Terraform, which I'm assuming you're referring to?

I don't understand why some people hate teraform so much.

Because it sucks. It has all the quirks and a whole lot of mess. That's terraform. Terraform Enterprise and Cloud must have been developed as a fuck you to enterprises, because it absolutely sucks.

I work with them daily, and have done so for 4 years now. It still sucks.

Unfortunately it's (Terraform, not TFE/TFC) the best in its class, and quite frankly I'm not smart to solve its issues.

If you're having a bad time with TFC/TFE, make sure to check out Spacelift[0].

We're striving to build a product that's both highly customizable and has a good UX, improving on many of the pain points. The migration path is straightforward and it'd be great to hear what you think about it!

[0]: https://spacelift.io

Disclaimer: Software Engineer at Spacelift

I have checked it out. Also checked out Scalr and some other one that I can't remember the name of. Issue isn't finding an alternative, it's convincing the team to move away from TFE and TFC.

Terraform is great... until you hit one of it's many limitations, or run into a bug in a provider that really ruins your day.

Plenty of people probably never run into those problems. But if you do, it's pretty terrible.

To be fair, there isn't really anything better. Pulumi and CDK address some of the limitations by using a "real" programming language, but introduce different issues. And bugs in providers are probably unavoidable, and are partly the fault of poor APIs from cloud providers.

Agreed. And nobody tries to replace it because nobody could replace all those providers without an army of engineers. And nobody is going to use a new project that only implements one or two APIs.

> run into a bug in a provider that really ruins your day

Most of these problems I encounter aren't even down to hashi themselves. The GCP provider consumes from google's magic modules repo, which is often out of sync with the GCP API. Or in plenty of cases, it is in sync, but the API does not expose features in the same way as the console.

> $630B Industry

They're not actually competing for the $630B

granted, not that I follow this all that closely, but I just remember vagrant. Then boom, all of this cloudy infrastructure stuff...

Reply on news.ycombinator.com