Chef to be acquired by Progress
blog.chef.io
blog.chef.io
While I can't comment on the exact problems because I didn't use it personally, there was a recurring mentions of losing states, seemingly destroying systems and configurations instead of setting things up. People were adamant that moving away from Chef was the best thing they've done lol, which I don't know about but I take at face value after the third person saying the same thing.
Plus, they have been diversifying their products in recent years with things like Habitat (application packaging/deployment) and security/compliance tools (InSpec).
I'm hoping it's the former, but I wouldn't be too surprised if the latter since the 'enterprise' ecosystem of cloud tooling has grown exponentially in the past decade.
Here's a simple test: if your runtime container infrastructure is immutable, you should be able to run your container with a file system being read-only. If you cannot do that, then you do not have an immutable infrastructure and hence you will need need something to manage the state. Ignoring it is akin to kicking housekeeping tasks to some point in future and hoping that before you get buried under the multi-year backlog of tiny corner cases before your entire product is replaced with something else.
But anyone could see many years ago that it’s easier to flatten and reinstall stuff than trying to painstakingly manage various moving pieces. We were limited by our technology unless one was using stuff like zones in BSD or Solaris.
The surprise for me has been how rapidly huge behemoth organizations like the DoD has shifted to technologies like K8S when they took so many years just to get onto VMware and some configuration management.
Ansible is/was straight forward to use and understand, very F/OSS, relatively easy to orchestrate (even without tower), and had a nice ecosystem. Ansible always won for me because it was simple, and had just enough (though I always felt that the roles/tasks analogy was kind of awkward) and was easy to hack. These things were mostly true about Chef, but outside of being "ansible, but ruby", I don't remember distinctly what made Chef worth doing other than if it was there before you got there.
Someone please correct me if I'm wrong but IIRC Salt was one of the first to start venturing into the infrastructure provisioning realm and ansible went there too relatively easily though it's somewhat obscure[0]. Ansible and Chef (and presumably puppet) were quick to follow/build similar tooling but we all know who won that race -- Terraform.
Funnily enough, Terraform is now making it's way back around to eat the post-deploy logic as well with it's provisioners[1]. Can't wait to see HCL grow into an even more grotesque monster.
- Your team is comfortable with at least one of the languages Pulumi has native support for, ideally your codebase/infrastructure repo is already written in it.
- Someone on your team isn't going to hang everyone else and themselves with the wide open nature of a fully featured programming language at their fingertips
- You've got some innovation tokens to spend
I think HCL is going to get more and more like a fully featured language as time goes on -- if they ever lean in to it (and introduce support for alternative "language runtimes" or something) then I would go terraform and never look back.
It's hard to recommend pulumi over terraform because of momentum, ecosystem, etc, but it does seem like a better base to start off of (and it's what I use for my own stuff).
> Today, we are pleased to announce the community preview of the Cloud Development Kit for HashiCorp Terraform which allows users to define infrastructure using TypeScript and Python while leveraging the hundreds of providers and thousands of module definitions provided by Terraform and the Terraform ecosystem.
> CDK for Terraform is in alpha and is in the early stages of development. This project is another way to interface with Terraform using languages like TypeScript and Python. We are working to support additional languages such as JavaScript, Java, and C#. In addition to language support, we plan on expanding the scope of CDK for Terraform project to support other first-class commands to provide a user experience similar to the AWS CDK.
This has to be a shot across the bow at pulumi, glad Pulumi innovated and took this direction (maybe CDK was on the roadmap at Hashicorp, not sure) and seemingly spurred TF taking the turn as well.
Those problems sound like people aren't taking the same care in their cookbooks that they are in their regular software. If you write bad cookbooks and don't at least give a cursory review of the community cookbooks you use, you'll end up with a mess. If you treat it like the customer facing software it is, I find it a pleasure to work with.
Our cookbooks have CI pipelines that run some basic tests and we have automation around rolling out changes. We rely on a lot of conventions to avoid storing state, but we still have nodes auto provisioning their own storage and load balancers where necessary. It's let us stand up a ton of infrastructure and make changes safely and quickly. That's not to say we couldn't have made Puppet/Ansible/etc work, because IMHO it's more about your development practices than anything else.
edit: we only use Chef Infra and some Inspec. Not talking about Habitat.
For example the different levels of attribute types. I saw "senior" developers writing nasty Ruby code (actually `system("rm -rf directory")` to delete directories (outside the chef DSL) because they didn't understand the appropriate chef resource.
The market decided to take away the power of a real programming language DSL as an interface. People are now forced to deal with terraform (HCL), YAML or JSON. It's a lobotomy.
Perhaps the company itself has improved since then. I certainly (for Chef's sake) hope so.
Investors aside, did anyone make money here?
Either way, I’m glad that they are continuing to exist. Chef was onto something with Habitat and (especially) InSpec.
The risk is that one of the later investors had a ratchet or something that would allow them to claim more of the proceeds in a sale. You can't just take $220M cash, subtract $105M in funding and pass that to the founders and employees. The preferred shares were probably "participating" meaning they get a portion of the common.
Here's an example of a ratchet: https://www.forbes.com/sites/petercohan/2015/11/07/unicorn-s...
Founded in 2008, that's 105M for some fraction (maybe a majority) of the equity... minus fees / taxes / legal costs, that's a pretty lousy investment, especially compared to market returns over this timeframe.
As one of those people, I just hope they don’t try to close the doors on the Cinc community.
https://github.com/skx/puppet-summary/
Simple to deploy, configure, and use. Although I always felt I should perhaps have based my work against the puppetdb to get a more dynamic "dashboard" it certainly replaced the old dashboard, and being only a single binary it is simple to install & manage.
(Though I don't know if it supported puppet at that time.)
Again, only casual observer, so my impression may be overly negative.
Meanwhile, their core technology is kind of a dead end. More and more people are looking to manage containers and other serverless type things, rather than managing fleets of VMs. They're not in the same boat as companies like VmWare, but they're in the same flotilla. If somebody does buy them, it'll probably be an act of dinosaurs mating and not much else.
There is less buzz around those tools now however, and people tend to gravitate to other tools which are easier to get started with.
They may end up "silent heroes" (much like Ruby itself, I would say) or they may end up replaced by other tools. The concept of state as text in version control is a timeless concept and continue to exist in one form or another.
One of the problems with configurations management systems is that they are conceptually easy, everyone has opinions of them, and they really need to control your whole world in order to shine. There was never really room for one clear standard to emerge.
Puppet Inc.'s problem was that they kept chasing shiny stuff (agentless you say? we have that too!) in order to woo enterprise customers. Their customers should have been developers and sysadmins, not their bosses. There is only room for so many VMware in the world.
Eh. While I won't say that it can't be used, I will argue that it's not the best tool for the job. I wouldn't call it a "great tool" for that purpose, although I suppose that depends on your workflow. I prefer to take the approach that containers are immutable once built. If you want to change the container, fine- make your changes, build a new instance, hotswap the connections over, and bring down the old one. There's no need for any sort of config management in my eyes, at least not in the traditional sense.
Now, Puppet could be useful for things like managing the VMs that the containers are running on, but I would prefer those stay immutable too, and with containers, they easily can. I'm (sadly) very aware that not all enterprise software can fit in this model though. Enterprise always lags behind, but it will get there. Hell, even SAP is heavily exploring containers. It's only a matter of time.
Even if that was workable, which is a question in itself, there is still configuration outside the packaged application. AWS keys, metrics and logs, dashboards, that sort of things.
Likewise, how you separate config data and deploy to production is an age old problem with many solutions as well. Sure, Puppet can even be a solution here, but you're just kicking the can down the road. How do you separate your Puppet code from data? Put it in Hiera? OK, how do you promote Hiera? It's the same issue.
Both of these issues were solved long ago before Puppet came along, and what we all learned from back then still applies: separate your data, config, and code. Version control as much as possible. There are many simple solutions that don't require Puppet. Hell, set up a bind mount to an external config store. Or something fancier like your CICD tool bundling the config data and signing it with a cert that only a certain environment can use so you never get dev code running in prod.
The argument is that when your infrastructure is immutable (which is the direction most of us are going) you don't need a tool to manage the configuration of said infrastructure. Because it can't change. When you do want it to change, you just follow the typical code SDLC and use the powers of version control and CICD.
Should you at some time replace all this with a common data format, then congratulations you have invented a configuration management system very similar to Chef.
The use of a version control system or a CI/CD system does not somehow negate the need to keep configuration, neither does a configuration management system do the job of the former. Nobody builds their software using Puppet, they use something like Jenkins. In fact, I would consider having VCS and CI/CD in place a prerequisite to any type of configuration management.
I am not sure what complex config state management solution brings to the table for immutable containers. The state is immutable, so there's nothing to manage after deployment. It's like using an atom bomb to take out a zit.
Puppet/Chef in their glory days, pre-cloud pre-container, excelled at managing desired state across numerous mutable static systems. With immutable solutions, Puppet/Chef are both cumbersome and expensive.
The deployment side also carries configuration, including things like desired amount of instances, request routing and filtering, log destinations, log retention, persistent volume sizes and location, backup rules, metrics and monitoring rules etc.
If anything, application deployments today carry more configuration, not less. Fifteen years ago, half of the above didn't even exist. Perhaps you pointed you application to a syslog server and that was it.
All of this configuration exists in disparate tools (JSON files, YAML files, firewalls, metrics dashboards, cloud providers proprietary APIs) and will over time slowly turn into a sprawling mess. Bringing control over this into a central repository is a good thing.
Not sure about expensive, as the aim is maintainability and reduced complexity, but these tools do tend to get cumbersome. After all, they want to do everything. It's a fundamental problem. It's not surprising that has led to a surge in less capable tools.
But Kubernetes solves most of this problem in an easier way with Pods, ConfigMaps, Secrets, Services and Endpoints.
Your whole infrastructure contains much more than that, and managing all that as a coherent whole is what these tools do.
Testing Puppet manifests locally is (was) THE WORST. Hiera and r10k really shouldn’t exist in 2020. IIRC you essentially need long-lived Git branches for managing different envs for different playbooks.
Ansible is just SSH and a binary. pip install and I’m done. This is great for building VMs, base Docker images, app deployments, whatever you need. Puppet has their own wire protocol and is very dependent on a client-server relationship. You need to edit config files and fake a puppet master. That’s work.
What kind of branching strategy is most suitable is given from how your testing is organized, not which particular tools are chosen.
Ansible is completely equivalent to Puppet or Chef in this regard. They are all the same family of tools. Many find Ansible easier to get started with due to its agentless defaults, but there are some (perhaps Spotify was most well known at the time) who run Puppet agentless too.
Having a server, something like Tower, is necessary as soon as you want to manage distributed configuration for clusters, and if you manage monitoring and backup systems.
In total over 12 years, Chef raised around 105M and puppet raised around 189M in funding.
They're both doing pretty bad in the end and are on a (steep?) downward trend. They'll never be a billion dollar business. They probably won't recoup their valuation. Chef sold out. Puppet tries to survive.
From my perspective Chef users were actively fleeing so I guess it was sold for scrap, puppet might be downward but maybe not as steep so maybe puppet is trying to continue to operate as a small medium company. Also Puppet might be a fairly harder sell because it has both a higher valuation and debt, doesn't look like a great buying opportunity.
On Puppet, I think they are a different beast to Chef at this point. They seem hyper focused on the larger enterprises rather than SMB or startup land. I came across them at both of the banks I recently contracted for, for example. Whilst I wasn’t hands on myself, there was a big project to roll out Puppet’s impact analysis and CD product at the last place.
I also ‘think’ Puppet are about double the size of Chef and saw their BlackRock debt round announced. You don’t get money from someone like that unless your at scale and you don’t get debt (especially at a time like this) unless you can afford the payments+interest. So, they must be at least growing if not profitable as well.
They have the Telerik suite of tools, which is used by .Net developers, but how many .Net developers use Chef?
Chef isn't just CM, it's also a lot of policy enforcement tooling these days. I don't have much use for it anymore but it does have its niche.
That being said, the experience on Windows is abysmal. The implementation with ruby on Windows is sloooooooow. You wait seconds just to get things like version.
While you could access the database via SQL that was very much the second class citizen; it's native 4GL was fully integrated, and long predated attempts like Linq to hide developers from SQL.
https://www.wikipedia.org/wiki/OpenEdge_Advanced_Business_La...
Since it seems to be really hard to find this information, want to make a copy of the content of the second link ([1]):
> A New Corporate Sponsor for NativeScript
> Progress is proud to introduce nStudio as the new corporate sponsor of NativeScript. After fostering a robust open source developer community, Progress looks forward to what nStudio will bring to the NativeScript community and customers.
To be fair, 4GL and OpenEdge work very well together.
Of course not - the entire business model behind OpenEdge (which is far from "open") is to have applications built with their laughably irrelevant ABL/4GL language and only working with data in OLTP / one-record-at-a-time because that makes it almost impossible to switch to the conventional (RDBMS + web-service + frontend) architecture that the entire industry shifted to 20 years ago.
OpenEdge (nee "Progress DB") was a cool platform in the late-1980s through to the early 2000s because it supported IBM's AS/400, MS Windows, and Linux - with their UI system that let you design an input form once and have it magically work in text-mode AS/400 terminals and the Windows desktop - but the fundamental design of OpenEdge is still based on its AS/400 roots and it really doesn't work well with modern systems (e.g. it has this design with multiple "broker" processes which is a PITA to configure).
Their anti-open-source article on Progress' website that the GP post linked to was infuriating to read: it felt like the same anti-GPL propaganda that did the rounds around 2003 when SCO was claiming Linux contained their copyrighted code and Microsoft was astroturfing articles about the "risks" of open-source code and seemingly intentionally confusing people about MIT/Apache vs. GPL licenses. Le sigh.
"We have a long history of supporting open source communities so I am proud to say that we will continue to nurture the vibrant and thriving Chef open source community."
Some of its products (e.g. Telerik Kendo, NativeScript) have open source versions. There's also Fiddler Network and JustAssembly/decompiler.
They might claim though - as in the link below - that this is because their products are too niche to support quality input from open source. Make of that what you will.
https://www.progress.com/tutorials/odbc/open-source-database
With that said I'm glad I moved onto a *nix based project and got to learn Ansible and docker
“100% growth in incremental recurring revenue” doesn’t say much if they are churning a lot of existing customers. (That they chose such a double-speak revenue number perhaps says a lot). My impression was that DevOps software like this is very hard to pull out once you’ve embedded it in the org.
Am I missing something?
I worked in a couple places which used Chef and paid for 1-2 years but eventually terminated the commercial relationship.
We don't know anything about the profitability or the growth. I can imagine that their growth is flat or even declining.
[0] https://investors.progress.com/news-releases/news-release-de...
All while Ansible is doing fine.
Chef tried to adapt with Chef Workstation to do push-based config, but has inability to extract state from the system configured. The target system would have to update state to a server, which is fetched indirectly. This doesn't scale, so this is another reason why Ansible and Salt are popular.
Puppet also experiences some of the same issues with Ansible eating their lunch and popularity of push-based config for immutable infra. They attempted to respond with Bolt, but Bolt is based on static hostnames or DNS names, which won't scale given dynamic nature of cloud native and transient ephemeral systems.
In the case of managing fleets of systems that are not atomic stateless nodes, where you need to maintain a state across nodes within a set, both Puppet/Chef do not scale, and create outages (though window is small), because they have to synchronously push state to a server, and rely on eventual convergence. This doesn't scale in cloud computing. With push based config, you can set the cluster into the proper state, and then use service discovery (asynchronous updates) to maintain the state of the cluster. In K8S, kubectl/helm would fill the push role, and etcd used to maintain state. Outside of K8S, such as lambda in cloud, pulumi/terraform could push state, and discovery through cloud metadata (labels, tags, etc) or service discovery like consul to maintain the state.
Chef and Puppet could have responded, but couldn't see past their own platforms that are based on managing desired state for groups of individual atomic systems. They also failed to monetize on things like inventory management that enterprises fork over a lot for such things.
You're proposing that he replaces a cross platform tool with an entire OS and on top of that with a tool that has a very peculiar package management system (I'm not saying it's bad, just that it kind of requires full buy-in to make it work).
Thanks a lot for building this!
I toyed with a simple puppet-alike, written in golang, called marionette (in hindsight a terrible name):
https://github.com/skx/marionette/
It isn't anywhere near as complex, or featureful, but starting with golden AMIs it allows the necessary changes that I need in a consistent fashion.
Well, yeah, especially as Puppet has a well known solution named Marionette Collective
* https://puppet.com/docs/mcollective/current/index.html * https://choria.io/docs/about/mcollective/
Historically there were 4 main configuration management tools: puppet/chef/ansible/salt.
The first two faded away. They're unarguably not the most usable and require to program in ruby. The next generation ansible/salt came out and arguably did better on every aspect.
I think it's fair to say that ansible is the safe standard (redhat acquired it few years ago, they maintained and extended it pretty well). Ansible is very easy to use and to integrate in anything as long as you have SSH access.
The only downside of ansible is that it can get slow with thousands of hosts to manage (gotta establish SSH connections). Salt can do better with large amount of hosts.
I don't want to orchestrate against hundreds of systems with SSH and tell them what to do. I'd rather tell them what I want, and let them sort it out locally.
Pass all of your variables and dependencies (role artifact urls, python requirements.txt, etc) to machines through user data and have the machine download the dependencies and run ansible-playbook to configure itself. Optionally, add a systemd timer to run it on a schedule and you have an "agent".
I've done it this way a large percentage of the time using ansible. The other way is by baking images and using ansible as the provisioner (using ssh but only during the image build).
That has been my experience/perspective as well. This was what I found industry...
* ~2012 Puppet golden years * ~2014 Chef golden years * ~2016 Surge of popularity for Ansible and Salt. * ~2018+ Kubernetes ubiquity, Terraform for cloud, Ansible for systems
Related to this, saw popularity in SSR (server-side rendering) with Rails before 2012-2016, and after 2016 rise of popularity in SPA (Angular, React, Vue) on top of micro-frameworks like Flask (Python), Express (Node), GoLang, others. Combined with this are ML and other backend infra that requires managing clusters that scale better on Kubernetes, where Chef/Puppet have little presence on either K8S or backend distributed clusters.
https://www.cvedetails.com/vulnerability-list/vendor_id-1294...
When doing immutable infra, where managing desired state is only at deploy time, then Ansible is by far more popular. Beyond that you get into container scheduling-orchestration platforms.
Others mentioned Terraform and Pulumi, which manage cloud resources, where Chef manages mostly system resources. Once upon a time, there was an attempt with Chef Metal, and later rebranded as Chef Provisioning, which are drive by chef-solo (or chef-zero) and use fog driver. The end result was something that was so gawd awful terribad, unstable, and unpopular. Instead of Chef investing to improve it, Chef started promoting Hashicorp Terraform over their solution.
CINC is the recent-ish community build of the code. It's literally a drop-in replacement as the only changes to Chef Client and other bits are branding. It's the CentOS to Chef's RHEL.
My biggest complaint is how unresponsive they are to fixing bugs. After reported it may be months or a year before you see it fixed.
What I dont know is if that is common in that "ui component" sector?
I think this is why almost nobody uses closed-source libraries. Having to pay isn't the biggest problem: it's not being able to fix issues like these yourself.
Basically, they're a business apps technology provider, and have been around since 1981, providing app development tools for enterprise scale applications. Progress 4GL was a database-backed application environment, quite widely used (albeit quietly) by big businesses all over the globe to build apps, and has transformed into OpenEdge.
As the Web evolved, Progress seem to have embraced all of it, and - as you say - somehow seem to be using smoke and mirrors to describe what they do - but actually their tooling is pretty awesome. Write an app in OpenEdge, deploy it on any web server. They were doing big relational database stuff on cheaper hardware (PC's, minicomputers) in an era when "big database" meant investing heavily in "IBM" or "Relational Software" (now: Oracle) resources.
Its pretty neat to see them still around. I had a lot of fun with Progress 4GL back in the day, but haven't touched it since mysql/postgresdb was released ..
Things "worked", but everything was OLD. CLI tools had bugs and warts that GNU figured out decades previously.
I remember having to use obnoxious workarounds when tooling broke because 'find' wouldn't even work properly once you exceeded a certain number of files. (Several thousand, IIRC)
And that curses-based wrapper for doing administration tasks... glad I forgot the name of it, to be honest.
[[muffled screaming]]
The culture was a lot more active in the 90's. I mean if you were trying to port something to AIX and having trouble with gcc, you could go on comp.unix.aix and IBM employees would actually respond and be helpful.
We lost a lot when that era died.
huh. That's not been my experience. To me, it looks like they stopped improving things circa 2005. Things that are new tend to be half-baked (.NET integration, OOP language features). No command line compiler (although you can write one yourself and a java-based community project exists), no linting or other static analysis tools. Their IDE is eclipse-based and frequently crashes when working with GUI for .NET. The autocomplete and in-IDE documentation that it provides is terrible- missing methods, not context aware, etc. They have a docker image, but only for their app-server product. I'm currently building a REST API using their app server product and I keep running into stumbling blocks. I've built similar APIs in PHP and Python and those were (largely) easy- I could focus on the problem and not the tooling. Here, I keep stumbling over things that work differently than anywhere else or that aren't well documented at all.
Their earlier stuff is quite impressive when you judge it by the time. They just seem to have fossilized.
They have a lot of other products now as well.
Their website is marketing puff but the products are great.
4GL (aka ABL or Advanced Business Language) is the language that works best with OpenEdge. You write 4GL code, compile and deploy it to a specialized Tomcat environment called PASOE (Progress Application Server for Open Edge). It's their container to run 4GL apps.
WebSpeed is their middleware to serve APIs using 4GL code. Companies stuck in the desktop world are using WebSpeed to refactor their code and expose business logic to other languages through WebSpeed API middleware.
They acquired Telerik some time back. Those controls are used by .NET web devs and desktop devs. They are cool controls. I like them. Telerik controls are also integrated with their 4GL desktop environment. Kendo UI are tools/controls for the web and are pretty neat as well. Fiddler is also their product now.
DataDirect is a suite of connectors to make it easy for devs to connect between different DBs from 4GL, and for other languages to use DataDirect to connect to OpenEdge or call 4GL methods.
Corticon is their version of Dell Boomi or Alooma or Prismatic.io. It's a tool to run business processes with no code or low code. It's neat if you have tech savvy business analysts who like to use tools like that.
Sitefinity is their version of WordPress/Drupal/content frameworks. Sitefinity has its place.
NativeClient is their product to build mobile-first application across various platforms. They whipped up a neat app at their conference and it really worked well. Yes, it may have been cowboy-coded a bit, but it worked as intended.
Now they acquired Chef. I am sure there was a demand for orchestration within their client-base. They are making improvements steadily. They have a strategy. They have a niche target market, primarily in LATAM and EU and less so in North America. They do not have a great version control, static analysis, dynamic analysis, orchestration tools. Companies like Intuit and Fiserv seems to have built their own ecosystem around 4GL tools, but many companies just have primitive means of development with 4GL, still. Progress is trying to stay current. Their customers are much slower at adopting to changes in engineering; perhaps what they have just works for them. Not sure.
Acquiring chef is a good move. I've worked with their products for 2-3 years. It wasn't fun. I don't like 4GL. I'd like if Progress made SQL a first class citizen and invite PHP/Python/Ruby/etc. communities to write code against OpenEdge. From a business standpoint, I feel they are trying their best to stay modern and relevant. They are acquiring products that are indeed making their ecosystem cohesive.