Comment by leg100 | Hacker News Reader

Comment by leg100 | Hacker News Reader

Parent Full thread

According to [1], "Native providers are built directly and automatically from the cloud provider’s API...", and are not hand-coded, unlike Terraform's providers.

I'm surprised this is possible. If it was, why didn't Terraform follow this approach a long time ago?

The cloud providers' APIs provide the endpoints for creating, updating, reading and deleting resources. Terraform and Pulumi's value is to provide an idempotent abstraction on top of that. And that abstraction is not straightforward to write, because it has to handle numerous nuances and anachronisms in the underlying APIs. For example, in the event of updating an existing storage bucket, the abstraction has to determine whether the bucket can be simply updated, or if it needs to be re-created (say if you were changing the name or location). And the underlying API will not necessarily reveal this kind of information.

Hence one would think completely avoiding handwritten code is incredibly difficult if not an insurmountable problem.

[1]: https://www.pulumi.com/blog/pulumiup-native-providers/

[caveat, Pulumi co-founder here]

You are right that it's not easy. Thankfully the cloud providers themselves have moved in the direction of auto-generation for their own SDKs, documentation, etc., which has forced this to get better over time. This is motivated by much the same reason we've done it -- keeping one's own sanity, ensuring high quality and timeliness of updates, and ensuring good coverage and consistency, when supporting so many cloud resource types across many language SDKs.

Microsoft, for instance, created OpenAPI specifications very early on, and standardized on them (AFAIK, they are required for any new service). Those API specifications contain annotations to describe what properties are immutable (as you say, the need to "re-create" the resource rather than update it in place). Google Cloud's API specifications similarly contain such information but it's based on the presence of PATCH support for resources and their properties. Etc.

The good news is that we've partnered with the cloud providers themselves to build these and we expect this to be increasingly the direction things go.

This is good news. This will reduce the barrier to entry to all sorts of software leveraging Cloud APIs, not only Pulumi. (I believe crossplane.io might be following a similar approach).

Much sweat and tears has gone into hand writing Terraform provider code. The vast majority of which has come from and continues to be maintained by volunteers.

To have had to replicate this manual effort all over again just to create a competitor would have been silly.

There will surely be a lot of wrinkles to iron out with this 100% automated approach. But indisputably this is a positive development.

The Terraform provider for Google Cloud uses partial autogeneration, here is the repo that does the autogeneration for multiple automation tools:

https://github.com/GoogleCloudPlatform/magic-modules

Magic modules is a strange beast. It relies on manually writing YAML, containing serialized Ruby objects. You then invoke rake tasks to generate Go code.

And even then, magic modules only covers some of the resources in the GCP provider. The remainder are completely written by hand. Presumably because the Ruby objects are not sufficiently expressive to cover all the edge cases.

This is an excellent comment. I think the answer is just that the cloud provider API's are getting more "stable" and user friendly with time, making the additional abstractions of e.g. terraform not worth the overhead of learning that entire world. The response from the Pulumi co-founder seems to indicate as such, that they're relying on the simplicity of the API's to not get into a bad state (although I'm still skeptical if that's possible).

Wouldn't this mean that Pulumi has better coverage than CloudFormation?

Some things in the AWS API aren't (yet) available in CFN.

It would mean Pulumi has better coverage than CFN and terraform. Indeed as their blurb states, Pulumi would have 100% coverage.

But I'm skeptical.

afaict the aws terraform module tends to have 100% coverage. i've never found a resource that didn't work, or didn't already have some way to use. i was able to deploy eks when it was still in beta through terraform, while i think the cloudformation stuff for eks lagged behind like almost by a year.

The terraform AWS provider lags from time to time too. I remember waiting the best part of a year for API gateway websockets support.

https://github.com/hashicorp/terraform-provider-aws/issues/7...

Reply on news.ycombinator.com