Hacker News Reader

Comment by jrudolph | Hacker News Reader

Getting a lot of docker hub vibes from this one. HashiCorp is course within their rights. Can't be cheap to run the registry given the obscene size of some terraform providers.

$ ls -lah terraform/providers/registry.terraform.io/hashicorp/aws/5.14.0/darwin_amd64/ total 368M

Anyone have an idea of the reasons terraform needs a 370 MiB binary just to call REST APIs?

View on HN

> Anyone have an idea of the reasons terraform needs a 370 MiB binary just to call REST APIs?

That's because Terraform fell for the Go trap. When space and bandwidth are cheap, why not go for an environment that only ships fully self contained binaries? Oh, and why not go for a language that attracts hipsters like fruits attract flies, but is a nightmare to develop in?

Bloody ridiculous, it's a miracle Terraform got as far as it did.

(Yes, I'm working with Terraform every day and it's pretty decent but I'd love to extend it for Atlassian Cloud stuff without having to add a sixth language to my already sized toolbelt. Why Atlassian doesn't offer Terraform integration on their own is beyond me in any case)

> Can't be cheap to run the registry given the obscene size of some terraform providers.

Some providers are also hosted externally. I guess if traffic is going to be a problem they might also just switch to hosting every provider that is build on GitHub to GitHub releases (and hope that GitHub won't change its policy)

  curl -s https://registry.terraform.io/v1/providers/carlpett/sops/0.7.2/download/linux/amd64 | jq .download_url
  "https://github.com/carlpett/terraform-provider-sops/releases/download/v0.7.2/terraform-provider-sops_0.7.2_linux_amd64.zip"

  curl -s https://registry.terraform.io/v1/providers/hashicorp/helm/2.10.1/download/linux/amd64 | jq .download_url
  "https://releases.hashicorp.com/terraform-provider-helm/2.10.1/terraform-provider-helm_2.10.1_linux_amd64.zip"

https://github.com/hashicorp/terraform-provider-aws/issues/3...

The size is what you get when you add every single AWS Go client into one binary.

Each service client like 1-2MB. But when you have 200 services....

I think the providers might actually come from GitHub because the registry insists you have them as release assets

That indeed seems to be the case. Seems that for community providers, HashiCorp is serving up JSON which refers to GitHub download links.

Here’s a sample size of 1: https://registry.terraform.io/v1/providers/spacelift-io/spac...

According to the provider registry protocol, which I have previously implemented for internal hosting (in an afternoon of writing a single file of Python): https://developer.hashicorp.com/terraform/internals/provider...

For their own managed providers they no longer provide binaries in GitHub releases, and serve them from their own servers instead. Which feels like a trap BTW.

I didn't realize that, and extra weird they publish the SHA256SUM file as a release artifact but it references 14 zip files and the manifest.json so, ... thanks?

But, in their defense, installing an "unofficial" provider (or build!) into TF is some JFC so there's that. We'll just add that onto the damn near infinite pile of "I hope OpenTF fixes ..." things

The AWS Go SDK is the vast majority of that bulk. In general go binaries can get pretty big but AWS has hundreds of services with thousands of APIs and it’s all going to have to get included in the AWS provider.

The AWS Go SDK is actually a case study on making everything into pointers

It’s really a case in the abject inadequacy of the Go type system.

Also AWS has so many services that the SDKs are mainly generated from json descriptions + nice wrappers on top. That leads to a different and less abstracted type of code than you'd write yourself - which leads to bigger compiled objects.

Ruby had this problem too and at some point split the SDK into multiple gems so you don't have to install everything.

The Azure sdks are the same. Auto generated from some underlying description. Then for backwards compatibility, every previous version is its own complete copy, all included in one single bundle.

Reply on news.ycombinator.com