Hacker News Reader

Comment by dingdingdang | Hacker News Reader

These two points alone are honestly just bad news for Golang on the whole

>> pkg.go.dev fetches the list of modules from proxy.golang.org: a closed-source proxy through which all of your go module fetches are being routed and tracked

>> pkg.go.dev hard-codes a list of centralized git hosting services [that are the only allowed ones]

View on HN

>> pkg.go.dev hard-codes a list of centralized git hosting services [that are the only allowed ones]

So how in the world did git.lukeshu.com get to be on that list?

https://pkg.go.dev/git.lukeshu.com/go/libfastimport

Likely cause Luke is providing some valuable code that other golang users depend on?

Also: hi Luke! And thank you!

Note that some features are absent here. You can't click on any of the declarations to view their source, for example. This is the degraded view for unknown hosts.

Some features being absent is not the same as what you said in the article. "Any GitLab instance other than gitlab.com [..] none of these are going to work unless every host is added and the list is kept up-to-date manually." makes it sound like they don't work at all, but in fact it's just a few minor features that don't work. sr.ht has always worked on pkg.go.dev from day 1.

Actually, pkg.go.dev needs to have some way of figuring out where to link to, so it's not necessarily easy to automatically make links that work for 100% of the sites. That's not "a blatant failure to comprehend the fundamentally decentralized nature of git hosting", that's having to deal with a large ecosystem spread out over thousands of domains. Go's model is a lot more decentralized than most other package systems (which also makes stuff like this harder).

They do have a way, and godoc.org uses it: go-source meta tags.

https://news.ycombinator.com/item?id=24024130

That doesn't address the main meat of the comment, which said that your claims in the article are highly misleading.

I don't think that my claims are misleading.

You suggested many code hosting sites don't work, while in fact they do work except for some (IMHO minor, but we can perhaps disagree on that) features. You don't think that's not a full and accurate representation of the state of things?

>they do work except

Except being the key word here. My commentary is on the quality of their engineering approach and I have demonstrated that their approach is flawed. Their approach is wrong.

None of that is relevant; you can feel the Google engineering approach is the worst in the history of humanity – all fine with me – but ..... that does not mean you get be misleading about facts. You can – and should – state that one feature doesn't work; rather than say that none of it works. Because the former is true, and the latter is not.

What is the goal of your post? To inform your readership, or to convince them that Google is bad? If it's the former, then you have done your readership a great disservice. If it's the latter – which seems to be the case – then congratulations: you've fooled your readship – or at least part of it – in to believing something that's not true.

My goal - my stated goal, that is - is to point out that the engineering approach is flawed. I don't have to explain the degree to which the feature is wrong in order to utilize the fact that it is wrong as a mark against the engineering approach. The approach is wrong, they were told, they didn't change it. End of story.

And you've been misleading yourself: it's not a "single feature" - several features depend on this code. It is a second-class experience, and in ways that build upon other problems I've put into the article.

Cut the bullshit, dude.

> I don't have to explain the degree to which the feature is wrong in order to utilize the fact that it is wrong

Ehm, there is a massive difference between something not working at all and one feature not working. My car's headlights not working is not the same as my car not working. My laptop's trackpad not working is not the same as my laptop not working.

> And you've been misleading yourself: it's not a "single feature" - several features depend on this code.

I use pkg.go.dev almost daily (including for sr.ht repos on two occasions) and have never noticed this. It's fine to point out that there's a degraded feature-set and be angry about that (some people probably care about these features more than I do), but claiming that the entire thing isn't working is – and I feel like a broken record with this – just not true. It's just not. No matter what you say.

This is easily fixed by just changing a few words. I don't get how anyone can defend this. If someone would point out such a thing on my own website I would drop everything and promptly fix it, no matter how strongly I believed in the main point I was making (and I have actually done so) as I want to be as accurate as I can reasonably be.

You know, you actually inspired me to work on open source full-time last year? True story, that's what I do most days now, for almost a year to the day. This is what I wrote a year ago:

> Drew DeVault’s sourcehut in particular was inspirational in actually making the step. If Drew can do it, then why not me? [..] I know it’s a risk and that there’s a decent chance this will fail. That’s okay. I’d rather take a chance to try to achieve a goal and fail instead of always playing it on safe. Not doing anything is also a decision – which you can regret just as much as any other – and the worst thing that can happen is that I lose some money and end up taking a job again. Doesn’t strike me as so bad, in the grand scheme of things.

But with this like this – and this is not the first time I've seen you do it – you've lost a tremendous amount of respect. I don't care that you write rants (okay, it's not my preferred style, but whatever) but when you're being misleading and then defend that on account of "but my point was correct" then ... yeah nah... It's hard to be inspired by someone like that.

I don't expect that you'll care a great deal, but just so you know regardless. Do with it what you will.

If you're at the point where you're writing things like "cut the bullshit, dude", you're past the point where you need to walk away and take a breath.

Your post states that their approach is flawed in large part because it only works for sites which have been explicitly declared in the service’s config. Given how key that is to the case that the engineering approach is flawed, it seems very relevant that the majority of the service does work for any site, and there are specific features which require the site being included in the service config.

There’s a big difference between requiring explicit site inclusion for full functionality and requiring inclusion for any functionality, and your post claims the latter.

How do you propose this problem be solved? I was partially responsible for the go-source meta tag thing in the first place. Is there some other mechanism to find a web-based view of the source of a git repo?

Well, it's complicated, and we could brainstorm a few ideas. The problem seems to be that this brainstorming session never happened.

One problem is that the import path doesn't really give you enough information to totally figure out how to fetch the package. It's treated like a URL, and then the go-source tag comes in to relate it to something VCS-controlled. If we encoded the VCS into the import, we could e.g. have "git+https://git.sr.ht/~sircmpwn/getopt" as the import path, or we could move the import details into a separate file (like go.mod).

But if we assume that the import path and go.mod formats are fixed, then we still have more things we could do. For example, why this:

    <meta name="go-import" content="git.sr.ht/~sircmpwn/getopt git https://git.sr.ht/~sircmpwn/getopt" />

When we could have this:

    <link rel="alternate" type="application/x-git-http" "https://git.sr.ht/~sircmpwn/getopt" />
    <link rel="alternate" type="application/x-git-ssh" "git@git.sr.ht:~sircmpwn/getopt" />

And why this:

    <meta name="go-source" content="git.sr.ht/~sircmpwn/getopt https://git.sr.ht/~sircmpwn/getopt
           https://git.sr.ht/~sircmpwn/getopt/tree/master{/dir}
           https://git.sr.ht/~sircmpwn/getopt/tree/master{/dir}/{file}#L{line}">

When the same tag could be "source-browser" or something similar? I'm sure many projects other than Go would be very happy to have features like this standardized and available for all of them to use, but the Go team sees no further than its own nose when designing this sort of thing.

And furthermore, with the specific case of hardcoded software hosts on pkg.go.dev, why isn't it using the go-source meta tags to look up how to create links to files & line numbers? You guys forced this upon us and then don't even use it? There's no reason to hard-code regexes for various git domains when you could just fetch it like godoc.org does.

I agree that it would have been better to design these a little differently.

In the first case the go-import meta tag is totally unnecessary. The go tool and module proxy can discover git repos from an import path just fine. It’s only required for “custom” import paths where the path is some domain but the code lives somewhere else, like github.

In the latter case the decision to use go-source was actually made by the original author of godoc.org, not a google employee, and was done in coordination with another non-google initiative (gopkg.in) to make their source links work. So nothing to do with the go team really, sorry.

https://github.com/golang/gddo/commit/864b1c0aba009e37d30136...

If pkg.go.dev isn’t respecting go-source meta tags then that should probably be fixed. It would also imo be worth considering devising a more general, well-known mechanism for doing this. Worth proposing I think!

>In the first case the go-import meta tag is totally unnecessary. The go tool and module proxy can discover git repos from an import path just fine. It’s only required for “custom” import paths where the path is some domain but the code lives somewhere else, like github.

The relevant docs are here:

https://golang.org/cmd/go/#hdr-Remote_import_paths

You can explicitly put ".git" into your import path, but no one does this and it's not explained to new users. In order to have predictable import paths like people have been trained to use, you need go-import tags.

>In the latter case the decision to use go-source was actually made by the original author of godoc.org, not a google employee, and was done in coordination with another non-google initiative (gopkg.in) to make their source links work. So nothing to do with the go team really, sorry.

And yet, godoc.org is what you're replacing. Not taking into consideration is how you end up with what you've got: regression. And this only further betrays Google's warped worldview of "us and only us": this person has made an amazing contribution to Go and yet you consider them an other and don't take their design into account.

>If pkg.go.dev isn’t respecting go-source meta tags then that should probably be fixed. It would also imo be worth considering devising a more general, well-known mechanism for doing this. Worth proposing I think!

No, it's not worth proposing: it's worth doing, and should have been done in the first place. It should not take an outsider - it should happen naturally from a good engineering ethos. Playing well with others is your burden, not everyone else's.

> And this only further betrays Google's warped worldview of "us and only us": this person has made an amazing contribution to Go and yet you consider them an other and don't take their design into account.

I don’t even know where to begin with addressing this criticism. To say that you have a warped view of the situation is putting it mildly. I’m just going to back out of this conversation while the going is good.

Wait, using the same tag as an existing project to maintain compatibility is proof that Google doesn't play nice and didn't take them into account?

What? godoc.org made up a new tag, and pkg.go.dev doesn't use it, breaking compatibility. They failed twice.

I've had a hard time following who's doing what.

That does not work in the old godoc.org either:

https://godoc.org/git.lukeshu.com/go/libfastimport

So that's not against pkg.dev. Did somebody look into why?

Anyway it seems that the repo was not added to pkg.dev automatically:

https://github.com/golang/go/issues/38326

and the automatic fetching should work now:

https://github.com/golang/go/issues/37002

> Anyway it seems that the repo was not added to pkg.dev automatically: > > https://github.com/golang/go/issues/38326

The way this works (or rather, worked) is that you first need to "go get" a module and then it gets picked up by the pkg.go.dev system. This works like that for every site, and was kind weird and confusing, although as you mentioned this was improved on recently. Either way: not related to that module being on sr.ht; that's just coincidental.

See my comment here which explains how "go-source" works:

https://news.ycombinator.com/item?id=24024130

It doesn't work on godoc.org not because they don't recognize your domain, but because cgit doesn't have go-source meta tags.

There was a proposal to add a new go-source-2 meta tag, but that was abandoned because a lot of code hosting sites would not have been supported yet:

https://github.com/golang/go/issues/39559

https://github.com/golang/go/issues/40477

That is why they decided to turn it into an internal/source package inside pkgsite first to have test cases for a later go-source-2. If your site does not show source links on pkg.go.dev then add a comment to issue 40477 above and an exception will be added.

- internal/source: https://github.com/golang/pkgsite/blob/master/internal/sourc...

Note1: Maybe mention this on your website because that took a little to find. It does not fit into the narrative though.

Note2: Requiring frontends to offer meta tags is the only way to make code discoverable with the url alone. cgit not offering this is on cgit and not on Go/pkgsite (haven't checked whether they do)

export GOPRIVATE="domain1.com,domain2.com"

[1] https://golang.org/cmd/go/#hdr-Module_configuration_for_non_...

Python PyPi downloads are also tracked https://pypistats.org/. The data is public, nothing wrong with that IMO.

If I host a package outside of the PyPi eco-system though as a wheel, pip while happily install it if I provide a full path to it, and at that point it's not part of pypistats.org

I think all the PyPi stats are available in BigQuery, if anyone’s interested in doing some analysis.

Edit: I had the wrong table. It does have the data

https://console.cloud.google.com/bigquery?project=the-psf&p=...

pkg.go.dev supports fossil-scm (and I believe any "go gettable" source repo). So, the idea that it "hard-codes a list of centralized git ..." needs a citation.

Whereas AFAIK godoc.org only supported git. IMHO support for a variety of SCMs makes pkg.go.dev a huge improvement.

Support for "go get" and pkg.go.dev via fossil-scm does require some golang-specific meta tags, as TFA exclaims. But why is that a big deal, it's a golang repository after all?

Reply on news.ycombinator.com