Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

While there are use cases where lightweight threads are a benefit, I think a web crawler is pretty much the ideal case for async/await, given you are almost exclusively waiting. Although I must admit I don't really understand this example, why is the fetching being parallellized, like you typically want to add sleeps, even with a single thread you've got yourself a gatling gun that can DoS a website if you aren't careful.

There's also some optimizations that could be made. Instead of having

    synchronized (seen) {
        if (!seen.contains(u)) {
            seen.add(u);
            // ...
        }
    }

you can get rid of the synchronization with

    // seen = ConcurrentHashMap.newKeySet()
    
    if (seen.add(u)) {
        // ...
    }

View on HN

I was surprised by the lack of ConcurrentHashMap. Does Go not have an equivalent?

Go does have a concurrent map in the standard library, but the documentation recommends not using it unless you have specific access patterns that it's optimized for:

https://pkg.go.dev/sync#Map

You lose type safety when using it. I'm guessing with generics in 1.18 now, we'll see an updated generic version of sync.Map whenever a bunch of new collection types are added to the standard library that take advantage of generics.

Yeah, I'm looking forward to Go's stdlib evolving useful collections now that generics exist.

Like maybe a set. That isn't a map[T]interface{} and can do things like unions and intersections so I don't have to implement it with for loops.

> That isn't a map[T]interface{}

FWIW while map[T]interface{} is somewhat more convenient to use, for efficiency reasons you should prefer map[T]struct{}: interface{} being a fat pointer is takes 16 bytes * the capacity of the map/set. struct{} is a zst, so takes 0.

It also signals much more clearly that it's a map to nothing, which is a set, a map[T]interface{} could be an actual map to a bunch of random things.

While you're not wrong, and that is how you implement a set in Go, it's still a workaround over a standard library Set type unfortunately.

But, with Go's generics, I hope they will add new collection types to the standard library.

It's actually much more common to use map[T]bool, because it allows easy membership testing just with

if m[x] { ... }

Not that I have seen. map[T]struct{} is more common, with if _,ok := m[x]; ok {} used for testing if a key is in the map. The space savings really add up with large sets.

At least at Google, map[T]bool is more common, by an order of magnitude. Probably because map[T]bool is recommended by our official style guide, which tends to favor readability over a negligible performance difference.

That is largely a distinction without a difference, the main gist of my comment was that using an interface{} value, while convenient in some ways, is way costlier than necessary (at least in memory) without any real advantage.

This is exactly why a generic Set collection should have been there since the start.

I don't disagree at all, I'm just pointing out a less than ideal pattern.

And I much appreciate your feedback :) Whenever I try to think of what was the best practice idiom, I always mix up map[T]struct{} and map[T]interface{}, due to the curly brackets at the end.

As you can tell, it's been a while since I've written Go.

Hah, 50/50 I'd get that wrong, cheers.

In Go, while it is possible to use a concurrent hash map, it's generally not recommended. In most cases, you should use a channel.

In one of my first Go programs, I used mutexes to avoid concurrent writes in a critical part. Then I moved the writing process into a goroutine that receives its input through a channel of size 1. In my case, the later code was easier to maintain and more performant than the former, mutex-based, code.

Yeah - ConcurrentHashMap makes the most sense. I left it off just because I thought it would be easier to see the parallel with the Go code.

Reply on news.ycombinator.com