zeeboo

There's no shadowing in the latter case. The second case is the same thing as

    a := 2
    a = 3

By definition you must have nested scopes to have shadowing. Within the same scope, it's only ever assignment.

This example is just to show that pauses are not in the same class as any other fully managed languages. I haven’t looked at the code to know how much it stresses the garbage collector, but I agree it does seem unlikely to do so.

That said, being able to work on the stack does influence the design of the garbage collector. Throughput becomes less important because you can choose to allocate less, and pause time is more important because latency is harder to optimize. Latency is affected by your whole stack and can’t be meaningfully reduced by just adding more machines in the same way that throughput can be increased.

No cited references, no answers about what “most” is, no answers about what “most” is in the specific context of Go, no answers about any “unnatural” patterns existing, no answers about the claim that it was “tuned” to reduce allocations, answering a question that wasn’t asked dodging the question that was (generational having better throughput vs when does throughput matter).

There’s no way I can believe you are acting in good faith. It’s obvious to me now that you’ve just been trolling for years every time you discuss the topic of garbage collectors. I’m no longer going to engage.

No, I meant what I wrote. The time spent in total in GC includes the time spent performing allocations. If you reduce the time spent in allocations with all else being equal, you reduce both latency and increase throughput. In that way, optimizing for your allocation latency also increases your total throughput.

And yet here’s an empirical benchmark providing evidence against both of those claims. What empirical evidence do you have supporting yours? Can you provide them?

Who defines when a construct is “unnatural”? Can you show examples of “unnatural” patterns in the provided benchmark programs supporting your hypothesis that you have to write unnaturally to control allocations? If not, why are you making those claims?

Why is throughput of allocations important for “most” applications? How much time is typically spent on allocations in those programs? What is a typical application? What about typical applications in Go specifically since that’s the only language that Go’s garbage collector matters for?

Also, why did you bring up the point about generational collectors? It seems like a red herring. This discussion has been about tuning for latency, and how in this benchmark, Go’s implementation did better on both throughput and latency than any OpenJDK collector under any tuning, and that tuning for latency is possibly a sound engineering decision.

I’ve also noticed that you’ve made no explicit attempt to answer my more difficult questions, and if you continue to do so I will no longer assume you are acting in good faith.

Sure, but that's not relevant to the discussion here. In this case the context was responding to the the statements that Go pundits claim the garbage collector has low latency but it always has bugs that will be fixed in the next version. I was providing evidence that the garbage collector actually does achieve low latency right now.

Also, there are ways to solve your throughput problems if you have control over your allocations, but there are not ways to solve your latency problems. Indeed, even if you don't have control of your allocations, you can often run multiple copies of your program to increase your throughput, but multiple copies will not help your latency.

Also, tuning your latency does in fact help increase throughput, because if your process spends a significant amount of time in allocations, its latency suffers. It's not a zero-sum knob between the two, especially in the presence of humans that care about tuning the performance of their applications.

Thanks.

Since most of the explanations that you gave involve things like controlling value types and data layout, it sounds to me like you might agree with the statement that in languages with better control over allocation and data layout, tuning a garbage collector for latency over throughput can be a good idea because your time spent allocating is less important. Is that fair?

I believe we misunderstood each other. In particular "the application in question has been tuned to perform as few allocations as possible" seems contradictory to "Go doesn't give you control over allocation in a meaningful way."

Allocation being fast isn't very important if you don't spend a significant portion of your time allocating. Isn't it possible that some languages (maybe even Go) by the nature of their semantics spend significantly less time allocating than other languages, and so tuning a garbage collector for latency is the appropriate decision?

Why do you seem to believe that the Go application has been tuned more to avoid allocations than the Java application? If it hadn't been, then you would have to agree that tuning the garbage collector for latency is better because you can invest some effort into your application to have it perform better on both throughput and latency.

Or is the argument that because "[throughput] is typically what applications want", this application is not a good example of most applications (according to what measure of "most")?

You're jumping to conclusions. I understand the distinction and that there are many options for Java. Please be charitable.

To rephrase the question more precisely: can you explain why OpenJDK, which has a multitude of garbage collector implementations tuned for both throughput and latency, performs worse on both throughput and latency than Go on this benchmark which has a garbage collector that, according to the original statement, is "tuned for latency above all else"?

Perhaps OpenJDK is problematic in other ways (the author of the benchmarks suspects it's JIT induced, even though that's attempted to be controlled for), or perhaps this test depends less on the GC for some reason. Or maybe typical Go programs don't require as many allocations making allocation time much less important making tuning for latency the correct engineering decision. Would HotSpot do significantly better as is claimed? That's the sort of interesting technical discussion I'd like to have, instead of pedantry.

How do you explain the throughput and latency numbers posted by https://github.com/ixy-languages/ixy-languages? Go is beating Java in both latency and throughput, and the author attempted many different Java GC tunings (https://github.com/ixy-languages/ixy-languages/blob/master/J...)

Maybe tuning for latency is the appropriate trade-off given Go's high level of control over allocation and data layout?

The memory leak due to channels not being cleaned up was almost certainly because goroutines themselves are not garbage collected when they're forever blocked. In other words, `ch := make(chan int); go func() { ch <- 1 }()` is a memory leak. This is analogous to spawning a thread that deadlocks by attempting to lock a mutex twice, and has nothing to do with garbage collection.

About the GC pauses, check out the latency graphs at https://github.com/ixy-languages/ixy-languages (being careful not to mix up the Javascript and Go lines). Go handily beat every other garbage collected language in latency, and is in the same ballpark as the two non-GC languages (Rust and C). The peak tail latency times are measured in the hundreds of microseconds even at the highest loads tested, so I think the pundits may have a point.

On the repo page, there’s a latency graph, and on it there are two green lines. One is for Javascript and has poor latencies. The other is Go and has some of the best latencies. Because the Go one is clustered with other overlapping lines, I think people often misinterpret the Javascript line as the Go line.

https://github.com/ixy-languages/ixy-languages

Maybe you should trust the judgement of the highly-qualified engineers then when they decided to go with their own toolchain, and not just assume that they’re doing it because plan 9 or some other vacuous reason.

It’s only slower at the highest batch sizes. I’d say their performance on throughput here is comparable, except the Go version has much better latencies (don’t be confused by the first graph like I was: that green line at the top is actually javascript).

Publish isn't necessary for the Go version, since the model is on-demand pull driven. godoc.org is the same way: there's no way for private code to end up there unless your private code is actually public.

The privacy concern is entirely that Google may see the name of some module you attempted to download.

> You talk about git.apache.org being down, while you're willing to bring down the entire ecosystem across the board with your own special server.

If the proxy goes down, by default the command falls back to directly fetching.

Those disallow data races, not race conditions. I find that this blog post explains the distinction well: https://blog.regehr.org/archives/490

You can’t naked return with a shadowed named return variable. The compiler disallows it. Thus, at any return site, you locally know that you’re either returning the only err in scope (the named return), or the specific value listed in the return will be assigned to the named return variable.

I don’t know what error types has to do with it. If you need to annotate differently for every exit point, sure, a defer doesn’t work, but neither would any other proposals I’ve seen. In those cases, do the more verbose thing because the verbosity is apparently warranted. In my experience, it is rarely warranted, and a stack trace with annotation about the package/general operation is sufficient.

The stack traces contained still include what line the return happened on when queried from inside the defer. They retain the information about which return executed. My, or any, helper could, if desired, explicitly skip the defer stack frame, and it would be indistinguishable from capturing at the return itself.

I totally agree that good error wrapping is essential. In fact, I've been using this exact system for wrapping my errors for years, and it's been great (https://godoc.org/github.com/zeebo/errs#WrapP).

I suspect it's only "somewhat weird" due to lack of familiarity, and that adding an additional mechanism to wrap errors when one already exists is not in the spirit of a language built from small orthogonal components.

That `Close` call only happens in the case that an error happened during either copying or closing. The point of checking errors on `Close` is so that you catch problems during flushing of any buffered writes. In this case, the `Close` is not to ensure the data is flushed, but to ensure that the file descriptor is released in error scenarios, justifying that the error need not be checked.

On the other hand,

  w := try(s.Create(dst))
  try(io.Copy(w, r))
  try(w.Close())
  return nil

this code has the problem where if there were an error in the `Copy`, the file descriptor would be leaked.

So, yes, one of the examples is wrong.

Indeed, I believe it's impossible to statically prevent race conditions in a turing complete language. Data races can be prevented, but that's just a subset of all possible race conditions.

To be clear, you can name the return value whatever you want. People just typically name their error values `err`. Also, the mechanisms to mess with named return values in defer already exist: https://play.golang.org/p/7nFyiuAa3Ra

An implementation with buy-in across the entire ecosystem and language so that you don’t have some systems using threads and other systems using futures and other systems using different reactors, etc.

Also known as the point of the article.

Additionally, that implication is entirely of your own creation. The article explicitly lists many languages besides Go:

> Three more languages that don’t have this problem: Go, Lua, and Ruby.

Perhaps you just have an anti-Go bias?

They're equivalent representations of the same value. There's no need to propagate anything to anyone.

I'll believe you once you tell me how the openness of a core internet router is important to the security of visiting a website over https. Good job keeping up the FUD!

Assuming the server will eventually get hacked is how you design secure systems. You assume the worst, and ensure you are still secure. The server code being designed well had nothing to do with the hardware running the system being hacked. For instance, no amount of good design in my server stops a remote 0-day against linux, for example.

This is a subtle point, but that thinking is misguided. You, as a client, have no control over what server you’re actually talking to. The only way to be sure you are secure is to be secure independently of if the server has been hacked, or is malicious, or whatever. Thus, you design your system such that the client only discloses information that is allowed to be public to the server (public keys, encrypted messages where the server can’t decrypt it, etc). In that way, you don’t care what code the server is running, and auditing the server makes no difference to security.

The whole purpose of cryptography is to reduce the set of things you need to trust. Including the server in the trusted base is not only a worse design but a false sense of security if your trust ends up misplaced (hacked).

So, this is different than how a bank would work, for example. In the case of a bank, you have to trust that their servers are secure, and their software being open would help with auditing that. In this case, keybase is not the endpoint you’re talking to: another keybase client is. The keybase server is just an intermediary, just like any router on the internet.

Enclaves are interesting, and I also look forward to all of the new things they allow. But all of that has nothing to do with open sourcing the server being important for security if given the ability to audit the client, and the client is not designed to require a cooperating server.

I'm tired of trying to get you to understand this point and have you respond with red-herrings and FUD. Please be intellectually honest when asking keybase to open source their server in the future, and don't claim that it's relevant to the security of the system.

I didn’t say you believe in the security of every application, I said you believe in the security of some applications. For example, websites secured by https do not require the security of routers to be secure.

There is no need to run the code that you expect if the client is designed properly, and you are only sure you're running the code you expect until your server is hacked, which also becomes a non-issue if the client is designed properly. There are other reasons to want to self host: to be in control of your (encrypted) data so that it isn't lost if keybase goes under, for example. But security is not one of the reasons.

Why would I not be able to make inferences about the software the servers are running if the chances on lying on the letter is low? I haven't read Signal's source code, and yet I believe with just as much confidence that they aren't logging extra information as if keybase had sent the same NSA letter. To me, Signal's source code is effectively closed, and reading it wouldn't increase my belief. (Have you read all of their server's source code? If not, why do you justify your belief?)

The article on attested contact discovery states "Of course, what if that’s not the source code that’s actually running? After all, we could surreptitiously modify the service to log users’ contact discovery requests. Even if we have no motive to do that, someone who hacks the Signal service could potentially modify the code so that it logs user contact discovery requests, or (although unlikely given present law) some government agency could show up and require us to change the service so that it logs contact discovery requests.", which is exactly the point I'm making. They choose to solve it by signing code and ensuring that exactly that code is running (seems like they just move the trust to Intel. Hopefully SGX never has any bugs like https://github.com/lsds/spectre-attack-sgx or issues with the firmware, as noted by the Intel SGX security model document), which is fine, but an equally valid way to do this is to make it so that the secure operation of the system does not depend on what code the server is running.

Doing that has some tradeoffs: there's usually overhead with cryptography, or an algorithm you need may not even be possible (Signal disliked those tradeoffs for this specific algorithm), but for some algorithms, it's entirely possible to do. For example, one can audit OpenSSL's code base, and determine, regardless of what the middle boxes or routers do, that the entire system is secure. Just replace OpenSSL with keybase's client, and middle boxes with keybase's servers, and do the auditing. Hence, open sourcing the server is not necessary for security. Would it be great if more systems could be audited? Absolutely. Is it always necessary for security? Absolutely not.

edit: Another quote from the article: "Since the enclave attests to the software that’s running remotely, and since the remote server and OS have no visibility into the enclave, the service learns nothing about the contents of the client request. It’s almost as if the client is executing the query locally on the client device." Indeed, open sourcing the code running in the secure enclave is effectively open sourcing more code in the client.