Go 1.5 max procs default
golang.org
golang.org
I'm glad that Go has matured to the point where this can be a historical footnote! Good work, Go team.
Great to see they have changed this to something that, without any other context, makes more sense.
You can still benefit from the "claims of the language", even if code isn't running in parallel.
However, I think calling my response "technically true" is selling Go's concurrency primitives a little short. They enable a really great programming model, which makes it much easier to express certain ideas in code. And I/O-bound programs benefit even if GOMAXPROCS=1.
I didn't understand that subtlety when I first started with Go, and it seems like GP commenter (and probably other HN readers) didn't either. "Concurrency is not parallelism" is very helpful in explaining that, which is why I linked to it.
EDIT: Verification and validation [1] are another pair of terms where the distinction is important for industry, but popular understanding often mixes them up.
[0] http://en.wikipedia.org/wiki/Accuracy_and_precision
[1] http://en.wikipedia.org/wiki/Verification_and_validation
Interestingly there is one example of parallelism without concurrency and with just one processor and just one core - vector instructions. Although it would be a breathtakingly good compiler that could emit vector instructions to schedule two goroutines in parallel
For instance, the subtle difference arising from MIMD execution and SIMD execution. The former allows for concurrent execution, whereas the latter does not. However, both are parallel execution models. This isn't theoretical either as GPUs are strictly SIMD machines in their execution.
* Parallelism is a feature of the algorithm, chosen to make it run faster. E.g. you can use a parallel algorithm to factor a matrix. From a technical standpoint, the challenge of parallelism is how to deconstruct the problem so that each processing unit will be able to take on a share of the effort -- parallelism implies cooperation.
* Concurrency is a feature of the problem. E.g. many users using Facebook at once. If at any one time you have 10K requests, then your concurrency level is 10K, no matter how many cores are used. The challenge of concurrency is how to allocate computing resources among the competing requests -- concurrency implies competition.
Parallelism is orderly; concurrency is messy. Parallelism is a choice; concurrency isn't.
My reaction is somewhat like this XKCD: https://xkcd.com/169/
I disagree, it makes total sense when you understand the consequences of GOMAXPROCS set to anything but 1.
Almost every person I know who started with Go got a long way in before doing some benchmarks and realizing that they were only using one thread. I know the scheduler had issues but I think setting the default to 2 would have avoided people going so far before learning that the language can't handle everything they need to care about.
“In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other. The computations may be executing on multiple cores in the same chip, preemptively time-shared threads on the same processor, or executed on physically separated processors.”
https://en.wikipedia.org/wiki/Concurrency_%28computer_scienc...
If you were to go back and reread my comment, note that I used the broader term concurrency at the start – which includes both classic async patterns within a single thread where operations may be interleaved but shared resources wouldn't be updated simultaneously as well as true parallel execution – and then specifically referred to parallel execution at the end as a source of potential pitfalls.
Data point: Inside Google we have been setting GOMAXPROCS=runtime.NumCPU for many years because most of the Go programs we run are highly concurrent network servers, for which a higher GOMAXPROCS value gives better performance.
As a consequence, answering the question of what other processes where on the same machine was not as easy as on an EC2 instance. We didn't mind though because Borg was so good at scheduling we pretty much didn't have to care about the machine level. Instead we thought at the datacenter level.
You mean that it was good for writing network servers capable of handling many tens of thousands of connections? You usually don't need more than one core to handle IO, and if you are doing processor intensive stuff a few more cores is not going to help if you have many thousands of requests - in this case a better pattern is a work queue and a way of letting the client know their work is done.
Concurrency/threading is generally considered a "hard" topic within software dev, and if you lack a solid grounding in the basic concepts you are destined to misuse or misapply the paradigm.
Additionally, even rendering HTML templates can take quite a bit of CPU time and when Go's concurrency is limited to a single-core, that means only one template can be rendered in parallel, blocking other requests.
For what definition of "many"? If you have 16 cores that is 16 requests being processed in parallel. Go hasn't even got out of bed by the time it is handling 16 connections. When Go is being used for it's intended purpose - handling thousands of concurrent connections, you need to be a lot smarter than just running on all the cores you have.
I'm not saying you wouldn't use all cores to maximize utilization - what I am saying if that if you are scaling beyond one core, then you would already be thinking about scaling across multiple servers behind a load balancer - i.e. you are doing ops work on a production scale cluster. And if you are doing production ops work, you are going to be tuning all the processes under your control in detail, not just making rash assumptions about how they may or may not utilization multiple cores.
The point I was arguing here is that someone who said "Back when I was learning Go" and who understands the "nature of the claims of the language" is not going to be running into these sorts of scalability issues, and would not be bitterly disappointed to learn of a default threading value of 1. Unless they had some misconstrued understanding of what Go's concurrency support was all about, and made an incorrect assumption that it was primarily about parallel computation, rather than it's true purpose of having many thousands of lightweight processes handling lots of network I/O.
And latency is important. I can't have a user performing a lengthy operation, while at the same time blocking all other users. Real (not fake) threads are needed to combat latency.
And if you have thousands of current connections (the kind of workload Go was designed to handle), and only a few hundred of them are performing "lengthy operations", you could have 4 cores, or 16, it won't make much difference compared to 1 core - which is the vast majority of the connections are going to be blocked for unacceptably long intervals. In this scenario, the only solution is to either queue up work and make the client come back later, or scale across many servers behind a load balancer.
The point I am making here is that people who are engineering scaled out systems across multiple servers to handle thousands of concurrent requests are usually high-end ops people who are not going to be naively assuming that any process will be doing threading in a particular way - they would have already tested Go in details first and would very quickly have seen you need to set the GOMAXPROCS variable to use more than one core. They would probably look at the GC as well. Tuning stuff to run well is what ops people do, and choosing how many cores a process should use is pretty standard stuff (it may be sharing the server with other processes after all).
In my opinion, it is perfectly legitimate to have the requirement of low latency, while running lengthy operations in the background (I'd show a spinner or a progress indicator, a very common approach). Yes, at a certain point this will break down, at which point I will add more servers. This is better than having no such system at all.
If a language is only good for serving >>1M users and only doing I/O, then I guess it should not be branded as a general purpose, mainstream language, even if aimed at the web.
That's only part of my argument, and I don't think it is a strawman argument.
> In my opinion, it is perfectly legitimate to have the requirement of low latency, while running lengthy operations in the background
Ironically, this is a straw-man argument - I never said nor insinuated it wasn't a valid requirement. All I am saying is for the kind of scalability Go was designed for (1000s-100,000s connections), if you are doing lengthy operations, you will never be just thinking about multi-core, you will be scaling across servers.
> then I guess it should not be branded as a general purpose, mainstream language
Well I honestly am not sure it was. It was branded very early on as a language that is good for writing highly concurrent, highly scalable network servers. For example, integrating with C/C++ libraries is a pain, and slow. I would say most general purpose languages would have good C integration as a basic feature. And the lack of good C integration is a direct result of specializing Go at running millions of little Goroutines concurrently (segmented stack and all that).
An error occurred during a connection to golang.org.
security library: DER encoded incorrectly formatted
message. (Error code: sec_error_bad_der)
I wonder if there is a Firefox-unsupported certificate extension in the new certificate?
EDIT: I wonder if it's related to this Firefox issue: Secure connection failed (sec_error_bad_der) due to certs with SAN dNSName entries incorrectly containing IP addresses[0]
However, it doesn't appear the golang.org certificate has any subjectAltName DNS entries using an IP address.
Firefox doesn't even give you a way to bypass the erorr, even the error itself gives absolutely zero indication of what the issue actually IS. It's extraordinarily obnoxious.
Great. I guess I'll just fork out $10k to satisfy some stupid technicality.
"We used to be able to do whatever and be marked a secure and now we actually have to do security right! UNACCEPTABLE".
Not very convincing.
Removing the 's' from 'https' in FF works for me.
Browser defaults have to be created to cater to the greatest (lowest?) common denominator. And if you can't figure out how to bypass the SSL warning you shouldn't bypass it.
This does not improve security in any meaningful way whatsoever.
openSSL doesn't complain though:
sielicki@wetdog ~ $ openssl s_client -connect golang.org:443 -CApath /etc/ssl/certs
CONNECTED(00000003)
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = *.appspot.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.appspot.com
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
<...>
Start Time: 1432839068
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
Looks like the Chromium source code had some "protection" against this, at least at one point in time: http://security.stackexchange.com/questions/6873/can-a-wildc...
It appears that the specification (RFC 2818?) doesn't specify what is acceptable with regards to allowed wildcard certificates - leaving the various implementations to decide what is "too risky" for themselves.
> Correctness. There is one non-performance concern. Increased parallelism could make bugs in racy programs more likely to cause crashes or other problems. The setting of GOMAXPROCS=1 may have thus far let those bugs go undetected. Raising it may therefore make buggy programs less reliable. We hope that both the race detector and the introduction of goroutine preemption and interleaving in Go 1.1 has already helped programmers identify and eliminate many such bugs. In any event, we can’t reasonably hobble the performance of working Go programs for fear of breaking buggy ones. If such buggy programs do turn up, the authors can set GOMAXPROCS=1 explicitly to keep them working until the bugs can be fixed.
That said: unlike, say, string counting idioms in C, the share-using-channel idiom in Golang is powerful, in the sense of: it's what most people reach for by default.
For a language as opinionated as Golang, it's extraordinarily pragmatic. It is designed first and foremost to be a tool, and then a lot of other things, before being a statement about how to design safe programs.
This probably won't cause anywhere near as many problems as >1 CPU, but still since VMs sometimes have 1 CPU and almost no developer computers have 1 CPU I expect it will not be unheard of.
Even knowing the reasons why this was done (which have been widely discussed for years), I still think the default should have always been to have it set to the runtime.NumCPU value and then have clear documentation on why developers might want to set it to 1 in your own program.
I recall that it took me a while to discover the race detector when I started using Go. Now I always set GOMAXPROCS=runtime.NumCPU() and use the race detector.
That's what Erlang does, too.
Do they test them on a regular basis, appropriately, with GOMAXPROCS > 1?
Personally, I've found the issue to be documented well enough, and it is easy enough to set GOMAXPROCS to whatever works best for a given program, that I'm content with how the team has handled it. And I'm thrilled at the work that is being done now to fix the issue once and for all.
For your own code, we trust you to fix your races or deal with the consequences.
> Correctness. There is one non-performance concern. Increased parallelism could make bugs in racy programs more likely to cause crashes or other problems. The setting of GOMAXPROCS=1 may have thus far let those bugs go undetected. Raising it may therefore make buggy programs less reliable. We hope that both the race detector and the introduction of goroutine preemption and interleaving in Go 1.1 has already helped programmers identify and eliminate many such bugs. In any event, we can’t reasonably hobble the performance of working Go programs for fear of breaking buggy ones. If such buggy programs do turn up, the authors can set GOMAXPROCS=1 explicitly to keep them working until the bugs can be fixed.
So the answer to "are there generics in 1.x release" will always be "no".
http://golang.org/doc/faq#What_is_the_status_of_the_project http://golang.org/doc/go1compat
Go 2 is years away (if it even happens).
I've never asked this. Certainly it would have been a big news. I asked if there are any updates in general - any new thoughts, plans...
Oh, I need self-modifying code and both computed and assigned gotos, too. I'm used to using them since FORTRAN.
Come on, Go team, fix your language so that we can start using it.
Don't hold your breath. it's unlikely the go team will add generics to the language, that's obviously a political issue not a technical one. Same for error handling. At least 2 people on the record, including Rob Pike said no new feature will be added to the language.
It doesn't mean you should stop asking for them, but it ultimately means that people wanting generics and other features in Go should consider a fork
Forks are ALWAYS healthy, the nodejs/iojs story is yet another proof of it. The thing people wanted (iojs) has won over the original project, which forced nodejs merge iojs back.
Like many I personally think Go isn't striking the right balance between minimalism and features. People who are against generics are in favor of trading compile time type checking(which is always safe) for run time type assertions(which can fail). But they refuse to acknowledge that fact. That's what make their argument against generics wrong and will ultimately damage any potential growth for the language.They are basically working against their own interests.
There are deep technical issues that must be solved to fit the idea of generics into Go in a way that works well with the rest of the system, and we don't have solutions to those. I wrote on my blog about one issue years ago (http://research.swtch.com/generic), but there are others too. Even supposing you get past the problem on that page, the next thing you would run into is how to allow programmers to omit type annotations in a useful, easy-to-explain way. As an example, C++ lets you write make_pair(1, "foo") instead of make_pair<int, string>(1, "foo"), but the logic behind inferring the annotations takes pages and pages of specification, which doesn't make for a particularly understandable programming model, nor something the compiler can easily explain when things go wrong. And then there's a princess in another castle after that one I am sure.
We have spoken to a few true experts in Java generics and each of them has said roughly the same thing: be very careful, it's not as easy as it looks, and you're stuck with all the mistakes you make. As a demonstration, skim through most of http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.ht... and see how long before you start to think "was this really the best way to do this?" (For example, http://www.angelikalanger.com/GenericsFAQ/FAQSections/TypePa..., but note that the latter page is only one part of the FAQ, not the entire FAQ.)
To be very clear, we acknowledge this fact: there are definite disadvantages to not having generics. You either use interface{} and give up compile-time checking or you write code generators and complicate your build process. But there are also definite disadvantages to generics as implemented in existing languages, and there is a very large advantage to not compromising today: it makes adopting a better solution tomorrow that much easier.
As I said in the interview at http://www.pl-enthusiast.net/2015/03/25/interview-with-gos-r...:
> Go is more an engineering project than a pure research project. Like most engineering, it is fundamentally conservative, using ideas that are proven and well understood and will work well together. The research literature’s influence comes mainly through experience with its application in earlier languages. For example, the experience with CSP applied in a handful of earlier languages—Promela, Squeak, Newsqueak, Alef, Limbo, even Concurrent ML—was just as crucial as Hoare’s original paper to bringing that idea to practice in Go. Programming language researchers are sometimes disappointed that Go hasn’t picked up more of the recent ideas from the literature, but those ideas simply haven’t had time to pass through the filter of practical experience.
I believe generics is one of those ideas. It certainly needs at least one more iteration, possibly a few more than that.
I understand that Go team are more "systems" persons rather than "languages" in general. Did you try to consult with Dart team? e.g. with Gilad Bracha. Maybe you should consider hiring some languages person in Go team.
And the lesson learnt from studying other languages is "Don't add generics unless it's right for the language, even if that means choosing, on balance, to _NOT_ implement them at all".
C++ compile type templating is the cause of egregiously long compilation speeds. One of the original drivers for the creation of Go was the speed of C++ compilation.
I don't think you can separate a desire that generics was implemented a different way in these examples from the regret that their current implementation causes.
I'd much rather have fast compilation and slower execution and some kind of generics, but the current situation is that people are insanely attached to their pride in fast program execution.
This is legal(-ish) C++:
template<typename T> T Add(T one, T two){ return one + two; }
It's completely illegal in C#. And Java. Both have "generics". Both are wildly different.
Then you have the Haskell camp which look at all of these "generics" and shake their heads and laugh bitterly since their static type system and "generic" blow all other ones out of the water anyway.
So when you say you want "generics" - which flavor do you want? How does it play with Go's interfaces? How deep down the rabbit hole do you go?
It's very easy, when you're not familiar with the problem to say "just do it" - but everyone seems to have their own idea of what "generics" is, how it works, how to constrain parameters, etc.
So even if the Go team picked an implementation from Camp A, I'm 120% positive that camps B, C, D, and X would declare it the worst implementation ever, and again, they've pleased no-one.
I'm glad they're sticking to their guns on this one.
Show them an implementation that works in Go, and they've said multiple times they might do it. Put up or shut up, basically.
Didn't someone fork Go and do exactly that only to get ignored? Perhaps it was a rumor, do you know? I'll see if I can find any reference to it.
https://groups.google.com/forum/#!topic/golang-nuts/L5Lothlv...
It also led to a real world example where the author tried using Go for a project and had a lot of need for Generics:
http://oneofmanyworlds.blogspot.com/2011/11/draft-2-of-propo...
I'm a C++ engineer more than a decade now.. And I'll be more than happy if Go had ANY kind of generics - even Ada-like, with explicit instantiation... Anything is better than current interface{} run-time checked stuff.
1. Expand on your idea with some examples. 2. Show how this can be done without breaking existing Go code 3. Showing where the performance hits are - runtime? compile time? binary size? memory? 4. Show how this plays along with Go's implementation of interfaces, as well as other already-generic types, such as channels and maps.
If you can do all that, realistically, then I don't see why it wouldn't be taken seriously?
Heck, if you can provide even an example implementation (talk is increasingly cheap) I bet you'll get lots of people on-board!
You seem, like many others, to think that Go core team members don't want "generics". All they have ever said is - "we don't want bad generics". That's all.
Anyone can say "I don't know what exactly Go generics are, how they look like, how they perform, how they change the syntax, all I know is - I want them". Like I said, talk is cheap.
I understand the caution, but this justification doesn't add up.
Generics + subtyping (inheritance) gives rise to a number of complex situations. Java needed to embrace bounded polymorphism in the type system in order to handle this. And we don't know about a simpler type system which is sound and has that particular mix. If you remove subtyping and replace it with another construct, type classes or implicits, then generics are somewhat easier to fit to the language.
CSP turns out to be far more orthogonal, as long as you use a typed channel primitive. Hence, it is easier to fit CSP into an existing model because its composition with other type theoretic constructs is far simpler.
Go is the new Java (<=1.5). It's slightly better....but it's not a language that will grow with you. If you're a serious programmer, you'll easily outstrip the usefulness and expressiveness of Go.They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the lang tl;dr: Go is uage that we give them has to be easy for them to understand and easy to adopt.
[1] http://nomad.so/2015/03/why-gos-design-is-a-disservice-to-in... [2] http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Fro...
Coming from years of doing both Python/JS and C/C++, Go is a nice balance between the stong points of each camp, and it fits amazingly well in some cases. I hope it will be the new Java, because I love writing in Go and hate writing in Java :)