I vaguely remembered minux suggesting a way to do this more efficiently, and dug it up: golang.org/issue/3035
Maybe it'll work for your use case.
228 karma · joined February 24, 2012
Maybe it'll work for your use case.
If you're going to judge this code based on both pathological haystacks but also pathological needle regexs, then you have to admit that this code, which has linear space usage in the pattern length is far better than PCRE, which has exponential time in the pattern length[1].
Considering its succinctness, I find that pretty impressive.
rather than this: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
(not to mention the 2 other Java files to setup the server).
I'm slightly confused by this question, because that's what the standard library does. If you've ever used net/rpc, or net/http, then it spawns goroutines for each request (or connection, respectively).
If you meant to say spawn a bunch of _processes_ to serve requests, then no, I don't think anyone has done it. I don't think it makes a whole lot of sense for anyone to write code to do this in Go, tbh.
> redundancy in case a process does hang
If you're talking about deadlocks, then only the deadlocked goroutines will be blocked. The rest will make progress, just as if you had multiple processes.
> rolling restarts to switch out to a newer version of the code seamlessly by starting multiple processes to handle requests before killing the old one
You can do this with only 2 processes, old and new. You can spawn the new one, tell your LB to add the new to the pool, wait 30 seconds, remove old from pool, wait 60 seconds, kill old one. You don't need an LB, you could, of course, use an nginx frontend or something instead. There's also some neat ways to do nginx-style zero downtime restarts that I've never tried, but heard good things about.
> I suspect you'd hit some limits of the scheduler
AFAIK, limits of the scheduler tend to be hit when you increase GOMAXPROCS to something above 8. At this point, you'll spend a lot of your time in the runtime managing goroutines. My solution is just to run multiple processes with GOMAXPROCS=8 and point your LB at both of them. Again, you can just use nginx.
Feel free to experiment with the model you proposed, but this is relatively non-idiomatic, and the context-switching cost will start to mislead you as to Go's actual potential. The advantage, btw, that you get when you use the model I spoke of is that in memory caching, connection pooling, context switch time are all close to optimal, and you have fewer processes to monitor/restart/update.
(I understand that you did because the original poster did it)
It is getting there. If you run godoc locally with tip, you will find many more examples than there were previously.
See http://golang.org/pkg/path/ for a current example. And the source for those examples can be found here: http://golang.org/src/pkg/path/example_test.go. Note that this technique will work for your own packages as well.
The technology is there, the implementation is still incomplete; but more progress has been made than is easily visible.
First, let me point out that we have moved on to a slightly different topic. The tools that json_decode gives you is available in Go (it would be syntactically awkward, out of a necessity born of Go being statically typed). An example: http://play.golang.org/p/OvshdY1oVQ
However, I would write it as follows http://play.golang.org/p/pODFi9Jrve. This is merely a different way of doing the same thing.
Now, to the topic that we are talking about is the claim that the latter is nicer. I was not comparing PHP and Go when I said that it was nicer (although I have some experience with PHP). I was drawing from my experience with working with JSON in statically typed languages. In particular, working with JSON in C++, Objective-C and Go. In all of these languages, the JSON libraries I used were essentially dynamically typed. I claimed merely that having the JSON library use reflection to deserialize into typed structures is nicer than using NSDictionary's various methods or maps in C++.
In PHP, the syntactic overhead of walking dynamic structures is obviously much lower than the languages I was thinking about, so that criticism is blunted when referring to that language. But I still believe there is some advantage there. A large part of what I find nicer about the latter is indeed the same reason I prefer static typing to dynamic typing. It is impossible to mistype the latter (try changing the ".Foo" to ".Fo" and trying to run the snippet). Consider that doing the equivalent in PHP will simply return NULL (I think), as it is really just doing a dictionary lookup. I fully admit that this is a static versus dynamic thing, not about JSON in particular.
As to the performance thing, I think you too easily discount the importance of low latency. But regardless, I was really comparing to Go's statically typed brethren. They essentially use hashtables for representing parsed JSON objects which is far less than ideal. Note that they don't employ the clever tricks that dynamic languages use. V8, for example, makes JS's objects (which are essentially hashtables) perform significantly better than a mere hashtable.
Why is this nicer?
Let's say you are working with the Reddit API. You can, of course, do it the way you suggested, and just decode the result into an untyped object. However, I find this incredibly brittle. You end up with strings indexing into arrays all over the place.
On the other hand, when you decode into a Go struct, you use the regular dot syntax to access parameters. You can't misspell these or use the value inappropriately (trying to loop over a JSON number, for example). You can write comments on the fields to document what you expect various values to hold. It is nicer in the same sense that using classes is nicer than just using arrays for everything.
Another benefit is of course the improvement of performance. Accessing an string value in a JSON object by a key is the equivalent of a hash table lookup in Python or PHP. It is incrementing a pointer by a constant number in Go (assuming you use strongly typed decoding).
I'll say it again, Go's encoding/json can do what you want (the equivalent of json_decode). There are better alternatives available to users of that package, and the GP was discussing how to exploit one of those better alternatives without having to write the struct by hand (which I personally find a trivial one time cost).
It also can decode directly into typed structures. An example from the documentation (you'll have to open the linked example): http://golang.org/pkg/encoding/json/#example_Decoder
Anecdotally, a typed version of JSON decoding is much nicer to work with.
This is in the same vein as TCP, TLS, or DNS. Developers use these technologies all the time, and it exposes a simple interface to them, but all of them are quite complicated under the covers. I don't believe that Nagle's algorithm or the UDP framing for DNS is common knowledge amongst the majority of developers.
Similarly, SPDY or HTTP/2.0 will expose a simple interface to developers, but internally transport the content in an efficient, if somewhat inscrutable, manner.
While I appreciate the sentiment that HTTP is easy to understand as it stands, I'd rather not optimize for "developers of almost any skill level". While it would be totally OK for you to write a HTTP/1.1 parser as a toy, it would be somewhat crazy for you to use that implementation in a production scenario.
There, everybody uses a small set of implementations that have been vetted: Firefox, Chrome, Apache, Nginx, etc. These guys work extremely hard to make sure that their implementation is solid not merely "compatible-ish". I'd rather optimize for them, rather than the beginners. I'd rather make it easy for them to write correct implementations. I'd rather jettison the "fault tolerant enough that compatible-ish is ok" design of HTTP/1.1 that makes their life so difficult.
Pushing onto a stack is equivalent to `s = append(s, obj)`. Checking if the stack is empty is equivalent to `len(s) == 0`. Peeking at the top (assuming its non-empty) is equivalent to `top := s[len(s)-1]`. Popping from the top (assuming its non-empty) is equivalent to `s = s[:len(s)-1]`
If you want to do your own thing, you're free to ignore the rest of the community in this respect, and create your own build mechanism. In fact, if you pass the "-x" flag to the go tool, you can even see how its accomplishing what its accomplishing; which might help you write this better build tool.
Personally, I keep 2; one for external packages, and one for ones I'm working on.
Gist of the task at hand: you'd have to arrange for data passed into your target application to point to structures in memory, causing them to be pinned. Ideally, these structures should be big, to maximize impact of the attack. Further, you'd have to ensure that the data you passed in isn't garbage collected, as this would allow the target structures to be collected.
No more so than with C++. There is a relatively high probability of you having left a memory leak in an average C++ server.
> Monitor memory usage, and just restart?
That would work, and is probably a good idea for all servers anyways. I've seen apache take upwards of 60 gigs due to some misconfiguration, so monitoring memory utilization of your programs and alerting or automatically restarting is a good idea.
> What am i missing? How does this not completely destroy anu utility of these languages? Why do people put conservative gc in languages outside of the esoteric or academic context?
There isn't really a good reason as far as I can tell. The only advantages are that its much simpler to implement, particularly when you have C interop. You can annotate Go objects with types to do precise collection, but as soon as you pass them to C, a lot of guarantees the Go typesystem makes go out the window. Other than that, can't think of anything.
As to the lack of a shared address space, I thought I had communicated that when I said "It might require you to restructure to reduce chattiness", but yes, there are performance advantages to having a shared address space.
You should note that there are security disadvantages to having a shared address space with a sandboxed plugin written in a relatively low-level language (bytecode). I won't post several links to exploits of the Java sandbox in the last year alone, but I am sure you can google for them.
This mechanism is most certainly not a hack. This is the approach that Chromium and the other multi-process browsers (all the modern ones?) use for sandboxing. It seems far more effective to me to push responsibility for security to the OS, rather than the language runtime.
> A full featured runtime reflection mechanism that does not drop "static info" on the floor
Package reflect is fully featured. It just requires you to have a value, or type to actually reflect upon. The tradeoff being that Go can do dead-code elimination and Java cannot. Same thing for Class.forName.
> @MetaInfo() ... Arguably not easy on the eye but necessary and enabling.
I think you are referring to annotations. Yes, Go doesn't have these on methods, functions, or types. But it has these for struct fields (tags); this gives you 90% of the functionality, without muddying up the syntax.
> Bytecode injest (beyond dynamic linking) -- the JVM is your oyster.
Sure. See my note about all of these points below.
> and related Instrumentation
Go has a CPU profiler, a memory allocation profiler. In tip, it has a data race detector. It can expose this data over an HTTP interface so you can profile a running server http://golang.org/pkg/net/http/pprof/.
> It is borderline fanaticism to insist on comparing Go to Java, and unfair to Go. Effective advocacy of Go should focus on its strengths (syntax) and not attempt to gloss over its inherent limitations. It is a capable and viable language within well defined limits. Java and JVM are in another league. Accept it and move on.
You continually give me solutions that are impossible to implement in Go, but not the corresponding problems. Like I said, there are things that are syntactically impossible to write down in Go. If you want me to give you a list of things that are impossible to do in Java, I can do that (structural typing, value types, methods on values, multiple top level definitions in a single file, first class functions, efficient array slicing come to mind). I have not done that because it is a nonsensical thing to do. That would be like me saying Javascript is superior to Java because Java does not have prototypical inheritance.
The only useful example I've heard is sandboxed plugins. That is a problem, not a solution. All the things you've listed are solutions. If you actually want to have a conversation about how to solve problems in Go, then list problems, not a laundry list of features that Java has that Go does not.
> If a new 'headius' feels up to it, Go on JVM would be a very interesting new language for the JVM ...
Why? If I want more performance than gc gives me, then I use gccgo. What would Go on the JVM give me?
Also, on an unrelated note, I find it a little sad and pathetic how both you and 0xABADC0DA have to resort to insults to get your point across.
I was responding to "you can't have an app that loads plugins much less one where the app can do IO but the plugin can't."; that is, I was merely providing a way to implement what you wanted (plugins that cannot do IO, in a host that can).
Like I said in my original post, "If you mean things that you can syntactically write down, like a Giraffe is-a Animal, or an Apple is-a Fruit, then yes, that is impossible to write down in Go." Similarly, it is impossible to dynamically load code, or enforce permissions at the runtime level (although implementations of this enforcement has historically been... buggy, to be charitable). I was attempting to show that the problem of having sandboxed plugins is solvable in Go.
Sure you can. Spawn a child process with lower security privileges, and communicate over a pipe. It might require you to restructure to reduce chattiness, but on the other hand, you will be much less likely to expose yourself to security vulnerabilities than the Java "sandbox".
If the child process is written in Go, you can even use the -u compiler flag to only allow safe packages (i.e. no assembly, C, only packages explicitly marked as safe and pure Go). You can go even further, and mark package os unsafe, disallowing file system access altogether. Or do what (I suspect) Appengine does, and provide your own neutered implementation of package os/time/net/etc, and mark those as safe.
The Go playground is an example of this, btw. It merely streams stdout and stderr, but you could imagine doing other things with the sandboxed program.
> I think Google Go advocates put blinders on like agentS because if they actually objectively looked at the situation they would have the same thing to say as others do about Google Go: "meh".
I think you are falling prey to the same fallacy of false cause as your GP. "agentS likes Go, so he must not be objectively looking at the language". It is possible to objectively look at the language, and think it useful.
I certainly don't have this philosophy, and I don't know thats its a thing amongst Go programmers. I think its just a standard library thing.
> I mean, how hard can it be to split the "big formatted I/O package" into a "basic formatted I/O" package and an "advanced formatted I/O package"
It would easy, but then you're cluttering your library to make implementing the std lib easier, rather than making the lives of people using the std lib easier.
> or maybe even make "basic formatted I/O" something like part of the language "built ins"
Things that are in the language built-ins would need to go in the spec, and would need to be implemented by two separate compiler suites. The authors seem to be doing their best to keep the core as small as possible. Having libraries written in Go is cleaner, reusable across compilers, and give newcomers a place to read idiomatic Go.
I would argue that both of these (or atleast the core of these) is addressed by Go.
For documentation, there is a simpler way to get at a library's public API; an automated program that extracts that API and presents it in a web page (or in your console). See http://golang.org/pkg/unicode/utf8/ for an example (for completeness, here is the file it is generated from http://golang.org/src/pkg/unicode/utf8/utf8.go)
For avoiding unnecessary recompiles, Go solves this problem by making compiles really fast, rendering this problem moot.
I'm confused. This is indicative of a deadlock, where no goroutine can make progress. Sleeps would mask the symptoms, yes, but would never actually solve a deadlock, the program would just do nothing for a longer period of time. What Go codebase(s) are you referring to?
> A concurrent Go program will likely behave differently given 2 bits (just 2 lousy bits) of difference in the object binary. (runtime.GOMAXPROCS(1) vs runtime.GOMAXPROCS(2)). Imagine someone touching those 2 bits in a "large codebase". It is practically impossible to do the same thing in a large Java codebase and fundamentally change the programs runtime behavior. (Happens all the time in Go.)
If you are trying to find a low number of bits, you can just use 0. GOMAXPROCS can be set via an environment variable, the function is just to override that value.
More to the point, I am convinced that you are wrong. You are saying that in a Java class file, there are no 2 bits I could change that would impact the behaviour of a program, and that is plainly false.
> It is very difficult to reason about a Go routine's behavior in a "large codebase" without global view and a mental model of the dynamic system e.g. which go routine is doing what and who is blocking and who is not.
I guess this could be true if you engineer a system where every goroutine depends on every other goroutine. But it isn't true for code I've seen. As an example, the http library in Go has a goroutine that accepts TCP connections, and a goroutine for every accepted TCP connection. This knowledge is not necessary to use it, and is not necessary if a different part of your program is using it, because it is exposed behind an abstraction (http.Handler). To paraphrase the OP, Go enables simple programming. It doesn't forbid bad programming.
> There is nothing, absolutely nothing, that you can do in Go that you can not do via libraries in Java.
Depending on what you mean by "do", I believe Turing would have something to say on this matter: http://en.wikipedia.org/wiki/Turing_completeness
> On the other hand, there are plenty of things you can do in Java that are simply impossible to do in Go.
Depending on what you mean by "do", I either agree with you, or think you are mistaken. If you mean things that you can syntactically write down, like a Giraffe is-a Animal, or an Apple is-a Fruit, then yes, that is impossible to write down in Go. If you mean a problem that can be solved in Java that cannot be solved in Go, then I think you are mistaken.
> Once we factor in the possibility for bytecode engineering, then Java is simply in another higher league as far as language capabilities are concerned.
Why is this in another league? You can use assembly from Go, meaning you can generate code and jump to it if you really want to. Not sure why this would be considered a special feature of Java, or even why you think Java pioneered this. Bytecode is just a portable assembly, but its just as portable to build per-platform assembly emitters (like compilers do).
> Most people who rag on Java are clearly diletantes Java programmers.
Right. That makes sense to me; no one who knows anything about Java would ever criticize it. Sarcasm aside, this demonstrates a fallacy in reasoning.
> If Go actually manages to be as effective as Java for concurrent programming at some point in the future
It isn't more effective now? News to me.
> when they fix the somewhat broken runtime
Sorry, remind me what this is referring to?
> It just works. (But it is "boring" because it's not bling anymore. Oh well, kids will be kids.)
This is dangerous thinking. This is the cry that the native programmers raised when Java was born. Try and keep that in mind.
Do like the http library does and separate out socket handling to be in terms of net.Listeners/net.Conns, and create a handler abstraction for yourself. Your socket code will be testable, because its trivial to write fake net.Conns that are backed with byte buffers. Your handler code will be testable because they are in terms of parsed objects.
I tend to write socket code once per server project, and then leave it alone for months, so this isn't a big deal for me. What makes Go nice for me is that I can block in client code, and retain the efficiency of using a select loop.
I don't have access to the source for the proxy I wrote for work, but here's one I whipped up quickly: http://play.golang.org/p/Fz19qSehCg
Go doesn't expose select, and has the runtime do that for you; but this allows them to make all Go libraries share a select loop, which has nice performance characteristics. Although, now that I think about it, it is probably possible to have a userspace implementation of select that works atop the runtime's shared select loop. Hmmm...
As to your example of io.ReadFull: what about it? If it gets an EOF before filling the buffer you passed in, it'll return ErrUnexpectedEOF per the documentation. If it gets any other error, it'll return it instead. I haven't actually looked at the source in a while but that's how almost all of the byte stream functions work.
For me, C++ memory management becomes challenging in the face of concurrency. Particularly when you write applications that are event-driven, instead of thread-based, and you've surrendered to an event loop. It becomes challenging to keep track of object lifetimes.
Just my 2c.
err := readMessage(reader)
if n, ok := err.(net.Error); ok && n.Timeout() {
// handle timeout
} else if err != nil {
// handle other errors
}
You're absolutely right. People should be able to use libraries without reading their documentation, and instead rely on compile failures to slowly iterate their code towards perfection. /s
Seriously, though, I believe expecting programmers to read documentation of a library they're using is a pretty low bar.
Also, no it doesn't list what all the possible error types are. To do this would require what amounts to checked exceptions in Java. The issues with these are relatively well known.
First, it complicates versioning as adding a new error type is a breaking change for all clients. When they get an error from an API, most clients either return that error verbatim, decorate that error slightly and return the decorated error, or handle a few specific error situations and return an error in all other cases. Declaring all possible error types makes your programs brittle, as it is easy for libraries you are using to break your code.
Second, they are a hassle for larger programs that touch many systems. It is easy to declare that you return an EntityNotFound error. But it is not so easy to declare that you return an EntityNotFound, MemcacheCASConflict, FileNotFound, FileExists, PermissionDenied, Timeout, InvalidJSON, ConnectionClosed, or TemplateRenderingFailed error. This is perfectly reasonable set of errors for a simple method that gets a value from datastore (possible caching it) decoding a field as json, and writing a template to an HTTP connection. Any 'simple' wrapper of this method then inherits all of these declarations. Now certainly with Java, IDEs will "fill in all the forms" for you, so this problem is a little more palatable, but Go does not require programmers to use (often heavyweight) tools to make them productive.
> code returns a singleton error (no state) presumably so the caller doesn't have to do a bunch of casts
This is a little disingenuous, as you don't really know why they made it a singleton error. I can't think of any state that would be useful in this situation, can you?
The Java version of this API throws an exception with a single field, the Key that could not be found. I think this is supremely unhelpful and just adds clutter to the documentation, as the user of this method clearly already has this information in hand.