What Python developers need to know before migrating to Go
blog.repustate.com
blog.repustate.com
Writing to a file, there’s File.Write([]byte) and File.WriteString(string) – a bit of a departure for Python developers who are used to the Python zen of having one way to do something
Simply because it's not currently feasible to write a function that can have multiple type signatures (via overloading, generics, etc.).
Going between []byte and string. regexp uses []byte (they’re mutable). It makes sense, but it’s annoying all the same having to cast & re-cast some variables.
There are actually two different versions of each regexp function: one for strings, one for []byte. Maybe they're using a different version?
No tuples, have to write your own structs or use slices (arrays)
Actually, you can declare a fixed-length array as a type. So if you need to return a triple of ints, you can declare the return type to be [3]int and have the type system check that you're actually returning an array of ints of length 3.
Similarly, a lot of the other bullet points are natural results of having a static type system.
I was expecting more points like:
* you can't monkey-patch functions into external libraries like you can with Python modules * no generics, so you can't write a generic "map" function, for example * no preemptive multitasking, so one goroutine can wedge the entire program * passing around arrays and structs is by value, leading to unexpected allocation and memory usage * pointers
That's different from a tuple as it is commonly known in Python and statically typed languages. Tuples can be heterogeneous in languages like Python, OCaml, and Haskell, which Go fixed length arrays can't be. For example, in Go you can't have `(int, float)`.
Think of a tuple as an anonymous struct. (Or perhaps think of a struct as a tuple with labels :P.) We want to statically differentiate between a pair of ints, a pair of strings and an int, string pair, which we simply can't do with interface{}.
switch x.(type) {
// cases
}You can add functions to existing types in Golang.
no preemptive multitasking, so one goroutine can wedge the entire program
That's pretty misleading. If all you write is go code, you can't "wedge the entire program" in Golang just from one thread. You could wedge a native thread if you were using C methods. Assuming you don't want to fix your broken code, you can just set GOMAXPROCS to something high and work around even that.
passing around arrays and structs is by value, leading to unexpected allocation and memory usage
Most developers pass pointers. Pointers are not really that hard in Golang, because there is no pointer arithmetic or undefined behavior.
- Function overloading is a major convenience that you will miss. There are differently named versions of every function and you will call the wrong version with the wrong arguments all the time. The number of functions in the standard library could be reduced by at least 1/4 if they'd got this right. The official FAQ (http://golang.org/doc/faq#overloading) explains that leaving out overloading is "simpler", meaning simpler for them.
- Default parameters are a major convenience that you will miss. Using strings.Replace() to remove some chars from a string? Don't forget to pass the -1 at the end, asshole! The -1 says don't put a limit on the number of replacements. In Python there would be a max=None default parameter and this would never bite anyone.
- No named arguments, because fuck readability.
- Forcing me to handle errors is great. Having 20 different ways to do it is not great. Examples: fmt.Errorf(), fmt.Fprintf(os.Stderr), errors.New(), log.Fatal(), log.Fatalf(), log.Fatalln(), panic/recover...
- Using && and || for logical operators in this day and age is just ridiculous. Why do people keep inventing programming languages as if Python doesn't exist?
- Don't think that just because the Unicode guys invented Go that Unicode is going to be easy. Their solution is not to create an airtight abstraction layer between chars (or "runes" WTF?) and integers. Their solution is to provide almost no abstraction and force you to deal with the inherent integer-ness of all characters. Example:
In Python:
len("нєℓℓσ") # 5, because there are 5 chars len("нєℓℓσ") // 12, because there are 12 bytes
utf8.RuneCountInString("нєℓℓσ") // 5, plz kill me i am an abomination
No it's not, it takes 10 minutes to learn that && means and and || means or (maybe a little longer to get the hang of it properly), and this knowledge transfers to many programming languages.
(This is a little like arguing "we shouldn't use + when English has a perfectly good word 'add'"; symbol reasoning is valuable.)
Python expresses the bitwise and and or using the & and | characters, the exact same characters as Go.
And that also explains why Go chooses to use && and || for and the logical and or operators.
No. Cognitive overhead. You pay for it every time you parse these words in your brain. You pay for it by reducing the number of nested/combined clauses that you can parse on the fly.
(This is far from the only readability issue with Go, by the way, and you're right in that it's among the more superficial ones. The language is designed so well in all ways except the one that matters the most, it hurts.)
&& is pronounced "and" but actually means "shortcircuit left-to-right-evaluated and".
If you're coming from a Pascal (or non-programming) background, you do not assume either left-to-right evaluation order, nor short circuit evaluation.
The cognitive overhead is always there, because whether you like to admit it or not, programming is applied math, and exact meaning is very important;
e.g.:
if a == 0.0 or b/a > 3 then launch_missile();
The hypothetical "newbie programmer who can write a working program but has cognitive overhead deciphering &&" is a mythical creature that does not actually exist.
len("нєℓℓσ") // 12, because there are 12 bytes
utf8.RuneCountInString("нєℓℓσ") // 5, plz kill me i am an abomination
Shouldn't UTF8-specific things live in the utf8 namespace? Some programs won't need any string handling, after all, and it would be a waste to include code they never used.
Assuming you can allow the utf8 namespace as sensible, would you feel better if there was a RuneLen() function aliased to RuneCountInString()?
If you are that upset about it, then my suggestion is to explain your rationale and submit a patch[1] to provide the alias. It's not like it would be hard to code. Perhaps you might convince people and get it in the next release.
But that doesn't matter. Let's say you are correct: when working with strings, you more often want the rune length. It still wouldn't be the right decision, given the other design decisions of Go, because it would have needlessly complicated things with only arguable benefits. Let me show you what I mean.
The len() function works with a whole lot of things: strings, arrays, slices, maps and channels. For the first three, len() returns the number of bytes involved. This is because all three are backed by an array, and so sensibly have similar semantics. It would have violated the principal of least surprise for anyone who knew the language to have an array-backed storage not return a byte count. Both the language developers and the users of it would have to special-case strings, in code and in their brains.
Now, they could have decided to do it anyway, but then another surprise awaits. What happens when you take a slice of a string? Oh no, more special casing and more complication for everyone.
The Go developers do special-case where doing so would clearly be a win for their users. Consider range, which iterates by runes over a string, potentially moving the index on the underlying array forward by more than 1 on each pass. That is clearly going to be the most common usecase the user is going to want and so was worth doing. It also eliminates many of the usecases where getting the length of a string in runes would matter to you. Not all, but a lot.
What happens when you slice a unicode string in Go is that it cuts multi-byte characters right in half, unless you get the byte boundaries just right. I know real programmers keep the byte boundaries for all the chars in all their strings in their head at all times, but for people like me this basically makes string slicing unusable for non-ASCII text.
Python somehow magically slices unicode strings without chopping characters in half.
In Python:
s = "нєℓℓσ"
s[1:4] # "єℓℓ"
s[1:4] // "�є"You need a byte offset to slice a string, and it's impossible to convert from a Unicode rune offset to a byte offset without parsing the entire string up until that point. I'm not all that familiar with Python, but if the language works as you implied, it is basically doing this behind the scenes in common string processing tasks:
1. The user uses some kind of pattern matching function or whatever to find where they want to split the string. Python returns a rune index.
2. The user tells Python to go split apart that string along a rune index. It promptly begins parsing the string all over again until it finds the right byte boundaries.
3. The language then actually creates the new string in between the byte boundaries.
1. User pattern matches blah blah blah and gets a byte index.
2. User tells their sane language to split the string apart at the byte index and it just does so.
When the hell would you have to remember the byte or rune boundaries for characters in the first place? Why would you be slicing up a string with magic number indices? If you're getting indices from pattern matching functions, you shouldn't care whether they're in bytes or bits or nibbles, you should just be passing them on to your language's split routines (or whatever else you wanted to do). Unless, of course, you're the one actually writing low-level string processing routines, in which case rune offsets are far less useful than byte offsets for the reason explained above.
This Python "feature" seems to exist entirely to keep newbies from getting confused when they attempt to slice up strings in their REPL, for I cannot fathom a reason why anyone would write "s[1:4]" in production code. IIRC, Python was designed for pedagogy, so I'm not surprised that it would take on such a pointless implementation cost just to spare teachers from explaining why "s[1:4] gave me question marks"
This might explain why my comments seem like heresy to you. I would point out that the OP is about Python programmers switching to Go.
Ok, so it (EDIT FOR YOUR BENEFIT: I'm talking about slicing strings with rune indices here, not slicing in general. Array slicing is a useful language feature, and, uh, it's not unique to Python or anything) is not just a language wart, but a fertile source of pointless inefficiency in everyday Python code, glad to know.
>This might explain why my comments seem like heresy to you.
You aren't challenging my beliefs or anything, I'm just trying to make you see that you don't understand how UTF-8 string operations work very well. If you did, you'd understand that Python is doing the exact same thing as Go here, but in a less efficient manner.
The fact that it's used all the time would suggest it's not pointless inefficiency, no? Maybe you should try Python before bashing it.
>Python is doing the exact same thing as Go here, but in a less efficient manner.
It's not doing the same thing. "�є" is not the same as "єℓℓ".
Dealing with a format where data elements are defined to be fixed length in characters that happens to be encoded in Unicode?
Well, that rather depends on what you mean by "character" and "in half".
>>> s = u're\u0301sume\u0301'
>>> print s
résumé
>>> len(s)
8
>>> for i in xrange(len(s)):
... print s[i]
...
r
e
s
u
m
e
>>> print ' '.join(s)
r e ́ s u m e ́I have and I generally like the syntax to be more explicit (since I so rarely work in bytes): string.getBytes().length
You make all fair points, and I guess it is my opinion that varies then but I think the rule of least surprise would involve returning the rune count for both strings and string slices. Also, wow, range iterates over runes but len returns byte count. That's messed up.
The implementation of strings in Go is a 2-word struct containing a pointer to the start of the string and the length (in bytes). Under this implementation, len(s) is O(1) and RuneCountInString(s) is O(n). It makes sense to have the default case also be the fast one, particularly since people appreciate Go for its performance.
Alternatively, you could store the rune-count in the 2-word struct to reverse the above runtimes. However, this is detrimental for the common operation of converting between []byte/string as well as writing a string to a buffer. Both of those operations are a simple memcpy with the actual Go implementation, but would be O(n) using this alternate implementation.
Perhaps you could make it a 3-word struct that contains both byte-length and rune-length; Then all strings take up additional memory as well as requiring more overhead when used as function arguments.
Consider a simple string: "école". How many runes does it contain? Possibly five:
LATIN SMALL LETTER E WITH ACUTE
LATIN SMALL LETTER C
LATIN SMALL LETTER O
LATIN SMALL LETTER L
LATIN SMALL LEtTER E
LATIN SMALL LETTER E
COMBINING ACUTE ACCENT
LATIN SMALL LETTER C
LATIN SMALL LETTER O
LATIN SMALL LETTER L
LATIN SMALL LEtTER E
Fortunately, you generally don't need to deal with any of this. If you're working with filenames, for example, you really only care about the path separator ('/' or '\' or whatever); everything else is just a bunch of opaque data. You can write a perfectly valid function to split a filename into components without understanding anything about combining characters. When you're dealing with data in this fashion, you rarely if ever care about the number of runes in a string; instead you care about the position of specific runes.
In Go:
fmt.Printf("%s", string([]rune("нєℓℓσ")[1:4]))
// єℓℓ
print("нєℓℓσ"[1:4])
# єℓℓ"Runes" were the original name, as implemented in Plan 9 by the same folks, for what the standards committee later decided to call the relatively blaze term "Unicode codepoints"--and which are not quite the same thing as characters.
(In fact, I would say that the notion of a Unicode "character" is ambiguous to the point of uselessness--there are glyphs composed from several codepoints (base glyph + combining accents), which should be treated as one "character"; there are ligatures that hold single codepoints, but which semantically are multiple "characters"; there are stacking languages where one "character", representing a whole word, will be composed together from several codepoint "radicals"; while in other ideographic languages, each pre-composed idea-part is its own "character" and has its own codepoint; and so forth.)
The solution is to use Normalization Form C (NFC) (which combines accents with characters).
> there are ligatures that hold single codepoints, but which semantically are multiple "characters"
OK, so use Normalization Form KC (NFKC) (which splits ligatures, and combines accents with characters).
You're right that "length" of a unicode string is very ambiguous. Arguably, you shouldn't be able to call "length" without supplying an argument about what you are actually asking.
This also means your program code is simpler, and therefore, faster.
Function overloading usually means virtual method tables, and therefore indirect method calls. Depending on how deep your inheritance / overloading structure is, these vtables can get really messy.
(I had a class in university where we were given a C++ UML class diagram, and told to draw the vtables that resulted when one instance of a subclass was instantiated.)
PROGRAMMER SEES INTERNAL REPRESENTATION
foo(int a, char b) foo_int_char
foo(int x) foo_int- If you are passing a number of related arguments, often a struct is a better data structure than default or named parameters.
- I am ambivalent about panic/recover, but have no problem with the rest.
- I do tend to agree that "and" and "or" would be more readable, but it's such a minor issue.
- Regarding your UTF-8 example, I imagine the Go authors believed most Go users would be spending more of their time dealing with bytes than characters. Go is not a language optimized for text manipulation, it is optimized for byte manipulation, like C. This is apparent in the lack of effort put toward optimizing the regular expression engine to date.
Is there any particular reason for this? Sets are so fundamental to mathematics and beyond that I am concerned right away about this. Sure you could use a map as a replacement, but what happens to "user experience"?
I do not get this for other languages as well. Data structures that are present in nearly all computer science books are often not built into the language or standard libraries, like trees, graphs. Reasoning given is that some other data structure could serve as a replacement (never mind that the programmer's intentions are not directly reflected in the code anymore), or because it is considered too removed from practical usage, or because some third-party library includes it. I do not buy any of these arguments.
Agreed that such wrappers are then easily built by the programmer himself. But then comes the added pain of including those definitions in everywhere within a project and in-between projects. They become one more dependency to handle.
Then come these wrappers or even whole data structures from third-party libraries. Nearly in every case I see, there is some impedance matching issue with some other libraries, requiring glue-logic to convert from one format to another. [Edit: Plus the added licensing/cost issues for those libraries.]
If a set is just a wrapper around some more generic data structure, it is OK to implement it like that in the library to avoid duplication of code in that library. However, by not making that wrapper, the system is resulting in much more amount of duplicated code in the user space and now with the mentioned impedance mismatch or with weak mapping of programmer's intentions to the code.
Basically there really isn't a general purpose way to make a set and it’s not a fundamental component of programming, it is a modified HashTables or what ever. So I argue sets don’t actually exist in CS because you can’t represent one as it exists mathematically. You are simply tacking union and intersect to an existing data structure.
Most of the time, you would use map[key]value rather than your own hash table or BST.
If you need an implicit linked list, there is one in the standard library: http://golang.org/pkg/container/list/
Sorry to be "that guy" who questions the question, but sorted lists aren't really a great data structure. As you probably already know, insertion is O(N), where N is the total number of elements in the list. It might be better just to use the GoLRB library, which provides some always-sorted tree structures.
Say I implement log, but not log2. If you understand what log is and how to use it, you have log2. If you don't understand this, you ask why log2 is not provided.
It is not just about having it or not having it. It is also about user experience. Why should I write log(x)/log(2) assuming that is a commonly used operation such that the library could have provided me with simply log2(x)?
Just yesterday there was an user experience related article on confirmation vs. undo. The message was that if most of the users are likely to take the action, then undo is better than confirmation, and if most of the users are unlikely to take the action, then a confirmation is better.
While it is bad to provide every option to the user, it is also just as bad to miss the options that the users commonly use.
The real question then is if a set is commonly required or not. I believe it is used often enough.
"Abstractions are useless, lets all just program with cmp and jmp statements"--seriously folks, have we not learned anything?
What I hate is that even after having a good understanding of software development and computer science fundamentals, and having a good picture of the solution to a given problem, I still cannot today program without having to perform multiple Google searches, reading Stack Overflow messages, etc. to deal with what should be trivial stuff.
Just for example, if I were to need a set, I would search for "set" in Go documentation. If the set itself is not included, on the very least the documentation should talk about what should I do instead. But they won't even have that in there. Result: Few more Google searches.
I agree that Go should include a Set type modeled after its List. Sets are simple and there's really just one way to implement them. My own is less than 40 lines; a simple set of tests are about 60 lines.
For trees, graphs, etc. I do think it is fair to establish these as the domain of the community. These are more complex, and there is more variety.
Could just load it into a map[string]interface{} and then just make sure he does type assertion on the value before using it.
In my mind, a map is a set. They are both Associative containers. It's just that the key and value can be different things in a map while in a set, the key and value are the same thing.
Edit: I speak from a strong C++ viewpoint, but maybe Go is not like that (I'm not sure why it would not be): https://en.wikipedia.org/wiki/Associative_containers_%28C%2B...
There are semantic differences between maps and sets and lists. Just because you can use one to represent all the others doesn't remove the benefit of having all 3 available.
Right now, optimizing Python code by replacing critical sections with C works really well and isn't too hard to write or distribute.
How well would tooling around doing the same thing with Go work?
I know the bright team over at bit.ly made a successful switch from python to go-- anyone else?
The more I hear about go, the more I like it.
"We identified early on that Go had all the makings of a language that could supersede some of the places we would have traditionally turned to C and some of the places where we wanted to move away from Python."
The ecosystem has a long way to go (compared to Python), but goroutines and channels are a pleasure to work with. It's nice being statically typed again, and I really like object composition versus Python's inheritance. Resource usage for my cases is much lower, latency is much lower, throughput is much higher, and my deploy/provisioning scripts are a lot more simple due to Go's static compilation.
At my day job, we may start mixing in Go for background tasks that are a bit too slow/inefficient in our current Python stack. I could also see it being a good fit for some of our more high traffic HTTP APIs.
"The code you write with Go just seems to be correct."
e.g.
- Having to always check errors (or at least explicitly ignore them) * By the way, do this anyway. It's really nice when every error in your app is handled. Since I've started with Go for a personal project, I've used a similar approach in my sigh PHP project so every error now has an explanation and suggestion.
-Can’t have variables/packages that aren’t used so to test simple things requires sometimes commenting out lines
-Python is more forgiving. You can take slices of strings using indexes that are out of range and it won’t complain. You can take negative slices – not Go.
Actually multi processing has a reasonable implementation.
Who would adopt a NEW google-ism at this point?
Is there a defined standard library that you can count on for the next decade?
Maybe in 15 years when Go has outgrown its lock-in it may be worth considering. At this point signing on with google is like playing russian roulette.
Really??? How long did you look?
I call bullshit; you did not look. GCC is an alternative compiler. Go away troll.
I'll gladly go away, fanboy.
You are either a troll or an idiot. Actually, I'm leaning towards both.
So you have the source code for the compiler and source code for the standard library and the source code for any package you might decide to use.
Now obviously you'd need to build a pilot to test all of these those components.
But other than the risk of that pilot not working as expected, where else is there a risk and how is that different to any other technology?