Goodbye Python, Hello Go
thinkfaster.co
thinkfaster.co
This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.
Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.
I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.
Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.
Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.
There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.
Another upside of a single binary is that it's not easy to modify; you can even sign it. Quite an upside if you want to audit what you deploy to your production that handles millions of dollars.
GOOS=linux GOARCH=386 go build foo && scp foo user@server:
That's a lot of time saved, especially when you do it on a daily basis, and not something you can do as easily with virtualenv and such.Usage would be
python foo/setup.py bdist_xar && scp foo/dist/foo.xar user@server
if my understanding is correct.
Personally I'm not a fan but I can totally see the appeal.
I also found this comparison of libraries pretty good: https://tech.townsourced.com/post/embedding-static-files-in-...
[1] https://github.com/shurcooL/vfsgen
[2] https://github.com/shurcooL/httpfs/blob/master/html/vfstempl...
Good times.
You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.
Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.
If you're on some other operating system, yeah, using python sucks for you.
While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.
Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.
In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.
Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.
And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.
The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.
Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
> pypi via pip, and especially into a virtualenv
I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
> Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
If you write software that will be run by others (which usually means open source, probably libraries), yes. If you write software that will only be run by you (pretty much all dynamic websites a.k.a. webapps land in this category), you don't want to have three different distributions in half a dozen different versions anyway, so you can pin yourself to the target environment just as well.
Sadly, there are a community of devs who want to rely on libflakey 0.0.1-beta, released an hour ago, and think that waiting for APIs to stabilise & distros to securely package things is unprofessional (!).
Raw npm-style clone-from-master seems to be prolific, though.
Yeah, my primary software development mode is 1) gather opensource software, 2) build web app or platform-specific app.
> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).
It's crazy that this even has to be said in this day and age. I suppose distro specific patches to maintain a secure and stable api has gone away in favor of some vendored static dependencies that may or may not ever be upgraded, and rebasing your dependencies introduces the same set of problems.
I foresee dynamically linked go with platform specific binary packaging becoming the future. Recompiling the same bits of software ad infinitum is probably going to get old.
Houdini is a fairly ubiquitous tool for creating dynamics; fire, water, destruction, dust and has been around since 1996. In general, Houdini users tend to prefer Ubuntu and most medium to large vfx houses use CentOS/RHEL (Pixar, DreamWorks, ILM).
For Linux and macOS, Houdini uses your system's Python but it has to monkey patch parts of the standard library in order to work smoothly. You can set a flag (and on Windows this is the default) where it will use the Python shipped with the application.
At my current job we're using slightly out of date versions of everything, which is quite common in production. We're running CentOS 7.2, Houdini 15.5 (released Nov 2017), and Python 2.7.5. If you call "httplib.HTTPSConnection('google.com')" it throws an exception because of ssl library changes across 2.7. I haven't tried any other versions, but the requirements for the current version of Houdini says "CentOS 6+ (64-bit)" (among other OSes).
This isn't unique to Python, but it's very hard to properly support real-world clients and dynamic libraries among operating systems, commercial applications, and less well funded open source or independent efforts.
My M.O. is if a technology is core to your business, you should control it. Which means not using the OS' Python. Upgrading Python and OS independently makes so many things much easier when you have a large codebase.
If the users of your software are end-user desktop types, I can see your sentiment. My software is typically for building software (aka, libraries) or users that know how to compile things.
Nuitka (nuitka.net) has been able to compile python programs to a single stand alone executable reliably, robustly and easily for a while now.
It's a real gem and deserves to be more popular. We are very far from the quirks of pyinstaller or cxfreeze : it just works, even with numpy or qt.
Yes, go is still superior for deployment :
- it can cross compile
- executables are smaller
- it's built in so it's just an easier experience
- it doesn't depend on libc
But it's important to spread the word: the time were your python program was hard to ship is over.
Just use nuitka.
> To be safe, you need to carry the vm and other dependencies with you.
You should make it easy as possible for users of your software .
The first point about inconsistent spelling mistakes, I 100% sympathise. I suffer from this terribly. Pylint and syntastic are a silver bullet for me. I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
Parallelism: I do look over the wall at Go and wish that python 3.x had spent more time on removing the GIL, but that is old news now.
However I enjoy threading with python(not its limitations), because its so trivially easy. The Queues primitive saves 90% of the headache. as for ctrl-c not doing anything set all your threads to daemon (<threadObj>.setDaemon(True)) When the main thread exits, so does everything else. Go shines by having built in true parallelism
If I need horsepower, then I have to run multiple processes (not multiprocessing, that has drawbacks.) which has the annoyance of serializing and message passing. (but that does mean scaling horizontally less painful further on). Again, Go really shines here.
Statically compiling thing: YES. This means that I can 90% of the faff with docker (xyz version not working any more, the build fails) and use a real scheduling system that runs on real steel(or AWS) and not the omnifaff that is kubernetes.
Numba on the other hand has a few nice use cases (where "nice" means close to Pareto-efficient improvement), but I tend to just be using PyPy if I'm going with JIT approach. If for some reason I had to use CPython but had a few bottleneck functions that could be sped up with Numba I'd strongly consider it.
I'm happy to report that JetBrains' VIM plugin is almost flawless (Block-based editing, window splitting, save/close, it's all where it's supposed to be).
I used to be in the same camp. But if you add a linter and parser and full blown backend to make lookups for code completion to your VIM configuration you might as well use an IDE. I've switched to IntelliJ for Java.
It dumps numpy arrays in /dev/shm which can loaded by other processes by the filepath, muuuuuch nicer than passing things around
Not even if X is better than Y for every specific task?
#!/usr/bin/env stack
-- stack script --resolver=lts-12.2
and run it as an executable if your machine has stack installed. If not -- compile it with a `stack ghc --resolver=lts-12.2 ./file.hs` and you'll have a `./file` executable. No project creation, no need to list the dependencies.Yeah, it's very much on those lines, though the syntax is a bit more ML-style (Rust pushes a bit towards a C-like syntax).
(I actually mostly use Scala myself these days, but I know the JVM is a dealbreaker for some people, and the design compromises for the sake of Java compatibility mean Scala is probably not the best choice for a first ML)
I really like OCaml, and you should learn it, but you should also know that concurrency is not a strength of it. It's being worked on...
(Tl;dr: It will be officially merged in 2019.)
I hope that it does get merged soon, people have been waiting quite a while, in my understanding. It’s a tough problem.
To give some context: When it comes to serious programs I use Go already, but often I just want to code a little script to watch a website for changes for example:
#!/usr/bin/env bash
url="https://example.com"
mem="memory.txt"
oldsum="$(cat "$mem")"
newsum="$(curl -sL "$url" | sha512sum)"
if [ "$oldsum" != "$newsum" ]; then
xmpp_notify "$url changed"
echo "$newsum" > "$mem"
fi
I haven't tried it yet, but in my mind it feels like writing such a program would be more time consuming/complex in Go than writing it in bash. Neverthelss, I feel like Go is overall a much better language than Bash (with all those quotes), so I would like to transition to writing more short programs in Go and for that I would like to learn how to do things like reading/writing/grepping files as easily in Go as I do it in Bash. Any idea if someone wrote some hints together already?Saying "Go is overall a much better language" is like saying a wrench is a better tool than a screwdriver. You should keep both in your toolbelt and switch out when the need arises.
For your question, start with the regexp and os package, learn how to open a file, read in the contents, and then while iterating over the contents, apply a regex to the text. That should get you to the point that you can search plaintext for character strings. Good luck.
Just a few months ago I replaced an almost 100-line Go "script" with two lines in the Makefile. The fact that Go is great for big projects doesn't automatically make it good for everything. Go is for programs, not scripts.
My problems come from the other end: My Bash scripts tend to become larger. A few weeks ago I wrote a 'cache' utility function (very useful to cache curl requests for example). A few days ago I wanted to use it in a second script was just about to write a package manager for bash modules... I think that is the kind of situation you are getting into when the line count of a bash script increases and you are trying to do proper software development ;-)
So I am not going to build every script in Go from today on, but I would like to learn a bit more on when Go is a viable alternative to Bash.
func main() {
url := "https://example.com"
mem := "memory.txt"
oldsum, err := ioutil.ReadFile(mem)
if err != nil {
log.Fatal(err)
}
// Download URL
resp, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
// Hash it
newsum := sha512.Sum512(body)
if oldsum != newsum {
err = exec.Command("xmpp_notify", url + " changed").Run()
if err != nil {
log.Warn(err)
}
err = ioutil.Write(mem, newsum)
if err != nil {
log.Error(err)
}
}
}
Don't listen to the naysayers. Go is fantastic for scripts like this - it's really good at network stuff, has a ton of built in functions, etc. And you get actual robustness and error checking that isn't a complete joke. Look how many ways your Bash script could fail! That's fine if you're literally watching the output with your eyes. If not it's going to cause issues when `curl` fails for example.The only downside really is that you have to compile the Go code, which is fast but not as fast as just interpreting a Bash script (the first time).
Another possible issue is executable size - if you have a lot of Go scripts they may take up a fair bit of space.
I wish Go had an interpreter / JIT mode for scripts.
Edit: Before someone says "but that's 3 times longer than the bash code!" please write the equivalent Bash code that has proper error checking.
trap "exit 1" ERR set -euo pipefailYou might be thinking "but... exceptions!". They are a pretty bad solution. The problems are:
* Error handling is hidden - it's impossible in most implementations to know which exceptions can be thrown by a function * Flow control context is lost. If you wrap lots of lines with try {} you don't know which one caused the error and can't add context information. You end up with shit error messages like "file not found" rather than "error opening config file foo.cfg: file not found". The solution is to wrap every line in a separate try/catch but that is just Go-style error handling but shitter and even more verbose.
Go is a good balance. If you want proper error handling you have to actually write it. Yes it is a little tedious but that's what it takes to do it properly.
If you don't want to do it properly then fine, use Bash and set -e. But accept that it will be shit.
I think for this small size of an example program the error checking in Go is pure overhead, but as soon as the program starts to have a few more functions it will be very useful to have more options than just to abort on fail (Bash trap style / set -e). So I don't have any problem with 'polluting' my source with `if err != nil` checks ;-)
What I am more concerned about are the different ways I have to call functions from different libraries like
resp, err := http.Get(url)
...
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
That is a lot more 'special' than the other function calls and the bash version and probably requires more documentation reading than '$ curl --help'. But hey, thanks to your example I have a starting point and probably are going to experiment with it in the new future.For all the others who wrote that Bash is the right tool for the job: I read your warnings, opinions and suggestions and will keep them in mind while exploring the possibilities of using Go for script like jobs.
I'd love to see some future bash like language that's strongly typed, has a consistent and simple syntax, but retains these advantages.
#!/usr/bin/env stack
-- stack script --resolver=lts-12.2
import System.Process (system)
import Control.Lens
import Crypto.Hash.SHA256
import qualified Data.Bytestring.Lazy as B
import Network.Wreq
url = "https://example.com"
mem = "memory.txt"
main = do
oldsum <- B.readFile mem
newsum <- (hashLazy . view responseBody) <$> get url
when (oldsum /== newsum) $ do
exitCode <- system $ "xmpp_notify " ++ url ++ " changed"
B.writeFile mem newsum
I didn't make use of it, but there's also the Turtle library which aims to provide a fairly solid shell scripting experience, providing a lot of coreutils as simple Haskell functions. You also get a REPL in the form of GHCI, which comes default in Haskell installations.Hmm, but those IDEs actually do solve most of the problems author has had with Python. If one regularly code in Python, spending a few hours configuring the IDE is justifiable. I for example never write scripts outside the IDE, because why?
They can use heuristics to get fairly okish. Pycharm is not bad and VSCode's new Python extension seems to work fairly well. But it's never going to be as rock solid as Java code completion for example.
If you need an IDE to solve common pain points of a language, isn't it fair to say that another language which lacks those pain points is a better fit for you? Using an IDE to aid you, to smooth things out and allow for more rapid development and better workflow is one thing. Being forced to use one because the language is lacking is a bad look on the language!
- I make stupid mistakes in Python constantly. I misname a variable or a function or I pass in the wrong arguments: Try mypy
- If the task is IO bound: Threads are the wrong the answer to that. You're going async by using Golang, why not in python?
- With Python, I have to ensure all the packages I need are installed on the remote machine: pipenv Like slezyr mentions
- Consistent Styling: Try autopep8 or black
- Intellisense: I would say "use Pycharm" but vscode is getting better I heard
Prospector does a better job and it prevents you from wasting time with what essentially are stylistic choices and BS like a hard 80 chars limit per line (remember, A Foolish Consistency is the Hobgoblin of Little Minds)
And yes, yes, automated stilling tools are better
Intelisense is needed but even the standard vim autocompletion plugin does an ok job
In Golang, you're still using a thread-based abstraction.
In Python, "going async" involves switching to a slightly different style of programming and using different I/O libraries. While that has advantages, using threads lets you stick with the standard style.
One minor plug `pyflakes` is very good for basic checks for mistyped variable and missing imports, etc. It's not "smart" and doesn't check type consistency, but very useful to catch stupid mistakes.
pipenv
This this this, a million times this. I work mostly in Clojure where such problems are equally as bad, maybe even more so. Dynamic, elegant languages can feel liberating in so many ways, but there are significant workflow drawbacks that get me nearly every day.
It's a tradeoff for using an expressive dynamic language, I guess.
But they don't use the tools created to catch these kinds of errors, and complain that they aren't protected against these errors? There's a fairly large disconnect there. Hell, I have protection against this built into my VIM install. Not running these basic tools is tying a hand behind your back.
https://github.com/facebookincubator/xar/
Then just build your XAR executable with:
$ python setup.py bdist_xar
I recently started using python as the primary language, the more I use, the more I like it.
Go's put everything in a static binary has its pros and cons.
This bores me, probably because I find everything to be the exact opposite (except for parallelism, I'd be foolish to pretend Go isn't better in that area) but specially because I find Go to be an ugly language to write. The error handling is only the top of the can of worms.
Python, in comparison, is a delight to write and specially to read (when it's well written).
In short: Oh, terrific, another one.
And the HN discussion https://news.ycombinator.com/item?id=10402307
numpy - https://github.com/gonum/gonum matplotlib - https://github.com/gonum/plot pandas - https://github.com/kniren/gota (API not finalized)
Pylint solves the undefined variable issue, but it is annoying as hell and always requires configuration. I wish there was a linter for python that didn't care about 80 character limit and was careful not to drown important errors with unimportant stylistic tics.
flake8 is a linter that doesn't need much config, and you can config it to give you only "serious" errors and not stylistic issues.
I gave a talk on these subjects last year at the Paris Open Source Summit: https://speakerdeck.com/sfermigier/python-quality-engineerin... (NB: Black didn't exist yet at the time).
Ideally instead of needing to be told whether to only show "serious" errors, for instance, it should just give you a project "score" (sum of the errors x their severity) and show the top 20 or so errors in order of importance and by default it should never exit with error code 1 with just stylistic errors unless explicitly configured to do so.
Or rather, I'd vaguely heard the name but I erroneously thought it was just another flake8 type thing.
(Nothing against golang, just wanted to point out it's not that easy for every usecase).
> Unit tests would catch most of these, but it’s hard to get 100% code coverage, and I don’t want to spend time writing unit tests for a one-off script.
Sorry, I actually do like golang but this is telling of a bigger issue. Someone else is going to have to delete/fix it