On Learning Rust and Go: Migrating Away from Python
blog.liw.fi
blog.liw.fi
Python has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense. I'm no huge fan of Python, I use it when I have to but Rust is from a bells-and-whistles point of view a step down from Python, it is more a better 'C' than a better Python and Go has it's own share of issues that has kept me away from it so far.
If you want to crank out some research project or application quickly building on the libraries that others provide (especially in machine learning or other data intensive fields) then Python would be my first choice, and something like 'R' probably my second (though I see 'R' mostly as a data scientist's workbench that has a programming language bolted on).
\*More than one, of course. But I use only PyCharm :)
While you can do reflection/code generation even in Java, for the non-framework code almost nobody does it.
That's why I feel lot safer with large codebases in Java, even though I love some dynamic languages too. I don't think they're as scalable for big programs though.
I support the notion that Python is unsuitable for larger codebases with all my heart.
I can write reflection based Java code that will make automatic renaming choke.
There are ifs and buts. They may be uncommon, but they're around.
* dependency injection errors,
* template errors,
* magic annotation malfunctions, etc.
The last one in particular actually seems less safe than Python's decorator mechanism. Java splits it in two parts, so that the annotation classes used at compile time are separate from the library that will process them to actually do something. If the latter part is somehow broken, the code will just do nothing, so you don't even get a runtime error.
In Python, decorators are just functions that serve as annotations as well as doing the actual work. There is usually no separate processing part that can go missing. I've never encountered similar problems in Python.
...you're doing it wrong.
It sounds like your loss of interest may be correlated with your level of discomfort with increased cognitive load.... the alternative explanation would be strange: you can only be interested in simpler problems/implementations. (Which I doubt)
I build things from scratch. By the time a project hits that size, I’m utterly bored with it. It’s been figured out and built. It’s been refactored. It’s been tuned in places that need performance improvement. It’s had new features added in. But fundamentally, the idea(s) behind the software have become completely boring. 5 years of working on the same thing through multiple iterations—including reducing LOC while expanding features. There’s a certain project size that I’ve found correlated with time ticking my internal interest clock to 0.
It’s a bit like how Professor Farnsworth feels when there are no more questions.
Edit: I’m not very good at describing my feelings. Delight was a poor word choice. For me, 15K is typically early in a project, when there is still delight to be found in figuring out the problem space. The unanswered questions are delightful, interesting, thought-provoking. Bringing Professor Farnsworth into it, i was reminded of him saying it was the questions that drive him.
There's your problem then! You don't write a codebase that size in such a way that you need to keep it all in your head at once. The whole point of 'getting organized' is to reduce the amount of code that is in scope for you to have to understand. If it's more than a page or so then you will always end up with hard to debug problems. Reducing the amount of code in scope is one of the most powerful tools you have to deal with large codebases.
The one thing that stands out to me is that it's heavy on the unit tests and light on integration tests which would make an application like this (at least for me) more difficult to handle. I'd go heavier on integration testing and lighter on unit testing.
1 - When working with a large enough code base that you don't remember exactly what every function you have written needs, or using third party libraries, function signatures lacking type specifications make it harder to know what is expected. if there is documentation that is good, you can of course load that up, but this slows you down. IDEs today can usually show you the function signature as you type it, but with dynamic types you get less information.
2 - When refactoring something in a statically typed language, changing a data structure or type name will cause the compiler to error every single place that you need to adjust for your change. With a dynamic language you need to rely on unit tests or you will be finding places you missed during runtime for a long time afterwards. If you do rely on unit tests for this kind of thing, then you have sort of built an effectively statically typed environment, at some large degree of effort, with worse runtime performance to show for it.
I understand some of the workarounds people use to deal with these problem but a lot of them seem to push you to the point that you might as well use static typing and get better runtime performance for it.
Either some weird support library hasn't been upgraded to 3.x yet or some expensive piece of software/hardware can't be accessed with Python 3.x
> Mypy mostly seems immature, buggy ...
That was my experience some time ago as well, but it's getting better from release to release. Look at the repo [0], it's continuously improving. I find Mypy deserves more recognition.
The same guys working on mypyc [1], which I think is very interesting too.
[0]: https://github.com/python/mypy/graphs/contributors [1]: https://github.com/mypyc/mypyc
If your project is just _your_ project, then dynamic languages let you start faster and produce quicker. Apart from anything else, these languages attract people who value productivity, so clunky tools don't survive in those ecosystems.
Of course, almost no successful project stays in the hands of just one or two people: new developers come in, or it's Open Source and more people show up, as well as the "drive-by" contributions.
IME, Ruby on Rails projects scale to about three core developers before you start to feel pain. The strict conventions of the framework help mitigate the communications issues of keeping people in sync. Beyond that, the implicit knowledge that is not in the codebase is harder and harder to scale.
This is almost satirical.
> Assembler has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense.
I can definitely see someone replacing python with any X language.
Just because you can do something, doesn't mean you should. I personally like opinionated languages that have made some decisions for you based on best practices and research.
I didn't read it that way - but your retort reads as quite definitely snarky.
Either way -- could you condescend to explain to us what was so outright ludicrous in the sentence you quoted?
The parent I was replying to said this:
> Python has all the bells and whistles you need to build large code bases,
but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense
I have a problem with this statement which I hoped to convey by satire (but obviously failed).
The argument being put forth by OP is that python's lack of static compile time typing makes it hard to build large code bases. This is a legitimate argument supported by several whitepaper research papers conducted by well known and trusted entities. The parent I replied to countered this with "but you do need to get organized", but neglected to specify any details about the said organization.
Obviously I have a problem with this because the statement is utterly meaningless.
You can make this type of statement about literally every language. I picked assembler to demonstrate this point. Also, it is low effort jab against Java programmers, as if they are clueless about organization.
> Assembler has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense.
That statement is equally applies here. Assembler obviously has all the bells and whistles you need to build a large code base, given that it's turing complete and that people have in fact built large code bases with it.
So in summary, the parent I was responding to:
1. Made a vague statement that can be applied to any language.
2. Made a low effort snarky remark about Java developers without providing any support/proof.
3. Said you need to get organized in order to write large code bases in python, provided no further details on the organization.
4. Contained no useful information at all that can actually be applied and used to write software in python or any other language.
5. Tried to counter a legitimate, research supported position with a vague meaningless statement.
I think given all that the satire was warranted.
IMO Python is still the best language to go with when you are writing anything "scripty":
- Stuff that's not expected to be long. Do one thing well, let some other program do another, compose the higher order functions from functions that work. Of course this isn't always possible.
- Stuff where the types are not going to confuse you. Often you just have a few types, and it's fairly obvious what you can do with a given object. If you get too many types, there's a good chance it will start to get confusing and then strong typing would help.
- Stuff where performance is not a problem. Don't write your HFT system in Python. You simulation though, might benefit from orchestration of some VMs.
The ultimate use case for Python is glue code. And guess what, a lot of code is just glue. A good glue language has a large ecosystem of libs, and some language features that make it easy to read. That's Python in a nutshell: whitespace formatting makes things easy to read, and you get built in list/dict syntax to keep everything short and sweet.
For me Rust and Go are interesting having come from C++. There's a lot of pitfalls in C++ that might be helped with Rust. At the moment I have a whole load of Valgrind suite tests that look for potential problems. Perhaps Rust can bake those into the language? I haven't had time to look at it yet.
Go seems to be interesting as a compromise between easy and fast, and by the sound of it things have turned out well. I have people on my team using it, and it seems to be the thing to do for web services that need to be faster than nodejs but you don't want to spend a load of time making sure a c++ version is correct.
Note that Python is actually a descendant of the ABC programming language developed to be human-friendly, not a shell replacement language per se.
[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
I prefer it to Korn and Bourne because it's compatibility is so widespread.
For this reason, the safest thing to do is write code for either the original Bourne shell, or ksh. Programs written in ksh are the most portable across systems, moreso since other systems have it too. And to top it all off, ksh sets the standard for POSIX, especially ksh93.
Shell portability issues are a thing. Also, why all the snark?
"If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date."
Python brings a ton of dependencies, thereby seriously limiting your deployment options. And, together with Python's not-so-great package management story, this spells a lot of trouble for as long as Python 2 and 3 need to coexist. Many useful Python 2 programs (such as Apple's old caldav server based on twisted) won't be ported over to 3, and the migration to Python 3 creates an unnecessary burden on apps that chose Python as user-scripting language, such as Blender. There might be reasons to use Python, but the aesthetics (or lack thereof) of a scripting language, or an artificial limit on the number of lines most definitely isn't one of them.
So he’s basically comparing 20+ years of Python usage with marketing material from Go and Rust. Right.
I wish people could be honest with their motivations, instead of making up excuses to justify this sort of change. Here it’s a classic case of “I got bored and I don’t like the new features, want new shiny”, wrapped in a bit of whining about problems that have well-known solutions (use a good IDE, use type hints or one of the many libs that enforce types...). I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.
Python is still one of the best ways to get stuff done, quickly and with a minimum of boilerplate. Its limitations are well-known and have not really increased for about 15 years - if anything they have shrunk in many, many areas. Adoption rates keep exploding, despite a lot of doom forecasted by some alpha geeks about 10 years ago: “There is no mobile story! We will all be using Ruby/Go/NodeJS tomorrow! Nobody will ever go to Py3!” Etc etc etc.
You got bored by it? That’s fine, it happens, no harm done. But you don’t need to convince anyone, including yourself, that you’re moving on because of this or that flimsy excuse.
This seems to be a rather shallow reading of the blogpost, and the way you phrased your reaction is not exactly polite either. If anything, the "adoption rates" of Python (like those of ECMAscript, a language of a similar vintage all-things-considered) are most easily explained by non-technical factors, so I have quite a bit of trouble making sense of your comment.
Python's typing story is far from perfect and rather annoying compared to other languages. Python is good for interacting with the operating system, networking and other things.
But wanting a proper compiler is an honest motivation.
R is a strang thing to me, though it has its moments (graphing). I still think pandas which gives python R like data frames is one of the great tools out there.
The sentiment that R is easier than python to install/manage is common among some R users but I disagree. With R I'm constantly facing dependency hell problems -- one package wants an old version of R, while another needs the newest. Conda/venv solves this problem very nicely in python.
Recently lI've been using Julia more and really like it. One nonstandard case where I've found it really shines is parsing large bioinformatics data, such as pileup files. Python is just so slowww here, but neither do I want to write a C program to do the text parsing. Julia is perfect in this case.
Ignore that reality if you want to, but it is a fact.
Big complicated python projects are seldom pure python, they are usually a friendly python frontend to a serious application written in something else.
It seems in no way remarkable that someone wanting to build a serious backend type piece of functionality would pick another language that was, just for example, multithreaded.
The point is that Python is the entry point for large majority of data scientists currently and its absolutely disingenuous to try to dispute that reality.
The key word here is "users". Python is fantastic for users. It allows users to get things done without having to worry about types or memory or the underlying hardware in any way.
I think Python made some really good choices with their types from UX that lead to you having to care more about your logic than the machine. Only one integer type, for example! For most users it’s perfect UX.
However, I think (possibly due to the early time when a Python was made) some of the decisions made practical but regrettable trade offs. Duck typing is ultimately just type inference with strong performance penalties and really weak static tooling, for example. _Many_ static type systems are still very far from the user’s actual domain, but the core techniques don’t have to be.
If you program in Python for performance-critical applications, as I do, then you will frequently hand-write C-level extension modules, write extension modules in Cython, use JIT tools like numba.
All of this stuff “is Python.”
For the 1% work that is actually engineering the model's deployment in production, it's not uncommon to use faster languages with the exported model.
If what you're doing is writing a vanilla CRUD server that gets thousands or hundreds of thousands of hits a second, Python is probably a bad choice.
Even a mere 1,000 req/sec adds up to 86.4M per day. That sounds like something at least as popular as Stack Overflow, for example. How many web apps are there, really, that have millions of users?
On the other hands, there's various APIs and tracking type of services that probably do handle well in excess of that, being run by not-large companies none of us have ever heard of.
If you need easy-to-use tools, R libraries cover significantly more ground than Python. You don't have to worry about 2/3 compatibility of niche libraries. Plus the functional language forces the libraries to have similar API's. You also get to use tidyverse and ggplot2 in all of their majestic beauty. At most you have to write a "%>% do() %>%" to get around weird library API choices.
However if you are in the frontier, you could avoid so much awkward notation with an array-first language like Matlab or Julia. You have "^" instead of "np.linalg.matrix_power", "/","\" instead of "np.linalg.lstsq", the . operator and so much more life-saving tools.
Personally I find type systems a lot more annoying because they lead to more verbose code and more cognitive load. There are studies that show bugs being proportional to the number of lines of code, regardless of language. And that effect seems to be more significant than the difference among dynamic and static type systems.
Defensive programming can be achieved through various means. Type checking is one of them and often can't replace Unit and Integration testing. Validation of Inputs or other data is another issue.
Other traditional languages are adopting type inference, etc. We could be near an inflection point where we get statically type languages that are as convenient as Python.
The popular statically typed languages which are actually Python's competitors are for example C++, C#, Java and the like.
Sure as hell they are more verbose. Even Typescript was extremely intimidating to me when I started to try and understand Typescript code bases.
And I am NOT going to try and teach Haskell or Ocaml as a first programming language...
I love Haskell computing model, but holy hell, why is so difficult using things like State Monad or even a trivial thing like a global counter [1] ? Why something so simple should be so difficult? Don't say it's because functional programming, Scheme/Racket is not that difficult!
Haskell and OCaml has their own problems, I agreed with you that both languages are more expressive but the code density is not necessary a good thing neither.
[1]: https://github.com/scotty-web/scotty/blob/master/examples/gl...
counter :: IORef Int
counter = unsafePerformIO (newIORef 0)
usesGlobalState :: IO ()
usesGlobalState = do
count <- readIORef counter
putStrLn (“the counter was “ ++ show count)
modifyIORef (+1) counter
Anecdotally, I find it incredibly rare that such a thing is desirable, when you could instead create it in an IO monad and pass it through to functions that need it. But, it’s there when necessary. And making this kind of anti pattern hard is a strength of these languages, not a weakness.
Of course they won’t replace unit or integration tests, but they will replace a lot of damn trivial tests you have to write to make sure you exercise all paths of the program, because there are no types to help you.
Also, at least when I think of defensive programming, I don’t think of how Haskell et al. allow me to more clearly express my intents in the type system. I think of the times where I write JS/TS and cannot rely on any data anywhere being in the format it says it’s in.
Have you forgotten the cognitive load of "what does this return and what can I call or access on it?"
> There are studies that show bugs being proportional to the number of lines of code, regardless of language
But that value isn't the same across all languages, only the trend.
To me it sounds like circular reasoning, because in my subjective perception, dynamically typed code is a lot easier to understand to begin with. Also, tooling has become a lot better, with and without typecheckers/annotations.
"Despite their importance, whether static type systems influence human software devel- opment capabilities remains an open question. One frequently mentioned argument for static type systems is that they improve the maintainability of software systems—an often used claim for which there is no empirical evidence. This paper describes an experiment which tests whether static type systems improve the maintainability of software systems. The results show rigorous empirical evidence that static type are indeed beneficial to these activities, except for fixing semantic errors."
https://pleiad.cl/papers/2012/kleinschmagerAl-icpc2012.pdf
Follow on:
An empirical study on the impact of static typing on software maintainability
"We further conduct an exploratory analysis of the data in order to understand possible reasons for the effect of type systems on the three kinds of tasks used in this experiment. From the exploratory analysis, we conclude that developers using a dynamic type system tend to look at different files more frequently when doing programming tasks—which is a potential reason for the observed differences in time."
https://link.springer.com/article/10.1007/s10664-013-9289-1
Note that I wrote "evidence", not "proof". Also, there is no robust evidence for safety improvements.
Are there potential confounders? You bet. Are there other aspects that can help? You bet.
I do find that types help me with older code, despite the fact that my preferred languages (Objective-C, Smalltalk), both have keyword syntax that helps tag arguments even if there's no types or documentation. Particularly protocols help me define the interactions that I expect in my system, which otherwise tend to be only defined implicitly through the objects. I'd prefer real connectors.
When I try to understand Smalltalk code, that's a lot of browsers open and digging etc. When I look at some of my older Objective-C frameworks, all the id types do leave me with a bit of head-scratching. A lot of the times the types are trivial to add, and just make things obvious at a glance.
However, I also totally agree that these languages help me write cleaner code, and that enforced type systems get in the way of useful architectural abstractions. Sometimes I need that id.
¯\_(ツ)_/¯
So the argument is analogous to saying "how can I trust the types in this Java program are correct, if it has never been compiled".
Well, whether some "old code base" has them correct or not, just run the type checker and find out.
In my opinion optional type checking is not comparable to statically typed language guaranties. So if I do not need types I would use python. If I need/want type specification I would use statically typed language.
"our central finding is that both static type systems find an important percentage of public bugs: both Flow 0.30 and TypeScript 2.0 successfully detect 15%!"
In fact there have been more than 1 study with the same conclusion.
https://blog.liw.fi/posts/2017/08/13/retiring_obnam/
I can really understand his:
"Obnam has not turned out well, from a maintainability point of view. It seems that every time I try to fix something, I break something else. Usually what breaks is speed or memory use: Obnam gets slower or starts using even more memory."
Also from his current post:
"I could perhaps have been more diligent in how I used Python, and more careful in how I structured my code, but that's my point: a language like Python requires so much self-discipline that at some point it gets too much."
If the issue is one of fundamentals it will only follow them to the next project or language
Really, changing a Python project is not pretty, because speed and memory usage guarantees vary wildly between releases. One factor is that CPython is constantly being rewritten.
And that does not even cover projects that depend on 20 PyPI packages, which introduces practically guaranteed breakage.
Exactly. I have an impression that the programmers as the whole already spent significant time only to develop and maintain different solutions only to manage different Python versions and dependencies in production environments, which were maybe much less needed exactly with some better "discipline" in the Python development itself.
On another side, Python won many hearts over Perl 5 which is much more stable, but "appears to be" harder. Interestingly, Perl actually forces a programmer to "care more" (i.e. be more precise): the reference to something requires different notation than the direct use of something, the use of the array requires different notation than the use of the scalar etc... it actually has "some kind" of typing enforced by the language and explicit in every line. In my personal experience, I have much more "trust" in a 'biggish' Perl program than in a that big Python program to behave "exactly how I'd expect it" (e.g. in the sense that Perl doesn't produce as easy "exceptions" and whoever wrote it had to think more about the correctness than for Python, especially if "use strict" was used, which has stronger guarantees about the typos in variable names than Python). Going further, however, C-like (actually Algol-like) compiled languages simply allow much more than the scripting languages, regarding the smallness of what is eventually produced. E.g. Busybox ( https://busybox.net/ ) has to be written in C or something very close to C. Free Pascal http://wiki.freepascal.org/Platform_list is actually also very practical language for many use-cases.
I like Python well enough, have been using it since mid-2001 and was very much a hardcore Python enthusiast for a number of years, but nowadays I prefer languages that encourage immutability and functional programming styles, as I find that they make it much easier to avoid bugs. I've had plenty of Python experiences where bugs were caused because something somewhere else was mutating stuff behind my back, messing up my mostly-unrelated code. Usually this other code was also written by someone else, so I had no way of knowing about it. This kind of thing can be prevented in other languages, but requires a lot of discipline to avoid in Python (and other languages, I'm only picking on Python here since that's the topic of the article). These problems won't follow you to the next language, if the next language is more suited to avoid them.
https://github.com/lukipuki/obnam/blob/master/obnamlib/blob_...
https://github.com/lukipuki/obnam/blob/master/obnamlib/chunk...
https://github.com/lukipuki/obnam/blob/master/obnamlib/encry...
https://github.com/lukipuki/obnam/blob/master/obnamlib/forge...
https://github.com/lukipuki/obnam/blob/master/obnamlib/hooks...
https://github.com/lukipuki/obnam/blob/master/obnamlib/lockm...
https://github.com/lukipuki/obnam/blob/master/obnamlib/metad...
https://github.com/lukipuki/obnam/blob/master/obnamlib/obj_s...
https://github.com/lukipuki/obnam/blob/master/obnamlib/pathn...
https://github.com/lukipuki/obnam/blob/master/obnamlib/plugi...
https://github.com/lukipuki/obnam/blob/master/obnamlib/repo_...
https://github.com/lukipuki/obnam/blob/master/obnamlib/sizep...
https://github.com/lukipuki/obnam/blob/master/obnamlib/split...
https://github.com/lukipuki/obnam/blob/master/obnamlib/struc...
https://github.com/lukipuki/obnam/blob/master/obnamlib/vfs_l...
> use type hints or one of the many libs that enforce types...
As if that's pythonic. Heard of duck typing? And those are things you have to do yourself, as additional work and maintenance rather than getting it free because it's a feature of the language. And you don't always have full control over the stack.
Is there an IDE that will guarantee that no runtime errors will take place due to typing?
> Adoption rates keep exploding
Because python made its way into the rapidly growing machine learning market. Python is great for writing glue code/scripts because it's easy to extend with C. Everyone and their grandma getting into data science use python.
> I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.
Looks like you are just not willing to accept or even recognize any valid criticism of Python. This type of fanaticism towards the language is not the first time I am seeing it.
pycharm also uses the type hints that are PART OF THE LANGUAGE (aka pythonic) as of python 3.x and enforces them.
You cannot do with heuristics what you can do with static types.
Why do you assume motivations other than what the author claims? I very much started on Python, but as my career progressed, I got tired of being frustrated with poor library documentation and having no static types, making it extremely difficult to figure out even the parameter types to pass in, especially after it's been a while.
After switching to static typing, things have been a lot less brittle and even faster to develop. It is often said that Python is fast to develop in, but this is really only true for prototyping. Making a program and making a robust python program are entirely different, whereas they're much less separate in a statically typed language.
Also, statically typed does not imply FactoryFactory or no type inference, especially for languages like Rust, Swift and to a certain extent Go as well. The code written in Rust, Swift etc. could often be as elegant & short as one in Python, without loosing the benefits of static typing.
I had read some stuff about Rust and it seemed like a interesting language, mature enough for my applications. I tried it out and came to like it more than I would’ve expected. In fact I tend to use it over python for many things nowadays. One of these things is definitly how the module system works, but I agree with you: if you fuck up in Python with your code structure, the problem is definitly between chair and desk.
However: Rust won’t let you get away for cheap if you don’t think about structure, and this could certainly help some to adopt better practise (which in the end is good for all of us).
Interestingly enough my appreciation for and interest in languages like C and C++ grew a lot, after I had learned Rust. It gave me a good new perspective on C++ and certainly learnd me ton of good patterns and concepts that will definitly end up beeing useful in other languages. Rust certainly is well thought out and good ideas are never bad to look at.
I've worked with the opposite who absolutely refuse to ever learn python/go/whatever, producing these awful thousand line bash scripts.
For me it's bit rot. My Python is approaching EOL on this version and needs upgrading to Python 3, and I have a lot of 3rd party code in here that made things simpler to create but over time they've changed enough or disappeared that I now have to take on work there too. Then Django and it's internals have changed a lot in the last 6 years.
My Go code written at the same time (6+ years ago) hasn't had any of it's libraries change so much that I've needed to do anything. Every now and then I pull the next major version of Go and recompile and it's done.
This is all side project stuff but a lot of people use it (monthly average visitors being +250k)... it's non-profit and I don't wish to have to maintain it much.
It worked once, if nothing has changed in the functionality it should keep working... but this isn't as true in non-compiled languages where you need to depend on so much of the environment, let alone build culture that doesn't fully vendor dependencies.
I like Python, but prefer to write code and move forward with other projects.
As to Rust vs Go. I code in both and only picked Go here because a lot of the code in another project I worked on can be used here. I'd probably try Rust if I was starting from scratch now. But I'd be a little concerned bit rot would affect Rust too, it's still quite early for the wider ecosystem.
It's perfectly possible to build a python application that is extremely reliably tied to its dependencies, you just have to avoid the default python package "management" solutions that get shoved down everyone's throat (pip, virtualenv, pipfile, tox and the like). I've been building python projects on top of Nix for years and have managed to maintain stable and reliable results.
And good look finding replacements for Python/Django packages in Go or Rust.
Don't get me wrong. You'll find a lot of libraries for those languages. But probably not everything...
They explicitly said they don't want to rewrite. It looks like they have some existing projects in Go that work in a self-contained way and upgrade with minimal effort.
9 years later, still works.
Go is mediocre as a a language, but it has one big edge - if you're doing web backend stuff, the libraries for that are the ones used internally by Google. So they're being executed billions of times per second on Google servers and tend to be well behaved. Python tends to have multiple libraries for basic functions like the database interface, all with different bugs. If you write web backend stuff in Go, it will probably work.
Having suffered through the Python 2 to 3 transition, I'm less of a Python fan than I used to be. At some point, they need to stop adding features. Currently, Python is acquiring unchecked "type hints", which look like a terrible idea. Optional typing, sure, but unchecked type hints? If the compiler enforced them, they might be worth something. (And no, running a separate type checking tool is not the answer. Especially since no such tool exists that's production grade and in sync with the language.)
Rust has no idea when to stop adding features. Rust started as an imperative language and then morphed into a semi-functional language. Mostly to get a sane error handling model. The borrow checker was a brilliant innovation, though. I used Rust for a while, but quit about two years ago in disgust. I never want to hear "that only compiles in nightly" again.
I don't understand how you can reconcile these beliefs. They stabilized too many features, but you also want more nightly-only features to be stabilized?
It's my experience as a Rust programmer since pre-1.0 that very few libraries require nightly; they tend to be niche things (e.g. Rocket) by authors that prefer to experiment on the bleeding edge of the language over having stability or many users.
All it takes is that one library you want to use requires Nightly. Since Rust adds features all the time, it's not uncommon for library authors to start using these features before they're in Stable. It's just the nature of a young language with an active community who likes being on the bleeding edge.
This also extends to the toolchain; it's not that long ago that RLS and Rustfmt were still Nightly-only, for example.
The presence of Nightly alone gives you a choice that usually doesn't exist with other languages. Not too long ago I went down the rabbit hole with some text stream stuff (Unicode is surprisingly poorly supported in places), and ended up with a temporary workaround because the std::io::Chars API wasn't in Stable yet and I just didn't want to deal with Nightly. But it was tempting. With other languages such as Go, the choice generally isn't there, even though you could theoretically download HEAD and go from there.
This is a classic case of our standards of success continually moving, and that some domains are better suited to stable Rust than others. For example, if you want ergonomic async/await, then you need to use nightly. But otherwise, the number of crates needing nightly has shrunk enormously over the last few years.
> it's not uncommon for library authors to start using these features before they're in Stable
I would say this is actually pretty uncommon.
So are Java and C++ programs at Google. golang is very mediocre for webdev, if anything, it can be used for small network tool kind of stuff. For anything that needs to be more complex, Java would be a better fit (e.g. golang doesn't have annotations that can be used for validations or automatic mapping between different internal and external types, which come in very handy and prevent bugs).
What does "production grade" mean w/r/t such a tool?
After working on a quite large application and having to use lots of assert(isinstance(arg, type) at the beginning of almost every function, I began to think that a strong type system is very much needed for large projects. I believe this was one of the reasons why Twitter moved from Ruby to Scala.
I still love to use Python for hacking something quick for myself. But I also look at some popular strongly-typed languages now and hope to get better at one of them soon.
Python's optional typing could also be used to get rid of a lot of your assertions.
You’re confusing static/dynamic and strong/weak. Python is a strongly typed language, but also is dynamicly typed language.
Need proof? Try doing this:
a = 3
b = “3”
c = a + b
> having to use lots of assert(isinstance(arg, type))
Python has type hinting now, try using a current version of Python and this isn’t needed.
Type hinting does not guarantee to catch everything, but for Python proponents 95% is apparently "good enough".
For example, it doesn't have a built-in way to make a dictionary-whose-keys-must-be-integers (so with your example above if you wrote thedictionary.get(b) rather than thedictionary.get(a) you'd get an error rather than None).
I've often seen learners have trouble with this sort of thing (you read a number from a file, neglect to call int(), and get mysterious lookup failures).
Technically, dict.get() returns None if the item isn't found and you don't provide a different value for its default.
thedictionary[b] would throw a KeyError on the other hand.
One could always subclass dict if they wanted to ensure the keys were always integers or whatever.
> I've often seen learners have trouble with this sort of thing (you read a number from a file, neglect to call int(), and get mysterious lookup failures).
Not sure if you're arguing for a stronger type system or having the language promote the string to an int as (I've heard) other scripting languages do.
Though, once you figure out the language doesn't automagically parse files for you you're well on the way to finding more fun and exciting ways to get python to throw error messages at you.
It's definitely a neat feature when writing an application from scratch that interfaces with few other modules or only the well-tested ones.
The most irritating thing I've found is trying to write a function that accepts either a path or the raw contents of a file.
Python 3 makes this easy, you can check against string. Python 2 of course treats everything as a byte string.
In the end I just check the length of the input if the interpret identifies as python 2. If it's a short byte string, I assume it's a file name. Otherwise the minimum expected file size is almost certainly larger than the maximum path length.
high_level_function(src, ...)
...
mid_level_function(src, ...)
...
mid_level_function(src, ...)
low_level_function(src, ...)
...
low_level_function(src, ...)
...
Another case is where there are two or more parameters that you want to be generic in this sense: if your only option is duplicating the function you can end up having to define inconveniently many variants.
https://pandas.pydata.org/pandas-docs/stable/reference/api/p...
Python should not be written like this; assertions can be ignored at runtime: https://docs.python.org/3/using/cmdline.html#cmdoption-o
A much better solution is to use qualified try/except blocks to handle potential errors.
For personal projects (and past consulting work) I have been loving Lisp languages since the 1980s but many languages have so many of the benefits of Lisp (thinking of Haskell and Julia) and there is definitely a ‘lisp tax’ on deploying and shipping products.
Anyway, for writing new books, writing open source projects, and perhaps some consulting, I would like to cut back to a single language, instead of the 4 or 5 I use now.
EDIT: I did some Go Lang codelabs when I worked at Google and have played with Rust (principally spending some evenings with an open source blockchain project) and neither language will be my home base language.
And enjoy your opportunity to work with whatever languages you like. The variety of tools available today is so much greater than it was in the 80s.
Elixir is perfect for servers and fault-tolerance. Also scales quite well to several tens of machines before needing any special tooling.
For a huge percent of most projects today Elixir would be ideal.
OCaml I want to use more for its really concise and strict (but extensible) syntax. And mostly for its strong typing and really well-optimized compiled native code.
Congratulations on retirement, relax and tinker at your own pace. ^_^
Anyone else agree with this view ? Programming languages should be choosen based on technical merits, rather than who is behind it. Is there a lesson from history that I am missing ?
This can be both a blessing or a curse. Google seems to be on the good side, but I’ve seen plenty of languages which are frustrated by its steward.
Go is quite unique in that it produces very independent binaries. This might make software written in it quite future-proof.
Personally I like Python's philosophy and the way the community is run. I like the trajectory. I understand the trade-offs. And I am sure as hell not going to teach Go or Rust as a first programming language to people who aren't computer science students. And even then...
What do you mean by "it produces very independent binaries."? Do you mean that Go implements system calls itself, instead using corresponding wrappers from the OS's underlying C library?
I read something like that recently, but haven't looked into the point yet.
Absolutely not. Technical merits are only part of the considerations.
Big companies, for example, where the stakes are higher, never chose languages on their technical merits alone. The also look at the owner/stewardship of the language, and in many case, roll their own languages, to avoid being at their mercy.
So it’s not a completely fair comparison.
It got fully open sourced?
Considering the motivations of the language owner is a valid concern. Many people, for example, jumped on to HHVM as a "faster PHP". Then it started branching away from PHP[1]. Predictable, if you considered Facebook's reasons for having it.
[1] https://hhvm.com/blog/2018/09/12/end-of-php-support-future-o...
[1] https://en.wikipedia.org/wiki/List_of_Java_virtual_machines#...
Rust is quite the opposite of Go in that regard - just like C or C++, it makes it really easy to write over-abstracted and clever code that is hard to understand, audit or review.
Rust is also an extremely powerful tool and a ray of hope for systems programming, but comparing it to Go is an apples and oranges comparison.
To me the main advantage of it is that the small number of primitives and simple syntax forces people to in general be very explicit about what you’re doing. The code almost doesn’t need comments.
That has little to do with 'simple syntax', and more with the language itself (and/or its community) setting clear standards as to what "idiomatic" code in the language should look like. And Rust is fairly good at that, albeit with the inherent limitation of having to support the additional complexity of ownership/lifetime management. (Go can do without this since it uses GC instead. Swift just uses ARC everywhere.)
I've seen people copy protobuf definitions from repo to repo because they couldn't get them building consistently from a different directory, never mind issues with different versions of the go protoc plugin.
If you are writing code that must work, you should be either formally prove the system (if feasible) or have a huge test suite covering everything.
While true, it's a big benefit to eliminate classes of known bugs (e.g. memory safe languages eliminate undefined behaviors, and Rust eliminates data races). That allows the programmer to pay attention to other aspects of the code, since there's only so much energy one person can spend.
You are talking about systems that SHOULD work, but aren’t required to always work. Typically, safety-critical software is the latter, the rest of software is the former. But the latter are entirely different beasts, and is what we were discussing.
By the way, Rust does not eliminate all race conditions; and memory safe languages may still have undefined behavior unrelated to memory (e.g. Rust)
I find it a pleasure to organize code in python.
What am I missing here?
Obviously this doesn’t replace the need for functional tests but writing those tests is as much prone to user error as writing the code itself. So having a compiler check for stupidity up front is always going to be a bonus.
Also the static type system helps a lot too. Having dynamic types can be awfully convenient at times and I completely relate to why some people like them but static types means your structures and classes effectively have written specifications defined in the code itself which the compiler verifies the rest of your program against. So if you’re accidentally trying to write text to a numeric type, your compiler will catch that.
In my experience the extra up front overhead (in terms of development time) of static typed AOT compiled languages end up saving you development time in the long run when working against larger projects.
Humans error. I’ve seen numerous occasions when stuff that should have been added into the tests wasn’t. Even from engineers who are normally thorough. I’ve also seen cases when competent engineers have written test cases that pass when they should have failed. Heck even I’ve done that and I’ve been writing software for 30 years.
The more a languages build tools can catch developer errors, the less you need to rely on developers remembering to check for them and thus the more time developers can focus on the project at hand.
If your devs are pushing code into production that doesn't pass the tests (or they aren't even running the test suite) then you're asking for trouble no matter what language you use.
As much as I love reading 20+ pages of errors c++ likes to spit out when you use a template wrong if I want to do something simple I just reach for python.
Clearly that’s not even remotely what I said. But I’ve posted the same thing twice already. I see no point posting it a 3rd time.
Sure, yeah, if you want to ensure type safety then use a type safe language...argument over.
...unless you want to do something like duck punch some function into someone else's library and you really don't want to rearchitech how the whole thing works because of that one special case which just doesn't quite work.
That said, I’ve been on projects debugging code that has had functions overridden like that and it wasn’t fun. Sometimes people can get too clever for their own good. But you can have that problem regardless of the language
In my experience, if I'm writing relatively decent tests and sprinkling of sanity checks these types of errors get caught quickly and are very obvious.
I can sympathize with somebody who has largely worked with untested statically typed projects and has moved to untested dynamically typed projects though - this is where the benefit of compiler type checking might seem to be most valuable.
I can also sympathize with somebody working with a dynamically typed language with a bad type system (cough javascript).
>In my experience the extra up front overhead (in terms of development time) of static typed AOT compiled languages end up saving you development time in the long run
Yeah, mine doesn't.
I do agree that tests are important; hence why I raised that point from the outset. But ultimately in Python you’re still having to add extra tests just to catch up with the default behaviour of Go and Rust. Which is the point I was raising. Go and Rust are better at catching stupid user error.
Clearly though you can write large projects in Python if you’re disciplined enough. Same could be said for C, BBC BASIC or even assembly. The question isn’t whether you can, it’s whether other languages makes it easier or not.
You don't though. The tests you'd write which would catch these "stupid user errors" are basic ones which you'd be remiss not to write in any language.
The fact that frequently developers do choose not to write any tests at all on large code bases does mean that they start to rely heavily on compiler catching 'stupid user errors' but it doesn't have to be that way.
>Go and Rust are better at catching stupid user error.
...if you don't have a basic level of tests, yes.
(although rust does provide certain type level protection against, say, data race bugs which python doesn't).
Positive being that the code does what you expect under normal operation. Those you’d obviously need in any language
Negative tests are, amongst other things, checking that your functions behave correctly when you put garbage in. If your compiler flat out refuses to send a string type to a function that is designed to accept an integer then you’re already covering a whole class of bugs without needing to write specific tests for it.
The straw man of my argument living in your head flip flopped. I did no such thing.
When I said "basic tests" I meant demonstrate the behavior under normal circumstances". If you're doing TDD on 60% of your code base you'll write enough of these.
>Negative tests are, amongst other things, checking that your functions behave correctly when you put garbage in.
This is a completely unsuitable use case for a test. If your function expects non-garbage it should raise an exception if it gets garbage with a sanity check.
I wrote one just a few hours ago like so:
assert no_nans(dataframe)The point of functional tests is they catch developer errors before your codebase hits production. Which you cannot guarantee if your error checking is only in the runtime.
Disclaimer: I write services used by millions of people (literally) each day. Some of those services are written in Python.
I agree. I said this too
> and the ones you shouldn't be writing (if you do, simply switch to another language).
That was my point
On other hand dynamic languages give you flexibility and speed when developing. In some cases that is worth a lot more then losses you suffer from them.
Having said that Rust is just terrible application language. It was built for a purpose use it for that purpose.
From typed perspective from what I have used recently Swift is probably my favourite.
Here comes the catch, libraries, community support all tramp the hell out of almost any language feature.
There is a reason we write metric ton of C++ for everything and basically everything Python has C++ and C under the hood. C++ has stuff to do it all, it has been tested polished, tested some more documented and works. Now even Pytorch has first class C++ API.. It’s about time!
Java traumatized me as a young engineer fresh out of school. I can’t bring myself to even look at it these days. I am sure it is much better now. I just can’t do it.
You'd be surprised.
Also, writing self-referential data-structures is very difficult in Rust (but not impossible) due to the type system. I suspect that a lot of developers who just want to get work done get turned off by fighting with the typechecker, and choose GoLang instead.
And also: GoLang is a very accessible language, that can be mastered in a couple of days.
Not really true (at least, not nearly to the same extent) since Pin<> was stabilized. I suppose that only serves to prove your point about Rust still being "in research phase", though!
Other things like the situation with async/coroutines is still it great in rust.
Go is, syntactically, a much simpler language than rust.
It all depends on what you need, what you want, and where then community around it is (e.g. Docker came first and was written in Go).
This is because what you call "systems" is not the "systems" that the lack of GC would benefit from. For all intents and purposes Kubernetes could be written in Python or any other general purpose programming language (and for example OpenStack is in fact written in Python).
On the other hand, systems like "writing an operating system kernel" or "running on a microcontroller" are things where GC might indeed be problematic and/or impractical.
Now I am not arguing one way or another. People can just have opinions or they can just write software or they can do both.
Kubernetes may as well have been written in Python. It's mainly IO bound, so the benefits of compilation as opposed to running on an interpreted VM (like Python) aren't that big.
Now that a lot of my scraping needs require me to render the DOM, I have had to run a headless web browser (previously PhantomJS/Splash and now Chrome). Chrome released the Puppeteer library which is the most straight forward way to interact with Headless Chrome. This is in Node so that is what i’m using.
On my last project we needed to parse through a large amount of HTML files to extract relevant information (about 70,000,000) and put it in a DB for further analyses. I have done similar things in Python before, but there are blocking/concurrency issues that can make it slow and frustrating to build.
My solution was to just do this in Go. I can then run my extraction in a goroutine. Go’s selling point for me was speed (relative to Python) and dead simple concurrency - which made it the right tool for the job.
Edit: Spelling, iOS’s autocorrect can be so annoying sometimes :)
Any idea if Go has an equivalent to Pythons Beautiful Soup? :)
Back when I used to do Python development, Beautiful Soup was awesome for extracting info from possibly-munged HTML. Haven't yet found a good equivalent for Go. :/
For HTML that's at least mostly valid, I like goquery: https://github.com/PuerkitoBio/goquery
Go went with simplicity to a fault. No generics (use code generation instead), just one way to make a loop, garbage collection.
Rust on the other hand has all the bells and whistles a language could have combined with the ability to micro manage every bit of memory you're using.
People tend to lean either way.
Go is meant to be a higher level server side language that replaces languages like php, python or Java on the server. It's simple so it's almost as fast to compile as a scripting language is.
I really enjoy the author's blog, which has been discussed here many times over the years https://hn.algolia.com/?query=liw.fi
https://www.reddit.com/r/rust/comments/acjcbp/rust_2019_beat...
And at least in startup-world which is the bulk of my career, the speed at which you can build a reasonably robust prototype can be the life or death of a business, and this prototype usually ends up being the production product anyway. Prototypes are seldom rewritten "properly".
I’ve found so many unexpected runtime errors in Haskell, behavioral bugs that require deep, stateful dependency injection to properly test for, type class designs that were set up wrong for some use case last year and now are horribly brittle and unmanageable to try to use it for the new use case that needs customer feedback immediately.
Plus, for all the syntactical niceties of Haskell, the code becomes unreadable the instant you start needing a dozen different GHC compiler pragmas all defining weird special behavior or syntax. Yet it’s just basic Haskell 101 these days to require new compiler pragmas for tons of basic functionality that you could get out of the box in other languages. Getting a whole team up to speed on how to mentally keep track of that from one section of code to the next is a very difficult chore.
My real life experience with Haskell is why I don’t use it professionally any more.
A few things are interesting to me here:
1. How we've managed to come to such different conclusions despite supposedly having similar experiences
2. The vigour with which you attacked my rather innocuous comment
3. I had a look at some of your previous comments about Haskell, and you seem to repeat the same talking points over and over, almost verbatim. In fact, this comment you made is particularly interesting:
> it becomes the same shit code mess as any other paradigm.[0]
If what you say is true, then what is your argument exactly? If everything inevitably produces the "same shit code mess", then why are you attacking me for essentially saying "I like Haskell and it is working for me"?
I do attest to this same criticism of Haskell from time to time because I believe it’s important to refute the very frequent, baseless claims that are often made on Hacker News that claim that functional programming or static typing can make any difference on the reliability or rate of defects in software: they cannot and do not.
I’m not at all trying to dissuade anyone else from using Haskell if they want to use it, and in fact I’d even go so far as to say Haskell is my favorite programming language.
As long as everyone checks the wild claims at the door, and we all admit that there is no difference in terms of safety, robustness, performance, reliability, readability, etc., between Haskell, Scala, Rust, OCaml, Python, Java, Go, Erlang, C++, C#, F#, C, FORTRAN, etc.
None of them is conceptually superior or dominant across a wide range of use cases. It’s just as easy to get hard-to-fix errors in all those languages, just as easy to categorically prevent other classes of errors, just as easy to design for compositional or extensible interfaces, etc.
As long as everyone is essentially willing to accept a No Free Lunch theorem about programming languages, then I don’t care what language people like to use or happen to prefer for personal reasons. I’m not attacking any such preferences.
But when there are unqualified comments about some paradigm or language (like functional programming, or Rust, or Julia, or Haskell, or static typing) being strictly better than some alternatives (like OOP or Python or C), then it’s worthwhile to counter that claim by pointing out that no, in fact, all those paradigms easily lead to the same problems and in no way offer any type of widely applicable solution to the usual problems of developing software.
You were recently warned by a moderator to stop making sarcastic and unsubstantive comments[0], so I don't think it's me who has trouble reading your tone.
> I believe it’s important to refute the very frequent, baseless claims that are often made on Hacker News that claim that functional programming or static typing can make any difference on the reliability or rate of defects in software: they cannot and do not.
Any comments I have made regarding the benefits of FP and static typing are anecdotal, and are therefore not baseless.
There's also anecdata like this[1], which further refutes the point you're repeatedly trying to make.
> and we all admit that there is no difference in terms of safety, robustness, performance, reliability, readability, etc.
That doesn't match my experience. My experience has been that it is cheaper for me to run my businesses on Haskell and Elm, as opposed to Ruby and [plain] JavaScript. I've tried both approaches, and I perceive a difference.
> As long as everyone is essentially willing to accept a No Free Lunch theorem about programming languages, then I don’t care what language people like to use or happen to prefer for personal reasons.
I'll accept there's no free lunch — my technology choices aren't a panacea. They do make a difference to my business however, and as a business owner I am perfectly within my rights to make that judgement call.
> I’m not attacking any such preferences.
You certainly seem to be.
[0]: https://news.ycombinator.com/item?id=19241069 [1]: https://twitter.com/rtfeldman/status/961051166783213570?lang...
I'm zachaysan@gmail.com if so. If you're not (or too busy, etc) that's totally understandable. Don't feel like you need to explicitly decline.
For his ends, statically-typed languages have benefits over Python:
Obnam is about 15000 lines of Python code. That turned out to be too much for me in Python: too often there were bugs that a static, strong type system would have caught.
> After all python is still an interpreted scripting language.
Is it? Or rather, is this even a well-defined category?
> interpreted
Can't any (OK, almost any) language be either interpreted or compiled? The fact that python is typically run in an interpretive runtime setting doesn't really make it "interpreted" as a language. In fact, since python is nearly always compiled to byte code at runtime, it's arguable just as compiled as it is interpreted.
But that's not even the point: there's nothing lingual about python that stops it from being compiled. It can be compared with other languages that are typical run subsequent to a compilation step without being the apple to an orange.
> scripting
This one speaks to me even less. For python to properly be understood as a "scripting" language, I think you'd have to convince me that the lion's share of community energy is dedicated to a paradigm of running a single module front-to-back as a script.
And this is clearly not so.
A better description would be “high level JIT compiled language”.
> there's nothing lingual about python that stops it from being compiled
Are you sure? Python is a very dynamic language. Some of the most dynamic parts of python might not be possible to compile fully ahead of time and might need to be JITed, because the behavior of the code will only be possible to determine at runtime.
We could say that the typical compiled language is used for scripting Unix syscalls :-)
In the broad swath of languages, go and python are more similar than, say, python and Haskell.
What makes languages apples and oranges is generally not their compilation method but things that are more approach-centric or radically different syntax (which is why I left Rust out. Its syntax has been the thing that's slowed me down most there)
We've found Go meets our needs better than Node.js for writing and debugging large programs. The languages are very different. Both are used for small and large programs. The experience in using them is what makes us choose one over the other.
We already have a largish (couple hundred thousand line) Node.js app written in Typescript and are looking towards other alternatives for the future product as I am not very happy with the quality of Node.js ecosystem.
Stuff goes in, stuff comes out. You don't need to wonder if something outside of the scope magically comes around and changes things in the middle.
Also Go's concurrency pattern with goroutines and channels fits really well to my mental models of how stuff tends to work in my field.
1. we don't force our customers to lay down a large Node.js blob just to run our software.
2. The old code was written for an old version of Node.js, v4.x, I believe. As such, it was full of code suffering from callback hell, and a lot of anonymous functions written inline. It was very difficult to reason about when tracking down bugs. I'm sure your Typescript code is far better organized.
3. The edit-compile-test loop is far faster. It takes less than a minute (20-50 seconds) to compile the new codebase, which is a mix of Go and C apps and libraries. The developers use a simple build script to rebuild everything from scratch. The old codebase, a mix of Javascript and C apps and libraries, was built with CMake to manage dependencies. Each rebuild would complete in a few minutes for small changes, but could take about 10 minutes if the "right" code was touched.
4. Like theshrike79, I find Go's concurrency patterns, channels, and goroutines a delight to use.
5. We use json files for configuration. Go has a simple scheme for converting between json and structs. Node.js allowed us to make poorly structured json files. When we converted them to follow some simple nesting patterns of structs within structs, converting between structs in memory and json files was just too easy!
It seems that the new codebase is significantly smaller, so it's easier to find where things are implemented. Part of that is getting rid of tech debt and being very familiar with the new code.
When we were still in beta, we ran some performance tests. They indicated the new code was as performant as the old node code (which has been worked on for years). We expect to be able to improve on that as we have time.
I hope that helps. It's been a major amount of work to rewrite our apps while maintaining the old code base - not something to do on a whim. I know the dev team is delighted with the results. Good luck.
As soon as I read that, I know immediately that I'm dealing with an object-oriented programmer. Always the same gripes, always the same problems! Your programming paradigm is busted and no language will ever be able to change that, no matter what lexicography and grammar it implements! There is a reason why procedural or functional programmers don't complain about those types of issues!
HN users need to follow those regardless of how bad an article is or they feel it is.
You can do 'cargo check' to do the analysis part without actually compiling a binary, it's super fast and excellent for development.
Considering clang generates LLVM IR and compiles much more quickly than Rust, this argument doesn't hold. The Rust compiler generates IR which chokes LLVM and makes it slow, but they didn't have to do that.
It’s a huge problem with a number of different causes.
Steve, this is a cop out. There's nothing in Rust that can't be implemented in C or any other Turing complete language, including stack unwinding, invoke, or anything else. It may be inconvenient for Rust to use or not-use some feature of LLVM, but that's not the same thing.
Sure, I can technically use Assembler to write a modern Windows program, but that's no argument against high-level languages.
Being able to implement something does not mean that by implementing it a different way, it will certainly be faster. And that's what we're talking about here, compilation speed, not computational equivalence.
We find tons of bugs in LLVM due to using certain features (like noalias) that are used a lot by Rust and very, very little by current C or C++ codebases. That shouldn't be particularly surprising. Nor should it be surprising that LLVM is mostly optimized for making C and C++ compilation times fast, and not necessarily other languages.
Again, as I said, it's a complex, multi-faceted problem. You can't just say "well, Zig compiles to LLVM, and it compiles fast, checkmate." That's not how engineering works.
(And, note that FastISel is "This is a fast-path instruction selection class that generates poor code and doesn't support illegal types or non-trivial lowering, but runs quickly." So, the thing I'm talking about is a thing that would help debug build times, but not production build times. I'm not claiming that this one specific thing is the root cause of poor compile times, in fact, I'm claiming the opposite: there are a number of different things that all come together to lead to compile times being not as fast as we'd like.)
> Being able to implement something does not mean that by implementing it a different way, it will certainly be faster.
Of course not, but if I point to implementing something which is equivalent but faster, you can't say it's impossible to do it fast any more.
> You can't just say "well, Zig compiles to LLVM, and it compiles fast, checkmate." That's not how engineering works.
Yeah, actually I can. The claim is that LLVM is slow, so it's not Rust's fault. But LLVM isn't slow when compiling a language with nearly equivalent features, so that's a poor excuse. That's how engineering, logic, and accountability work. There is nothing in Rust which doesn't have an analog in C++, and C++ compilers aren't as slow as the Rust compiler.
> note that FastISel is "This is a fast-path instruction selection class that generates poor code and doesn't support illegal types or non-trivial lowering, but runs quickly."
You brought FastISel into the discussion. This is just another part of the shell game. Compare clang++, g++, and rustc with optimizations on or off.
I never claimed this. I claimed that there are a number of interlocking issues that make Rust compilation slow, and pointed out one that I haven’t seen a ton of discussion about before as an example of how it’s not a simple problem.
I also started this conversation by acknowledging one of the things that you said, which was that we generate a lot of LLVM-IR. That’s one that does get talked about a lot.
I don’t think we’re really gonna make any progress here so I’ll leave it at that.
No, the person I first replied to did. Then you jumped in and defended his position with your "yes and no" comment.
> I don’t think we’re really gonna make any progress here so I’ll leave it at that.
I agree. I always appreciate your friendliness and thoughtfulness, but I've never gotten you or anyone else on the Rust team to budge an inch on the problems with Rust. (which is a shame, because it's close to being a great language...)
It seems we’re in full agreement about the problems: compile times are too slow! Everyone acknowledges this. It seems like our disagreement is about the why.
I believe the original poster meant generated code when (s)he wrote about compiler performance; the compile time is rather irrelevant for something that gets done once; what's relevant is the speed of the generated code which then runs the next billion times (or more).
So let's get back to the performance of the code generated by the Rust compiler, shall we?
https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/fortran.html
I'm not going to beat myself in C when I calculate a 100 million by 100 million matrix in Fortran. ifort will generate SIMD code with CPU autodetection and use machine code seqencing best for that processor, plus in Fortran a matrix is a normal variable type like any other. I'm not beating that with C (and I really know C).
And just so you know, my information isn't old, I come from supercomputing.
Random anecdotes are meaningless. I've looked at plenty of generated code, and I've written plenty of SIMD assembly too. If you can't measure it, you can't compare it or improve it. There's nothing to discuss.
If there is nothing to discuss and you plain refuse to do what I told you, why reply? Until you do exactly what I told you, there is nothing to see here. I have zero tolerance for people who won't listen. Move along, move along.
"Move along"? You're very full of yourself, and I think that explains why you aren't as smart as you think you are.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
Where and to whom do I send the bill?
If you wish to contribute then please follow the instructions; if you don't wish to contribute then have a nice day.
> So let's get back to the performance of the code generated by the Rust compiler, shall we?
"Shall we" just stick our heads in the sand too? No, you don't fix problems by ignoring them or pretending they aren't a problem. Slow compilation speed is a problem if you care about developer productivity and quality of life.