Rust – A hard decision pays off | Hacker News Reader

Rust – A hard decision pays off

pinecone.io

474 pointsmuizelaar367 comments

This doesn't surprise me and matches my own experience. Rust literally makes a codebase nearly void of most bug classes with the exception of logic bugs (Unfortunately, in a huge codebase, there can still be tons and tons of logic bugs).

Still, when I migrated my Python codebase to Rust I got rid of whole classes of bugs and honestly code faster in Rust on a "per debugged line of code" basis. In Python, every line MUST be run or could trigger an exception (and so many do and it is hard to see). This now requires a test harness that tests every single line which is a huge pain. In Rust, the things that can obviously panic are pretty easy to see and are limited (unwrap, expect, indexing, etc.) making it much easier to write robust code, even with a more limited test suite.

That’s why Elixir developers like to make NIF callouts with Rust. It’s dangerous to do anything outside of the BEAM but Rust negates all of those concerns.

Maybe not all concerns, you're still allocating memory outside of BEAM and you still need to be careful to yield control on longer-running NIFs.

Is it possible to use Ada instead of Rust?

I believe you can made a NIF with just about anything.

I wonder if there are any examples though. Would be helpful. I suppose I could just make Ada bindings.

>with the exception of logic bugs

this is one big exception, because logical bugs are most common and most problematic.

If your main concern is security bugs, then according to Microsoft[0] about 70% of their bugs were memory-safety related.

[0]: https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe...

Most people don't write operating systems or write software in C/C++, so memory safety bugs are not big concern.

> ... or write software in C/C++

The article we've all supposedly read (but clearly some have not) is specifically talking about software written in a mix of C/C++ and Python.

I strongly disagree. The vast majority of bugs I see are unhandled None or exceptions and typos in dictionary keys and class attributes.

The unique safety features of Rus are all about memory safety, which is not something you have to worry about at all in Python. I can only assume that the bug classes you are referring to would have been eliminated by using essentially any language with a type system.

I find Rust to have a particularly expressive type system, and a quite strict one to. Not all languages with type systems such have strict enforcement of mutability or null safety.

> I can only assume that the bug classes you are referring to would have been eliminated by using essentially any language with a type system.

Rust has a very expressive type system[0], which you won't otherwise find before hitting the more functional and research-y side of things. Modelling data in terms of enums (sum types), move types (affine types), etc... makes it a lot more reliable and a lot less faillible as it just removes entire swathes of edge cases. See concepts like "making invalid states unrepresentable".

[0] though it's still lacking in many ways, the more you get the more you want after all

Other than Rust the only other "pragmatic" language I can think of that has as much of a robust type system is Typescript. As you say anything else you'd have a hard time convincing the rest of your team to use.

Typescripts type system is anything but robust. It's escape hatch build upon escape hatch. Don't get me wrong, it has to be like that due to it allowing gradual typing and essentially beeing a superset of js.

I now introduce you to Haskell. Welcome to the dark side.

To some extent, possibly, but it is a very rich type system, and does not have null which in and of itself gets rid of a set of errors. I also find Rust culture tends to be more guarded on panicking due to having such a rich error type (Result), so panic is reserved for the worst possible "can't continue" scenarios (unlike languages with unchecked exceptions). Also, enum pattern matching is exhaustive which gets rid of errors where you simply don't account for new variants. The list goes on and on.

> I can only assume that the bug classes you are referring to would have been eliminated by using essentially any language with a type system.

That's not entirely true. As an easy example, Go has a type system but it's relatively anaemic and recreates the "billion dollar mistake" of nil. I have personal experience with Go applications that have broken in production due to nil dereferences, to bugs related to type-switching off `interface{}`, and to the language's inability to enforce exhaustive case statements in general.

While the unique safety features in Rust are generally about memory safety, it still takes a much more safety-conscious approach in general than most languages that are statically typed.

Agreed, also consider C++ whose type system uses implicit conversions and mutability by default, leading to a number of non-memory related bugs.

> The unique safety features of Rus are all about memory safety, which is not something you have to worry about at all in Python

Rust gets safety with its ownership system. The ownership system brings strong guarantees around who is mutating / reading any given object.

Python has uncontrolled ownership. Any function can generally mutate / store any variable you pass in. This can definitely cause bugs if a rouge function starts modifying input.

Rust makes this impossible without the mutation “permission” being part of the method signature. Strong ownership controls force functions to be extremely explicit with their intentions

For those not familiar with ownership concepts, it does apply to many things beyond just memory management. Most obvious example of this is file handles. In python you have to close() them or scope within a context manager to auto close when the block exits. If you need more complex logic around, such as transfer the handle into another function, the context manager is no longer sufficient and you quickly loose track of who should call close(), is it the caller or the callee.

Rust keeps track of this, no matter how deep you twist your logic and it does so for every variable, no context manager blocks required.

wrong, rust is about exclusive ptr access, nothing more. you cannot do ReadOnly<T> in rust.

Why not? Isn't that &T like a read only wrapper of T?

try to do ReadOnly<T> yourself, and then put Cell into it, or RwLock, or Mutex, or Atomic, or crossbeam channel, ...

it LOOKS like it's possible but it can be broken easily. again, because rust is not really about (i)mutability, it's about mem-safety - and all of those can be safely mutated with &T

Rust has no exceptions and enforces correct propagation of errors at compile time. There are few if any strong typed languages in current use that offer a similar experience, for this mix of [memory safety + strong types + error correctness + close to machine performance] you would have to go to some very exotic language route.

Unlike those exotic options, Rust also comes out of the box with a very powerful free linter (clippy) which you definitely want to use, and has a wide community, library, cargo ecosystem, is very stable and tested etc.

When you total it up, you get a fast, safe program that exhibits 60-70% less runtime bugs than something written in either Python or C++ for a similar development effort. You still need to test of course, but you can focus on major integration and logic bugs instead.

While the big one for rust borrowing system is memory bugs it als make it extremely specific what can be mutated. And there can only be a single mutator at any time. Unless you explicitly write something to work for multiple mutators. This makes many logic bugs much more obvious even though you could still make them.

And of course you don't have nullability which is another big one.

Due to the pain that concurrency is in Python, the codebases I've seen mostly shared data only via external processes (queues, databases).

I was very surprised to read the article. I think in the code I have in mind the borrow checker would only add overhead.

The most frequent errors I see in the Python webapp code are: 1. logic errors, especially around concurrent DB transactions, 2. type errors (missing values, improperly modeled data), 3. performance problems.

It's not only threads. It's also explicit in whether a function can modify an argument because it's type signature says this with the mut keyword.

You're right. It just doesn't seem to create much issues in the application code I have experience with.

Well the article is about writing a database, which is definitely going to need concurrency and communication via shared memory to deliver the performance that it needs to (which is why the original implementation was mostly C++ and the new one is Rust). But yeah, I think in general if you're running into issues with mutability in Python, you're either using Python for something it should not be used for, or you're just programming in a really weird way.

I find exception handling in Python to be one of its weakest spots. I really like Python and I use it often. But I'm never quite sure which exceptions a library function might throw. This is rarely documented well and so you end up encountering new exceptions at run time, which is exactly when you do not want to be encountering new thrown exceptions.

Usually I end up looking at source of libraries and making a list of all exceptions that can be thrown. But of course this is also terribly error prone.

I think unchecked exceptions in Python probably make sense. I think the real issue is people trying to use Python to write large systems instead of scripts. I love python for 100-500 line scripts, but beyond that the ease of use features get in the way of writing robust code.

However, I think unchecked exceptions are a mistake in any language for writing _serious_ code. Interestingly, everyone hated checked exceptions in java so now everyone makes new exceptions unchecked. I would say this is not because people disliked the idea of ensuring errors were handled, but the verbosity required to do so meaningfully was annoying, so people took shortcuts to avoid pain by wrapping in unchecked exceptions and in the process introduced "exceptions gone wild" (and now everyone who runs a known java app is familiar with the exception stack trace in logs and in the console).

Python definitely considers itself a serious language, not one intended for 500 line scripts. They have libraries supporting server creation, GUI and desktop applications etc.

It's just that in practice, the trade-offs the designers made for easy development made writing large applications next to impossible, unchecked exceptions being one of the major sources of instability. You write very fast buggy code and get excellent productivity initially, but as the program grows you fave a factorial growth of interactions between components, and, because the interface points are not well defined and checked at compile time, what you gained during development is lost under an uncontrolled torrent of subtle runtime bugs.

When I went from Java to Ruby on Rails it was an incredible breath of fresh air, and a massive, massive time saver, and truthfully Ruby can do many things Rails can not, and it worked well.

On a language level though (I still hate java) but the type safety was nice and did save you a lot of worry. In retrospect I realize most of the issue was the frameworks available and all the terrible configuration stuff.

I love that today you have so many more options because for a lot of time you do save time on a per line level..

> In retrospect I realize most of the issue was the frameworks available and all the terrible configuration stuff.

Also that Java’s type system is… not great.

So you have to put in a lot of effort (especially if it was before java 8, to say nothing of Java 5) and you just… don’t get much out of it.

Even more so given the syntactic overhead of defining java types before records (or without Lombok), to say nothing of the performance cost.

> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust.

I don't doubt this in the least. I've been a professional Python developer for 15 years, and I can't believe Python ever had the reputation for "high dev velocity" beyond toy examples. In every real world code base I've worked in, Python has been a strict liability and the promise that you can "just rewrite the slow parts in C/C++/Numpy/etc" has never held (you very often spend at least as much time marshaling the data to the target language format than you gain by processing in the faster language and you have all of the maintainability problems of C/C++/Numpy). Python trades developer velocity for poor runtime performance.

I don't think Rust is the paragon of high dev velocity languages either, but it seems to be more predictable. You don't have something that works as a prototype, but then you run into a bunch of performance problems (which virtually cannot be worked around) when you go to productionize, nor do you get all of the emergent quality and maintainability issues that come about from a dynamic type system (and last I checked, Python's optional static type system was still pretty alpha-grade).

I strongly recommend avoiding Python for new projects. Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling. Use Rust if you're writing really high performance or correctness-is-paramount software. I'm sure there are other decent options as well (I've heard generally good things about TypeScript, but I'd be concerned about its performance even if it is better than Python).

I think Rust deserves a lot of credit for popularizing the notion of “constraints as power”, but excessive enthusiasm around it does seem to obscure the rich traditions of PLT that it has more than liberally borrowed from.

There are more rungs on the ladder.

"Constraints as power" describes the proposition of all statically typed languages. But the compilers of popular languages weren't smart enough to cover many common scenarios, forcing language designers to offer many unsafe escape hatches.

It’s a very valuable thing to be able to separate the mental model one uses from the target machine language.

I worry about the more ardent Rust enthusiasts because it seems popular in those circles to embrace an attitude that we’ve somehow arrived at the right conceptual framework.

Rust is a competent, pragmatic embedding of a few of the big ideas from serious PLT into a well-optimized C++ compiler toolchain, and it’s cool and useful as a result.

But I hope to God it’s not the endgame on fast ML/Haskell/Lisp.

I'd be disappointed if the best language people are writing software in when I retire (perhaps a decade or at most two in the future) is Rust. We can surely do better.

But I'd be even more disappointed if the best language people are writing software in when I retire isn't even Rust.

To be successful a language still has to be approachable to most dev. I just hope you are not conflating PLT with Functional. Look what functionalist did to Scala... It had an interesting balance and they destroyed it.

Interestingly Object Pascal was my introduction to data type modelling for constraints, followed by Caml Light, Modula-3 and Ada.

Naturally one can argue they weren't popular, with exception of Object Pascal.

I say this as someone who has had a lot of criticism for Rust: Rust definitely borrows a lot from other languages, but Rust deserves praise precisely because it curates the right features and composes them in the right way. There are way too many languages from the PLT space that add some neat type system innovation, but they fail on syntax or performance or tooling or etc. That's not particularly laudable IMO, but Rust has great tooling (cargo, rust-analyzer, rustfmt, rustup, etc), great performance, great safety, and a bunch of type system innovations (including its approach to memory management) which were probably picked from the PLT world. The individual features are neat, but Rust is much greater than the sum of those borrowed innovations. Choosing what features to use and which to omit is a much more valuable task than any individual innovation.

Credit where credit is due: Rust has done a far better job than most of free-riding on Haskell’s type class system and clang’s optimizer, but rust-analyzer is a dumpster fire next to clangd, that’s not an example I’d use.

Um what?

Maybe it's just that I don't understand what clangd provides that's so great, but rust-analyzer, despite a few hiccups I've had with it, seems completely amazing at what it does.

Well we could start with the fact that it crashes process IDs more often than Justin Bieber crashes Maseratis: https://github.com/rust-lang/vscode-rust/issues/890 (it’s always LLVM’s fault, I know). Or move right along to the insane configuration hacks that are mandatory if you want anything upmarket of VSCode (who doesn’t need a little practice with their Emacs Lisp), or the N^2 whatever that happens when you point it at bindgen output.

What clangd does is work.

You linked to an issue from over a year ago to support your case? And the solution in that issue was just "switch to the right extension"? That's hardly a compelling argument. By all appearances, that person’s environment was completely broken since they reported the other extension did nothing at all, and a broken environment isn’t really a reflection on the tool. I can set up a broken environment for any tool and then complain about it. The absence of a bunch of other people encountering the same issue in that thread indicates this was not some big issue with rust-analyzer.

If I search for "Rust" in the VS Code Extension Marketplace, it shows the extension you linked to... but it is clearly marked as deprecated, and VS Code won't even let me install it. So, it’s even more of a non-issue today than it was a year ago.

The actual rust-analyzer extension works fine for me. I think Go's tooling is slightly better in my experience, but both are far better than any experience I've had with C++ tooling.

> it crashes process IDs more often than Justin Bieber crashes Maseratis: https://github.com/rust-lang/vscode-rust/issues/890

So -- this guy used an extension (named Rust) with rust-analyzer, a configuration known to not work, it didn't work(!), and the extension author recommends he tries the extension made for rust-analyzer.

Hardly a case for the ages. Guy uses unsupported config and things don't work?

> What clangd does is work.

Don't doubt it. I'm just saying -- I haven't had any problems with the rust-analyzer extension since it became the Rust default. But, yes, I had a few hiccups and crashes beforehand, no doubt. I just have to imagine it's both younger, and doing more/different things than clangd.

The other comments have pointed out this was a simple misconfiguration that was easily fixed.

What I take exception with is "it’s always LLVM’s fault, I know". If you're going to trash other people's hard work in such a glib manner, it behooves you to know the details of what you're talking about. Tell us, why would a language server that doesn't generate any code use LLVM?

This thread is a bit stale but I try to check once or twice a week to tie off loose ends. I appreciate that probably no one will read this, but, you never know.

One of several issues around engaging with the Rust community at least as personified on HN is that it's very difficult to see any other outcome than being backed into a pretty extreme corner: I started my involvement with this thread saying that I like Rust and want to like it more, but found taking a piece of software that seems interesting on its own merits and headlining the Rust part (whether due to the original post authors or the submitters, makes no difference) contributes to a skepticism-inducing sense that there's an agenda other than "cool software" sitting close to "Rust" in the embedding space.

And, if you care to, you can watch me go from "Rust is cool, great stuff written in it" -> "hey some of these claims are seeming extravagant" -> "ok let's not get carried away, the cited examples are mostly Haskell semantics and clang optimization" -> "alright I'm calling bullshit that all the tooling is superior, that's fucking stupid" give or take. It's a pragmatic cool programming language that people write great software in. That's plenty, that's more than most languages ever achieve, and Rust is far from done achieving things.

`rust-analyzer --version` reports: "rust-analyzer 2022-04-11" on my primary Nix shell. `clangd --version` reports "clangd version 13.0.1" on the same env. I drive both of these things on the command line, in CI, and via `emacs` (on `eglot`) every month at a minimum, every week almost certainly, and usually every day.

I have seven nontrivial elisp `defun` and/or `defadvice` (deep into `rustic-mode`) stanzas for keeping `rust-analyzer` working ok. I have to defeat it's naive `Cargo.toml` search in interesting projects, I have to turn off big blocks of my buffer going RED ALERT RED while a brace remains missing which is statistically most of the time, and most importantly, I have to manually restart it with save excursion trickery so that I can survive a crash without blowing my session. Some of these defects may be as old as earlier this year when I last audited all that.

For `clangd` I have zero such nonsense. Which is completely understandable, it's an older and therefore more mature piece of software. That's not a knock on Rust or `rust-analyzer`.

But I made the fatal mistake of posting the wrong link to the issue tracker, for which my reward was to cause the "Flying Fucking HN Rust Brigade Gangbang Comment/Downvote Battalion" to put the Batman light on.

If you folks are comfortable being the "extremists people are too tired to fuck with", then do you. But I know for absolute certain that I am not the only person doing some level of Rust adoption in spite of this bullshit rather than because of it.

I haven't used clangd since the early days, but rust-analyzer hasn't been anything other than reliable for me (which is pretty impressive considering how often Rust changes as a language and how new rust-analyzer is).

I would respectfully submit that time spent pushing your toolchain from more than one direction confers many benefits, of which a realistic appraisal of the strengths and weaknesses of the various tooling options is but one.

".. but rust-analyzer is a dumpster fire next to clangd.."

Sorry, but this is patently false and is just straight-up misinformation. rust-analyzer is an extraordinary piece of work with careful, well-thought out idiomatic software design, easy usage and relatively low productive defect count.

Rust, the language, itself might be rather over-hyped here on HN - its most loved status will drop significantly with larger real-world adoption. However, the rust-analyzer project isn't overhyped and is a clean and elegant product of software engineering. I wish I was good enough to design and code language software like this.

Yeah, specially when people refer to stuff taken from ML as Rust "features".

If people only used Python as originally intended (as a language to write small scripts in), this kind of problem never would have happened.

Sure, but the Python community itself promoted the idea that Python was suitable for general application development and so on. And it probably was a really good option back in the day when people were using C, C++, Perl, and PHPv4. And I'm sure there are still some areas where it's a fine option (maybe heavy data science stuff?). But I would only use it as an absolutely final resort.

Greenfield Python programs that had strict type checking from day 1 can be pretty reasonable, as long as you know you never plan to write heavy data-munging code.

I often work on codebase that involves typed async Python plus a collection of Rust CLI tools that do most of the heavy lifting. And it's really not bad at all.

The only use case that makes sense for Python.

> I've heard generally good things about TypeScript, but I'd be concerned about its performance even if it is better than Python

I wouldn't recommend Typescript for CPU-bound tasks, but for I/O-bound tasks with complicated business logic it's type system is on the order of magnitude better than Go, and makes you able to get really close to the "make invalid states of the system a type error" ideal of functional languages like Haskell, while still being a lot more approachable.

Talking about Haskell and stuff. You what I want? Rescript to replace entirely Typescript. That will be so much better.

Personally, I love ML-family languages and would enjoy writing in Rescript more, indeed.

But I write software not to enjoy the process, but to solve business needs. And using Rescript, Scheme or Haskell drastically limits the talent pool and rises the cost. Which, BTW, can be very good for some businesses, that would want to hire only the best programmers anyway and are ready to pay for it. But that's some, not all.

Yes I agree. Also if you know your business won't need a lot of dev, like stuff like Instagram where I think they were still at 60 engineers even after hitting 1B active users, it can also be a good thing to use a niche (but viable) language to attract some bright engineers and they will be thrilled to be able to have a job to use it. Now I don't have such startup (yet) so I can't do it :)

I just wish ReScript would have gained traction and replaced Typescript. But who knows maybe it can still happen.

I would endorse that.

TypeScripts's type-system soundness holes kill the fun with that language very quickly, imho. That was a really big disappointment for me.

A "static" language that compiles fine and than crashes at runtime with complete WTF-bugs is not much better than a dynamic language. You just can't trust the compiler and need to double check every line of code anyway. So there is not much additional value in this kind of "type safety".

I would prefer Rescript of Scala.js anytime to TypeScript. Got really disillusioned by TS.

I sort of agree, but I also think type systems are somewhat overrated compared to performance, static compilation, native compilation, easy cross compilation, good tooling, expansive ecosystem (esp devoid of C dependencies). I think TS and Go are on par with respect to tooling and ecosystem, but I would rarely trade Go's static, native compilation model (distributing a single static binary is pretty awesome) for some extra type system features.

Specifically, I posit that returns on type system features diminish--there's a ton of value in going from a dynamic type system to Go's pre-generics type system, but every addition thereafter offers decreasing value.

> distributing a single static binary is pretty awesome

It is, but does it really matter when you write backend? I've written and operated Node-based backends at scale, and I can't honestly remember any bugs or outages that would have been prevented by this.

At the other hand, there's a lot of potential bugs and problems that didn't happen because proper usage of TS's type system prevented me from committing them.

> I've written and operated Node-based backends at scale.

I'm interested in producing fullstack typesafe web projects using typescript on both backend and frontend.

My question is how do you scale nodejs since it is single-threaded? Say I need 10k concurrent connections. These connections doesn't have to be all hammering the system at once but I need to accept 10k connections and perform something like 2k requests/s.

How do I approach this? One server with clustered nodejs? pm2?

I ask because there are a ton of outdated info on how to scale nodejs.

Some rudimentary benchmarking with clustered or pm2 nodejs tells me that 95 response times go from 30ms to 1000ms once I hammer the server. Whereas Go and C# keep it under 100ms without any effort or extra tooling, just using standard library.

I'm not claiming a single static binary is going to prevent bugs or outages. It's just a surprisingly nice property (though less-so on the backend). For example, because Go distributes as a single static binary, we can make scratch images that weigh in at only a couple of megabytes. This (alongside fast compilation) means we can iterate more quickly in the cloud (pulling images onto nodes and starting them is almost instantaneous). This is particularly helpful when you have an outage and you're trying to get things un-stuck (you can try a lot of things in a short amount of time). It's one of those things that aren't obviously valuable until you've tried it.

Again, it's not the end-all-be-all, but it's a surprisingly nice property.

I am launching my startup on Typescript (backend and frontend). So far, it's been an amazing experience - I can define the problem in terms of types even before I start writing any executable code, and refactoring is a breeze if you make your types hard enough. There are mature libraries, you can share code between the two sides, infinite options for PaaS providers who can "just deploy" a typescript codebase. As a solo dev, it's saving me an incredible amount of time to go to market.

I've had experience with large codebases in Python, Java, Go and Javascript before.

Python - as everyone already said, just sucks. Refactoring is a nightmare, you have to test every single line of code for any sort of dependability and that makes refactoring even harder. It allows for too much meta-programming and within 3-6 months your codebase is legacy.

Java is just not fun to write for me, but Kotlin is kinda fun and I could see myself choosing that. The problem is that my product deals with a lot of deeply nested, user-defined objects, and converting JSON (or YAML, or XML) to POJOs is a verbosity nightmare.

I just fucking really hate Go. I can't explain it, but I find the language infuriating. It somehow manages to be less expressive, more verbose and more straight up boring than all the other options.

> I just fucking really hate Go.

As a Go fanboy, this made me chuckle. :)

> I can't explain it, but I find the language infuriating. It somehow manages to be less expressive, more verbose and more straight up boring than all the other options.

I sort of get it. I think people fixate too much on the for loops and the `if err != nil` boilerplate, but there's definitely some validity with respect to "how to properly annotate errors" and emulating enums (in the Rust sense of the word) is pretty error prone and it still doesn't get you exhaustive pattern matching.

The stuff I like about Go:

* Single static, native binaries - being able to just send someone an executable is pretty nice, not needing to make sure people have the right libs and runtime installed is phenomenal.

* Great tooling. I love that the Go tool takes a list of dependencies. Unlike Gradle/CMake/etc I don't have to script my own build tool in a crumby/slow imperative DSL.

* Compilation speed. Go compiles really fast.

* Small language, easy learning curve. Any programmer can read just about any Go project with very little experience. Any Go programmer can jump into any other Go project and be productive immediately (no extensive configuration of IDE or build tools or anything like that). You can also write surprisingly abstract code without reaching for generics, but you will likely have to change your programming habits.

* Value types done right. Most other mainstream languages don't have them, and the ones that do very often implement them poorly (e.g., IIRC in C# you can define a type as a struct or a class and only structs can be value types, and classes are more idiomatic).

* Go eschews inheritance and prefers data-oriented programming over object-oriented programming (although OOP is poorly defined and some people think it means "anything with dot-method syntax").

* I'm just more productive in Go than any other language

Stuff I don't like about Go:

* No enums

* I wish there were fewer tradeoffs between abstraction and performance

* Needs a better system for annotating errors

* Doesn't run on embedded systems (although I've heard good things about TinyGo)

* Doesn't make me feel as clever (this is a nice property for professional software development, but for hobbies not so much)

On balance, I think Go is the better language, but I can also understand why it chafes people. Different strokes. :)

I am still fairly early in my career. I've worked mostly in Java, but my current job is a mix of Python for E2E testing, PHP and JS for application work. Go is my preferred language for tools.

My favorite language is by far Java with Go a close second. I dislike the others. The only thing I wish Go had is more functional support, but only in the style of Java's fluent API. I strongly dislike PHP, JS, and Python functional style and would almost prefer they not exist.

I on the other hand like the fact that Go keeps things simple and primitive.

Don't need those functional constructs at all, allthough I'm using them quite often in Typescript.

But for Go - please no. Go is like C. Adding things just makes it worse.

> I can't explain it, but I find the language infuriating. It somehow manages to be less expressive, more verbose and more straight up boring than all the other options.

I don't hate it like you, I just find it amazingly backwards, and I think I can explain why:

The community has the same persistence as certain teachers when it comes to locking things down and making things harder than necessary for no good reason, all for our own good.

Prime example: announcing a supposedly cutting edge language without generics in 2009, launching version 1.0 without it in 2012, 8 years after notoriously conservative Java got it, then waiting until 2022 before listening, that is some hardcore teacher mentality IMO.

Yeah, honestly I would respect them more if they never added generics. Still wouldn't use it. I haven't tried go-with-generics but I can't imagine it can be a good addition so late in the game.

Was absolutely a brilliant addition to Java.

I haven't used Go for years but it was sorely lacking last time.

Are you sure you aren't suffering from the Stockholm Syndrome?

> I can define the problem in terms of types even before I start writing any executable code

This is how code is supposed to be written. Sort out the logic first, don't write the actual implementation details until later (models, http handlers, file parsers).

The support for this type of SOLID design is much more common in Go than in Typescript projects which often seem to inherit bad practices from Javascript developers.

We have similar experiences. I absolutely love Typescript. But I don’t like Node very much. So I keep TS in the browser.

I’ve been looking at Nim or Scala for the backend. It’s either that, or just dive into Rust.

It really seems if you don’t want to use Node, Java, or Go the available choices for a statically typed backend get quite slim.

What don't you like about nodejs? seems like a solid choice for most things, the only use one language (nodejs/browser) is very attractive (besides SQL).

Look at Kotlin if you haven't. Sure, "it's just Java" but really it's not. A good example is the collections all have an astoundingly uniform API and everything can map into everything else. It's really quite enjoyable.

If you like doing compile time things with types like typescript enables then Nim is fantastic. I've been learning Rust recently and I'm continually disappointed with how little compile time or type based stuff you can do.

With prior C++ experience it's easy to equate C++ templates and Rust generics (similar syntax - usually does similar things - quack like ducks etc.); I know I did. And through this lens generics will soon seem very limited compared to templates. But the underlying reason for this is because C++ templates aren't generics - they're much closer semantically to hygienic, declarative macros. C++ templates are based around rules like SFINAE and allowing multiple alternatives and the compiler will work out which is the most-specific match that works. C++ templates (even with concepts) are unconstricted - when you give a template a type, there aren't any checks enforced by the language about the types; many templates have a list of requirements and you, the programmer, have to check them, unless the template itself has guards and checks the types.

Through this lens, the semantic equivalent of C++ templates in Rust isn't generics, but macro_rules! - but for most uses of C++ templates you would rather use generics and traits.

Yah the type level programming in Rust really is limited to traits which provide little compile time functionality. It's pretty straight jacketed which is nice in some ways as you wont have un-anticipated interactions with user types. On the downside that limits un-anticipated usage of libraries or types.

Even macro_rules! aren't as capable as modern C++ templates with types. I am let down with how lacking Rust is when doing compile time type based programming. There's no CTTI or RTTI in Rust. The procedural macros don't have access to type information, just syntax. The constexpr equivalents are so limited, etc.

Yep, true. Stuff like Zig's famous MultiArrayList, where you write pretty simple Zig code to take apart and reify types and functions just doesn't seem to be much of a thing for Rust by design. Procedural macros also have build time issues, because the macro must be fully compiled before any code using it can be compiled (hence why they must be put in a separate crate), and as I understand it procedural macros only get TokenStreams, so if they want to work with Rust code, they must re-create the AST from tokens with their own parser. Unfortunately, derive macros also happen to be procedural and subject to those limitations, handy as they may be.

What I really like about Rust generics on the other hand is that, while limited in many ways, those limitations allow them to be complete in the sense that as far as types concerned, the generic signature is the law of the land; it'll compile with any type fulfilling the trait bounds, and the implementation behind the signature is fully limited to exactly what's provided by the trait bounds.

I think Python shines as an interactive calculator, eg an iPython REPL. Calculations, plotting etc. Or for website backends since Django is very nice. Or scripting OS tasks like file manipulation etc.

For full projects outside websites, Rust for almost every case.

Funnily enough, a REPL environment is in the works for Rust: https://github.com/google/evcxr

I like Octave (gnu matlab) for that.

I could get down with that, although my experience with the Python REPLs has been a mixed bag. Specifically, you can't easily deal with async stuff IIRC (not sure if that's been fixed or not).

Agreed. I still give Python credit for allowing us to write software in an extremely efficient way. But after I wrote the initial version of the software with Python, I always procrastinate on testing, fixing existing issues, and as the software grows larger I simply want to give up as the different issues keep piling up. In comparison, Rust code usually just works out of the box due to the error handling and type system designs. With rust I develop much more consistently, and even more efficiently as the project progresses due to IDE hints, rust's built in `cargo check`, `cargo clippy` commands etc.

Don't you find thinking about the borrow checker unnecessary overhead while prototyping?

Of course. That's rarely debatable (unless making sure that you have a very small memory footprint is one of the project's critically important goals).

The point is that you pay more upfront but can then iterate much more confidently and quickly.

No. Not anymore. 4 years in and I maybe run into a borrow checker issue very rarely in a given week.

Once you understand how the borrow checker works and how to write/structure your code in a way that the borrow checker is happy with it sort of becomes second nature.

I think by side effect coding in this style does lend itself to simply writing better to understand code. Nowadays I just write my code this way without much thought. It happens effortlessly. I find myself writing programs in other languages like I would in Rust nowadays.

This is compared to when I started and ran into borrow checking issues many times a day. Additionally the borrow checker has improved since then and NLL borrow checker will accept more programs as valid than the previous one.

Interesting. I tried rust maybe 6 years ago and the combination of the BC and its slow compile times drove me away. Actually my main gripe was something to do with it not being obvious when calling a function that it needed a reference (I forget the specifics), but maybe that's been improved with better code completion in intellij. I may take another look.

Yes re: Python.

Story -- I was a very early adopter of Python, back in the mid-90s. When other people wrote their CGI scripts in Perl, I always reached for Python. The first paid gig I ever had was a CGI script ("resume builder") I wrote in Python in 1996. But almost nobody was using it back then, and I'd get quizzical stares from people in job interviews etc. when it came up.

So for many years back then I really really wanted to get a job working in Python. And I was super excited to get a job around the 2001 time frame in it on a pretty cool embedded Linux project.

But my first experience working in a large codebase was disappointing. I could see right away that once many developers started working in that codebase together, best practices were really hard to enforce, and things started to fall apart into a bit of a mess, and lots of problems just showed up at runtime in bad explosions. That was the beginning of my disenchantment with dynamically typed, late-bound languages. They only give you the illusion of fast prototyping. You just shift your time to fixing type and binding errors later in your dev cycle.

Python really improved a lot with the 2->3 transition, and the community has much better best practices now. But I think the core problem with late bound languages remains.

All that said, I'm currently in another window working for my current employer wiring an embedded CPython interpreter into a Rust-based runtime...

I agree with most of this, but I would argue that the 2->3 transition was on balance a mistake. Python 3 is better, but not better enough to justify the huge amount of time and resources that otherwise could have been spent on better performance, typing, packaging, tools, etc. I'd gladly trade the Python 3 improvements for Python 2 equivalents of TypeScript and V8.

You are probably right, I'm not invested in enough in that ecosystem to know one way or another. I do know lots of things remain disrupted by this transition in ways that still personally inconvenience me.

> They only give you the illusion of fast prototyping. You just shift your time to fixing type and binding errors later in your dev cycle.

That is the perfect way to describe it! And sometimes you’re out of the dev cycle and into production if your test suite didn’t hit all cases.

I read "CGI" as computer generated imagery, python was a strong glue language in this field in the 90s.

Also I think people misread 'python speed', few c++ masters said it was more about prototyping to converge on good overall design rather than suffering the efforts of early iterations in c++ (90s c++).

Definitely in the context of the times, Python was more pleasant overall for initial prototyping than any static typed alternatives, including Java.

I still use it for quick one-off scripts to "do things" with bits of data and so on. And as a calculator.

I think this is a bit unfair. As always, blanket statements in software engineering aren't really meaningful.

As I understand it, the article talks about switching from python to rust in a non-trivial database service that is required to be fast and robust. I can't imagine python to do well in this regard, especially when C/C++ extensions are required to enhance the performance.

I also agree that 'Python has been a strict liability and the promise that you can "just rewrite the slow parts in C/C++/Numpy/etc" has never held'.

I just don't agree those points necessarily imply "avoiding Python for new projects". Even in the case of the original article, was it, in hindsight, a bad decision to start with python? I'd argue not necessarily. Python is still great for quickly hacking together something that works, not necessarily fast or maintainable, but if you're under time pressure or you're still trying to feel out the market fit for your product, it's not a great idea to waste brain cycles to deal with borrow checks for example.

Python's weaknesses only show when you need to scale, both on the performance and on the lines of code / number of collaborators metric, which is a problem you have only when the project succeeds. Avoiding python because of these problems reeks of premature optimization, unless you know it's how the project will end up in advance.

As other sibling comments point out, python is still great for small scripts and weekend projects. Perhaps you do mean 'avoiding Python for new projects that are expected to grow non-trivially', which is fine advice. I'm just a bit tired of all those over-reaching blanket statements people make regarding software design that only really makes sense in a limited context (here's a funny one I've heard recently: never write nested-for-loops because it's sloooww). It would be unfortunate to have the message "don't use python for large projects" be misinterpreted as "python sucks no matter what".

> As I understand it, the article talks about switching from python to rust in a non-trivial database service that is required to be fast and robust. I can't imagine python to do well in this regard, especially when C/C++ extensions are required to enhance the performance.

The article specifically talked about following the conventional Python advice to use C/C++ for the fast parts and Python for the "glue". This is precisely where Python people say Python shines.

> I just don't agree those points necessarily imply "avoiding Python for new projects".

You should avoid Python for new projects because you never know at the outset whether you will run into a performance bottleneck that can't be resolved by "just rewrite in C/numpy/etc" and there are other languages that are better than Python at both its strengths and weaknesses. The odds of painting yourself into a corner with Python are very, very high, and there are other languages that are better than Python at just about everything (the only area where Python still dominates is data science exploration libraries). Even if you don't paint yourself into a corner, you'll be dealing with lots of surprising papercuts: if you want reproducible builds, you'll have to wade through an ecosystem of package managers each with their own pitfalls including long build times (e.g., I saw pipenv take 30-45 minutes just to `pipenv lock` on a small-medium sized project); developing with docker basically doesn't work on mac (and windows?) because the VM uses all available CPU marshaling filesystem events over the guest/host boundary; your developers accidentally pulled in an API SDK that makes sync I/O calls (or else does some CPU-intensive task) which seizes up the event loop bringing the process down (and the failure cascades to remaining instances bringing the whole app down); and then there's the "regular" dynamic type system errors like forgetting to "await" things or missing/incorrect type documentation.

> Even in the case of the original article, was it, in hindsight, a bad decision to start with python? I'd argue not necessarily. I'd argue not necessarily. Python is still great for quickly hacking together something that works, not necessarily fast or maintainable, but if you're under time pressure or you're still trying to feel out the market fit for your product, it's not a great idea to waste brain cycles to deal with borrow checks for example.

My point isn't that Python is bad for literally every project--I actually think this project was probably a best-case scenario for Python (they didn't describe many of the issues I described in my previous paragraph) and they knew ahead of time where there performance issues would be so they were able to start with an architecture that was amenable to "write the slow parts in C/C++". My point is that there are languages that are better at fast iteration than Python (namely Go or TypeScript) while also excelling at Python's weaknesses (performance, tooling, distribution, type system, package management, etc etc etc). I think it's incredibly reckless to start a new commercial project in Python in almost all cases (with a possible exception for data science, but even then I would look hard at everything else first).

> Python's weaknesses only show when you need to scale, both on the performance and on the lines of code / number of collaborators metric, which is a problem you have only when the project succeeds.

This might be true for certain Django CRUD apps, but it's patently untrue in the general case. I've seen Python work fine for a prototype data science app, but fall over when real customer data was introduced. Similarly, I'm currently an SRE attached to a Python project which falls over regularly because of some event loop issue or another (often someone pulls in an SDK that makes sync calls under the hood, or else some CPU-intensive task blocks the event loop). I see stuff like this regularly, and it never happens in our Go apps.

> As other sibling comments point out, python is still great for small scripts and weekend projects.

Granted. Do what you want in your free time, but it's risky to use Python for new commercial apps.

> Perhaps you do mean 'avoiding Python for new projects that are expected to grow non-trivially', which is fine advice.

Yes, this is what I meant. I didn't mean "hobby projects".

> I'm just a bit tired of all those over-reaching blanket statements people make regarding software design

I don't usually make blanket statements, but Python really is so bad that this is a blanket statement I'm pretty comfortable with. I've scarcely ever heard of someone getting burned for picking Go over Python (there was some Python2 veteran who wrote an angry blog post once about Go back in the early days, but never since). Python just refuses to adequately progress because it wants to maintain compatibility--that's fine, but as professional engineers we should understand that the rest of the ecosystem is moving beyond where Python is willing/able to go, and it's no longer an appropriate tool for the overwhelming majority of new projects.

> I think it's incredibly reckless to start a new commercial project in Python in almost all cases...

> I don't usually make blanket statements, but Python really is so bad that this is a blanket statement I'm pretty comfortable with.

Speaking as someone who agrees with you that something static is very likely better than python for a large project, how do you square this with the existence of a lot of Python projects, even "very commercial", that seem to go on just fine?

It's a fine theory and something I instinctively support, but I don't see measurable evidence in support of it.

Survivorship bias. You don't hear about the failures. Moreover, many of those things (e.g., lots of Python at Google) was written at a time when there weren't better options, so you weren't struggling to make Python work while your competitor is writing Go or TypeScript.

Those projects were started when python was the least bad choice and grew out quickly enough that they could hire legions of developers to work around pythons problems.

Just fine, you say? I had dozens of colleagues being in medium to large Python projects and every single one of them was terrified of every small change in the projects because inevitably something fails in an unexpected way.

These guys and girls are heroes. They are like the plumbers and electricians working at 3:00 AM so your water and electricity never stop while you're awake.

"Just fine" is likely only your external observation. I'd give them medals and help them retire at 35 for all the horrors they have endured. It's "just fine" because they tirelessly made sure of it.

Things were just fine despite Python. Not because of Python.

Plus, none of us has statistics that prove a theory beyond any doubt. Forget about that; this scarcely (if ever) existed in the programming world and likely will not exist anytime soon. We should apply our experience and the intuition that grows out of it.

Finally: a pile of anecdotal evidence can be treated as an objective evidence with, say, 50-60% probability. Insisting that the real world must comply with academically pure practices is not going to get us anywhere (and is misguided).

> This might be true for certain Django CRUD apps, but it's patently untrue in the general case. I've seen Python work fine for a prototype data science app, but fall over when real customer data was introduced.

The cognitive dissonance embedded in this statement is rather striking, considering Django itself is written in Python and is known for its ruggedness and code quality. Django manages all of this without even taking advantage of somewhat newer features in Python like typing.

There's no cognitive dissonance, you just misunderstood the claim. :)

> Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling. Use Rust if you're writing really high performance or correctness-is-paramount software.

I'd add one more addition to use Rust (speaking from an ex-Go dev): Use Rust if you want a robust std lib. Go is good, i used it for ~5 years, but man Rust was a breath of fresh air with the amount of tooling that helped me solve problems. Iterators, are a great example.

I also preferred Crates vs Github Repos, but i think that's just personal preference.. not objective.

I think this comes with the caveat that there are a lot more mature libraries for Go just as a function of the age of the language.

Agree, you can see this in https://github.com/rust-unofficial/awesome-rust vs. https://github.com/avelino/awesome-go

I've noticed however that there has been an uptick in great libraries over the last 2 years, with examples like pola.rs, rust-bert, tokenizers etc. starting to build momentum in the ecosystem.

That might have been true 2-3 years ago but nowadays whatever I can think of, Rust already has a library for it.

Most recently I've run into Hyper not having tracing support

When performance and correctness are paramount, I personally stick to Ada/SPARK. Proven to be perfect for the job, and to be honest, Rust is quite a difficult language and I am not sure it is worth learning it considering that Ada/SPARK ticks all boxes when it comes to both low-level and high-level programming for critical systems.

I have fond memories of using Ada in college. Unfortunately it didn’t have much of an open source footprint and the community was downright hostile toward newbies (more so than other communities that I had experienced). I’m sure Ada has the stronger story for real-time embedded systems today, but (as I have learned Rust in the interim) I look forward to the day when Rust breaks into the real-time embedded space. In particular, I’m excited about ferrocene and the broader Rust certification efforts. I would really like to write embedded Rust professionally, and I hope certification will boost the number of embedded Rust jobs (no the to Ada, I just don’t particularly want to re-learn it).

My favorite thing about learning Rust is how helpful the community is.

Do you work in Ada/SPARK professionally? Curious what industry if so.

I actually don't care as much for iterators as I thought I would. Beyond some simpler map().reduce() stuff they tend to fall over pretty fast, and I end up spending too much time trying to make iterator chains work before defaulting to for loops. I also dislike how anemic Rust's stdlib is. I'm sure there are good reasons, but I like that I can just reach for Go's stdlib for annotating errors or dealing with JSON. Other than that, I like the API design in Rust's stdlib.

Here's an example where the iterator-chain version is way more complex:

    let posts = read_dir(source_directory)?
        .filter_map(|result| {
            (|| {
                let entry = result?;
                let os_file_name = entry.file_name();
                let file_name = os_file_name.to_string_lossy();
                if Self::is_bundle(&entry)? {
                    self.parse_post_bundle(&file_name, &entry.path())
                        .map(Some)
                } else if file_name.ends_with(MARKDOWN_EXTENSION) {
                    self.parse_post(
                        file_name.trim_end_matches(MARKDOWN_EXTENSION),
                        &mut File::open(&entry.path())?,
                    )
                    .map(Some)
                } else {
                    Ok(None)
                }
            })()
            .transpose()
        })
        .collect::<Result<Vec<_>>>()?;

Here's the imperative version:

    let mut posts = Vec::new();
    for result in read_dir(source_directory)? {
        let entry = result?;
        let os_file_name = entry.file_name();
        let file_name = os_file_name.to_string_lossy();
        if Self::is_bundle(&entry)? {
            posts.push(self.parse_post_bundle(&file_name, &entry.path())?);
        } else if file_name.ends_with(MARKDOWN_EXTENSION) {
            posts.push(self.parse_post(
                file_name.trim_end_matches(MARKDOWN_EXTENSION),
                &mut File::open(&entry.path())?,
            )?);
        }
    }

I guess that's subjective, not knowing rust I see a few more lines in the first example but beyond that it does not jump out at me as being "way more complex."

Hah, it took me and a bunch of other people in the Rust discord a fair amount of time to figure out how to get the first example to compile correctly in the first place, and I find the “transpose()” business to be particularly convoluted. I think even Rust people would prefer the imperative version. I also suspect there’s some hindsight benefit at play in that it’s easier to make sense of the first snippet than it was to write it.

Yes, the imperative version looks like the sane choice to me. Rust's iterator documentation does suggest that this is not a language which favours iterator chains everywhere.

However one reason to prefer an iterator chain in some cases is that the chain might imply an optimisation you'd otherwise have to write manually and might not think of.

For example if I have N things, and I map them, perhaps more than once, but I'm definitely getting N of whatever the output of the last map was, the iterator chain might very well see that I don't actually have any way to change N and so when I collect() that into a Vec the Vec it gives me is constructed with capacity N, saving the (amortized but non-zero) cost of growing it during insertions.

Imperatively I can remember to write Vec::with_capacity(N), and that's safe of course but it's an extra step to remember.

The imperative approach does particularly shine on the opposite edge of this, if I know that I'm getting no more than 100 items out of this pipeline, despite putting N in, Rust almost certainly won't see that and won't make Vec::with_capacity(100) for a collect() call whereas my imperative code can just explicitly write Vec::with_capacity(100) or whatever.

Depending on how you write it, iterators are much better at this since most iterator methods produce iterators with size hints.

If you use "take" for example, collect should do the right thing: https://doc.rust-lang.org/src/core/iter/adapters/take.rs.htm...

Mere hints aren't enough for collect()

It will only care about your hints if you implement the unsafe trait TrustedLen (which promises those hints are correct)

Take is indeed TrustedLen if the Iterator it's taking from is TrustedLen (as in this case it can be sure of the size hint)

If you get to ~100 via some means other than take()ing 100 of them, chances are you don't have TrustedLen as a result.

TrustedLen will mean it can safely take the upper bound, but Vec for example will still use the lower bound of the hint when something isn't TrustedLen

https://doc.rust-lang.org/src/alloc/vec/spec_from_iter_neste...

Ooh nice, I hadn't seen that. OK, so there are probably places where I assumed the iterator won't do as nice a job of collect() as I did manually but I was wrong.

I wonder how the hell I didn't see that when I was tracking down how collect() ends up caring about TrustedLen.

Thanks.

Honestly the closure + `transpose` is really ugly as sin. I had to do a double take and reason through the steps which you took to get that particular snippet (while not reading the imperative one).

Thanks, I feel validated. (:

The thing is in Rust this is totally ok. Is NOT un-idiomatic to mix imperative + functional paradigms in the same block.

Agreed, but I think people make too big a deal about iterators. The simple for loops that they’re fit to replace aren’t that hard to read or write, so they aren’t causing real problems, and on the more complex end you’re writing imperative stuff anyway. Iterators certainly aren’t a bad thing, but they’re not the game changer that Go-critics made them out to be in all of the Rust-vs-Go debates.

I disagree. If I can avoid an index variable, I’m going to have less bugs since I won’t have to index.

I'm not sure what you mean. You don't have to deal with index variables in a for loop:

    for item in iterator {
        vector.push_back(foo(item));
    }

You're not explicitly indexing the source iterator or the destination vector at any point.

Agreed. That's why I enjoy iterators, and use them whenever possible. I think they're important.

But earlier you claimed the for-loop version required index variables and indexing into iterators/vectors/etc ... ?

I clearly said the opposite, in response to, what appeared to be, your suggestion that indexing isn't so bad:

"I think people make too big a deal about iterators. The simple for loops that they’re fit to replace aren’t that hard to read or write, so they aren’t causing real problem"

It seems like you imply that iterators = range-for and "simple loops" = indexing, which is clearly not what the previous poster has in mind (which was "range-for = imperative loops, iterators = functional style, simple loops = trivial loop body").

Yes, I was responding to:

> I disagree. If I can avoid an index variable, I’m going to have less bugs since I won’t have to index

Which (given thread context) sounds pretty clearly like he's advocating against `for item in iterator {...}` and in favor of iterator chains. If there's another way of reading that comment, I'm not seeing it. :)

> If I can avoid an index variable, I’m going to have less bugs since I won’t have to index.

Not only that. In safe Rust vector indexes are always bound-checked that slows execution a little bit. Iterators over a vector, on the other hand, do not suffer from bound-check overhead.

> Agreed, but I think people make too big a deal about iterators.

I may just have more time to dick around, but I've been forcing myself to write everything as an iterator in Rust (instead of a for loop) just to learn the paradigm and, honestly, it's been pretty great.

First, there is a performance impact. You may think this would be small, but, in my experience, it can be relatively large. There must be several reasons of which I am ignorant, but they probably include mutable no alias inside closures, bounds check elision, etc.

Second, there is a mutability advantage to just using filter/flatten/map/collect for 90% of your use cases.

Third, the more advanced iterator methods will cover the other 5-10% of your use cases, and will make your code much easier to read, once you understand how to use them (if you're trying to learning something/anything more complex at the same time, save your brain power!). I don't have a single traditional for loop in my personal projects.

My take on iterators is, for iterator/Rust beginners (like me!/perhaps not you) -- the best way to learn is to get it working as imperative code, and when you have the time, go back a rewrite as an iterator. In the beginning, it's super frustrating. But you'll get better and better and, after awhile, you'll much, much prefer writing as an iterator. You won't even think of using a for loop.

I mean, I definitely have tried to rewrite stuff as iterators even when it isn't convenient. I only got to that working example by dedicating a few hours with the good people on the Rust Discord trying to make it work (a lot of seasoned Rustaceans also weren't able to get it working, it was a collaborative effort that took a fair amount of time and rhapsodizing). And even then, the result is pretty ugly (I don't think many seasoned Rustaceans would call it "idiomatic" or advise the first version over the second).

Agreed. Your example is a pretty pathological case, but yeah.

The selling point of iterators isn’t that they replace for loops, it’s that you can make new iterators by implementing a trait. That at way downstream users can use your iterators in a for loop. Of course you also can use a more functional style if you like.

The first version isn't what I'd write in a purely functional language either, a recursive version would be much saner. Isn't recursion idiomatic in rust?

Rust doesn't guarantee TCO and so recursion is a bit dicer than in many functional languages.

In addition to Steve's sibling comment, I'm responding narrowly to iterators versus for-loops. Recursion is out of scope.

Now, compare a parallel version of both.

It's not obvious to me that the parallel version of the first is going to be cleaner (or more performant) than a parallel version of the second, but even if it is, I don't agree with mangling (to put it nicely) single-threaded code on the off chance that it might be made parallel one day.

Yep, agree wholeheartedly with this. I came to Rust from Scala where writing an explicit for loop is looked at as one step above using goto :) and it took me a while to just give in and write the imperative version.

There is no need to force something iterator style. By all means use a loop if it is more readable. Also, if your iterator style closure body gets very verbose, you can always pull the logic out into a function or closure and pass it in. In the end, both styles are idiomatic, and the best choice is which makes it more readable (subjective of course).

I personally use iterator-style for 1-3 simple ops, but go into a loop for anything more complicated. The functional part of me wants to force iterator style but I fight it for complex use cases because it is irrational on my part, and I find loops easier to read for stuff like that.

Fair, but i'd probably write your iterator example differently. Though, besides the point perhaps - as i agree there are plenty of cases where imperative approaches are cleaner.

I however will often have a dozen maps/filters/etc and that style in imperative is what, nesting repeatedly or using `continue`. Meh.

If your Iterator consists of a single `filter_map` i can see why you don't get much value out of them in that case. It's because.. well, there's not much value to get out of it.

Iterator-chain version would be a lot better looking if it was actually was a chain and not everything pushed into a single `filter_map`. "Ifs" are weird looking, one branch is using `entry`(I take it's a `DirEntry`?) and another is using `file_name` I suspect both could use the and look like `Self::is_` ?

If you're going to write an iterator chain in an imperative way, then you might as well right it imperatively.

A few notes:

- you can compare `&OsStr` with `&str` without `to_string_lossy` any explicit conversion (`&str -> &OsStr` done in `PartialEq` is basically free)

- you can get extension from `Path` by using `Path::extension`

- you can get filename without extension by using `Path::file_stem` or `Path::file_prefix`

- making your parse functions take the same arguments makes it a lot cleaner

Something like this:

    read_dir("")?
    .map(|e| e.map(|e| e.path()))
    .map(|path| {
        path.and_then(|path| {
            if Self::is_bundle(&path) {
                self.parse_post_bundle(&path.file_name().to_string_lossy(), &path).map(Some)
            } else if Self::is_markdown(&path) {
                self.parse_post(&path.file_stem().to_string_lossy(), &path).map(Some)

            } else {
                Ok(None)
            }
        })
    })
    .filter_map(Result::transpose)
    .collect::<Result<Vec<()>, std::io::Error>>()

    // Ignore this, just an example
    fn is_bundle(path: &Path) -> bool {
        path.extension().map(|ext| ext == "bundle").unwrap_or(false)
    }
    
    fn is_markdown(path: &Path) -> bool {
        path.extension().map(|ext| ext == "md").unwrap_or(false)
    }

Forgive me for asking instead of looking, I'm still getting into rust; why do you map instead of getting the result?

    path.extension().map(|ext| ext == "bundle").unwrap_or(false)

    (path.extension() == "bundle").unwrap_or(false)

or further up

    .map(|e| e.map(|e| e.path()))
    vs
    .map(|e| e.path())

Is the .map keeping any value wrapped in an iterator so you can map again? I thought it was interesting what you wrote looks a lot like what I'd write in powershell.

Not your parent, but I do know the answer.

extension() returns an Option<&OsStr>, because not every file has an extension https://doc.rust-lang.org/stable/std/path/struct.Path.html#m...

map on option says "run this closure on the value if it's Some, or do nothing if it's None https://doc.rust-lang.org/stable/std/option/enum.Option.html...

So after the map, we now have an Option<bool>. The unwrap_or method will say "give me the value if it's Some, and give me the argument if it's not https://doc.rust-lang.org/stable/std/option/enum.Option.html...

That means we finally end up with a bool.

  (path.extension() == "bundle").unwrap_or(false)

This code wouldn't work because you're comparing an Option<&OsStr> to a &str. Even if that worked, you'd have a boolean, and unwrap_or doesn't make sense on booleans, you'd just leave off that part.

Awesome thanks, I wasn't on a machine to play around with it at the time.

`extension()` gives you an `Option`, remember `Path` might not be a file or file could be without extension.

Why not `((path.extension() == Some("bundle"))`? I don't know, it didn't work, and I didn't want to spend too much time on it.

`read_dir` gives us an iterator over `Result<DirEntry>`, so `e` in that map is a `Result` and not a `DirEntry`. It's two different "map" functions - on Option<T> and Result<T,E> it maps "T", but on Iterator it maps "Item" from that iterator.

Here is an explanation why `read_dir` give you a Result - https://doc.rust-lang.org/std/fs/fn.read_dir.html#errors

Thanks for the insight

I'm sure there's plenty of room for improvement. I spent several hours with a bunch of Rustaceans just trying to get the short-circuiting to work right, and we sort of gave up on it beyond that.

> but man Rust was a breath of fresh air with the amount of tooling that helped me solve problems

The tooling is the #1 reason I'd like to learn rust. I have not kept up with C++ and I'm not sure I ever will... sometimes plain C seems more straightforward. But those languages (C++ especially) have suffered from lack of standard tooling, in my opinion.

Even C# being a few years younger than Java seems to have made a huge difference in tooling availability (I'm sure having a single corporate driver during its lifetime made a difference, too).

I'm excited by the prospect of a compiled language with modern tooling.

Tooling was the reason I left C++. Having to do your own package management and script your own build system was deeply dull work. The IDE story was also pretty miserable, although I think clangd has improved it a bit. Getting decent information out of core dumps also sucked a lot. The language actually didn’t bother me too much after c++11, but everything else was such a bear that I jumped ship.

Rust has been very rewarding in that it largely fixes all of C++’s tooling and language problems. It very much is a significantly improved C++.

I actually filed a ticket at Carbon project saying c++'s top 1 problem is tooling, modern c++ is pretty nice to use already, no need for Carbon there in fact, at least not as needed as tooling.

Rust std lib is subpar compared to the Go one, yes you have iterators but that's it, basic things like async are not even provided just the interface so everyone has to use tokyo. Then for real use cases you're missing http/json/compression/crypto etc ...

https://pkg.go.dev/std

Overall tooling and std lib are better on Go, actually there are not many languages that are on part with Go to that regard.

When you see what you can do with the single go command, it's pretty powerful.

Quick correction for anyone searching: it's `tokio`, not `tokyo`. [0]

[0]: https://tokio.rs/

I find it really curious how people consider "async" as a "basic thing". Coming from C++, asynchronous runtimes were a thing companies and large open source projects were built around. It's not at all a basic thing.

I'd argue it's almost certainly something that should NOT be in the standard library. If anything, the feature was added to Rust too quickly and has way too many rough edges.

Agreed 100%.

Go doesn't have a standard library problem, it has a userland language problem.

* No sum types / algebraic data types in 2022.

* No exhaustive pattern matching in 2022

* No move semantics / Uses GC

* No borrow checker

* Still suffers from nil problem / No type-safe nil / No type-safe Optionals in 2022 -- (I can show you all the nil panics in kubernetes logs if you like)

Current go users are already sold so they don't expect more, and that's fine, but further evangelism will require a better host language.

Seems a bit odd to criticize Go for using GC rather than a borrow checker to manage memory. There’s surely a place for GCed languages, even if they’re not the right choice for every domain.

And yet Go has delivered very useful software used by millions of people.

So has C, and most other popular languages. Doesn't mean they don't have issues, often dangerous issues.

I don’t mind GC. I think it’s the right choice for the sort of applications Go targets. I wish it had algebraic data types (with exhaustive pattern matching), but the “in 2022” meme is just vapid snark.

Notably, despite its shortcomings, Go is still perhaps the most productive programming language for general purpose applications. It turns out that type system features are neat, but tooling, ecosystem, simplicity, etc outrank type system features by quite a lot. Rust does very well here too, but it’s choice of borrow checker over GC amounts to trading off a lot of productivity in the general case (but it makes sense for Rust’s goals!).

Docker is built with Go. Compare doing things with Docker api between Go and Python. I usually recommend people to "script" things with Go but I start to think that Go has to go...

https://docs.docker.com/engine/api/sdk/examples/

Exactly this, in rust it will even more complicated than in Go

Wow. That's quite something. I like Go, but more often than not I keep stumbling upong these kind of situations (N LOC to do something in Y, 10xN to do it in Go).

In addition to a emulating certain teachers dogged insistence of making sure students can't have nice things (an absolute no-brainer feature like generics only showed up recently...!), last time I tried to use Go for a web project a few years ago I was also amused by the lack of templating libraries - the way to make pages was to string together header + main + footer, the old PHP way...! Not a single library supported template composition.

Umm... what are you even talking about?

Go has had templating built into the standard library since... forever?

And Go templates can use other templates, as long as they're all loaded into the same context, if that's what you were getting at.

simple examples of Go's built-in templates: https://gowebexamples.com/templates/

templates using other templates: https://levelup.gitconnected.com/using-go-templates-for-effe...

I've found Go's templating to be pretty great

That doesn't look too bad on a quick glance, but the possibility to say that "this content goes in that template" still seems to be missing.

Look at this page: https://quarkus.io/guides/qute-reference

and read about “Template inheritance” in 3.5.6 to see what I mean.

In this example, I'm passing an object with a single value into the header template, but it could have as many values in it as needed: https://go.dev/play/p/kC8pm4Z4WrH

Go doesn't really do template inheritance in the way I think you're meaning. Go prefers composition over inheritance, and that carries over to the templating system. Go templates let you pass a parameter to the template you're calling, which I believe addresses the need you linked to. Calling another template is not just simple string concatenation; the other template can act on input provided by the calling template.

There are a lot of different ways to approach templating pages in a web app. I think Go's templating system is one of the least objectionable ones I've used.

I have no experience with Quarkus, so I can't say how good or bad that one is, but my experience with Ruby templating systems is that people often start mixing complex logic into the templates, including database calls, and that causes things to get messy. Go's template environment is very minimal by default, and you can extend it with custom functions if you want to give more power to the templates, but Go encourages you to compute your data ahead of time, then pass it into the template... and just have your template be a template for formatting the data.

> people often start mixing complex logic into the templates, including database calls

Sometimes you can solve social problems with technical interventions, but at this point, someone needs to sit down and talk :-)

> Go encourages you to compute your data ahead of time, then pass it into the template

Same elsewhere including modern PHP and Java.

The thing that is missing in Go is a natural way to group together what goes around the content.

Sure, stringing together the templates (yes, I know the other template can act on input provided by the calling template, but they are still stringed together) feels weird when what one really wants is to fill in slots in a design, which is my thought process.

I would say in this case it's more the API designers' fault than Go's fault. Looks like the Python API is operating at a higher abstraction level than the Go API - using the Python API you can just call a "macro" that does everything required, in Go you have to call each operation individually.

It probably is Go's propensity to return error values whereas Python would bubble them up. Every action in Docker involves an I/O error at a minimum (since you're talking to the Docker daemon) so this is basically a worst-case scenario for error handling boilerplate. I don't doubt that Python is better for scripting in this case, but if you want to make an application that interfaces with Docker, Go is probably the better of the two (despite its boilerplate). In fact, I vaguely recall running into a bunch of Python-related bugs in docker-compose (it was at an old gig, and I haven't used docker-compose since, so I don't recall the particulars).

Still, creating a method that calls several other methods one after another is programming 101, so I refuse to entertain the though that creating a "containers.run" method similar to the one from the Python API is not possible in Go. The error handling is definitely not the culprit, you can check the error after each "subcommand" and return early with this error. The question is just, why didn't they do it?

Fair enough. I didn't look deeply at the API differences.

Event the C# version is simpler :D

https://github.com/dotnet/Docker.DotNet#example-create-a-con...

Python has high dev velocity if you only measure how quickly you can get started on a project. The dev velocity drops exponentially with the size and complexity of your codebase.

But it is indeed quite fast when you are just trying to get your first file of code written.

If Go replaces your Python this easily, you have always been writing Go. The abstractions available in Python are in a totally different class than that of Go.

Nothing you said is verifiable.

> Nothing you said is verifiable.

Right, I was pretty explicitly speaking from experience. See my first paragraph.

> If Go replaces your Python this easily, you have always been writing Go

Go is only a 10 year old language, and I've been writing Python professionally for 15 years. I definitely have more hours on Python by a wide margin.

> The abstractions available in Python are in a totally different class than that of Go.

Python definitely is more abstract than Go, but I don't think that's a feather in its hat. Moreover, most of my criticisms of Python had nothing to do with its abstract nature.

I think the intent of saying "you've always been writing Go" was along the lines of "you can write FORTRAN in any language." E.g, implying that you are not making use of Python's abstractions, but writing it in the same procedural style you would write Go (or C) code in.

Fair enough, but that's still not the case. :)

That is exactly what I am saying. I not talking trash about Go. But if one isn't using Python for what it is good at, don't use it, use the language you are actually using.

> Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling.

Why not D?

I haven’t used it. As I mentioned, I’m sure there are other languages that fit the bill besides Go and TypeScript.

Started a small project with rust and serde(toml config). Having static checking of config an its parsing felt really good.

We started just writing configs in typescript which has the benefit of static checking in your editor with suggestions/underlines. Could likely do the same in rust.

I'm afraid by typescript "typesystem" even more so than rust borrows/lifetime. It's so versatile and stringy .. that said it's still good to have some.

What I'm proposing is rather than have Rust validate yaml/toml files, just write the actual config itself in Rust so that your editor can verify/show issues before running. Unless you require ability to edit configs without recompile.

I would recommend Rust even if writing a simple tool, the kind of thing I might previously have used python for. We had some data munging to do and someone started it in python. On the real dataset it ran for an hour then ran out of memory. I took a crack in rust, as a n00b rust programmer, and the rust version runs successfully in less than a minute. I haven’t tried server side yet, because the learning curve for that is more of a learning cliff, but for small things I am already faster in rust than with my decade of python.

Completely agree. Rust shines as a language for small tools that need to be performant and correct, like Unix utilities. Exa and fd are great examples of this.

I had the same experience with go, 100x performance improvements on data crunching scripts with roughly the same code. Worst part of writing it in python was simple misspellings in the last stages of the computation breaking things.

> Worst part of writing it in python was simple misspellings in the last stages of the computation breaking things.

I'm not sure I understand. Was the typo in something like a dict key? Or did you add two of the wrong variables together? How did another language protect you from this? I fail to see how you couldn't also add two of the wrong variables (with the correct type) together in a different language.

Python (out of the box) will only complain about missing variables or methods at runtime.

>for small things I am already faster in rust than with my decade of python.

That seems like a bold claim? I've also been writing Python for a decade and it takes no time at all to whip up a small script. I have no doubts the Rust version will be much faster, but stepping into a new language would take me at least longer to write the script than an equivalent version in Python. Plus there's all the other stuff to learn.

Once you get past learning the basic syntax (creating variables, loop structure, assignment/etc.), you still have to pick up new things, like executing your work with parallelism, downloading files, parsing and dumping JSON, connecting to databases... Each new topic introduces a bunch of stuff that takes a little bit to figure out. I guess you could get through most of it in a few hours though if you had the right problem to work on.

But still, there's the muscle memory aspect. Can a new language really sink in so fast that it's faster to use than Python after 10 years of writing Python?

I've seen the transition from a Javascript codebase to a Typescript codebase with all strict checks in place to have a similar step change in dev velocity. The speed of debugging stupid mistakes, confidence in larger changes, and removal of a need for a lot of unit tests that were poor-person's type checking, is really significant. And runtime errors are a lot less weird than they were.

> I've seen the transition from a Javascript codebase to a Typescript codebase with all strict checks in place to have a similar step change in dev velocity.

The important factor here is "with all strict checks in place". Whenever I hear fellow FE devs complain about how they dislike Typescript, the complaints often seem to center around how it doesn't really offer that many guarantees... only to find out that their typechecker is set to the least strict settings!

I'm not gonna lie, though: the transition from JS to TS absolutely comes with growing pains. This obviously includes actually updating the codebase and adding types, as well as the developer onboarding/learning process. Personally speaking, I found it very jarring to go from a permissive, lightning-fast JS dev cycle to a more methodical Typescript one.

And don't get me started on how frustrating it can be to diagnose weird type errors related to generic types :D

Dev velocity in Rust is directly proportional to how well you grok lifetimes and ownership.

Eh, I like Rust and started my career doing embedded C and C++, so I definitely "grok lifetimes" but I'm probably always going to be more productive with Go than Rust (for general app development, anyway) if only because I don't have to think about memory at all with GC (again, general app development, not talking about performance optimizations). Even if I do a bunch of unnecessary cloning + Rc/etc in Rust, I'm not going to move as fast as I would do in Go.

Also, +1 for Go. I think that one of the things that makes Rust and Go such great language is how strict their compilers are. This moves the cognitive load of writing same code from the developer/team, to the compiler.

Go is also simpler because of the GC, given that Rust can get a bit tricky with lifetimes. Go routines and channels are a delight to use together with contexts.

Rust allows for a lot more ways to solve one problem, i.e. you can iterate over an array, of use a loop. Go only provided a for loop and that is pretty much the only way to iterate over something.

Some people might find this limiting but I find it refreshing given that I can focus on what I need to do instead of how I should do it.

I'd argue that C# (dotnet core) is a much better option than Go for an easy GC language with max productivity and great performance ceiling.

Not only that, but greater tooling too.

C# 2022 is much improved over earlier versions. For starts it's cross platform and compiles to reasonably sized single executables (touted as a strength of Go). Moreover, C# designers have started reducing boilerplate as much as possible, so for example, you don't need to declare a namespace, a class or a main to start coding.

That said, Go is a fine language. It doesn't have exceptions last I checked, so that could be a deciding factor. Exceptions are handy in a lot of cases.

I haven't worked with C# since ~2013, but I mostly liked it. I'm sure it's a different beast now though. Some things I prefer about Go:

1. Tooling. `go build` just needs a `go.mod` file with a list of dependencies. There's no MSBUILD stuff (I think dotnet had a similar JSON file format, but then went back to MSBUILD?)

2. Runtime. I love that Go's default is static, native binaries, and that the ecosystem revolves around that. I especially love that this enables ~2mb Docker images. (I'm of the impression that you can do this in dotnet, but it's not clear to me how easy it is in practice or if there are performance tradeoffs)

3. Value types. Go doesn't have a classes; everything is a struct, and polymorphism and so on work on structs.

4. No inheritance. I'm pretty militantly opposed to inheritance. I know you don't have to use it in C#, but in Go I also don't have to deal with APIs or coworkers that assume inheritance.

That said, these aren't major things; I'm sure C# is a great language in general, and I'd definitely reach for if I wanted to make a Windows GUI app.

1. `dotnet build` requires a .csproj file. It's XML based (they tried a JSON based format a few years back, but reverted to XML).

C# and .NET's main strength, IMO, is the web API framework and general purpose utility; everything from web apps to Unity and Godot. It's easy to start with `dotnet new webapi -minimal` will give you a scaffolded project which is pretty similar a barebones Express or Flask app. But you get much more headway and scalability and the room to layer more complexity as needed.

It's main detriment is, as you mentioned, the need for a somewhat heavy runtime and the relatively slow cold starts which make it a poor choice for things like CLI tools and serverless functions.

Garbage collector is still a steaming pile though

Source? I thought .NET GC was tuned for high throughput.

Their performance is, impressive, to say the least.

https://www.techempower.com/benchmarks/#section=data-r21

It has terrible latency characteristics

IIRC it also doesn’t do much/any escape analysis, so ~everything ends up on the heap as an individual allocation (except for values), putting even more pressure on GC.

I disagree.

Most startups that YC has funded that became successful (Series B or higher) were written in Python or Ruby. Now you can say that this is a tradeoff for post Series B, and for that I don't know. I've never worked on a massive Python mono-repo for a company that size. But I know what I've done in Python and, yes, it includes performance optimization in Numpy / Scipy / Cython, and other than Ruby, no other language comes close to the developer performance I see with Python and the ecosystem around it.

That said, there is a good and a bad way of writing Python. You absolutely need tests and lots of them since there is no compiler for vanilla Python. Mypy helps a ton too and so do the linters and even Black. Also, get pdb++ and customize it. The tooling helps a lot.

Now I haven't ever written production Rust, but I have for Go and many other languages, and so Rust may be faster, but other than Ruby I haven't seen anything close.

> Most startups that YC has funded that became successful (Series B or higher) were written in Python or Ruby.

Is there data on startup lifespan ranked by tech stack? Genuinely been looking for this for awhile now.

> I disagree.

> Most startups that YC has funded that became successful (Series B or higher) were written in Python or Ruby.

It depends on what your goal is. If you want to get rich off of VC money, Python and Ruby might be a good fit. If you, on the other hand, want to write good, performant, maintenable and (relatively) bug-free software, then there are better choices.

Dev teams that are unable or willing to properly architect and test their projects will end up with buggy and difficult to maintainable code regardless of language.

That is unrelated to what the person you're responding to said.

If you, on the other hand, want to write good, performant, maintenable and (relatively) bug-free software, then there are better choices.

You can do that part when the money comes in.

Once you invite the investors in, the company isn't truly yours, and they expect 1000x growth. There will always be higher priorities than a rewrite.

If you don't take investors' money, then it's theoretically feasible, but, in practice I've seen companies stuck with their Python/Ruby stack for 10 years or longer, always wanting to rewrite but never having the time. Microservice architecture helps (not that it's not a can of worms of its own), and these companies tend to write new functionalities in their desired language (such as Go or Scala) and deploy them as separate microservice. The old code remains in Python or Ruby though, and now they have a multilingual mess that is near-to-impossible to maintain for a small team. Why not start with a better language in the first place?

> Most startups that YC has funded that became successful (Series B or higher) were written in Python or Ruby

So, what was the percentage of YC startups that started with Python or ruby and what was the share that became successful?

A lot of people seem to believe that dynamically typed languages make development easier, because you don't have to worry about writing type annotations or going through the extra steps to fix compile-time errors.

This is arguably true for small scripts, but for anything non-trivial I find that static typing means I can be more productive, because type errors are caught straight away (rather than potentially in production), and I can be much more confident in refactoring code because I know certain kinds of errors will be caught by during type checking. This doesn't remove the need for automated testing, but does reduce the number of test cases you have to write.

Things have improved a lot in the Python ecosystem of late with the introduction of type annotations and mypy, so to the extent possible I treat Python as if it's a statically typed language, putting type annotations everywhere and always making sure the codebase passes type checking before pushing a commit.

However, the fact that this is optional sadly means that not everyone does it, and if you're working with parts of a codebase written without this level of discipline, you're basically back to square one. I've come to the conclusion that for medium to large projects, dynamic typing encourages sloppy programming, because it doesn't force people to think as carefully about what exactly their data types are, and whether they're being used correctly.

> I find that static typing means I can be more productive, because type errors are caught straight away

I agree completely. I feel like there’s a pretty common arc among programmers:

1. learn to program using verbose statically typed languages

2. discover fun dynamically typed languages, eschew statically typed languages

3. discover dynamically typed languages are a shitshow for large real-world projects

4. re-discover fun/less verbose statically typed languages that make working on large projects tolerable

Except keep in mind that Rust adds more constraints than your typical statically typed language.

It is still more fun and efficient to program in a language that is garbage collected and statically typed.

About no. 4: That often translates into noticing C++ has gotten `auto` in C++11 with a later improvement in C++14:

  auto x { get_something_complicated() };

  foo(int x) -> auto {
      return get_something_complicated(bar(x));
  }

so it's less "not-fun" these days.

I feel the same. Java and C# have an equivalent called 'var'. Unless the scope is tiny, I always avoid it. Also, in Java when you create a lambda, it is possible to exclude the parameter types.

    foo(x) -> get_something_complicated(bar(x))

It is not good for readability in large source bases.

With C++ lambdas, it's similar, but the syntax is a bit uglier:

  [](auto&& x) { return get_something_complicated(bar(x)); }

and you can decide whether you want a value (no &'s), an lvalue reference (&), or a forwarding reference (&&). I think with C++20 you can also use a concept instead of auto for a constrained parameter type.

I hate that some universities teach dynamically typed languages as first language in their curriculum. I think for CS students the first language should be as "strict" (not in the mathematical sense) as possible, so the student is forced to think very carefully what is going where. This is of course a controversial opinion, especially among proponents of SICP (which is, in my opinion, not a good book for beginners. Yes, downvote me).

(Obviously, I am talking about CS students here, not about the general public who just want to learn what writing a program means)

What language do you recommend for beginners?

I think Go would make a pretty good language for beginners, but I haven't tested that theory. TypeScript might be the safer bet, especially since you can run it in the browser and see things, which is probably somewhat more gratifying than printing text to a console.

I would go on a limb and say Rust (or more specifically, a limited dialect of it) could become a good language for beginners if:

- the compiler had better life time inference, so that you wouldn't need to annotate simple functions and introduce beginners to complex issues related to lifetimes, i.e it should "seem" to work like python, initially.

- better learning resources for absolute beginners are developed, step by step tutorial, an easy to install full IDE with friendly error messages etc.

The thing I like about Python is that I can decide, even on a line-by-line basis, but more likely on a program by program basis, if it makes sense to be type oblivious, or type checked.

Exactly. For one-off scripts or code you might throw away, there is nothing wrong with skipping the types.

> I can be much more confident in refactoring code because I know certain kinds of errors will be caught by during type checking.

I'll add to that: static typing also guarantees that your annotations are correct and complete. Prior to Mypy, in the Python world, we would document types in docstrings, which meant that docstrings gradually came to be incorrect--a function that returns a string might sometimes return `None`, but you called it assuming it always returned a string. In other cases, people (including the Python maintainers via the stdlib) will add annotations like "argument 'foo' is a file-like object", which is ... insufficient (do I just need a "read()" method? or do I need to implement "seek()" and "close()" as well?). Ultimately, in the absence of static analysis, a lot of time is wasted reading source code trying to understand the actual type signatures.

> Things have improved a lot in the Python ecosystem of late with the introduction of type annotations and mypy, so to the extent possible I treat Python as if it's a statically typed language, putting type annotations everywhere and always making sure the codebase passes type checking before pushing a commit.

This hasn't been true in my experience. Last I checked, Mypy still couldn't express things like recursive types (think "JSON" or "recursive linked list"), and a lot of things still required jumping through obscure hoops (defining a callback with kwargs). I also couldn't figure out how to export type annotations in my packages, and I had a lot of trouble getting mypy to pull in type stubs for packages that didn't provide their own annotations. In general, Python's typing story still feels very shoe-horned, although it's been a minute since I last fumbled with it, so maybe some of this has improved.

> for anything non-trivial I find that static typing means I can be more productive, because type errors are caught straight away (rather than potentially in production), and I can be much more confident in refactoring code because I know certain kinds of errors will be caught by during type checking.

Fully agreed. It's not quite the same as Rust vs. Python, but I've pulled off several sweeping refactors and rewrites in Swift codebases that I wouldn't dream of even attempting with Objective-C.

Python is fine, use the typing module, don't be weird with the return types and you will be fine. Sure, you will actually have to think about your algorithm when writing for speed, and naive loops won't work most of the time. But "avoid python in production" or "don't write new code in python" is way too harsh.

Having recently used it, my impression is Python's type system is pretty damn powerful. It might not be as crazy as TypeScript, but it can handle recursive types, dictionaries with a fixed set of keys, functions, union types and generics. Types can be re-used or exported. There are generic types for things like 'Iterable'.

If a package does not supply its own types (increasingly rare now), you can generate stubs and put them in typings/$package.

It's picked up a bunch of bugs for me already. And if runtime performance is not a priority, I'd rather write Python than Go or Rust any day. It's more fun and more expressive.

Does it support annotating numpy arrays with their dimensions yet? Last time I checked I was incredibly surprised the status quo was basically all numpy arrays are np.ndarray.

I think so:

https://numpy.org/devdocs/reference/typing.html

Which is cool, even strongly typed languages don't usually support that.

There is no example there that show that. I mean something like np.ndarray[1024, 1024, 1024, type=Int] which would mean a 3D array where each dimensions has 1024 elements. Basically all statically typed language support it. Even in C you can do it.

I said this for years... no type checking by the compiler? Then you are the compiler.

python i think is still good prototyping. I can do numerical work, visualisations, animations, game dev, web dev, etc. for me python is just the biggest collection of tools while not being the best at anything. which is an ok trade off a lot of the times

So I was one of the 3 hackers who wrote the LSM that predated Rocks at FB, and as a result I distinctly remember when Rocks overtook it in features by trading off performance, and again when it met and exceeded the performance while keeping the greater capability. RocksDB kicks ass and has for a long time, I’d recommend it to anyone in the market for that sort of thing.

FAISS is also great, and has been for years. I don’t doubt that someone has done a meaningful rev on it.

But as someone who likes Rust and wants to like it more: don’t lead with that. The software speaks for itself. Or should.

> But as someone who likes Rust and wants to like it more: don’t lead with that.

The original title appears to be “Inside the Pinecone”, but seeing as they submitted it themselves I’m guessing they wanted the uptick on the Rust.

The article itself doesn’t discuss their transition to Rust until the end. So I wouldn’t say they led with it originally, but perhaps didn’t get traction on the original title.

> seeing as they submitted it themselves

I'm grateful to muizelaar but I don't know who that is, so we can't take credit for this one.

Oh, sorry. I misattributed that. I saw the domain origin and confused that for the submitter. In that case, the submission actually is against the rules of HN, as I understand them (use the original title rule).

I guess I’m rooting for Rust in the long game sense, which is a different set of imperatives than the hang the hashtag on peripheral stuff sense.

I use a lot of great software written in Rust, it’s demonstrably a good vehicle for great software.

But too many of its fans are advocates, this can start to seem like an agenda. Don’t take engineering advice from people with an agenda.

Thank you for pointing this out. Lately I've been getting a culty feeling around Rust which has been turning me off to learning it.

but it's the same with every hype: there's something at its core and might be worth checking out

Oh I think a hype like this comes along once or twice in a career. Java was every bit as revolutionary in its day and had Sun Microsystems (they were a Big Deal) pouring every spare dollar into fluff pieces about it and it still never achieved a true Jehovah’s Witness vibe.

Well, Java had at least one aspect of a religion: a purity test. Remember "100% Pure Java" from the late 90s?

100% Pure Java is because Java is actually about four things. One of those things is a Virtual Machine, which means you don't give a shit that the implementer has never seen an ARM CPU, because you've got the Java VM, and so their code works fine on your ARM CPU.

But if their "Java" software is actually 10% i686 Machine Code then that won't run on your ARM CPU, you will need to undertake an expensive port or move to Intel. So hence 100% Pure Java == Write Once / Run Anywhere actually delivered.

You can argue that Rust is more than one thing, but certainly not to same extent. Rust's culture is crucial to its "A language empowering everyone..." slogan, but it's not really separate from the programming language, in the way that the Java Virtual Machine doesn't need the Java programming language or the Java security model (now obsolete) or the Java browser plugin (remember that?) and so on.

IMO that "culty feeling" you're getting is the end result of a lot of very experienced engineers encountering Rust, going "holy shit", and trying to preach about it from the rooftops because it feels so much better than the things that have come before it.

I find this particularly compelling because Rust generally does not market itself towards novice developers. At least until very recently, I've seen Rust communities actively discourage novices from trying to learn Rust. Not that it's inherently a bad language to learn as one's first, but there just haven't been great resources for people to learn it if they don't already have some previous software experience. The net effect of this is that the enthusiasm you're seeing is coming from a much higher proportion than usual of people who've been in the industry awhile, and not from as many fresh college or bootcamp grads.

No, the culty feeling is this paragraph long explanation about how Rust is objectively good, paired with the downvotes I got from saying I feel like Rust is culty.

Pretty sure most experienced devs will say "use whatever works, there's some art out there written in erlang/JS/C++"

Case in point I think the most valid critique is that Rust is new and lacks a developed ecosystem. As an ML person myself it's not even useful for me to learn Rust unless I want to roll alot of my own stuff. So if your "experienced developers" are preaching about the best languages to promote, to learn as a hobby, etc.. they're not preaching about the best languages for prod unless it's for some specific application.

Also, fucking lmao, get over yourself, Rust isn't that hard to learn

And even if it were, being able to understand what is essentially wacky C++ doesn't mean you've "been in the industry awhile"

> No, the culty feeling is this paragraph long explanation about how Rust is objectively good

So to make sure I have this straight, based on your reply to the sub-thread, your objection is simply that I didn’t repeat the exact reasons why many believe Rust to be an improvement over languages that have come before it?

That wasn’t even the damn point of my reply, which was that given the enthusiasm you perceive as cultish behavior, here’s my take on the underlying reason.

Something tells me you’d have been even less happy had I launched into a lengthy breakdown of all the comparative advantages. It’s a no-win situation you’ve constructed. Congratulations, I guess?

> Pretty sure most experienced devs will say "use whatever works, there's some art out there written in erlang/JS/C++"

There’s art in Ruby and I love the language but it’s a terrible choice for long-term projects that grow to large team sizes. A language doesn’t have to completely replace all others to warrant enthusiasm. It doesn’t have to be better along every possible axis to warrant enthusiasm.

> Case in point I think the most valid critique is that Rust is new and lacks a developed ecosystem.

Ah yes, the favorite criticism of bitter and cynical engineers, right up to the moment they switch to “it’s old and crufty and the industry has moved on”. Oddly, none of these critiques ever seem to apply to the languages of one’s own choosing…

> As an ML person myself it's not even useful for me to learn Rust unless I want to roll alot of my own stuff.

Nobody anywhere is arguing any language is the best choice for all use-cases. Nobody anywhere is arguing for ML people to rewrite-it-in-rust. These are straw men of your own making.

> Also, fucking lmao, get over yourself, Rust isn't that hard to learn

More straw men! I literally never said this. Only that the community has made few if any attempts to promote it to complete novices, thus skewing the experience level of those who you do see promoting it (compared to popular languages that do promote themselves to novices).

I don’t think Rust is hard to learn, but I do think Rust’s ownership model is hard to truly appreciate if you haven’t spent at least some time in the trenches of languages with manually-managed memory.

There's enthusiasm, and then there's this

> a lot of very experienced engineers encountering Rust, going "holy shit", and trying to preach about it from the rooftops

Preaching is like a religious thing. Why are people getting religious over a programming language?

People are obviously just making a living off of writing Rust and that's fine. I just find it so hilarious, dumbfounding and sad that people would stoop to such levels of dick-suckery over a programming language. Like, you all know it's going to replaced by Malbolge in 15 years, right?

> Preaching is like a religious thing. Why are people getting religious over a programming language?

My brother in christ it was a turn of phrase meant to be in the spirit of your original complaint.

You seem in need of an axe to grind (not an actual axe, this is another idiom). I’ll leave you to it.

How would you expect the response to something genuinely better to be different, in a non-culty way?

Well there you go, if it's genuinely better it will speak for itself.

See when I'm being an apologist I say shit like

"Python-like syntax but it compiles to assembly"

"You can mess with the AST like in LISP, to make everything differentiable"

"You can type the code for 10x speed, or leave it untyped for readability, multiple dispatch is basically magic"

You don't see me going around saying "Julia is objectively better"

And not only because I don't want to be a cultist but because it isn't and can't be true, something about no free lunch, something about Gödel incompleteness and expressivity, something something...

I mean, it's only natural.

I have been using C++, Python and Rust professionally, the developer experience is simply one order of magnitude better with Rust. The tooling is excellent, a sane compilation model brings you a lot, the type system is very helpful. Compared with C++, I measured x3 productivity on writing initial code on a project (sometimes C++ was the rewrite, sometimes Rust was, so not this kind of bias). Compounding maintenance, I expect it to tend to 1 order of magnitude in time.

Meanwhile, using Python can sometimes feel magical, but I have to maintain a library written in the language, and I can feel that it hasn't been optimized for this use case. It is terribly hard to keep backward compatibility, adding typing annotations is somewhat useful, but has terrible ergonomics, ensuring no regression is a pain that translates to endless suites of unit tests even for trivial matters ("be your own compiler" will never be a great idea IMO, although tooling like pylance does help a bit).

Meanwhile the rearguard is defending for new projects some kind of "No true Scotsman"ish modern C++ delusion, that I am still looking for and that moves like goalposts in every discussion I have with its proponents. My only explanation for the phenomenon is that it is fueled by hubris and sunk cost biases, because my experience with the modern idioms is that while they bring significant improvements, they are still very subpar compared to the expectations brought forth by a modern language, and memory safety related CVEs are still being written everyday in C++.

So, what should I do? It is true that, in more than 15 years of coding, Rust represents a revolution in programming in my eyes. Should I sit idly while others are missing on it and someone is wrong on the Internet?

I don't believe I've been an over-the-top Rust advocate so far. But I use Rust, I like it so far, and I want to use it more. For me to be able to use it more, it would help if it had the good attributes of a popular language, including a broad ecosystem of libraries and relative ease of hiring developers. So I want Rust to be a popular language. That means winning at the short game. Maybe other Rust advocates have a similar motivation to mine.

Sometimes an agenda is as simple as "I really like this thing, and I think you'd like it too".

I believe this is off-putting primarily due to the mental antibodies people must develop in an advertising-saturated culture. The death of enthusiasm.

> ..complex runtime issues which were almost impossible to reproduce or isolate

> That’s when internal murmurs about a complete rewrite started brewing…

> We decided to move our entire codebase to Rust

> there was still one minor problem - no one on the team knew Rust

"We have runtime issues so we will fix them by a complete rewrite in a language that no one on the team knows."

I would not call that a "hard decision", rather "rolling the dice". It awesome that they managed to make it work.

I mean when the tools you know are failing you, you might as well roll the dice.

They could have had some team members learn Rust before deciding that Rust was the right language for the re-write. It feels really risky to do such an important decision only on a high-level view of what you think the language will provide.

I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of.

Rust fit the need they had pretty well (on paper anyway since they didn’t have experience with it), and the industry is showing it to be a good choice for solving these types of problems. That’s not rolling the dice, it’s taking a chance based on research.

> I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of.

I think hoping a Rust rewrite would fix "complex runtime issues which were almost impossible to reproduce or isolate" as rolling the dice. If you cannot reproduce or isolate an issue, how can you know a rewrite in another language will fix the issue?

"Almost impossible" is not quite "impossible", and once you have done the almost impossible a few times and isolated the cause of some of the problems, you may have gained some apprehension of the scale of the overall problem you are facing.

> I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of.

Rolling the dice just means relying on getting lucky (instead of being skillful or controlling the outcome), usually on an unlikely outcome. When you literally roll the dice the odds are completely transparent (1/6 chance of each d6 value).

In this case (and in general with big rewrites), the thing you can never know about up-front - and therefore are relying on luck for - is whether you can build the new version within the time/resource budget that you allocate to it. If you get the estimation wrong, or hit unexpected problems/delays, then the rewrite can easily balloon from an ROI-positive project to a massively ROI-negative one. For a startup, this is particularly important because you could run out of runway before your rewrite pays off. This is why most people regard it to be a bad idea (or at least a risky one) to do a full rewrite of a product instead of incrementally evolving it. (Spolsky's essay on the subject I think is the canonical list of concerns with a rewrite: https://www.joelonsoftware.com/2000/04/06/things-you-should-...)

If you read it carefully you'll see that they "rewrote" it by replacing components of it. Which:

1. Allowed them to evaluate if it would actually work. 2. Allowed them to train the team on it in a scalable way.

The result is that the team learned Rust and Rust fixed a bunch of problems. They were appropriately pragmatic in their approach and Rust was appropriately pragmatic in it's solutions to their problems. If you are ever considering migrating your codebase for some reason this is how you should go about it.

TLDR: You are heavily mis-characterizing their team and this article.

I read it as Rust forced the developers to be specific with their implementations and problems went away as a result.

I say this because I'm often _AMAZED_ at just how undisciplined the majority of developers I work with are wrt stability and error propogation.

Yeah, I am really surprised they managed to make it work, actually.

Rust is one of those languages it takes time to master... if you write a lot of code in it while you're learning, you're likely to make all sorts of mistakes that later will need to be cleaned up... at least, looking at Rust code I wrote when I was early in the learning process, I can't even imagine any of that code going to production... so I would say it worked for them either because they are highly skilled and can learn difficult things like Rust quickly and efficiently... or their previous code was so completely broken than anything else would be better :D...

As a relative newcomer to Rust, I'm curious to hear about those early mistakes

Rust is very good at telling you, the developer, that you've got some flaws in the way you've laid out your program.

What it doesn't tell you is that this is frequently a result of having a design that doesn't naturally fit into the kind of more restrictive paradigm that Rust wants you to write programs in. This is a natural consequence of moving from very forgiving languages where almost anything goes (e.g., garbage-collected languages). You'll try to write your software using paradigms that are familiar to you, and many of those don't fit well into the set of restrictions Rust forces upon you.

There are escape hatches that let you get around these limitations, and people become very accustomed to using them early on. This happens because the Rust compiler tells you what's wrong and that you can use one of these escape hatches to fix it. But as you continue to build, you have to use more and more of these to overcome the mismatch in your design and the kinds of designs that fit well into Rust's ownership model.

I believe it's very important to—early in your Rust career—try and understand if and when the compiler is trying to tell you about a deeper problem with your design than simply throwing surface-level ownership nits at you. I think for some people who are very used to RAII in C++ or very strict C styles, this can come somewhat easily as a lot of the patterns are common (though not enforced at compile-time as in Rust). It is much more difficult for people who are used to garbage-collected languages that tolerate program designs that are entirely unsuitable for languages without a GC.

What the sibling comment said is spot on! Rust requires you to design programs very differently than what you do in GC-languages.

You may try to avoid using things like lifetimes for a while, but those are completely undispensable to writing Rust... same with macros. They are really important and widely used in Rust.

One very good way to avoid simple mistakes is to configure Clippy to run on your IDE (or just run it manually): https://github.com/rust-lang/rust-clippy

Writing Rust at scale, I concur with some of these findings. Nowadays, my team exclusively writes Rust. Our rationale is two-fold.

1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency.

2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language.

Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases.

The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.

I am curious which web framework do you use? Also do you use an ORM like diesel?

You could probably get a pretty good idea from reading the Discord engineering blog.

Yep. I checked his bio only after commenting.

We use a mix of axum, and tonic/tower, depending on if the service is internal (grpc) or external (http).

And no, we do not use an ORM like diesel, since we primarily use ScyllaDB. We wrote our own - it's very nice. We have a fully type-safe scylla/cassandra client, that does static validation of the queries (which we define as annotated structs), and also, when the service starts up, it validates that the schema that the service has agrees with the schema the database has. Insofar a making sure that if you say `WHERE id = ?`, and you pass a i32 but it wanted a String, the service will not start.

It also does request coalescing and tracing/telemetry, and will soon do rate limiting, circuit breaking and speculative retries as well.

> We have a fully type-safe scylla/cassandra client, that does static validation of the queries

Is this on a crate by any chance? Thanks

Would love to see how your ORM compares with Catalytic, the Rust ORM for ScyllaDB.

https://www.scylladb.com/2022/02/15/introducing-catalytic-an...

I like this "and an overzealous compiler increased engineers’ confidence in pushing changes".

If you think about it, you can try to optimize a Rust program through trying brute force stuff and trying weird things that you are not even sure will break something or not. But you can try ~without 100% knowledge or confidence~ and boom the compiler will tell you if you are shooting yourself in the foot. Now try to that in C++. Even with 95% knowledge and confidence I will be very worried as a Lead or CTO.

> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. Built-in testing, CI/CD, benchmarking, and an overzealous compiler increased engineers’ confidence in pushing changes, and enabled them to work on the same code sections and contribute simultaneously without breaking the code base. Most impressively though, real time operational events dropped almost to zero overnight after the original release.

I think Rust is really shaping up to be in a sweet spot.

One thing that is not talked about enough with Rust, is that you do not need to start with fighting the borrow checker from day one.

Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python.

As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc.

Using Rc unnecessarily is a code smell commonly called Java Disease.

> Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python.

> As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc.

I've always found this advice to be infeasible in practice, despite sounding like a reasonable pitch.

Arc allows you to avoid thinking about ownership, which is largely what dictates the architecture when writing idiomatic Rust. Plastering on ownership later is going to change the code so much you might as well rewrite it .

Moreover, if you want to prototype (with Arcs, clones) a new part of the code, while integrating with a more mature part of the code base, you still need to adhere to the ownership structure.

In fact, this is one of my main issues with Rust, that it's a terrible language for prototyping. I don't have a solution, and I'm not blaming Rust – it's still a great language. But you have to have a very solid design/mental model before you start coding.

I'd disagree. It depends on the use case a lot but I encourage people to reach for reference counting when they have complex lifetimes. The cost is almost always trivial and for most use cases the lifetime analysis is clear and unlikely to become a problem in future refactoring.

There is obviously always some nuance, but I find that there is a wide useful space between Rcs that can be avoided quite simply and making dramatic changes to your code to avoid an Rc.

I don't want to be dismissive but how much of this success belongs to the team of rock solid senior engineers with enough experience under their belt rather than rewrite in Rust?

I would let the same team of "rock solid senior engineers" be the judge of that. They seem to think it mattered significantly enough.

Or ... since they are now betting fully on Rust, they now want everybody else on Rust too. Because more people == a higher probability that if they run into issues with Rust someone will fix it.

That's a really cynical take on the matter...

The writer seems to give massive credit to the team.

You can give some credit to the fact they were writing it a second time certainly, but as far as we know at least a lot of the people were the same ones who were central to writing V1. So certainly there was some lift from already doing it once, but if they went from a lot of subtle bugs they struggled to track down to having far fewer of them, and those bugs tended to be the types of issue the rust compiler and/or linter normally point out to you, it will certainly make it easier to mitigate the issues.

Rust cannot make a bad programmer a good programmer, but the fact it points out several classes of mistakes (sometimes too aggressively, hopefully someday we get Polonia and other improvements to the borrow checker to remove some things that are hard only because of the current implementation of various checking systems) allows far more confidence. I don't care how good someone is, having an automated way to validate large scales of issues aren't there is a big deal, and unlike unit/integration tests you don't have to write these.

It’s likely they could have chosen to stay with Python, C, and C++ and achieved similar results. I’d bet a pretty penny that the fact they were building on lessons learned with the rewrite accounts for almost 100% of the success here.

A team with no rust experience

This mirrors my experience: getting buggy software out is easier and faster in JS (or whatever you're familiar without compile guarantees), getting something out to production is easier with rust.

Sometimes you have enough senior developers and discipline to reproduce and catch obscure bugs (which happen more or less based on your software complexity) and you can fix your mess but that's not always feasible.

The more senior I get (decades of experience now), the more I appreciate languages that simply remove the mere possibility of entire classes of bugs. Even in Objective-C — a dynamic language with compile-time type enforcement — if something very weird is happening you have to start with “is that object really what I think it is?“

Writing something for the second time is a significantly different experience so that makes the conclusion much less definitive to me. Did they need python at all the first time - maybe not.

Python "velocity" is, to me, more about the ease with which you can make changes without breaking other code. You can still break other code and if you don't have a proper test suite then you're wasting your time - which is something that smells slightly in the story.

Yeah. Having runtime errors in a non-trivial Python code base is indicative of poor test coverage.

This is my feeling as well. The biggest problem with new projects is that you don't know what the product will look like in advance, even if you think you do. Restructuring code, especially with full 20/20 hindsight, will make that code much better.

The title is "Inside the Pinecone". "Rust – A hard decision pays off" is only one of the three major subsections of this article.

Interesting to see the part about Rust getting the most attention. We're giving a talk about it in NYC next week[1] and will definitely follow up with an in-depth writeup.

[1] https://www.meetup.com/rust-nyc/events/287821884/

The submitted link auto-scrolls me to the section on Rust, so I guess the submitter wanted to promote that specific section of the article?

Gotta mention rust, how else will you get upvotes on HN?

The title of the section posted is "Rust - A hard decision...".

It's been posted here with the anchor included making it clear that it is that section that has been submitted.

Indeed, I didn't notice. Sorry!

Maybe it’s just me but it seems irresponsible to move your entire team to a programming language none of them knows.

Rust is great, but modern C++ is great too.

> modern C++ is great

Great in the way that Alexander the Great was great. Achieved big things and killed a lot of people.

Though in the case of C++ they're just dead inside.

There are better threads to trash C++ hackers on than this one. It’s literally about insanely successful software written in C++, maintained successfully for a decade, that seems to have survived a port to another language.

>"Though in the case of C++ they're just dead inside."

I think this is pure and memory safe BS

> it seems irresponsible to move your entire team to a programming language none of them knows

Based on my experiences, I would agree. There seems to be a very important difference between my experiences, which probably align with most of the industry, and the Pinecone team. My career has been working with average developers doing ordinary business and internet things. Stuff that takes organization and teamwork, but not necessarily advanced or esoteric knowledge. So the COBOL programmers that went to the "Java in 21 Days" training that I consulted with, or the C# programmers who went with Go, they did poor-to-OK, but that was Good Enough.

The Pinecone team, though. They were already writing C/C++ to solve Hard Problems. As experienced programmers, they probably already understood the landscape of systems languages with pointers and having to manage their own memory. Rust, while imposing, would not be a huge leap for them.

Another aspect worth noting: they did they re-write incrementally, never completely abandoning the existing system. At least it seems so. I'll be interested in seeing the presentation from the upcoming talk about the rewrite that's mentioned in the story.

If you start the article at the top instead of where the hot link points (which is its own discussion) it starts by talking about mathematically proving why this form of database is faster than theory suggests it to be. Definitely not your average boot camp devs working on this one.

Precisely. Also, it wasn't one of those cases where management decides to move people off of the language they know to one they read about in InfoWorld or CIO Magazine. Or worse, had a golf buddy tell him how great it is.

In the words, the team wasn't moved, the team moved itself.

>but modern C++ is great too.

No, modern C++ is still cobbled together from tools built in a different era. It doesn't even come with a package manager/build system.

Just on the fact that cargo/crates.io exist and it has modules - it would have to be really bad to make me go back to C++ if I ever need to write something at that level.

There are package managers and build systems. MS ships a build system with visual studio and msvc and they have a fairly capable package manager in vcpkg.

I like cargo better than any c++ tooling but those tools don't really make a ton of sense for c++ where every platform has a different compiler.

Ultimately, I agree. I don't think I'll be going back to c++ any time soon, but for me it's more about the tooling fragmentation than the tooling itself. I actually like the language itself.

I mean sure, there are many environments you can use for any particular platform - but that's not how I develop software these days - I need my code to run on my Mac, my coworkers Windows machine and our deployment servers on Linux - with the least ammount of fuss possible.

I wince when I think about how much time I spent dealing with that crap back when I was doing C++.

Yes. This comes across as "we are so smart, this project wasn't hard enough to demand our full attention".

Coding in a manifestly immature language signals that you are not really serious about being used where long-term support across a variety of platforms may be important. The language is still more likely than not to fizzle. If it does, it makes your flagship project a niche player.

> The language is still more likely than not to fizzle.

What makes you think that?

Fizzling is the normal fate of any new language, absent a miracle. Only a tiny handful of languages get a miracle.

In any given week, more people pick up C++ or Javascript to use professionally than the total number now employed to code Rust. Not to fizzle, it has to increase its adoption rate by orders of magnitude, but its fans are almost uniformly hostile toward any measure that could make it easier to adopt.

> its fans are almost uniformly hostile toward any measure that could make it easier to adopt.

Do you have an example exhibiting what you mean here?

I would self-describe as a fan of Rust, but I also think I'm pretty realistic about its limitations and that every language has its niche. I will say that as far as miracles go, being the most-loved language on SO for seven years seems like a bit of a (minor) miracle here, and at about 87% loved vs Python's 67%, it's not exactly eking out a win. It'll take time for it to grow in market share, but it seems likely it's on its way, given that it is, for example, breaking into the Linux codebase - something C++ wasn't able to do.

> breaking into the Linux codebase

Is Linux really your go-to example? Linus is noted for his irrational hostility to C++. His complaints are proved silly or, at best, obsolete by the example of Serenity OS.

An example of a measure to make the language easier to adopt is to allow borrow violations to be made a warning when building in debug mode. All violations would need to be patched up before release anyway, so it does not change the language any, but the violations might be in code you will delete before that. Forcing borrow cleanliness in dead code before you can work on important code feels just spiteful.

People invent the most ridiculous objections, including that the compiler would not know what code to generate, or that it would split the language into two dialects. My conclusion is that the loudest Rust fans like that they are a select few, and don't want it adopted mainstream. They might win.

It seems like a bad idea to allow what is arguably the main selling point of the language to be turned off by a compiler flag so someone can build the rest of their program on top of that shaky foundation.

Whole hog or no hog, this is the rust way.

I.e., a ridiculous objection that demonstrates wholesale incomprehension of the topic.

This is an example of why I expect the language may still fizzle.

You seem quite certain of your opinions, especially that Rust will fizzle out. I'll bet $100 that in 2025

- it's still used at Meta, Google, Amazon and other large companies. Once a language gains critical mass at a large, profitable company it will be maintained one way or another.

- It will be used in the Linux kernel as well.

If you have any other objective measures of usage/popularity/usefulness, I'd like to hear it.

> In any given week, more people pick up C++ or Javascript to use professionally than the total number now employed to code Rust.

If you can tell me how you measure this I'll take a bet that this is wrong as well.

Every language you can think of, and many you can't, are in use somewhere at every large company. That is no distinction at all. So, no bet.

If it gets into the Linux kernel in any capacity, obviously it will still be in there as long as anybody still uses whatever it does. That, also, is no distinction at all. So, no bet.

I am not expressing opinions. I am expressing observations, and expectations that seem to follow. If you have other observations that bear on the question, bring them.

You say this language will fail because most languages fail. Ok, make a falsifiable statement and I'll take a bet on that.

You seem very confused. I have not, in fact, said this language will fail.

Bets are not interesting. I would like for Rust not to fail, but do not see the things happening that would prevent it.

You said, "The language is still more likely than not to fizzle." What's the distinction between "fail" and "fizzle"?

Not much. But there is a very large difference between "will" and "might".

Search your heart, do you think this story ends similarly if they decide to all do "modern C++" instead?

My guess is that even with initial discipline in the form of code review and style enforcement by one or two people, the C++ descends into a riot of different opinions about, as usual, which are the good bits.

A similar piece of software could be written in C++ but I doubt it gets written successfully, in similar time, by this team. I reckon the post mortem of such an attempt would be written off as "Don't do rewrites, duh"

You might want to take that up with the authors of RocksDB and FAISS. They seem to have shipped some cool stuff.

RocksDB, as I understand it, is a pretty old piece of software, forked from a Google product, which was already a pretty old piece of software, all written in C++. Doesn't seem to have previously been written in Python, doesn't seem to have a team of non-C++ programmers who decided to just learn as they went, and so it's not clear what parallels you expect to draw.

Maybe they're doing a Rust rewrite too? I'd be interested to read about that.

This is an exceedingly silly conclusion. "Similar pieces of software" are coded in C++ and shipped every single day, without anyone saying "and we managed to do it in C++, too!" The emphasis should be on what the software does well, and how, not on what hoops the developers were jumped through on the way.

> Search your heart, do you think this story ends similarly if they decide to all do "modern C++" instead?

https://github.com/facebook/rocksdb

> Search your heart, do you think this story ends similarly if they decide to all do "modern C++" instead?

It sounds to me they already had a good chunk of it written in C++ with python glueing it all together.

From a ‘dev velocity’ standpoint one would think having the senior devs learn a new language and rewrite an entire database wouldn’t make sense when they could just systematically replace the python parts with C++, which they already knew.

Or, who knows, maybe all their initial issues stemmed from the locations of semicolons around the else keyword?

> Search your heart, do you think this story ends similarly if they decide to all do

The main difference would be there wouldn't be a blog article on the front page. :)

Seems more like the team moved themselves to Rust

Something I still struggle to understand with these systems is why indexing 1-10M things is hard / their sweet spot, when that is basically a small dataframe in pandas or a small SQLite, vs the 10B+ rowa // GBs+ I was expecting them to be discussing

Frankly on larger projects overall time to create and test the whole thing in modern C++ or (I assume) in Rust should not take longer than in Python. I do not understand this idea of mixing languages where one is enough.

In my own case I rewrote decent size Python app in C++ and it had taken me about the same time as for original developers. And no I was not porting code but rather working with the specs so did my own design.

Sidenote, been using pinecone for semantic search recently, and it's been a joyful experience.

I'd be interested in a bit more detail on the technical migration strategy. Did they just cut over to a version of the project built in Rust? (I.e. vN is Python+C/C++, vN+1 is Rust?) Or something more gradual?

How did they verify that the new code was conformant with the old app's behavior/logic?

> To make matters worse, we would discover issues only after deploying (or in production!) due to Python’s run time nature.

I don't want to infer too much, but this makes it sound like perhaps they didn't have a very robust set of E2E/Acceptance tests, which would make a full-cutover migration scary to me. If you're finding Python bugs only after deploy, how can you find the inevitable rewritten-Rust-code incompatibilities before deploying/production?

I've been digging into Rust/Python interop recently using https://github.com/PyO3/pyo3 and maturin, and this points to an interesting migration strategy; PyO3 makes it quite easy to write a Python module in Rust, or even call back and forth between them, so you could gradually move code from Python to Rust. A migration path to full-Rust might be:

1. Incrementally replace all your fast-path C/C++ code with Rust, still having Python call these compiled modules. End-state: You now have a Python/Rust project instead of Python/C/C++.

2. Gradually grow the surface area of your Rust packages, moving logic from Python into Rust. Your existing Python tests can still run, and your existing entrypoints are the same Python.

3. At some point, you presumably need to cut over the entrypoint layer (the API?) to use pure Rust, instead of Python. This should be a much less scary migration since both versions are calling into the same underlying Rust library code. Depending on your architecture, if you have an API Gateway you can split your service backends to migrate one endpoint at a time to the new Rust API service, while keeping the old Python API service around to fail back to. (They are using k8s so you can do this with your Ingress for example).

I'm interested in others' experiences with Rust/Python interop, are there any rough edges worth knowing about?

Nice to see that Rust is working for your particular usecase!

Dev velocity with new projects is often higher so I am not convinced by that argument.

With that said. I learned Rust recently. I think it is a viable choice for a business core. Rust is not at all that difficult to learn as some claim. The tuts are great. Some things are very different though.

Any plans to open source the kv store?

> we would discover issues only after deploying (or in production!) due to Python’s run time nature

I love Rust but seems like they just needed to write some test before shipping to prod?

> Built-in test, CI/CD

Does it have anything to do with the language?

Python has built in testing and C++ has libraries like catch2. I would have liked some initial comment that said "we had a good suite of tests". Python has plenty of benchmarking tools available.

IMHO - a db is all core. Everything should be fast. Even the CLI tools - Rust is pretty good at cli tools. Why is he building a database out of python? How much python is in this repo anyway?

Also this is a Python + C++ app - not python. And the article critiques python, not c++. Which is weird, because I would have thought the source of problems would have been C++ not python.

Don't get me wrong - Rust is great. Cargo is great too. Compiled programs are great to deploy rather than python (pip hell) and C++ ({CMAKE_HELL}). But this is apples to oranges comparison.

The things that make Rust better than C++ / Python are not the things this article talks about.

Would you guys start a GraphQL Api today using Rust? I'm very tempted to hint at that but am still insecure about my 'options'. This is coming from a very comfortable python + fastapi + sqlalchemy ecossystem

The steps some people to go to to avoid writing tests is shocking. (I say this as someone who has written Python programs and been tired of deploying them only to find a runtime bug, and solved it by writing tests)

We're rusting at the moment too. Encouraging. :)

Has anyone ever compared Rust with Modula-2/3 or Oberon family of languages? I would think it would be an interesting comparison.

This very closely matches my own experience nearly 4 or 5 years ago now rewriting an ad server in rust from another scripty language.

This is all directly in line with my experience having ported several production applications to Rust.

Leading with several pages of what amounts to smug self-congratulation does not inspire finishing.

Rust deserves mention in the title only if actually getting it done in Rust was particularly difficult.

so how are graphs used for vector search?

See: https://www.pinecone.io/learn/hnsw/

Although our graph-based vector index does not use HNSW, the concept is similar.

;lkj

> We decided to move our entire codebase to Rust (and Go for the k8s control plane).

> there was still one minor problem - no one on the team knew Rust.

Is this real or satire? I can't tell.

How hard can it be for C/C++ devs to learn Rust? Python devs on the other hand..

That's not the point. If no one knows the language, no one knows if it's appropriate. All you know is the language marketing claims

Even though it's software engineering, which supposedly isn't really engineering, you can still apply engineering principles. Disregarding all the bullshit debates about whether Rust is better than Go, there are actually differences between the languages that you can objectively research.

Maybe no one in that specific company really knew the language, but there's loads of people actually using the language, it's not like it dropped from the clear blue sky. It's not like there's a Rust salesperson that knocked on their door with a leaflet. They're not even the first persons to write a database engine in Rust.

I wonder if other engineering domains have this same silliness. Do the architects have intense debates over steel H beams vs wooden LVL beams, and at some point newbie architects believe you should only switch from one to the other as an architecture firm if your architects have experience with it, because the only way to know if it would work is reading the marketing claims.

A compiler is just a tool. If you're doing a rewrite you better pick your tools correctly, and it seems like they did. You could theorise they were lucky somehow, or you could recognise that they're experienced, educated and skilled and made the right choice based on their researched.

> Do the architects have intense debates over steel H beams vs wooden LVL beams, and at some point newbie architects believe you should only switch from one to the other as an architecture firm if your architects have experience with it, because the only way to know if it would work is reading the marketing claims.

That is one way to put it, which is convenient for the point you're making. A different way, more convenient for the point I'm making is that if you're budding roads and you need a bridge, you probably wouldn't hire a firm that only has experience building tunnels.

So you see, your analogy is meaningless, because making an analogy is not a convincing way of making a point. Anyone can make an analogy.

My argument on the other hand that experience is a very important factor, is a lot more convincing because anyone who has any kind of professional career already agreed with it.

I think at this point Rust has more than marketing points. Then if you have some bright people, and I mean doing what they are trying to do and feeling that RockDB is a slow database... They should be able to pick it up. I don't think it's an unreasonable decision as a CTO/CEO.

It's probably exaggerated. I doubt they committed to the re-write before anyone knew Rust.

Hopefully what they meant is that Rust was only being considered when they started having the team learn it, or that the team was not extremely proficient at it when they picked (but had sufficient knowledge of the language).

Not saying that this part didn't surprise me as well. And if they did commit based only on the marketing material... what a gamble.

Expectation: this article.

Reality: It's insanely hard to find talented developers, but Rust makes people excited, so they are just using it as a marketing tool.

> Simultaneous fetches of thousands (sometimes tens of thousands) of objects started becoming inefficient, especially when fetching from collections of tens of millions of objects

Why not try leveldb. We're doing random reads of 170,000 vectors (/130M) a second. No startup needed.

Isn't RocksDB a fork (reimplementation) of LevelDB? :o

RocksDB started as a fork of LevelDB (Dean/Ghemawat).

The performance imperatives have changed with the hardware, but NAND flash at that time had an asymmetry between reads and writes in terms of the amount of data: once you were writing even one byte you had effectively paid for writing a whole block and were therefore incentivized to get your money’s worth and write to a “log”, which then would be “merged”, in some “structured” way, hence LSM.

This is old news these days but it was quite the novelty at the time!

Reply on news.ycombinator.com