Hacker News Reader

Comment by Nelkins | Hacker News Reader

So, I've been eyeing Rust for a while and have messed around with it a bit in my free time. Here are some things I'd want to know before trying it out on a large project:

1. What development environment are other developers using with very large code bases? Is the tooling responsive?

2. Using a language for a large project without some kind of async/await notation seems painful. How bad is it using only combinators for async (assuming you want to stay on stable)? And is there a date yet when async/await will be stabilized?

3. Most of the code I write is in F#. The nice thing is, there is basically .NET library for everything under the sun. For example, over the weekend I was looking for a library that could decode OBD-II data from a car, and with a quick DuckDuckGo search I was able to find three or four. In a large-ish project, how often do Rust users typically find themselves coming up short for that kind of stuff? Obviously it varies a ton by domain, but maybe someone could tell me whether the answer is "you'll probably write all wrappers for 3rd party APIs yourself" or "there's a decent chance there's a library out there that you can adapt."

View on HN

>2. Using a language for a large project without some kind of async/await notation seems painful

And yet most of the software we use, billions of lines of code, from Chrome to Linux, from Photoshop to bind, and from Nginx to Skype, has been created without such notation. Somehow we've managed.

"Managed" and "seems painful" don't disagree with each other.

That doesn't make the assertion that you need async notation for any large project to not be painful any less ridiculous.

Of course you don't NEED it. It's just that I've become accustomed to the bells and whistles (and accompanying productivity) of a modern programming language. If I'm considering a new programming language I'd like the feature set to be comparable.

1. I personally use VS: Code and Vim. IntelliJ has an officially supported plugin that I haven't used, but heard good things about it. The Eclipse project is considering making a Rust IDE but hasn't decided yet. VS non-Code should work now, but I haven't given it a go personally. On the "responsive" question, it's still actively being worked on, generally, so "it depends". We're actively investing in things being more robust.

2. We expect async/await to be stable before August, but as always, It Depends.

3. As you say, varies a ton by domain :) What I will say is, two years ago, crates.io had ~3000 packages, last year, 7500, today, almost 14,000. So we're getting there...

Steve, just so you know (for future answers), the IntelliJ plug-in is stupidly good considering its relative newness. You shouldn’t hesitate to recommend to existing IntelliJ users. :D

Understood :) I usually try to be really clear about what my opinion is based on, especially when it's second-hand, like with IntelliJ.

Have you tried debugging with CLion? I don't own a license so I haven't tried myself.

CLion debugging works pretty well for most cases, though there are some small issues every so often. I always make sure to report them to the appropriate place if they're not known already, though.

Great, thanks!

> Using a language for a large project without some kind of async/await notation seems painful. How bad is it using only combinators for async (assuming you want to stay on stable)? And is there a date yet when async/await will be stabilized?

Can someone explain to me what async/await is, in the context of a language like Go? I've used Rust a fair amount, but Go has been my native language for ~5 years now.

With that said, aren't both languages synchronous languages where you can drop into another thread (or green thread) whenever you want?

What UX is improved by async/await?

Eg, in Go I never felt the need for async/await (as I've used in JavaScript at least). If anything, Go's is better than async/await to me, because if I call `foo := bar()`, I don't have to know if `bar()` is an asynchronous function.. it just works. Is there something missing there?

Again, I use Rust and Go similarly in this post because Go is my frame of reference. I thought they were the same on this front, though. No?

What's missing here? What am I missing here?

Yeah, you are missing some details. But this is a huge area, and it's hard to explain in a single HN comment. Basically, yes, the convenience of not needing to write async/await is useful, but in order to accomplish that, you need a bunch of other supporting decisions. These decisions make sense for Go, but do not make sense for Rust.

The first thing to say on this topic is that virtually all of this (including async/await) is a library in Rust, not a part of the language. You can build whatever model you want on top. The one I'm describing is the futures/tokio model, which is becoming the de-facto, default one. But if you want something else, you can do it! This already is a divergence from Go, where you have what the runtime gives you, and that's it.

Fundamentally, Go's model is cooperative coroutines, where the runtime does the pre-emption for you. (You can also call it yourself but as you note, this isn't usual.) It'll do this when you request something from the network, when you sleep, or when you use a channel.

Tokio's model, on the other hand, is single-threaded. Instead of a ton of gorutines, there's an event loop, and you place a chain of futures onto it. Each chain of futures can sort of be thought of as a goroutine, but it's also different. For example, Rust can know at compile time exactly how big each futures' stack is, and so can do a single, exactly correct allocation. Anyway, each future in the chain handles yielding for you.

Async/await, in Rust, is about making those chains easier to write.

Anyway, I hope that helps. Maybe someone else will have a good comparison too. There's a lot of options in this space, and similar terminology that means the same, but slightly different things, so it can be hard to get your head around.

Technically you can get the convenience of not needing to write async/await, with the same runtime implementation decisions underneath as Tokio/futures. Kotlin is an example of this approach- its coroutines are state machines but with implicit awaits.

This can even be extended to "awaiting across function calls" with effect polymorphism. I haven't seen this done in this context, but it would allow things like passing an async closure, or a generic type with an async trait implementation, to a normal function and having it automatically become an async function instead.

The real choice between explicit and implicit suspension points is thus syntactic, not implementation-driven. The reasoning there is more along the lines of "I like to see where my function might suspend" vs "I don't want to pepper my code with `await`."

> I haven't seen this done in this context, but it would allow things like passing an async closure, or a generic type with an async trait implementation, to a normal function and having it automatically become an async function instead.

I've done that a fair bit in Scala using HKT. E.g. superclass is written in terms of a generic monadic type, one subclass implementation uses Future (actually EitherT with Future), another uses identity, a third uses Writer to track statistics...

Ah, right. I guess I have seen this sort of thing in Haskell then as well. When implemented with HKTs it usually gets really messy with things like monad transformers, so I prefer to think of asynchronicity more in terms of continuations.

I find monad transformers much clearer than continuations; a monad transformer stack tells you exactly how your effects are going to be interleaved, which is necessarily complex, whereas a continuation could be doing anything at all, the way I see it.

No, continuations can be controlled similarly through effect polymorphism, which is what I was trying to describe.

The difference is not that continuations allow anything at all, but that your effects are commutative rather than layered like a monad transformer stack.

> continuations can be controlled similarly through effect polymorphism, which is what I was trying to describe.

Fair enough, but you're effectively talking about a secondary, novel type system, right? One of the things I like about monads is that they can be just plain old values of plain old datatypes.

> The difference is not that continuations allow anything at all, but that your effects are commutative rather than layered like a monad transformer stack.

Yeah, monad transformers do feel overly cumbersome for the cases where effects do commute. The trouble is that some effects don't commute, and I've yet to see a formalism that had a good solution to distinguishing effects that commute from those that don't. (I read a paper about a tensor product construct once, but couldn't really follow how it was supposed to work in practice)

> you're effectively talking about a secondary, novel type system, right?

Yes, exactly.

The main reason I prefer language-level effects rather than monads is for composability with normal imperative control flow. Monads are written in terms of higher order functions passing closures around, and that gets really messy in a language where you not only have early return and loops with break and continue, but you also have destructors that need to run as you enter and leave scopes.

You can add that kind of stuff as monads but it gets really messy, and is basically completely untenable in a language like Rust that also cares about memory management. Even if Rust did have HKT, it would still be impossible to write a Monad trait that supports them all, for example.

This article makes some great points on the subject: http://blog.paralleluniverse.co/2015/08/07/scoped-continuati...

I have been meaning to dig into Kotlin's implementation, thanks for that! Is there any good reference documentation to dive into? Most of the stuff I saw was from a user's perspective.

I used this: https://github.com/Kotlin/kotlin-coroutines/blob/master/kotl...

Adding a recent detailed take on the situation, by another Rustlang contributor: https://manishearth.github.io/blog/2018/01/10/whats-tokio-an... . Hope that helps :)

> Can someone explain to me what async/await is, in the context of a language like Go?

As far as I know:

    c := make(chan int)
    go func() { c <- 1 }() // async
    i := <-c               // await

Goroutines allow you to write async code ergonomically. Without those, you'd have to write callbacks, which is very unergonomic.

Here's a before and after diff of some prototype async/await usage in Rust: https://github.com/mehcode/shio-rs/commit/8078a34c075bf2f52c...

async/await means you mark all your yield points explicitly; rather than having the runtime implicitly preempt you whenever it chooses, you mark the points at which task switching can happen, and at every other point it's impossible (equivalently it's as though any block of code that doesn't contain an "await" were a critical section). The syntax strikes a nice balance, making these markers as lightweight as possible but no lighter: you don't want the yield points to be completely invisible, but you don't want them to distract too much from reading through the straight-through happy-path control flow.

I find that having one implicit, pervasive, uncontrolled effect in a given piece of code is ok, but having multiple uncontrolled effects is very much not ok, because the ways those effects will interact can be very surprising. E.g. https://glyph.twistedmatrix.com/2014/02/unyielding.html gives an argument for using explicit async/await rather than go-style implicit (green) threads, in terms of how task switching will interact with state mutation. You can make similar arguments in terms of how task switching interacts with error handling or logging or database transactions or... - one uncontrolled effect is fine, multiple uncontrolled effects cause chaos when they interact. Unfortunately the kind of examples where one can see multiple effects in a single application tend to be quite big by their very nature (and if your app is small enough to only need to have one kind of effect, then using a language in which that particular effect is uncontrolled is probably fine, perhaps even a good idea).

> async/await means you mark all your yield points explicitly; rather than having the runtime implicitly preempt you whenever it chooses, you mark the points at which task switching can happen, and at every other point it's impossible (equivalently it's as though any block of code that doesn't contain an "await" were a critical section).

That's not how I understood it at all. I always assumed that it just promoted explicitly acknowledging when the effect of an async block would be present, but that the normal yield points (traditionally syscalls and IO requests in OS threads) generally still held for the actual execution of the other code.

I mean, if we aren't allowing separate code paths to execute (and thus reducing wait on resources such as IO), what's the point?

> That's not how I understood it at all. I always assumed that it just promoted explicitly acknowledging when the effect of an async block would be present, but that the normal yield points (traditionally syscalls and IO requests in OS threads) generally still held for the actual execution of the other code.

Put it this way: the language implementation won't preempt you except at your explicit yield points (some standard library functions for things like I/O will and should be yield points that you have to call with async/await). If the language implementation happens to be running as a userspace process on a preemptive multitasking OS then it will still be subject to the same preemption rules as any other userspace process on that OS, but if anything this is usually counterproductive (e.g. priority inversions are almost guaranteed); recommended practice when working in green-thread-based systems is to run with one thread per CPU core and maybe even pin threads to cores if the OS lets you do that, because the language's own M:N scheduling will keep those threads fully occupied and OS scheduling is only going to get in the way.

I think I see what's going on here. You're answering "in the context of Go" (which is correct, and what was corrected) and I'm interpreting it as a general statement about what async/await mean as general concepts (i.e. in other languages as well). My mistake, I wasn't paying close enough attention. :)

s/and what was corrected/and what was requested/ Ugh, so many typos or word mistakes lately. :/

Having used both, I'd be pretty shocked if anyone found Twisted to be easier to reason about or less prone to bugs than Go...

Twisted predates async/await syntax in Python.

Isn't gevent still better than async/await ? (if it had core support it would be, kinda like golang)

Async/await makes sense in python or JavaScript because it’s syntactical sugar that improves readability/maintainability and mitigates against concurrency errors stemming from the event loop (JS) or the GIL (python). I haven’t worked with Go beyond playing with it a bit, but it seems async/await would be utterly useless in Go since goroutines already accomplish the same thing in a much more performant manner.

C# was, as far as I'm aware, the first major language to implement async/await atop of Task<T> (akin to Java's Future<T>). The CLR uses native threads.

Go channels are also orthogonal to async/await. Message passing is not a substitute for futures/tasks, though it can be used to achieve similar goals. I would be extremely cautious about claiming that Go channels would be "more performant" than an otherwise-equivalent futures implementation, too.

Goroutines are not the same thing as Go channels.

Sorry, yes. Fibers (Go didn't invent them) also aren't directly equivalent because they don't allow explicit yields and composition.

Fibers don't, but goroutines + channels + closures do. They permit composition using the same call/return syntax and semantics as normal function calls.

Futures and promises don't. Async/await is closer, but only by creating a second class of function incompatible with normal functions.

> Again, I use Rust and Go similarly in this post because Go is my frame of reference. I thought they were the same on this front, though. No?

No. Rust used to have a green threads runtime like Go (and Erlang/BEAM, Haskell, etc.) but that was removed before the 1.0 release. So today, a thread in Rust is a heavyweight OS thread, like in C/Java and most mainstream languages.

The intro Rust book says you can easily find crates for green threads of you really want them. Is this not the case?

I am not entirely sure, it appears like due to the present status of std in rust, any green threads lib is prone to Undefined Behaviour whenever something uses Thread Local Storage.

Yea, I'm aware - but you can spin off a async execution and then send data back over channels (I forget what Rust calls), as well as wrap that whole thing up in a function with a return value so that the caller has no idea of the threaded execution taking place. Right?

Sure the implementation of the threading differs greatly, but I was mainly referring to that I can spawn async behavior, and use synchronous blocking methods to get data back from that async behavior - AND encapsulate it.

All of these are very different than, say, what JavaScript goes through anytime async is involved.

Rust also has channels.

> 1. What development environment are other developers using with very large code bases? Is the tooling responsive?

It’s solid. If you’re coming from Visual Studio, you’ll dig IntelliJ with the Rust plug-in.

> 2. Using a language for a large project without some kind of async/await notation seems painful. How bad is it using only combinators for async (assuming you want to stay on stable)? And is there a date yet when async/await will be stabilized?

It’s painful. I don’t know if there’s a timeline. But I’m positive someone can say more.

Edit: See Steve's response. He would know.

> 3. Most of the code I write is in F#. The nice thing is, there is basically .NET library for everything under the sun. For example, over the weekend I was looking for a library that could decode OBD-II data from a car, and with a quick DuckDuckGo search I was able to find three or four. In a large-ish project, how often do Rust users typically find themselves coming up short for that kind of stuff? Obviously it varies a ton by domain, but maybe someone could tell me whether the answer is "you'll probably write all wrappers for 3rd party APIs yourself" or "there's a decent chance there's a library out there that you can adapt."

If you’re expecting the F# (and .NET) ecosystem, you’ll likely be satisfied. In fact, it might be better. If you’re expecting the Python ecosystem, you’ll need to wait (or contribute back).

> If you’re expecting the F# (and .NET) ecosystem, you’ll likely be satisfied. In fact, it might be better.

Not even close (100k unique packages on nuget, 13k crates on crates.io). On the other hand, you have easy interop with C (which doesn't give you stuff like safety, idiomatic error handling or nice APIs, but it's occasionally better than coding it yourself).

It's not about total packages, but about quality of packages and whether they provide good coverage of most problem spaces.

NPM claims to have close to 500k packages. Should we assume it solves 5x the problems that the .NET ecosystem does, that that perhaps by the 50th implementation of leftpad, there's some cruft on there?

That's not to say that Rust's package ecosystem is large enough or sufficient in comparison to .NET, just that raw package count gets to be an extremely poor indicator of quality after a certain level has been reached.

As a point of reference, I approach this from Perl and CPAN, which is one of the oldest large fully featured package networks. At this point, the problem is usually not finding a package that provides a solution, but finding the right package out of what's available. Different solutions exist, but a good curated list of well implemented solutions to common needs can help quite a bit.[1]

1: https://metacpan.org/pod/Task::Kensho

Well, an order of magnitude of difference is a hint that, whatever the quality of the existing crates, there are definitely areas for which you are going to be hard-pressed to find a good solution (say, localization), or a solution at all.

It is not a sign that there is anything wrong with the Rust ecosystem, it has a very active community and things are moving in the right direction, it is not just as full-featured as more mature ecosystems out there.

See for instance https://github.com/ErichDonGubler/not-yet-awesome-rust

There are quite a few use cases not listed here, obviously.

> raw package count gets to be an extremely poor indicator of quality after a certain level has been reached.

Broadly speaking I think the best indicators are always going to be domain specific -- which set of packages is most used within the context of what you're doing? A million awesome packages for Excel automation aren't much help, per se, for huge file chunking or BigData work.

From that point of view it's more relevant to look at the size of the successful projects in that market that resemble your technical goals.

IntelliJ and a workspace based project will get you a modern IDE experience. However don't expect a responsive edit/compile cycle, compilation times just doesn't scale for large projects, even with incremental compilation. Linking against shared libraries for stable dependencies can help though.

No the ecosystem is not mature enough to get the "enterprise" libraries. Wrappers to c/c++ libraries are often needed.

I just switched from VS Code to InteliJ and it's great. There's been huge improvements in IDE support recently.

Can you elaborate on the differences you've seen between VS Code + RLS and IntelliJ?

Sure, VS Code keeps randomly asking me to choose my toolchain even though a toolchain is chosen, closing it down and re-opening it fixes that.

InteliJ seems a little snappier, it's code completion a bit better.

The biggest benefit in day to day usage is the inline type hinting.

1. VS Code with the Rust plugin along with rustfmt and the rls is great - when it works. I've had some issues now and then, but the tooling is pretty great.

2. Not qualified to answer this

3. There's a ton of rust libraries out there. The whole cargo ecosystem is quite nice.

Coming from a C background (and dabbling in lisp), I love Rust. It's C with batteries and concepts from functional programming - iterators, immutable data. The only times I've struggled with rust is when trying to mess with pointer arithmetic and evading memory safety (e.g. trying to program a garbage collector for a Lisp interpreter)

I've been using VS Code and WLS to execute the Ubuntu version of Rust from the bash console in VS Code. I'm not sure if this is a good method or not... but it feels very natural.

Nightly rust can use this async / await crate, but it will likely be deprecated once language support is available (later this year iirc)

https://github.com/alexcrichton/futures-await/blob/master/RE...

Neat, what is your use-case for OBD-II data? I've been investigating ways to get data from a race car back to the pit crew via something like a raspberry pi.

For the radio component - I suggest that you build a "block" of useful data and send that several times using very good Hamming (error correction) instead of error detection. If using wifi then consider UDP.

You'll get better range out of something proprietary, I've had success in the 915 Mhz band. A spread-spectrum approach is better if possible.

Ideal would be if the car assumes it cannot receive anything and attempts to send each block several times before beginning to transmit the next block. Another approach could be to send a window of blocks each time.

Receiving outside the car is likely to be much easier - a large antenna can be used away from all interference. An option could be a human operator holding a directional antenna, visually tracking the car.

You might also store the blocks within the car for later 100% accurate download.

The microcontroller component of that is going to be a walk in the park compared to the radio component.

Wifi with good antennas, or a cellular hotspot seems pretty easy.

Race engines are not known for their lack of EM emissions :)

Vehicles are not friendly environments to begin with, race cars are outright hostile territory. Vibration & G forces, interference, fluctuating power, huge temperature variation, aerodynamic effects on antenna and cabling are all factors that will make this quite a bit of work to get to acceptable reliability levels.

I like your idea of a cellular hotspot, that might be the easiest way to get to something that works. Compact, no external cabling. That's good.

Edit: I tried running a webcam in a racecar in the mid 90's but it never got beyond the planning stage before the obstacles became insurmountable for the budget available, now, obviously a lot has happened in those years and F1 and other racing sports prove on a daily basis that the bandwidth is definitely there if you have the budget, you make me wonder how easy it would be to pull this off today, mostly likely you could just plug it together from some consumer components, a small tablet with a SIM card in a waterproof and vibration isolated enclosure would probably get you 90% there.

Right now I just want to get more fine-grained data about my mileage. But maybe add some automation into the mix later on (remote start, etc.).

Ensure your remote-start doesn't turn into a remote-stop and a crash.

Reply on news.ycombinator.com