Hacker News Reader

Comment by orf | Hacker News Reader

Is anyone else impressed with the quality (and speed!) of some of the tools written in Rust? I'm an avid user of fd and bat, the former being ridiculously fast. Often I find something on github, I'm impressed by the quality of the documentation, features, UI etc, then lo and behold it's written in Rust.

Another one potentially for this list is tokei[1] I was trying to count the code in our repos at work and used the venerable 'cloc' utility. It took over 5 minutes. Looking around I found tokei, written in rust. Same-ish results (more accurate actually) took 10 seconds.

1. https://github.com/Aaronepower/tokei

View on HN

I had a case where tar was too slow for my needs. Rust did let me cobble the right syscalls and threads together to make it more HDD-friendly and faster while forcing me to handle all the gritty filesystem error cases from the start. Lo and behold, about 4 times faster on an idle system and orders of magnitude faster on a busy system. It just would not have been possible to do it by combining shell utilities and too finicky in C.

Do you have the code posted online? I would like to take a look through it.

https://github.com/the8472/fastar

Only 117 lines of code? Impressive.

I have moved the meat of the optimizations to separate libraries, they're linked in the readme.

That's the thing with Rust, it makes you work hard right from the start to make everything right, but when you have something working it's usually fast, too.

I'm surprised they said ack || ag over ripgrep. I'm a total convert. It's AMAZINGLY fast.

In the comments, the author admitted he hadn't heard of ripgrep, and would give it a try.

IIRC, loc (in rust) might be even faster than tokei at times? I always forget, and I’m sure I saw an old benchmark...

We have a whole working group this year focused on making the experience of writing CLIs awesome, so hopefully we’ll see even more great tools in the future!

>We have a whole working group this year focused on making the experience of writing CLIs awesome

Can you explain what you mean by this? Something like the people are going to focus on language and library features that help with writing CLIs? Or something else?

Asking because CLIs are one of my interests.

Here is the Rust CLI working group announcement: https://internals.rust-lang.org/t/announcing-the-cli-working...

Rust has different community working groups that focus on improving the Rust ecosystem in different ways this year: https://internals.rust-lang.org/t/announcing-the-2018-domain...

Thank you, will take a look.

We've tried to dig into various problematic areas of writing CLIs in Rust [0], worked to create or improve libraries [1], and are working on writing a "book" for Rust-based CLIs [2].

You can reach out to us on gitter [3].

[0]: https://github.com/rust-lang-nursery/cli-wg/issues

[1]: https://github.com/rust-clique/ and https://github.com/assert-rs/

[2]: https://rust-lang-nursery.github.io/cli-wg/index.html

[3]: https://gitter.im/rust-lang/WG-CLI

The areas we're trying to finish up for Rust 2018 Edition:

- Get clap (arg parsing) to 3.0 [4]

- Get assert_cmd [5] / assert_fs [6] to 1.0

- Finish work on man-page generation [7]

- Make it easier to package binaries [8] and document the CI for it [9]

[4]: https://github.com/rust-lang-nursery/cli-wg/issues/41

[5]: https://github.com/assert-rs/assert_cmd/

[6]: https://github.com/assert-rs/assert_fs

[7]: https://github.com/rust-lang-nursery/cli-wg/issues/38

[8]: https://github.com/crate-ci/cargo-tarball

[9]: https://github.com/crate-ci/crate-ci.github.io

Thanks for the detailed answer. Will check out some of those links.

Loc (mine) is usually faster in my tests but doesn't handle comments that start in strings, so if there's something like x="/*" it can be way off, so I usually still point people at tokei.

I need to try implementing the string thing. It'd be a lot more fun to try to compete on speed if we were the same on accuracy.

Well luckily I recently published a new comparison benchmark[0]. :)

The TL;DR is that loc is faster by a few hundred milliseconds depending on repository size, but as cgag mentions doesn't have comment in string detection so can be quite off in its metrics, for example on the Rust repo Tokei says it has 643,754 lines of code where as loc says it's 635,849.

[0]: https://github.com/Aaronepower/tokei/blob/master/COMPARISON....

Awesome! :)

I wonder if there's a name for this phenomenon.

New and not-popularly adopted languages tend to have more senior people learning and developing software using them; this leads to people (read: recruiters) looking for the people with experience.. People equate the quality of software with something innate with the language, or that the developers are exceptional. Then after being popularly adopted it slumps in perceived elegance and goes on a downward spiral until it ends up like PHP, Ruby or JAVA. (not to undersell the folks who predominantly use those languages, I'm just picking languages that I've seen follow this pattern)

The Python Paradox (2004): http://www.paulgraham.com/pypar.html

The name may not have aged well...

A more generous interpretation might be that Rust has sparked a CLI renaissance, since it lets developers write CLIs that have that satisfying zip that previously was only possible in C. None of PHP, Ruby or Java is responsive enough for a good CLI tool (also, static compilation is a must for wide deployment).

Not to mention you get a language that is enjoyable but also compiles to native code. My first Rust project was a Haml parser and I created a CLI for it too that is similar to the Ruby version. It is nice being able to ship the 8Mb executable and not have to worry about users having the right runtime.

Just like any other compiled language.

GCC's static linking issues with glibc don't apply to other compilers.

No, not like any other compiled language. Just like any other native compiled language, maybe, but Java and C# are also compiled yet require a large runtime in order to run any application. An 8Mb executable written in Rust is going to be 8Mb total. An 8Mb executable written in C# or Java is going to weigh much more than that when you take into account the framework. Now, one could argue that it is almost a given for the JVM to be installed on a given computer but you can't say the same for .NET when doing cross-platform development.

Java and C# are also native compiled languages, with linker support.

You just have to look properly for those options.

The only thing is that not all of them are free beer.

Also don't forget Cargo. It makes it so much easier to work with than installing C/C++ dependencies.

Cargo is awesome too. Cargo reminds me of Mix (from the Elixir ecosystem). The one thing that Mix has on Cargo is that it is super simple and straight forward to extend Mix. Cargo may be the same way but from what I've seen it seems more complicated.

That’d make sense given the common ancestry in Bundler :)

Do you have any docs on what you mean by extensibility here? I’m curious!

Sure! Take a look at Mix.Task [0]. You basically name your module Mix.Tasks.X where X is whatever you want the command to be and then include "use Mix.Task" and implement the run function and you have now extended Mix. You call the task my running "mix X" and it will run the code in the run function. I created a small task that generates ORM models for Ecto (Elixir's de facto ORM library) using Mix tasks [1]. So the user types mix plsm and they can generate the code.

[0] https://hexdocs.pm/mix/Mix.Task.html [1] https://github.com/jhartwell/Plsm

Ah ha! Thanks. So yeah, if you have an executable in your PATH named cargo-foo, then “cargo foo” will execute it. But we’re planning on having that style of functionality in the future as well, the cod name is “tasks”, but it still doesn’t have an RFC.

I'll definitely want to check out that RFC when/if it opens up. I didn't realize that cargo did that with the executables, that's good to know. Thanks!

FreePascal has been around for a long time and supports CLI programming. Maybe not very much support in the basic language and stdlib, but the essentials are there (CLI arg handling and file I/O). Third-party libraries may have more and you can always write your own. Both speed and size of binaries are good. I had compiled some simple CLI programs and they were under 100K, maybe under 50K. It's also supposed to be quite cross-platform, though I have not checked that out.

Also, I don't know Rust (but have read that it is somewhat difficult to learn); for the basics, FreePascal (FP) may be easier than Rust to learn and start using, because although it (FP) has advanced language features, for many basic CLI programs, the simpler procedural features should be enough.

https://freepascal.org/

You would be very, very pleased with Nim.

https://nim-lang.org/

Thanks. I had seen some HN threads on it, but had not really checked it out. Will do so.

This could actually be caused by disk caching effects. Have you tried running cloc second time afterwards?

I tried cloc multiple times. tokei much faster. Try it yourself.

tokei is fast but isn't the smartest tool (at least for python). I compared it to pygount (and looked on individual files). Total lines is correct but it's grossly miscalculating code vs comments resulting in almost 2x total actual SLOC for a semi large project I manage.

I manually counted a few files for comparison, I believe tokei doesn't understand python-doc-comments and think it's code.

Since we're on the subject, I would ask this question I've been meaning to ask for a while. I love writing command line tools and I've written a few in Python but the performance isn't there.

Would you suggest me to learn Rust in order to write CLI programs? I'm also looking at haskell for the same but after this thread, I'm really thinking of going the rust way. Ideas?

I personally would strongly recommend going the rust route having spent a decent amount of time learning Haskell. I think youll spend more time working on real problems and learn more about low level programming, in addition to the code being much faster.

I would say it depends on your goals. If performance is your primary concern, then go with rust, one of rust’s primary focuses is performance.

I don’t think of Haskell as the best language for command line dev, but I’m far from expert in either language.

Also try cython.

Have you done any comparison (even informal) of similar tools written in Rust and Go?

I'm asking because although I know Rust is likely to be faster than Go, I'm interested in the amount of difference.

You might be interested in Ben Boyter's journey to write a source code line counter:

https://boyter.org/posts/sloc-cloc-code/

https://boyter.org/posts/why-count-lines-of-code/

Thanks, will take a look at those.

A very informal analysis of bat and ccat is that ccat (written in Go) has choked on large files multiple times (which resulted in panics), while bat hasn't crashed on me yet. Of course I've only tried bat on two files so far ;)

tokei is actually the very first Rust program that I found 'in the wild' while searching for a tool. I had run many Rust programs prior to then, but they had always been in the context of experimenting with Rust.

Reply on news.ycombinator.com