Hacker News Reader

Comment by microtonal | Hacker News Reader

Overall, besides the extremely long compile times

Out of curiosity: fresh builds or also incremental builds? If the latter, how much of it is linking?

Both :) But what bothers me most, is the slow incremental debug builds. But that's a general Rust problem, although seems extra bad with Tauri.

My project has ~1700 lines of Rust, ends up pulling in ~700 dependencies and changing one line in main.rs takes about ~20 seconds for the debug compile to finish (on a 5950X CPU). Fresh build takes about 1m30s but that only happens on CI, so not really a problem day-to-day for me. Full CI build on Codeberg/Woodpecker takes around 5 minutes from push to release created, but that's including other things too.

I wonder if there are any plans to buttress this issue with Rust. Last time I checked the issue was that there is/was no ABI stability which meant you couldn't have precompiled libraries to build against. The only way around it at the time was to expose C compatible APIs, but that means of course giving up much of Rust's strengths.

> [...] My project has ~1700 lines of Rust, ends up pulling in ~700 dependencies [...]

I have no experience with Rust. Having 0.4 dependencies per line (so every 3 lines you are dependent on a 3rd-party) sounds very extreme. Can you elaborate a bit what those dependencies are (like many std libs?)?

That will be including transitive dependencies. Consider how many dependencies making a HTTP request will pull in. Ones for network IO, ones for parsing HTTP request. Possibly separate ones for HTTP1, HTTP2, HTTP3. Ones for type definitions of HTTP requests. Rust convention is to publish these bits as separate crates even if they are published by the same maintainer(s) so as to improve compile times (separate crates can be compiled in parallel) and modularity (if you're writing a competing library then you can share the internals of the existing implementation when possible).

This isn't like C++ where a single dependency is a huge project. Some crates are literally just trait (interface) definitions.

You can see for yourself here: https://codeberg.org/ditzes/ditzes/src/branch/master/src-tau...

My project have ~30 dependencies defined for various features in the client. The rest are transitive.

Here is the output of cargo tree for your leisure: https://pastebin.com/aep1bX4v

A quick scan of cargo tree seems to indicate most of them comes from tauri and actix projects.

Edit: oh, and also tantivy (a search engine) ends up pulling in ~300 dependencies which is a pretty big chunk of the total

Something for people to keep in mind that this is pulling in the source for:

- An electron-like framework

- A web framework

- Multiple HTTP clients

- An HTML parser

- An image editor

- A search engine

Some potential dependency savings I noticed:

How come you pull in both isahc and reqwest? On the surface, don't they fill the same role? Similar for rustls and openssl.

If you upgrade your readability dependency, you will only pull in reqwest once instead of twice. Might be useful to run `cargo tree -d`.

I'm not familiar with savefile? What does that get you over serde_json which you also pull in?

Thanks a lot for taking a look at the dependencies! I'm gonna be honest and say that I haven't spent much time "optimizing" anything, I haven't even arrived at a "first" version yet with Ditzes. But with that said, you have some good points :)

> How come you pull in both isahc and reqwest? On the surface, don't they fill the same role? Similar for rustls and openssl.

Yes indeed. I started with using reqwest, but it's no longer used and only isahc is used, so reqwest can be removed from the list.

Same with openssl/rustls. Started with rustls, but not longer used, so can also be removed.

Simply forgot to remove them at one point I guess.

> If you upgrade your readability dependency, you will only pull in reqwest once instead of twice. Might be useful to run `cargo tree -d`.

Ah, that's good to know. I'll do that once I put in some other changes, thanks!

> I'm not familiar with savefile? What does that get you over serde_json which you also pull in?

Rust savefile persists structs as binary data on disk, and also have versioning. Started out persisting everything as JSON, but loading thousands of posts (or even hundreds) from disk and deserializing them from JSON turned out to be quite slow. So using savefile mainly for performance, but I like the versioning/migration aspect of it as well, although I haven't used it yet with Ditzes.

serde_json is mainly used to parse responses from the HN API and to output JSON from the HTTP API, if I recall correctly.

`cargo udeps` tries to uncover unused dependencies. I wish we had more reliable ways to detect it so we could integrate it into cargo.

I compiled popular Rust projects before. It never pulled less than 300-400 dependencies. :/

Indeed. Here I took a random sample from https://github.com/topics/rust and listed the amount of dependencies from running `cargo tree` (full command: `cargo tree | awk 'NF' | wc -l`)

- denoland/deno (nodejs-like runtime) - 1550

- alacritty/alacritty (Terminal) - 303

- sharkdp/bat (`cat` clone) - 249

- meilisearch/meilisearch (search engine) - 1128

- starship/starship (command line prompt) - 427

- swc-project/swc (JS/TS compiler) - 1869

- AppFlowy-IO/AppFlowy (Notion clone) - 1146

- yewstack/yew (Rust/WASM web framework) - 942

- rustdesk/rustdesk (remote desktop) - 648

- nushell/nushell (shell) - 858

- ogham/exa (`ls` alternative) - 61

So yeah, seems even things as basic as a `cat` replacement ends up with 249 dependencies. Lowest amount of dependencies is a `ls` alternative, which ends up with 62 dependencies.

cargo tree | wc -l tends to over-count dependencies a lot because there are many very common crates.

Use grep package -F '[['package Cargo.lock | wc -l instead. This is doubly true if you pull in multiple crates using the same framework behind it, e.g. in async projects. E.g. for exa this is 45 instead of 61. For deno this turns into a 3x difference (515 vs. 1571).

Other reasons, besides what has been mentioned (vendoring being very uncommon, crates existing for common type and trait definitions, crates being generally scoped more narrowly resulting in more smaller crates), a lot of crates have additional crates for macros. Bindings often end up being a whole bunch of crates for this reason (i.e. one xyz-sys crate for the FFI bindings, plus one or more crates for a rusty wrapper and perhaps one or more crates for convenience crates). Case in point from deno's dependency tree:

    ├── futures v0.3.21
    │   ├── futures-channel v0.3.21
    │   │   ├── futures-core v0.3.21
    │   │   └── futures-sink v0.3.21
    │   ├── futures-core v0.3.21
    │   ├── futures-executor v0.3.21
    │   │   ├── futures-core v0.3.21
    │   │   ├── futures-task v0.3.21
    │   │   └── futures-util v0.3.21
    │   │       ├── futures-channel v0.3.21 (*)
    │   │       ├── futures-core v0.3.21
    │   │       ├── futures-io v0.3.21
    │   │       ├── futures-macro v0.3.21 (proc-macro)
    │   │       │   ├── proc-macro2 v1.0.38 (*)
    │   │       │   ├── quote v1.0.18 (*)
    │   │       │   └── syn v1.0.93 (*)
    │   │       ├── futures-sink v0.3.21
    │   │       ├── futures-task v0.3.21
    │   │       ├── memchr v2.5.0
    │   │       ├── pin-project-lite v0.2.9
    │   │       ├── pin-utils v0.1.0
    │   │       └── slab v0.4.6
    │   ├── futures-io v0.3.21
    │   ├── futures-sink v0.3.21
    │   ├── futures-task v0.3.21
    │   └── futures-util v0.3.21 (*)

Deps are a transitive tree. The number of deps for enduser code for a framework like this will always be high. That is kinda the point. A small amount of end user code uses the work of thousands.

I recently developed a similarly sized APP with tauri + rust (3kLOC+~250 deps) and found that using mold[0] as a linker helped quite a bit for incremental builds. There were also a few other bits, like decoupling the build step in tauri from the frontend build, that helped.

[0]: https://github.com/rui314/mold

Yeah, separating things is what I ended up doing as well. Ditzes is currently four parts: API, CLI, frontend (CLJS though) and desktop application. Most of the time when developing, only the API/CLI/frontend changes, while I just do desktop builds to check it from time to time.

What do people compare with when they complain about 20 second build times? That doesn't seem like anything for a thing that one doesn't need to do very often. You can still do cargo check without taking so much time, right?

`cargo check` is what my editor runs automatically to find type errors, that runs automatically on every save. It doesn't allow me to actually "check" that the application does what I want it to. I run `cargo run/build` when I need to "humanly" test the application, and waiting 20 seconds for it is pretty bad.

But then I come from a dynamic application development background (mostly Clojure), so maybe I do have a pretty unfair view on how fast I should be able to make a change and see the effects (sub second is the usual wait time in Clojure land).

Right, for that use, anything above 1 second is pretty bad. 20 seconds obviously breaks the flow completely.

> (on a 5950X CPU)

Incremental builds are unlikely to benefit from multiple cores. I have a 5900x Desktop and recently bought a Laptop with 12700H. For some incremental builds, I'm getting the same performance. The Desktop Ryzen shines with full multiple builds (where the laptop gets hot and has to sabotage itself); or with running multiple programs without noticeable performance impact.

Depends on the linker, no? My understanding is that mold is able to parallelize much better than the the default Rust linker.

I haven't tried mold. I already face lots of difficulties with the current linker since I need to deploy to multiple architectures, so I can't imagine how that would be with Mold (but then maybe that's why I should try it!)

Reply on news.ycombinator.com