How do you get anything done when the cost of testing an idea is several hours?! I’d feel like the sunk cost fallacy would kick in for even the most lackluster of changes just because you took the time to try it.
I just learned to work without compiling for a long time. Over time my productivity increased and the number of bugs fell dramatically.
Working this way requires you to really think about what you are doing, which is always a good idea.
This was over a decade ago and now I work mostly on Java backends and I am happy that I typically spend days or even weeks without ever compiling the code and that it usually works the first time I run it.
I can't think how I could get back. It looks really strange to me to observe other developers constantly compiling and running their code just to see if it works. It kinda looks as if they did not exactly understand what they are doing because if they did, they would be confident the implementation works.
The only time when I actually run a lot of compile/execute iterations is when I actually don't know how something works. I typically do this to learn, and I typically use a separate toy project for this.
As I mentioned, if I don't understand a dependency or external system I make separate toy project where I can rapidly experiment and learn.
Think of it as having fun in a aircraft simulator. You play with it so that you are prepared to fly the actual plane.
Also checking your assumptions by trying to see if the code works is a problem in itself. A lot of these broken assumptions will not break your code immediately but maybe sometime later. Maybe when your application is under load or maybe when clock on the server is moved back by an hour or maybe when the connection breaks in a certain way.
Base your work on knowing how something works and not assuming.
Best way to limit the risk of your application failing due to broken assumption is to limit your reliance on assumptions in the first place.
Specifically in Rust, you can use the language to guide you through things like refactoring, thread synchronization, correctness verification and a lot of other things. But you have to run the compiler.
But you can say the same for Java. IntelliJ IDEA internally does equivalent of compilation and tells me exactly where my code would fail to compile.
So in a sense I am not strictly practicing my approach, but I also don't see reason to do so if the tools are reliably giving me hints when I made mistake writing something that will not compile.
Developing in an IDE that compiles almost continuously is about as far from the development philosophy you're advocating for here as one could get :P
This isn't about throwing away tools for some idealized goal. It is about using the tools that are available to achieve best results without making you reliant on the tools to the point you don't know what your program is going to do without compiling and running.
IDE helps catch a lot of stupid simple mistakes and that helps save time. Why would that be bad?
> It looks really strange to me to observe other developers constantly compiling and running their code just to see if it works. It kinda looks as if they did not exactly understand what they are doing because if they did, they would be confident the implementation works.
Explain to me how this statement doesn't apply to your use of an IDE, but the other engineers you've observed don't understand what they're doing.
I’m looking forward to someone making a legit IDE/suite with support, no indication of it yet but I assume some day!
Rust, well, "works". But there is still a bunch of issues so I keep developing using C until I get the kinks ironed out.
I don't remember the specifics but while watching the job queue on the terminal screen I discovered that the job priority was editable. So I bumped it up, my compile job ran, and I was happy. I only did this a few times before I got a lecture from the system operations guys that was quite explicit that I should never do this again.
Yes, you figure out how to run code mentally, how to check carefully for typos and syntax errors, etc. No Intellisense then either, that's another modern crutch.
Less so than when working with JavaScript. But please teach me your ways hahha
This is a problem for many reasons. One is that this may help you get something working, but with lack of deep understanding the outcome will likely be subpar if only because not all problems that you could have predicted will show themselves on execution.
Another is that it basically shrinks the part of brain that is necessary for understanding and predicting behavior of your code (figuratively). Sort of like driving with GPS makes me helpless without it.
Try to write larger stretches of code without compilation.
Try to focus on modularizing your application so that you can reason about modules separately. This is always a good idea, but it is even more important when you need to be able to predict how something works without trying it.
When you have finally compiled your code and it failed, do not immediately go to fix the problem. Try to spend a moment to learn from the failure and improve your process so that you minimize chance of this happening in the future.
Ask yourself, what you could have done to prevent this problem from happening? Could you have specified some function or module better? Could you have simplified your code to be able to better reason about it? Would it help if you have spent a bit more time getting acquainted with this internal or external library before you decided to use it?
From my experience, most of this comes down to following things:
- defining your modules and APIs correctly -- badly defined modules make it difficult to predict the behavior,
- finding simple solutions to problems -- complex solutions tend to make it difficult to predict behavior,
- using only tools you understand,
- only interacting with existing code after you have understood how it works (I typically at least look over the code that I plan to use),
- thinking hygiene (make sure you base your work on hard facts and not beliefs),
- refactoring, refactoring, refactoring -- first solution to the problem is rarely optimal. I write something that works and then immediately keep refactoring it removing any unnecessary complexity until I am satisfied. Don't leave refactoring for later -- when you have just written a piece of code it is easiest to change it.
- as much as possible, writing your code in a way that it is not even allowed to produce wrong result. This is very large topic so I won't explain. There is a lot of techniques that you can research.
For example when I am interacting with a codebase for the first time and I want to implement something I just keep bashing and throwing shit at the wall untill something sticks. After that I start working more in line of what you described.
Just make sure you keep actual development separate from learning the tool if you care for your results and especially reliability.
Now, I use various ways to learn the tools and codebase. Running PoC for my idea or maintaining separate toy project helps me with maintaining hygiene.
For example, for the past year I have been spending a lot of time learning reactive programming with RxJava and Reactor. I have created a dozen small projects illustrating various ideas for making reactive APIs, processing pipelines, separating business logic from infrastructure, composing reactive modules, etc.
I did this with aim of purposeful learning rather than writing production code, even though some of these in the end migrated to be part of production codebase.
I am now at a level where I can, again, write large swaths of modules, libraries but now using reactive paradigm, with a very good chance of it working correctly, which for me validates that I more or less understand what is going on.
Comparatively speaking Java compilation was very fast at the beginning, so for instance Red-Green-Refactor (RGR) works pretty well. There's a parallel to other creative jobs where sometimes shuffling around the things you already have reveals a pattern that leads to a breakthrough.
But there are other feedback loops that, with J2EE in particular, the cycle times started to creep up, and up, and at some point if you haven't stopped and looked at what you're doing you don't see how crazy things have gotten. RGR still has a place there because you are typically not recompiling everything and you aren't spooling up the application to run unit tests. But making one line changes to see how a page loads is just bonkers amounts of busy work.
One of the bad dynamics is that people more like you also tend to memorize the code, which is both bad for new hires (circular logic does not reveal itself when you introduce one assertion at at time, but does when you get hit with all of them at once), and also incentivizes you push back on refactoring. Because those damned smartasses keep moving things around and they were Just Fine where they were. If that happens you have cemented the entire codebase and anything that is really wrong with it is going to stay wrong until someone proposes a rewrite. And having learned the wrong lessons the first time, we repeat them again in the second iteration.
I also presume you were compiling in release mode, which is good for producing extremely fast programs but not something I bother with in regular development (in contrast to Go, which has no distinction between debug and release optimization levels).
> How do you get anything done.
The vast majority of my workflow just involves seeing if my code typechecks, for which I don't even need to build the program (so no codegen, no linking). This I do very frequently, as a sanity check on my work. The command for this is `cargo check`. This takes less than a second for small projects, one to five seconds for medium projects, and one to twenty seconds for large projects.
Good point. I recently upgraded my machine to 64 GiB RAM, because 16 GiB filled up pretty quickly with parallel compilation, several VS Code/rust-analyzer instances and a couple of browser tabs open.
I guess you can have projects that are just huge and/or run into some pathological case that increases compile time a lot (just like with C++), but for any subsequent compilations you should get very fast incremental builds.
(Though I guess one might argue that slow compile times on typical code and rarely-used computationally hard features have the same root cause of not treating fast compile times as a top priority.)
One cost of being embarrassingly parallel is the One Definition Rule. If we can compile N different code units in parallel, but they're all allowed to define things in the same namespace, obviously those definitions might contradict each other and we wouldn't notice. So, the C++ language explicitly forbids this, knowing you'll probably do it anyway, at least by mistake. If (when) you do, that isn't a valid C++ program, but the compiler isn't expected to produce any diagnostic (warning or error). So, you get a binary, but the language doesn't care what that binary does. Maybe it does exactly what you expected. Maybe it does almost exactly what you expected. If not too bad, the One Definition Rule means your compiler was fast and that's what matters.
It will fail at link time if you link everything into the same library. Even here there is an escape: there are ways to mark something as a weak symbol and than the linker won't complain about more than one definition.
See your OS documentation for how this works on your implementation. (though don't be surprised if the documentation is wrong...)
did you use LTO ? It always catches ODR issues for me. There is also GNU Gold's --detect-odr-violations switch.
Though I've been tempted to write a paper for C++ to make it defined behavior. I know it works in some form on most implementations even though it isn't legal. Thus there seems to be some other use cases for it that could/should be formalized. If anyone can give me other examples of why you want to do this and what the rules are on each platform I'll take a shot at it.
it definitely does not, every time I had an ODR issue that caused actual bugs. For instance, dynamic_cast not working because a typeid was defined in two shared objects, etc.
What would be the behaviour you expect if you have
a.cpp:
int constant() { return 123; }
b.cpp:
int constant() { return 456; }
c.cpp:
int constant();
int main() { return constant(); }
e.g. here with gcc, if a and b are put into shared libraries, the return value depends on the order in which the libraries are passed to g++ when linking. e.g.
g++ c.cpp a.so b.so
g++ c.cpp b.so a.so
Gcc also has the concept of weak symboles which if invoked (and the linker supports it) would allow you to make one of the two weaker than the others and then the whole doesn't depend on link order. Visual C++ also seems to have something like this, but I'm sure it is different.
Like I said, I want to write a paper to make it defined - but the paper will be a lot longer than would fit in a response here, and depending on information that I currently don't know.
This use case isn't an ODR violation. Its just using the linker to mock an interface.
> It will fail at link time if you link everything into the same library.
That is an ODR violation, although there are variations on this pattern that are not required to be detectable. Template instantiation is an easy way to get a silent ODR violation.
I do have memories of the days when compiling Chromium or AOSP was a 4 hour battle, though :)
Sometimes it is just that someone drank the Kool-aid and made everything a template. Then split their code across hundreds of tiny files and did the one big massive include at the top of every file that pulls in everything.
I haven't used rust in production, but usually you can just use `cargo check` to only run the typecheck. Using `cargo build` is also usually much faster than `cargo build --release`.
Having said that, at least in my toy projects it was not uncommon to have to compile using `--release` to run the tests (since otherwise the tests would take forever).
Maybe you're already aware of that, but if the reason why your tests are slow is not “your code not being optimized” but “your dependencies not being optimized” then Cargo profile override[1] can save your life. You can have one specific dependency (or all of them) being build in release mode even when you're building you own code in debug mode. The first development build is going to be quite slow, but after that, you're going to have the best of both worlds.
[1]: https://doc.rust-lang.org/cargo/reference/profiles.html#over...
Also, I'm assuming it was a full (as opposed to an incremental) release build? But even then, I've never had to wait for that long.
In 90s and early 2000s if you worked on a large desktop application (the size of Office or Photoshop) you could expect to spend a business day waiting for compilation results.
Because they gained popularity roughly around the same time many people think they are competing languages, but they're really not.
Ruby is my main programming language, if I need something to be fast and/or light weight I switch to Go, sacrificing some comfort. And if I need absolute lowest possible overhead, I switch to Rust, sacrificing a whole lot more comfort. Rust is more expressive than Go, but if your Go project is so large that it needs a stronger typesystem in my opinion you should switch to a language like C# or Scala.
Sure the tooling around it (besides the compiler itself) was the dumbest 21st century tooling I've seen (this was ~6 years ago), but it was quick to learn, quick to write, quick to compile and quick to run so who am I to complain.
Also a project is likely to be split up over multiple crates, so a lot of the changes you make will require building and testing just that one crate.
For debug builds, where incremental is usually on by default, it was disabled for 1.52.1 due to bugs, and then kept off in 1.53 out of an abundance of caution. It should be back on in 1.54.
It was a bit slow in CI/CD build environments with no cache, but so are the k8s Go projects I work on (go mod pulling in the world).
The only thing approaching 2 hours I've ever seen is building the compiler from scratch.
> How do you get anything done when the cost of testing an idea is several hours?
Incremental compilation. Depending on the project you can get the edit compile cycle down to somewhere between 1 second and 5 minutes. It's nowhere near as fast as Go but it's not like anyone is actually making edits then waiting 2 hours for them to compile.
Subsequent rust builds for linkerd-proxy were quite fast. For envoy most time was spent in Bazel itself. Probably because I just did a Bazel build instead of specifying a more specific target.
One: my local dev machine has 16 gigs of memory and 16 cores. What takes the tiny docker container with 1 gig and 2 vcpus 30 minutes takes my computer about 1.
Two: Incremental compilation and testOnly make build/test cycles maybe a second / twenty seconds max, and most of that is test runtime on complex property based tests.
You just get by without a clean compile most of the time after you build the project the first time. And really, a lot of builds spend an inordinate amount of time just pulling in external dependencies (which are also cached on my local machine, but not on the containerized builds a lot of the time).
initial compile has to download and compile all dependencies. after that compiling is incremental. still slower than go though
My Rust compile times are usually between 10 and 30 seconds. Some tricks that help:
- Get a good development machine. I prefer a Dell Precision laptop from the last couple of years, with plenty of cores. A 5-year old laptop with 2 cores will be a lot slower.
- Use Visual Studio Code and the rust-analyzer plugin. This will give you feedback in the editor while you're working.
- Rely on the type system to identify most problems before ever generating code.
- Minimize the use of slow-to-compile libraries that rely on complex generic types. Diesel, for example, is a great library but it's slow to compile.
- Install cargo-watch, and use it to automatically re-run the tests.
Also, remember that incrementally re-compiling a single crate in debug mode will be much faster than building an optimized application and all its dependencies from scratch.
TL;Dr: Rust compilation times are less than ideal, but with a fast laptop and rust-analyzer, it's possible to spend very little time actually waiting for the compiler.
I've gotten by with much less for personal projects. But Rust will make very efficient use of (say) an 8-core i9 if you have one.
In a normal Rust application you break it up into crates. Crates are the unit of compilation, and are only recompiled when something in the crate changes. In a "normal" developer flow you only touch 1 or two crates at a time so normal developer compile time would be a couple of minutes at most. Even for a new build on a machine most developers would never see the two hour compile time because they would have gotten precompiled dependencies.
Obviously this depends on your computer, but generally incremental Rust builds should typically be on the order of seconds, not minutes.
> Even for a new build on a machine most developers would never see the two hour compile time because they would have gotten precompiled dependencies.
Cargo doesn't do precompiled dependencies (even between projects on the same machine).
rust's compilation times are as terrible as c++'s?
Incremental C++ builds can be lightning fast, or it can be ridiculously slow. I've worked on large projects where change + incremental compile + test was less than 5 seconds. And I've worked with project where no effort was made where the same could take 10 minutes.
However C++'s compilation units are smaller than Rust's which helps it to have faster incremental compilation times for small changes.
So, same order of magnitude basically.
Edit: I quickly realised, this might only because of Rust's—in my opinion—superior type system and tooling, so maybe it just requires less waiting for compilations than when working C++.
Edit 2: I've never actually measured Rust compilation times, but even for large-ish graphical codebases (wezterm, alacritty, bevy games) I've compiled from scratch, it felt noticeably faster even then.
AAA games need less resources than those tree walkers
With Rust, procedural macros add a lot of overhead when compiling, so I try to avoid them.
At work, we have a C++ project which, without using distributed builds with distributed ccache, running on powerful cloud computers, takes 3h+. Code quality is adequate. Debugging anything is a nightmare in such a codebase.