You basically need a compiler farm for fuchsia. I tried compiling it once and it took 3 hours, only to find out to need to recompile to add anything
During university, our computer security class involved finding exploits in Firefox. It took 8 hours to compile on our average student grade compute. There were probably faster incremental ways to build, but navigating the Firefox open source code base was hard to entry point as a student. Effectively, most people got to compile Firefox a few times, but no one went further than making any code changes or discoveries. Navigating the Firefox build and tool chain is probably an undergrad course in and of itself, much less make any meaningful merge request or finding a vulnerability.
Overall, horribly designed curriculum. The expectation was for undergrads to find a zero day in Firefox. The professor was a researcher from Microsoft, but didn’t seem like he had realistic expectations for undergrads. Maybe they were throwing darts, hoping one student finds an novel exploit, and they can be cited. In the end, no student found an exploit or made any code changes. I was the only who found a DoS code payload for Firefox, but it was neither a zero day (poking around a known less developed API) nor high risk. It merely crashed Firefox on any webpage which contained this 2 lines of JavaScript. In the end, I got the lowest grade in the class because the professor changed the rubric after no one else found anything. Finding this exploit went from 80% of the grade to 5%, and participation credit went from 10% to 70%.
I'm sure it was not intentional at all, but it is pretty funny.
Complete with lower pay (or at least less career opportunities) for working extra hard on the Death March project about to be cancelled.
I had a similar but not as bad experience. The lecturer was brand new and wanted us to design a programming language after a single vague PowerPoint and the worst introduction to yacc or lex or whatever they are. We were also undergraduates.
After several weeks of complaints he provided an example language and parser etc. Still it was simply too hard and in the end nobody could do it. I believe I managed to add a minor feature to his example language as my submission. I managed to get a reasonably high grade simply because I freely admitted I struggled with the entire task but could explain language theory and made comparisons with other languages I know and what features I would have added if I could.
Some people totally wrote that whole module off and planned around the scoring system where one failed module in a year would just be discounted and an averaged score created.
Same gig though. Set a programming assignment that's way too hard for the class, because they hadn't been taught programming properly (in this case, implement a few simple graph algos so hardly Firefox 0-day level). Realize too late that nobody can pass the assignment and retroactively change the marking criteria to be entirely based on writeup. Result: as one of the only people in the class who actually submitted the full three working solutions, I got one of the lowest grades, because "You explained the algorithm in 'detailed code comments'? Do you think I have time to read student's code? It needed to be submitted in a Word document alongside the submission."
This was at a UK Russell Group university, supposedly one of the elite. IMO universities are jokes. They were crap at teaching decades ago and that was before they were overrun with radical ideological activism. They're simply good at hiding the truth because academics are treated like gods - nobody believes students who say their teachers are terrible at their own subjects.
"University" typically means TAs the entire undergrad, right?
What, the 3 hour compile time? That's an OS+utils, there are projects that take more... Ever tried compiling QT+KDE?
I just tested on my machine (i7-6900k, 8C/16T from 6 years ago), building 5.15: qtbase, qtdeclarative, graphicaleffects,multimedia,quick controls2, serial port, svg, wayland and websockets takes 8 minutes total... it's hardly the end of the world and that's already more than needed for 99.9% of Qt apps. And I don't even use ccache.
make -j16 5343,12s user 307,95s system 1198% cpu 7:51,67 total
~/libs/qt5/configure -nomake tests -nomake examples -debug -confirm-license -opensource -platform linux-clang -linker lldBut adding the Yocto overhead (entire bootloader, kernel, and then fetch/decompress/build all the packages) takes a bit more time. And Qwebkit isn't trivial either.
Once Yocto has it all in sstate-cache then it's only like 15 minutes for a total rebuild.
Not too bad for an OS and all the utilities and programs that it comes with. The Linux equivalent will be compiling Gentoo (probably a lot longer than 3 hours, even on good hardware).
In 2003, as part of a university assignment, I used a CORBA ORM called 'mico' (or 'micro', not too sure) written in glorious C++ with as much use of templates that the developers could use, and that took a full 4 days to compile on my aging 1998 laptop[1].
When I switched to an expensive 2003 desktop a few months later[2] the compilation of the same package with the same flags on the same OS (Slackware) took less than five hours.
It is amazing what speed improvements we saw between 1995 and 2005. Just between 1995 (when I bought my 486 desktop) and 2000 (when I was using a pentium pro or something better than that), machines had roughly doubled in performance at the same price.
I really doubt that machines from five years ago are going to be that much better than one from today.
[1] As a student, I counted myself lucky to even have a laptop of my own, instead of booking time at the university lab.
[2] The compilation experience convinced me to save for a few months to get something expensive with lots of RAM
Yocto is not entirely unlike Gentoo - recipes describe components of the system, which is built from source - but with the addition of a fairly sophisticated caching scheme, where the input to a recipe is hashed and the hash used to store the outcome, which is then reused in future builds (and the cache can be shared by different builders) unless the input changes. The other key feature of Yocto is that the system is composable in layers, where upper layers add, extend or override recipes from the lower layers. Layers can and are provided by different parties (e.g. BSP layers from HW vendors or their SW partners).
Yocto Linux is used by projects which need to be able to guarantee that they can bootstrap their OS build, customize the build (e.g. filter out use of certain licenses or use a specific toolchain) and to manage SW supply chain (the layer idea, and then in a business contact something like RASIC for the layers).
To sum it up, "rebuilding the entire Linux OS w/ caching" is absolutely the norm in embedded dev, as is having a compiler farm doing this hooked into your CI. Edge nodes like developer laptops then access the CI build farm's caches to make a local build bearable, with the caveat that you don't get incremental builds at the component level this way, so usually you still want to either dev separately against an SDK (or reuse the build root) or at least keep your components small and modular enough to not make it painful.
Automotive, network equipment (e.g. routers), stage/production equipment (mixers and other networked A/V gear, etc.) and many other parts of the SW world work this way.
Hacker News is mostly exposed to web development and desktop Linux/mobile app development, which are pretty different. Indeed, perhaps the most surprising thing is how different desktop Linux development is from embedded Linux development and how little cross-pollination between these communities is taking place.
If you develop your for desktop Linux, the distro on your box also serves as its SDK - you just install a bunch of -dev packages from your package manager and build against the host. Or perhaps a Docker image as build env in some cases. But you generally never rebuild/bootstrap the OS, which in embedded is the primary unit of work (with mitigations such as caching).
One side-effect of this is that embedded systems and desktop systems tend to approach updates/OTA differently. In the package/binary-based systems, OTAs come in via the package manager. Embedded systems historically tend to go for full system image updates with again some mitigations such as binary deltas, and then A/B partition update schemes. Or a partitioning of the update content that is orthogonal to the partitioning that goes into the OS image build. Lately there's a trend for seperating applications out into container images that get deployed seperately from the base OS image, and thoughts about containers that can move between embedded devices on the edge and the cloud infra in the back.
I work on the android operating system and very rarely compile the whole thing from scratch in development environments. Incremental builds plus adb sync (think rsync between compiled artifacts in host and device) make it into a manageable workflow.
Even incrementally, it takes a few minutes to see your changes and that can be a source of frustration for newcomers who are used to instant feedback. Being productive requires making good decisions on how often and what to compile as well as strategies for filling up compilation time.
The data thing though, that sounds like your engine is poorly optimized for development time. There should be a way to work with data without repacking everything.
It is the language those managed language runtimes are written on, the main language for GPGPU programming, two major compiler building frameworks, and works out of box in all OS SDKs while providing me a way not to deal with C flaws unless forced by third party libraries.
I could be spending my time creating libraries for other ecosystems, but I have better things to do with time.
It wasn't because it was lacking in capabilities, rather the authors decided they would spend resources elsewhere.
Naturally it has the side effect to reinforce the use of the languages that are already being used.
For things like AAA games and OS development, I'm not convinced simply picking another language solves the problem. At least not while keeping all the same benefits. C builds faster, sure, but it doesn't have the same feature set.
I'm getting 150 ms of iteration time on small cases, 200-300 on average ones
>>It's surprising people are willing to put up with this shit at this in this day and age
All major APIs, the SDKs for PS5/Xbox are all only provided in C++ so it's almost a necessity. Same reason why we all use Windows - the platform tools are only provided for windows.
Some of the packing steps are also probably lossy (eg. Take this super high poly count model, and cut away 99% of the polygons). If you skip that culling step, the game probably won't run.
And yes, the client itself usually can't read the raw data, and even if it could there is not enough ram on consoles to load everything as-is. The workstations we use for development all have 128GB of ram just so you can load the map and all models in-editor.
One day, I stepped away and had a particularly intimidating voice say "your build has failed" and apparently knocked out my headphones. I came back just in time to hear that, and see a couple coworkers jump at the sound.
After that, I was much more consistent at disabling sound when I stepped away. I got a little teasing about that day, but generally it worked great.
Lately I've actively tried not to do that (it's a hard habit to break, and I still feel guilt sometimes), though.
With slower cycles, I think more about how much to try before submitting work. Some times I feel comfortable pounding out quite a bit of code. Other times, I know there's some subtlety so I need to double check things. I don't want to stumble on forgetting a const declaration, or something silly like that. Iterations are slower, but you can spend time in flow thinking harder about each loop.
Although, sometimes, I do just stare at the console waiting for feedback. That's usually a good time to go to the bathroom and maybe grab a snack.
Not necessarily multitasking. Just being careful about what plates are spinning, and which I can set down or pick up between steps.
Type checking is still a very quick part of compile, so it can still support a "fast cycle" workflow if most simple errors are detected via incompatible types. You just need to go all-in on type-driven development, rather than simple reliance on unit tests.
but yeah, types are great. Quickcheck is great too. But you'll have to pry my oracles from my cold dead hands. Computers show me over and over how stupid I am. Yeah, if I have a regression, I'm adding a specific test for that.
I know at least for me, I sit in quiet contemplation and think while the project is compiling. I expect the context switch of going between writing docs and writing code on every compile would be too much for my brain. Is it managers making you feel like you need to be doing something else while your code is compiling? I guess I just never felt like I was being unproductive while my code was compiling.
just 3 hours would be very nice. clasp also needs about a day.
And I think this is mostly a windows problem with the synchronous filter drivers? On linux you can hook filesystem accesses asynchronously.
Nobody said it was.
> If the AV slows down productivity then it's up to office politics to decide whether the perceived security/compliance is more important than developer time or whether they should talk to their AV vendor to optimize scanning or whatever.
I agree and everyone here is assuming the GP hasn't already tried that. Sometimes, and particularly in enterprise orgs, making minor quality of life improvements for developers is an impossible task. Sometimes you don't realize just how these places can operate until you've worked in one.
> And I think this is mostly a windows problem with the synchronous filter drivers? On linux you can hook filesystem accesses asynchronously.
Yeah it's very specifically a Windows issue. Windows IO is a lot more event driven from what I understand which makes Linux faster at randomly accessing files but virus scanners more effective (not that I have a particularly high opinion for them to begin with) on Windows.
Unfortunately, Windows is what 99% of enterprise orgs IT teams provision for their staff.
Those enterprise AV have macOS, GNU/Linux and even iOS/Android versions for a reason.
At Microsoft, our massive servers churn out nightly Windows image overnight usually 5pm-10am next morning.
10 years ago, you would be able to compile your whole BSD kernel and userland (plus KDE) in a week or so, while just checking occasionally whether it's completed or not. This was on a dual core, off the shelf desktop computer.
A week also seems pretty long for BSD. I'm sure I spun up FreeBSD with Gnome 2 in ~24hours once (it was definitely less than a week because I had a weekend to prepare it for a house party). Though admittedly I didn't compile base. But this was a machine from around 2005 sort of era, maybe even earlier.