Software Disenchantment (2018)
tonsky.me
tonsky.me
I think the analogy here is backwards. The better question is "how much would you prioritize a car that used only 0.05 liters per 100km over one that used 0.5? What about one that used only 0.005L?". I'd say that at that point, other factors like comfort, performance, base price, etc. become (relatively) much more important.
If basic computer operations like loading a webpage took minutes rather than seconds, I think there would be more general interest in improving performance. For now though, most users are happy-enough with the performance of most software, and other factors like aesthetics, ease-of-use, etc. are the main differentiators (admittedly feature bloat, ads, tracking, etc. are also a problem, but I think they're mostly orthogonal to under-the-hood performance).
These days, I think most users will lose more time and be more frustrated by poor UI design, accidental inputs, etc. than any performance characteristics of the software they use. Hence the complexity/performance overhead of using technologies that allow software to be easily iterated and expanded are justified, to my mind (though we should be mindful of technology that claims to improve our agility but really only adds complexity).
At my current place of employment we have plenty of average requests hitting 5-10 seconds and longer, you've got N+1 queries against the network, rather than the DB. As long as it's within 15 or 30 seconds nobody cares, they probably blame their 4G signal for it (especially in the UK where our mobile infrastructure is notoriously spotty, and entirely absent even within the middle of London). But since I work on those systems I'm upset and disappointed that I'm working on APIs that can take tens of seconds to respond.
The analogy is also not great because MPG is an established metric for fuel efficiency in cars. The higher the MPG the better.
I never liked this view. I can't think of a single legitimate use case that couldn't be solved better than by hiding your true capabilities, and thus wasting people's time.
> they probably blame their 4G signal for it
Sad thing is, enough companies thinking like this and the incentive to improve on 4G itself evaporates, because "almost nothing can work fast enough to make use of these optimizations anyway".
Consider a loading spinner with a line of copy that explains what's happening. Say it's for an action that can take anywhere from 20 milliseconds to several seconds, based on a combination of factors that are hard to predict beforehand. At the low end, showing the spinner will result in it flashing on the screen jarringly for just a frame. To the user it will appear as some kind of visual glitch since they won't have time to even make out what it is, much less read the copy.
In situations like this, it's often a good idea to introduce an artificial delay up to a floor that gives the user time to register what's happening and read the copy.
It's a real case, actually. I don't remember a name, but I've encountered this situation in the past, and that brief flash of a "in progress" marker was what I used to determine whether me clicking a "retry" button actually did something, or whether the input was just ignored. It's one of those unexpected benefits of predictability of UI coding; the less special cases there are, the better.
e.g. back in the 90s with Resi 1, the loading screen was hidden by a slow and tense animation of a door opening. It totally fit the atmosphere.
Plenty of games add an elevator or a scripted vehicle ride, or some ridiculous door locking mechanism that serves the same purpose without breaking immersion, especially as those faux-loading screens can be dynamic.
It's pretty much the exact same technique used in cinema when a director wants to stitch multiple takes into a single shot (e.g. that episode in True Detective; that other one in Mr Robot; all of Birdman).
I see this argument coming up a lot, but this can be solved by better UX. Making things slow on purpose is just designers/developers being lazy.
Btw users feeling uneasy when something is "too fast" is an indictment of everything else being too damn slow. :D
IMO it can be attributed more to bad UI than optimizations.
I'll prioritize the 0.005L per 100km car for sure. That means the car can be driven for all its expected lifetime (500k km) in a single tank of gas, filled up at the time of purchase! That means there is a huge opportunity to further optimize for many things in the system:
- The car no longer needs to have a hole on the side for filling up. A lot of pipes can be removed. Gas tank can be moved to a safer/closer location where it is used.
- The dashboard doesn't need a dedicated slot for showing the fuel gauge, more wirings and mechanical parts removed.
- No needs for huge exhaust and cooling systems, since the wasted energy is significantly reduce. No more pump, less vehicle weights...
Of course, that 0.005L car won't come earlier than a good electric car. However, if it's there, I'd totally prioritize it higher than other things you listed. I think people tend to underestimate how small efficiency improvements add up and enable exponential values to the system as a whole.
I've worked with a Preact SPA where the time to initial render was faster than the HAML templates it replaced.
But then, again, that was an outlier. If your target is speed, traditional SSR or static pages are the best bet, anyway.
I'm struggling to think of recent (or not-so-recent) software improvements that have had a similar impact though. It seems like many of the "big" algorithms and optimization techniques that underpin modern applications have been around for a long time, and there aren't a lot of solutions that are "just about" ready to make the jump from supercomputers to servers, servers to desktops, or desktops to mobile. I guess machine learning is a probably contender in this space, but I imagine that's still an active area of optimization and probably not what the author of the article had in mind. I'd love if someone could provide an example of recent consumer software that is only possible due to careful software optimization.
That would be the meaningful software improvements I can think of. Curiously, the key enabler here seems to be performance - we had the capability to write web apps for a while, but JS was too slow to be useful.
--
[0] - They may or may not have been developed earlier, but I haven't seen them used in practice before the modern web.
In this thought experiment we are talking about a 2 orders magnitude improvement - hardly a small nudge!
It's a nice idea but it wouldn't work. The gasoline would go bad before you could use it all.
Plug-in hybrids already have this problem. Their fuel management systems try to keep the average age of the fuel in the tank under 1 year. The Chevy Volt has a fuel maintenance mode that runs every 6 weeks:
https://www.nytimes.com/2014/05/11/automobiles/owners-who-ar...
https://www.autoblog.com/2011/03/18/chevy-volts-sealed-gas-t...
Instead of having a "lifetime tank", a car that uses 0.005L per 100km would be better off with a tiny tank. And then instead of buying fuel at a fuel station you'd buy it in a bottle at the supermarket along with your orange juice.
If you had pure gasoline, you could store it for years (and in past, countries and armies did exactly that for their reserves).
Secondly we talked about 0.005l cars, not 0.05l, so it would be a few hundreds times more efficient than train transportation.
I’m willing to bet that a significant percentage of my accidental inputs are due to UI latency.
I want all interactions with all of my computing devices to occur in as close to 0ms as possible. 0ms is great; 20ms is good; 200ms is bad; 500ms is absolutely inexcusable unless you're doing significant computation. I find it astonishing how many things will run in the 200-500ms range for utterly trivial operations such as just navigating between UI elements. And no, animation is not an acceptable illusion to hide slowness.
I am with the OP. "Good enough" is a bane on our discipline.
0ms is great, 4ms is very good, 16ms is minimally acceptable, 20ms needs improvement (you're skipping frames), 200ms is bad (it's visible!), 500ms is ridiculous and should have been showing a progress bar or something.
Responding to input doesn't necessarily mean being done with processing, it just means showing a response.
The usage story requires you to switch to turn-by-turn, and there’s no way to have bird eye map following your location along route (unless you just choose some zoom level and manually recenter every so often.)
It’s awful, distracting and frankly a waste of time... just to show a bit of animation every time I accidentally fail to register a drag...
Damn Ui
It does fill some other requirements that a regular car doesn't.
That's not really true. Slack could be just as pretty and a fraction of the weight, if they hadn't used Electron.
One, phone OSes are being designed for single-tasked use. Outside of alarms and notifications in the background (which tend to be routed through a common service), the user can see just one app at a time, and mobile OSes actively restrict background activity of other apps. So every application can get away with the assumption that it's the sole owner of the phone's resources.
Two, given the above, the most noticeable problem is now power usage. As Moore's law has all but evaporated for single-threaded performance, hardware is now being upgraded for multicore and (important here) power performance. So apps can get away with poor engineering, because every new generation of smartphones has a more power-efficient CPU, so the lifetime on single charge doesn't degrade.
A UI where each interaction takes several seconds is poor UI design. I do lose most of my time and patience to poor UI design, including needless "improvements" every few iterations that break my workflow and have me relearn the UI.
I find the general state of interaction with the software I use on a daily basis to be piss poor, and over the last 20 or so years I have at best seen zero improvement on average, though if I was less charitable I'd say it has only gone downhill. Applications around the turn of the century were generally responsive, as far as I can remember.
I was forced to use a monitor at 30 fps for a few days due to a bad display setup. It made me realize how important 60 fps is. Even worse, try using an OS running in a VM for an extended period of time...
There are plenty of things that are 'good enough', but once users get used to something better they will never go back (if they have the choice, at least).
Optimizing for microseconds when bad UI steals seconds is being penny-wise and pound foolish. Business might not understand tech but they do generally understand how it ends up on the balance sheet.
Note also the potential deadlock here. Optimizing core calculations at μs level is bad because UI is slow, but optimizing UI to have μs responsiveness is bad, because core calculations are slow. Or the database is slow. This way, every part of the program can use every other part of the program as a justification to not do the necessary work. Reverse tragedy of the commons perhaps?
- Opening multiple tabs in a browser will kill your battery, and it's not the fault of a single page, but of all of them. Developers tend to blame the end user for opening too many tabs.
- Running a single Electron app is fast enough in a newer machine but if you need multiple instances or multiple apps you're fucked.
- Some of my teammates can't use their laptops without the charger because they have to run 20+ docker containers just to have our main website load. The machines are also noisy because the fan is always on.
- Having complex build pipelines that take minutes or hours to run is something that slows dow developers, which are expensive. It's not the fault of a single software (except maybe of the chosen programming language), but of multiple inefficient libraries and packages.
I do that for most of my hobbyist Linux dev work. It's fine. It can do 4k and everything. It's surely not optimal but it's better than managing dual boot.
I actually do this for development and it works really well.
Ubuntu Linux VM in VMware Fusion on a Macbook Pro with MacOS.
Power consumption was found to be better than running Linux natively. (I'm guessing something about switching between the two GPUs, but who knows.)
GPU acceleration works fine; the Linux desktop animations, window fading and movement animations etc are just as I'd expect.
Performance seems to be fine generally, and I do care about performance.
(But I don't measure graphics performance, perhaps that's not as good as native. And when doing I/O intensive work, that's on servers.)
Being able to do a four-finger swipe on the trackpad to switch between MacOS desktops and Linux desktops (full screen) is really nice. It feels as if the two OSes are running side by side, rather than one inside another.
I've been doing Linux-in-a-VM for about 6 years, and wouldn't switch back to native on my laptop if I had a choice. The side-by-side illusion is too good.
Before that I ran various Linux desktops (or Linux consoles :-) for about 20 years natively on all my development machines and all my personal laptops, so it's not like don't know what that's like. In general, I notice more graphics driver bugs in the native version...
(The one thing that stands out as buggy is VMware's host-to-guest file sharing is extremely buggy, to the point of corrupting files, even crashing Git. MacOS's own SMB client is also atrocious in numerous ways, to the point of even deleting random files, but does it less often so you don't notice until later what's gone. I've had to work hard to find good workarounds to have reliable files! I mention this as a warning to anyone thinking of trying the same setup.)
But perhaps the answer is VMware Fusion instead then.
Having your VM stored inside a file on a slow filesystem is bad. Having a separate lvm volume (on linux)/zvols (with zfs)/partition/disk is much more performant.
That's about human-scale experience, rather than measured latency. It won't be as fast as native, but it seems adequate for my use, even when grepping thousands of files, unpacking archives, etc, and I haven't noticed any significant stalling or pauses. It's encrypted too (by MacOS).
(That's in contrast to host-guest file access over the virtual network, which definitely has performance issues. But ext4 on the VM disk seems to work well.)
The VM is my main daily work "machine", and I'm a heavy user, so I'd notice if I/O latency was affecting use.
I'm sure it helps that the Mac has a fast SSD though.
(In contrast, on servers I use LVM a lot, in conjunction with MD-RAID and LUKS encryption.)
I've only given Linux 6GB RAM at the moment, and it's working out fine. Currently running Ubuntu 19.10.
I picked VMware Fusion originally because it was reported to have good-ish support for GPU emulation that was compatible with Linux desktops at the time. Without it, graphics can be a bit clunky. With it, it feels smooth enough for me, as a desktop.
My browser is Firefox on the Mac side, but dev web servers all on the Linux side.
The VM networking is fine, but I use a separate "private" network (for dev networking) from the "NAT" network (outgoing connections from Linux to internet), so Wifi IP address changes in the latter don't disrupt active connections of the former.
My editor is Emacs GUI on the Mac side (so it integrates with the native Mac GUI - Cmd-CV cut and paste etc, better scrolling), although I can call up Emacs sessions from Linux easily, and for TypeScript, dev language servers etc., Emacs is able to run them remotely as appropriate.
Smoothness over SSH from iTerm is a different thing from graphical desktop smoothness.
When doing graphics work (e.g. Inkscape/GIMP/ImageMagick), or remote access to Windows servers using Remmina for VNC/RDP, I use the Linux desktop.
But mostly I do dev work in Linux over SSH from iTerm. I don't think I've ever noticed any smoothness issues with that, except when VMware networking crashes due to SMB/NFS loops that I shouldn't let happen :-)
>And build times? Nobody thinks compiler that works minutes or even hours is a problem. What happened to “programmer’s time is more important”? Almost all compilers, pre- and post-processors add significant, sometimes disastrous time tax to your build without providing proportionally substantial benefits.
Anecdotally, in my career I've never had to compile something myself that took longer than a few minutes (but maybe if you work on the Linux kernel or some other big project, you have; or maybe I've just been lucky to mainly use toolchains that avoid the pitfalls here). I would definitely consider it a problem if my compiler runs regularly took O(10mins), and would probably consider looking for optimizations or alternatives at that point. I've also benefited immensely from a lot of the analysis tools that are built into the toolchains that I use, and I have no doubt that most or all of them have saved me more pain than they've caused me.
>You’ve probably heard this mantra: “Programmer time is more expensive than computer time.” What it means basically is that we’re wasting computers at an unprecedented scale. Would you buy a car if it eats 100 liters per 100 kilometers? How about 1000 liters? With computers, we do that all the time.
The point is that we are wasting all the resources at every scale. We are supposedly burning computer cycles because developer time is more important. Yet we are also burning developer time with compiling, or testing for interpreted languages, at a rate that is starting to approach the batch processing days.
I use webpages for most of the social networking platforms such as Facebook. I am left handed and scroll with my left thumb (left half of the screen). I have accidentally ‘liked’ people’s posts, sent accidental friend requests only because of this reason.
Guessing along with language selection, it might be helpful to have a selection of hand preference for mobile browsing.
Moreover, the same cost-equation that produces software that is much less efficient than it could be produces software that might be usable for it's purpose (barely) but is much more ugly, confusing, and buggy than it needs to be.
That equation is add the needed features, sell the software first, get lock in, milk it 'till it dies and move on. That's equation is locally cost-efficient. Locally, that wins and that produces the world we see every day.
Maybe, the lack of craftsmanship, the lack of doing one's activity well, is simply inevitable. Or maybe the race to the bottom is going to kill us - see the Boeing 737 Max as perhaps food for thought (not that software as such was to blame there but the quality issue was there).
I think for webpages it is the opposite: non-orthogonal in most cases.
If you disable your JS/Ad/...-blocker, and go to pages like Reddit, it is definitely slower and the CPU spikes. Even with a blocker, the page still does a thousand things in the first-party scripts (like tracking mouse movements and such) that slow everything down a lot.
Wait, are you implying they don't ? What world do you live in, and how do I join?
or... something.
The car analogy does remind me of one I read a while ago, comparing cars and their cost and performance with CPUs.
1/ What didn't seem to get mentioned was the speed to market. It's far worse to build the right thing no one wants, than to build the crappy thing that some people want a lot. As a result, it makes sense for people to leverage electron--but it has consequences for users down the line.
2/ Because we deal with orders of magnitude with software, it's not actually a good ROI to deal with things that are under 1x improvement on a human scale. So what made sense to optimize when computers were 300MHz doesn't make sense at all when computers are 1GHz, given a limited time and budget.
3/ Anecdotally (and others can nix or verify), what I hear from ex-Googlers is that no one gets credit for maintaining the existing software or trying to make it faster. The only way you get promoted is if you created a new project. So that's what people end up doing, and you get 4 or 5 versions of the same project that do the same thing, all not very well.
I agree that the suckage is a problem. But I think it's the structure of incentives in the environment that software is written that also needs to be addressed, not just the technical deficiencies of how we practice writing software, like how to maintain state.
It's interesting Chris Granger submitted this. I can see that the gears have been turning for him on this topic again.
There's ways to develop working software, but not if it's all locked behind closed OSes and other bullshit.
In my experience, a lot of performance problems boil down to really stupid problems, like simple code using the wrong data structure out of convenience (e.g. linked lists instead of arrays for lots of randomly-accessed data), or structured in a bad way (e.g. allocating a lot of small pieces of memory all the time). Often times, there are cheap performance wins to be had if you occasionally run the product through a profiler and spend couple of hours fixing the most pressing issue that shows up. Couple of hours isn't much; there's enough slack in the development process to find those hours every month or two, without slowing down your regular work.
But I have a different experience (probably because we work in different areas):
Most the performance problems of the products I ever worked were purely systemic.
They boiled down to technologies and architectures having been chosen for "organizational" rather than technological reasons.
And "organizational" is in quotes because sometimes it was just blackmail: I worked with two developers who quit in protest after the prototype they wrote in Scala was deemed not good enough and dropped for... being too slow, ironically.
In the end the backend was pure and beautiful, but the the frontend devs had to perform joins in the the client and make 21 API calls in a 20-item list and then everything goes to hell.
I find it really interesting that no one in the future of programming/coding community has been able to really articulate or demonstrate what an "ideal" version of software engineering would be like. What would the perfect project look like both socially and technically? What would I gain and what would I give up to have that? Can you demonstrate it beyond the handpicked examples you'll start with? We definitely didn't get there.
It's much harder to create a clear narrative around the social aspects of engineering, but it's not impossible - we weren't talking about agile 20 years ago. The question is can we come up with a complete system that resonates enough with people to actually push behavior change through? Solving that is very different than building the next great language or framework. It requires starting a movement and capturing a belief that the community has in some actionable form.
I've been thinking a lot about all of this since we closed down Eve. I've also been working on a few things. :)
One thing I found was that surprisingly the C# code I write outperforms the C++ code I used to write at equal development times.
I was good at C++, but the language has so many footguns and in general is so slow to develop in that I would stick to "simple" and straightforward solutions. I avoided multi-threading like the plague because it was just so hard to get right.
Meanwhile in C# it's just so easy to sprinkle a little bit of multithreading into almost any application (even command-line tools) that I do it "just because". Even if the single-threaded performance is not-so-great, the end result is often much better.
Similarly, it's easy to apply complex algorithms or switch between a few variants until something works well. In C++ or even Rust, the strict ownership semantics makes some algorithm changes require wholesale changes to the rest of the program, making this kind of experimentation a no-go.
The thing that blows my mind is the "modern" approach to programming that seems to be mostly young people pretending that Java or C# just don't exist.
Have you seen what JavaScript and Python people call "easy?" I saw a page describing a REST API based on JSON where they basically had thousands of functions with no documentation, no schema, and no typed return values. It was all "Just look at what the website JS does and reverse engineer it! It's so easy!"
I was flabbergasted. In Visual Studio I can literally just paste a WSDL URL into a form and it'll auto-generate a 100K-line client with async methods and strongly-typed parameters and return values in like... a second. Ditto for Linq-2-SQL or similar frameworks.
Because it's never problems really, it's perceived that way though.
A certain challange needs a specific set of personalities to solve it. That's the real puzzle.
Great engineers will never be able to solve things properly unlessed given the chance by those who control the surroundings.
We seek how we should develop, what method should be used, is it agile or is it lean? But maybe the problem starts earlier and focusing on exactly what methods and tools to use we miss out on the most simplest solution even beginners can see.
For example I am an architect, I tend to not touch the economics in a project. It's better fitted for other persons.
While not having read much about team based development I do want to be directed to well read literature about it. Maybe it's better called social programming, just another label of what we really do.
The one I miss the most at work is my wife. She clearly is the best reverse of me and makes me perform 1000x better. I find that very funny since she does not care about IT at all.
And by "social and inventive structures", I'm assuming you're talking about change on the order of how open source software or agile development changed how we develop software?
While agile did address how to do software in an environment for changing requirements and limited time, we don't currently have anything that addresses an attention to speed of software, building solid foundations, and incentives to maintain software.
What would a complete system encompass that's currently missing in your mind?
Writing performant, clean, pure software is super appealing as a developer, so why don't I do something about the bloated software I write? I think a big part of it is it's hard to see the direct benefit from the very large amount of effort I'll have to put in.
Sure I can write that one thing from that one library that I use myself instead of pulling in the whole library. I might be faster, I might end up with a smaller binary, it might be more deterministic because I know exactly what it's doing. But it'll take a long time, might have a lot of bugs and forget about maintaining it. Then end of the day, do the people that use my software care that I put in the effort to do this? They probably won't even notice.
To me, where libraries get a bit more questionable is when they exist in the realm of pure abstraction, or when they try to own the flow of control or provide the structure around which your program should hang. For instance, with something like Ruby on Rails, it sometimes feels like you are trying to undo what the framework has assumed you need so that you can get the functionality you want. A good library should be something you build on top of, not something you carve your implementation out of.
> While I do share the general sentiment, I do feel the need to point out that this exact page, a blog entry consisting mostly of just text, is also half the size of Windows 95 on my computer and includes 6MB of javascript, which is more code than there was in Linux 1.0. Linux at that point already contained drivers for various network interface controllers, hard drives, tape drives, disk drives, audio devices, user input devices and serial devices, 5 or 6 different filesystems, implementations of TCP, UDP, ICMP, IP, ARP, Ethernet and Unix Domain Sockets, a full software implementation of IEEE754 a MIDI sequencer/synthesizer and lots of other things.
>If you want to call people out, start with yourself. The web does not have to be like this, and in fact it is possible in 2018 to even have a website that does not include Google Analytics.
https://www.reddit.com/r/programming/comments/9go8ul/comment...
> Today’s egregiously bloated site becomes tomorrow’s typical page, and next year’s elegantly slim design.
This is Twitter, not some random framework!
Edit: found a link with the same story: https://www.folklore.org/StoryView.py?story=Saving_Lives.txt
The software world needs more of this kind of thinking. Not more arguments like "programmer's time is worth less than CPU time", which often fail to account for all externalities.
In my country, SW engineer is one of the best careers in terms of income, and I bet it is similar in most of the other countries. Why do we deserve that much buzz/fame/respect/income if the work we are doing is NOT making the society better?
These thoughts just haunt me from time to time.
Edit: in other simpler words, it's useful and scales fine.
I understand that you're asking a theoretical question, not a practical one, but in practical terms the answer is fairly simple. Our economy is not built to (indeed, is built not to) reward individuals in line with what they contribute to society. An entirely different set of incentives are what structure our economy, and therefore the jobs and lives of most people.
In some sense, David Graeber's Bullshit Jobs is all about the widespread awareness (and denial) of this phenomenon, and what caused it. I wouldn't say it's a perfect book but it's the best one I've read on the subject.
Meh this is manager-speak for "saving human lives" which they definitely were not. They weren't saving anybody. I mean, there's argument that, in modern day, 2020, time away from the computer is more well-spent than on a computer; so a faster boot time is actually worse than a slower boot time. Faster boot time is less time with the family.
Good managers, like Steve Jobs was, are really good at motivating people using false narratives.
As it is, software can largely free ride on consumer resources.
As I write this, I've been trying to get my Amazon seller account reactivated for more than a year, because their reactivation process is just... broken. Clicking any of the buttons, including the ones to contact customer support just take you back to the same page. Attempts to even try to tell someone usually put you in touch with a customer service agent halfway across the world who has no clue what you're talking about and doesn't care; even if they did care, they'd have no way to actually forward your message along to the team that might be able to spend the 20 minutes it might take to fix the issue.
The "barely working" thing is even more common. I feel like we've gotten used to everything just being so barely functional that it isn't even a disadvantage for companies anymore. We usually don't have much of an alternative place to take our business.
I don't mean to shit on Khan Academy exactly because it's not like I'm paying for it, but those lessons may as well not exist for a 4 year old with an interface that poor. It was bad enough that more than half my time intervening wasn't to help him with the content, nor to teach him how to use the interface, but to save him from the interface.
This is utterly typical, too. We just get so used to working around bullshit like this, and we're so good at it and usually intuit why it's happening, that we don't notice that it's constant, especially on the web.
we need a solution to this mess. so far i've seen popups (of all things) letting users know they should disable the ad blocking. but that's not a solution. ideally websites should not break when ad blockers are enabled, but i've seen sites where their core product depends on ad blocking being disabled. strange/chaotic times we live in.
That's because the shotgun approach(sick 40 developers on a single problem idc how they dole out the workload) works well for most low stakes, non-safety-critical software.
So like a reactivation portal for your Amazon seller account is very low stakes. But Boeing treating the 737-MAX the same way, would be(and was) a very bad idea.
Because that low-stakes approach is extremely bug prone.
* Measure whether the service you provide is actually working the way your customers expect.
(Not just "did my server send back an http 200 response", not just "did my load balancer send back an http 200", not just "did my UI record that it handled some data", but actually measure: did this thing do what users expect? How many times, when someone tried to get something done with your product, did it work and they got it done?)
* Sanity-check your metrics.
(At a regular cadence, go listen for user feedback, watch them use your product, listen to them, and see whether you are actually measuring the things that are obviously causing pain for your users.)
* Start measuring whether the thing works before you launch the product.
(The first time you say "OK, this is silently failing for some people, and it's going to take me a week to bolt on instrumentation to figure out how bad it is", should be the last time.)
* Keep a ranked list of the things that are working the least well for customers the most often.
(Doesn't have to be perfect, but just the process of having product & business & engineering people looking at the same ranked list of quality problems, and helping them reason about how bad each one is for customers, goes a long way.)
I've lost count of the number of monitoring systems I've opened up just to see a wall of red tapering off to orange after scrolling a couple of screens further down.
At times like this I like to point out that "Red is the bad colour". I generally get a wide-eyed uncomprehending look followed by any one of a litany of excuses:
- I though it was the other team's responsibility
- It's not in my job description
- I just look after the infrastructure
- I just look after the software
- I'm just a manager, I'm not technical
- I'm just a tech, it's management's responsibility
Unfortunately, as a consultant I can't force anyone to do anything, and I'm fairly certain that the reports I write that are peppered with fun phrases such as "catastrophic risk of data corruption", "criminally negligent", etc... are printed out only so that they can be used as a convenient place to scribble some notes before being thrown in the paper recycling bin.
Remember the "HealthCare.gov" fiasco in 2013? [1] Something like 1% of the interested users managed to get through to the site, which cost $200M to develop. I remember the Obama got a bunch of top guys from various large IT firms to come help out, and the guy from Google had an amazing talk a couple of months later about what he found.
The takeaway message for me was that the Google guy's opinion was that the root cause of the failure was simply that: "Nobody was responsible for the overall outcome". That is, the work was siloed, and every group, contractor, or vendor was responsible only for their own individual "stove-pipe". Individually each component was all "green lights", but in aggregate it was terrible.
I see this a lot with over-engineered "n-tier" applications. A hundred brand new servers that are slow as molasses with just ten UAT users, let alone production load. The excuses are unbelievable, and nobody pays attention to the simple unalterable fact that this is TEN SERVERS PER USER and it's STILL SLOW!
People ignore the latency costs of firewalls, as one example. Nobody knows about VMware's "latency sensitivity tuning" option, which is a turbo button for load balancers and service bus VMs. I've seen many environments where ACPI deep-sleep states are left on, and hence 80% of the CPU cores are off and the other 20% are running at 1 GHz! Then they buy more servers, reducing the average load further and simply end up with even more CPU cores powered off permanently.
It would be hilarious of it wasn't your money they were wasting...
[1] https://en.wikipedia.org/wiki/HealthCare.gov#Issues_during_l...
https://www.youtube.com/watch?v=pW-SOdj4Kkk
His point is basically that there have been times in history where the people who were the creative force behind our technology die off without transferring that knowledge to someone else, and we're left running on inertia for a while before things really start to regress, and there are signs that we may be going through that kind of moment right now.
I can't verify these claims, but it's an interesting thing to think about.
https://tonsky.me/blog/good-times-weak-men/
Another take: rewrites and rehashes tend to be bad because they are not exciting for programmers. Everything you re about to write is predictable, nothing looks Clearly better and it just feels forced. First versions of anything are exciting, the possibilities are endless, and even if the choices along the path are suboptimal, they are willing to make it work right.
Anecdotally, a lot of rewrites happen for the wrong reasons, usually NIH or churn. The key to a good rewrite is understanding the current system really well, without that its very hard to work with it let alone replace it.
Nobody has any fucking idea what’s going on in their react projects. I work with incredibly bright people and not a single one can explain accurately what happens when you press a button. On the way to solving UI consistency it actually made it impossible for anyone to reason about what’s happening on the screen, and bugs like the ones shown simply pop up in random places, due to the complete lack of visibility into the system. No, the debug tooling is not enough. I’m really looking forward to whatever next thing becomes popular and replaces this shit show.
Reading Computer Lib/Dream Machine over the holidays and I wonder where it all went so wrong.
On both ends.
Software developers hate boring software for pragmatic HR-driven career reasons and because devs are apes and apes are faddish and like the shiny new thing.
And commercial hegemony tends to go to the companies that slap something together with duct tape and bubble gum and rush it out the door.
So you get clusterfucks like Unix winning out against elegantly designed Lisp systems, and clusterfucks like Linux winning out against elegantly designed Unix systems, and clusterfucks like Docker and microservices and whatever other "innovations" "winning out" over elegantly design Linux package management and normal webservers and whatnot.
At some point someone important will figure out that no software should ever need to be updated for any reason ever, and a software update should carry the same stigma as...I don't know...adultery once carried. Or an oil spill. Or cooking the books. Whatever.
But then also it's important to be realistic. If anyone ever goes back and fixes any of this, well, a whole lot of very smart people are going to go unemployed.
Speaking of which...
Recursive solutions are more elegant but you still use a stack and while loop to not smash the stack.
So there are reasons to prefer more imperative languages and their systems, but stack-smashing isn't one of them.
Literally anything built with NPM: megabytes? tens of megabytes? in size, totally inscrutable, code being pulled in from hundreds of megabytes of code in tens of thousands of packages from hundreds or thousands of people of unknown (and unknowable) competence and trustworthiness, if it breaks not only do you not know who to blame but you probably have literally no idea what wrong.
Yeah, jQuery was probably better.
React is just so, so much nicer to work with. It's easy to be dismissive if you've never had to develop UIs with jQuery and didn't experience yourself the transition to React which is a million times better in terms of developer experience.
I feel like people that don't build UIs themselves think of them too much in a completely functional way as in "it's just buttons and form inputs that do X", and forget about the massive complexity, edge cases, aesthetic requirements, accessibility, rendering on different viewports, huge statefulness, and so on.
And then if you're still feeling cocky try finding someone else who uses one without the other.
So initially an equivalent React and jQuery app would have React look a lot faster, due to smart / batched DOM updates. However, because React is so fast it made people create apps differently.
As always in software development, an application will grow to fill up available performance / memory. If people were to develop on intentionally constricted computers they would do things differently.
(IIRC, at Facebook they'll throttle the internet on some days to 3g speeds to force this exact thing. Tangentially related, at Netflix (iirc) they have Chaos Monkey which randomly shuts down servers and causes problems, so errors are a day to day thing instead of an exception they've not foreseen).
I'm with you, but motivation to really learn a system tanks when there's something else on the horizon. And what happens when new-thing appears really great for the first 1-2 years, but goes downhill and we're back to asking for its replacement only 5 years after its release? That tells me we're still chasing 'new', but instead of a positive 'new', it's a negative one.
This was also reinforced constantly by people claiming you'll be unemployable if you aren't riding the 'new' wave or doing X amount of things in your spare time.
It's a natural consequence of an industry that moves quickly. If we want a more stable bedrock, we MUST slow down.
I think it's good to keep that in mind as a reference point.
--
When it comes to developer experience I'd say that React and company are ahead of most desktop UI technologies, and has inspired others (Flutter, SwiftUI).
I like Svelte, the simplicity of programming in it is great, and it has several advantages compared to React. But I have no idea how it works, past a point of complexity. Like, yes: I can run the compiler and check out the JS it generates, same as I can do in React. For simple components, sometimes the compiled code even makes sense. But when I introduce repeated state mutations or components that reference each other, I no longer know what's going on at all, and I don't think I'm alone in this.
Svelte might be an improvement in ergonomics (and that's a good and much needed thing!) but it does nothing to answer the obscurity/too-far-up-the-abstraction-stack-itis that GP mentioned. The whole point of that is frameworks/abstraction layers that tell you "you don't need to understand what's going on below here" are . . . maybe not lying, exactly, but also not telling the whole truth about the costs of operating at that level of both tooling abstraction and developer comprehension.
I devised a way of using plain closures to create DOM nodes and express how the nodes and their descendants should change when the model changes. So a component is a function that creates a node and returns a function not unlike a ReactComponent#render() method:
props => newNodeState
When called, that function returns an object describing the new state of the node. Roughly:
{ node, className: 'foo', childNodes: [ childComponent(props) ] }
So, it's organized exactly like a react app. A simple reconciliation function (~100 lines) applies a state description to the topmost node and then iterates recursively through childNodes. A DOM node is touched if and only if its previous state object differs from its current state object - no fancy tree diffing. And we don't have to fake events or any of that; everything is just the native platform.
I implemented an in-browser code editor this way. Syntax highlighting, bracket matching, soft wrap, sophisticated selection, copy/cut/paste, basic linting, code hints, all of it... It edits huge files with no hint of delay as you type, select, &c. It was a thorough test of the approach.
Also, when we animate something, we can hook right in to the way reconciliation works and pass control of the update to the component itself, to execute its own transition states and then pass control back to the reconcile function... This has made some really beautiful things possible. Fine grained control over everything - timing, order, &c. - but only when you want it.
Sorry for the wall of text.
I have been through this struggle too. Of wanting to get rid of bloated tools and tools I don't understand, and the best I've found for this is Hyperapp. I've read the source code a few times (was thinking about patching it to work well with web components), so I feel it falls into a category of tools I can use. But I'm genuinely interested in understanding what you've done if it offers an alternative (even if more clunky).
The object returned by the function expresses a tiny subset of the properties of a DOM node. Often just {className, childNodes: [...]}. Only those explicit attributes are checked for update or otherwise dealt with by my code. My code has no idea that a DOM node has a thousand properties. By contrast, a ReactComponent is more complex from a JS POV than a native DOM node.
In other words, if my code returns: {className: 'foo'} at time t0, and then returns {} at time t1, the className in the DOM will be 'foo' at t0 and at t1. That is not at all how exhaustive tree diffs work, and not at all how react works.
With 5,000 nodes, you might have 8K-15K property comparisons. Per-render CPU load thus grows linearly and slowly with each new node. I can re-render a thousand nodes in 2-5 milliseconds with no framework churn or build steps or any of that. But more importantly, we have the ability to step into "straight-to-canvas" mode (or whatever else) without rupturing any abstractions and without awkward hacks.
This is unidirectional data flow plus component-based design/organization while letting the DOM handle the DOM: no fake elements, no fake events -- nothing but utterly fast strict primitive value comparisons on shallow object properties.
EDIT: Earlier I said that a node changes if and only if its state description changed; that is not strictly true. "if and only if" should just be "only if".
To do this, I imagine you will have to do certain things manually. I guess you can't just have functions that return a vdom, because, as you say, the absence of a property doesn't mean the library will delete it for you. So do you keep the previous vdom? Patch it manually and then send it off to update the elements? ... I guess it's a minor detail. Doesn't matter.
Interesting approach, thanks for sharing! I will definitely spend some time looking into it. Encouraging that it seems to be working out for you :)
If you're thinking, "wait, compare it to the DOM node? That will be slow!" - notice that we're not querying the DOM. We're checking a property on an object to which we have a reference in hand. I can run comparisons against node.className (and other properties) millions of times per second. Recall that my component update functions return an object of roughly the form:
{ node, className: 'foo', childNodes: [...], ... }
That first property is the DOM node reference, so there's no difficulty in running the updates this way. Things are slower when dealing with props that need to be handled via getAttribute() and setAttribute(), but those cases are <10%, and can be optimized away by caching the comparison value to avoid getAttribute(). There are complications with numerical values which get rounded by the DOM and fool your code into doing DOM updates that don't need to happen, but it's all handle-able.
Here's quick gist: https://gist.github.com/jeffmcmahan/8d10c579df82d32b13e2f449...
I wrote a little bit more about my thoughts on the problem here: https://blog.usejournal.com/you-probably-shouldt-be-using-re...
> Nobody
Speak for yourself
(which is not specific to React but more like an issue for any framework that tries to do everything)
React is huge, it’s unlikely you’ll implement the synthetic events, lifecycle hooks, bottom up updates, context, hooks with their magical stacks, rendering “optimizations” and all of react-specific warts.
There are simple reimplementations like hyperapp and preact and I completely recommend using those instead. I really meant React the library and ecosystem are at fault, not the general model.
Add to this the modern way of being able to hotfix or update features and you will set an even lower bar for working software.
The reason an iPod didn't release with a broken music player is that back then forcing users to just update their app/OS was too big an ask. You shipped complete products.
Now a company like Apple even prides itself by releasing phone hardware with missing software features: Deep Fusion released months after the newest iPhone was released.
Software delivery became faster and it is being abused. It is not only being used to ship fixes and complete new features, but it is being used to ship incomplete software that will be fixed later.
As a final sidenote while I'm whining about Apple: as a consultant in the devops field with an emphasis on CI/CD, the relative difficulty of using macOS in a CI/CD pipeline makes me believe that Apple has a terrible time testing its software. This is pure speculation based on how my experience. A pure Apple shop has probably solved many of the problems and hiccups we might run into, but that's why I used the term "relatively difficult".
> iOS 11 dropped support for 32-bit apps. That means if the developer isn’t around at the time of the iOS 11 release or isn’t willing to go back and update a once-perfectly-fine app, chances are you won’t be seeing their app ever again.
but then he also says:
> To have a healthy ecosystem you need to go back and revisit. You need to occasionally throw stuff away and replace it with better stuff.
So which is it? If you want to replace stuff with something better, that means the old stuff won't work anymore... or, it will work by placing a translation/emulation layer around it, which he describes as:
> We put virtual machines inside Linux, and then we put Docker inside virtual machines, simply because nobody was able to clean up the mess that most programs, languages and their environment produce. We cover shit with blankets just not to deal with it.
Seems like he wants it both ways.
Otherwise it's a tradeoff if you add constraints like cost, effort, time to market, and so on...
And speed isn't the only metric that matters; having both the 32-bit and 64-bit versions of DLLs uses a non-trivial (to some people) amount of disk space, bandwidth, complexity, etc.
If Android, Mac OS, etc were super slimmed down systems in comparison to Windows, I would understand the argument much better. Instead, it feels like we're in the worst of both worlds.
Yeah, didn't say it's impossible. I said it's a tradeoff.
Windows does it and pays for it with slower releases, more engineers, bugs, strange interaction between old and new, several layers of UI and API code for devs to decode and for users to be confused with, less ability to move to new paradigms (why would devs bother if the old work), 2 versions of libs loaded (32/64 bit), and several other ways besides...
E.g. I've stopped using Windows for a decade or so, but I read of the 3 (4?) settings panels it has, the modern, the Vista style, the XP style, and so on, with some options in one, the others in the other (if you click some "advanced" menu, etc).
I don't quite know what's going on inside Apple, but it doesn't feel like they're choosing which features to remove in a particularly thoughtful way.
---
Twenty years ago, Apple's flagship platform was called Mac OS (Mac OS ≠ macOS), and it sucked beyond repair. So Apple shifted to a completely different platform, which they dubbed Mac OS X. A slow and clunky virtualization layer was added for running "classic" Mac OS software, but it was built to be temporary, not a normal means of operation.
For anyone invested in the Mac OS platform at the time, this must have really sucked. But what's important is that Apple made the transition once! They realized that a clean break was essential, and they did it, and we've been on OS X ever since. There's a 16-year-old OS X app called Audio Slicer which I still use regularly in High Sierra. It would break if I updated to Catalina, but, therein lies my problem with today's Apple.
If you really need to make a clean break, fine, go ahead! It will be painful, but we'd best get it over with.
But that shouldn't happen more than once every couple decades, and even less as we get collectively more experienced at writing software.
Even with our priorities in order, there will still be contentious, hard choices (to deprecate so-and-so or not; to sacrifice a capability for consistency of interface or not), but the author's point is that our priorities are not in order in the first place, so the decisions we make end up being arbitrary at best, and harmful/driven by bad motivations at worst.
Before we fix performance, bloat, etc, we really need to make software reliable.
The reason for unreliability is probably the same reason why things are slow: developers and project managers who don't care about the users and/or who are not incentivized to improve performance and reliability.
If you think that "not caring about the users" is too harsh, consider that users do suffer from e.g. unoptimized web pages or apps that use mobile data in obscene quantities. This has a direct consequence on people's wallets or loss of connectivity which is a huge pain.
As developers we can all try to instill "caring about the users" into our team's priorities.
Apple have totally forgotten how to test and assure software against what appear to be even stupid bugs. macOS Catalina has been fraught with issues ranging from the minor to the ridiculous. Clearly nobody even bothered to test whether the Touch Bar "Spaces" mode on the MacBook Pro 16" actually works properly before shipping the thing. Software updates sometimes just stop downloading midway through, the Mail.app just appears over the top of whatever I'm doing seemingly at random and Music.app frequently likes to forget that I'm associated with an iTunes account.
Microsoft are really no better - Windows 10 continues to be slow on modest hardware and obvious and ridiculous bugs continue to persist through feature releases, e.g. the search bar often can't find things that are in the Start menu!
My question is who is testing this stuff?
Telemetry.
Companies seem to be increasingly preferring to use invasive telemetry and automated crash reports in lieu of actual in-house testing, and they use that same telemetry to also prioritize work. I have a strong suspicion that this is a significant contributing factor to the absurdities and general user-hostility of modern products.
But this means that anything that's not represented in telemetry gets completely ignored. The numbers won't show you how many of your users are pissed off. They won't alert you to the majority of bugs. They won't tell you if you have a bloated web application that's stuffed full of advertising. They won't tell you if your UI is incoherent.
I really do think that large companies are looking at the numbers instead of actually using their software, and the numbers say that everything's fine.
It's understandable people want to base their decisions off empirical evidence. But it's important to remember that what you measure (vs. what you don't measure) and how you measure will determine the path you're going as much as the results of these measurements.
I can comfortably play games, watch 4K videos, but not scroll web pages?
I think this is one of the more important points that the article tries to get across, although it's implicit: while the peak of what's possible with computing has improved, the average hasn't --- and may have gotten worse. This is the point that everyone pointing at language benchmarks, compiler/optimisation, and hardware improvements fail to see. All the "Java/.NET is not slow/bloated" articles exemplify this. They think that, just because it's possible for X to be faster, it always will be, when the reality couldn't be further from that.
Speaking of bloat, it's funny to see the author using Google's apps and Android as an example, when Google has recently outdone itself with a 400MB(!) web page that purports to show off its "best designs of 2019": https://news.ycombinator.com/item?id=21916740
Where I differ a bit from your take: Languages and platforms that target high performance are providing application developers an elevated performance ceiling that allows them the luxury to use CPU capacity as they see fit. Application developers using high-performance platforms may then elect to make their application high-performance as well, yielding a truly high-performance final product, or they may elect to be spendthrifts with CPU time, yielding something middling on performance. And yes, a truly wasteful developer can indeed make even a high-performance platform yield something low-performance.
What benchmarks and the resulting friendly competitiveness help us avoid is a different and worse scenario. When we select a language or platform with a very low performance ceiling, application developers continuously struggle for performance wins. The high water mark for performance starts out low, as illustrated by how much time is spent in order to accomplish trivial tasks (e.g., displaying "hello world"). Then further CPU capacity is lost as we add functionality, as more cycles are wasted with each additional call to the framework's or platform's libraries. When we select a low-performance platform, we have eliminated even the possibility of yielding a high-performance final product. And that, in my opinion, illustrates the underlying problem: not considering performance at key junctures in your product's definition, such as when selecting platform and framework, has an unshakeable performance impact on your application, thereby pulling the average downward, keeping those peaks as exceptions rather than the rule.
Probably because a browser like FF has the goal to load and display arbitrary dynamic content in realtime like a reddit infinite scroll with various 4k videos and ad bullshit, whereas the game has the goal to render a known, tested number of pre-downloaded assets in realtime.
Also on shitty pages the goal is different-- load a bunch of arbitrary adware programs and content that the user doesn't want, and only after that display the thing they want to read.
Also, you can click a link somewhere in your scrolling that opens a new, shitty page where you repeat the same crazy number of net connections, parsing ad bullshit, and incidentally rendering the text that the user wants to read.
If you want to compare fairly, imaging a game character entering a cave and immediately switching to a different character like Spiderman and inheriting all the physics and stuff from that newly loaded game. At that point the bulk of your gameplay is going to be loading new assets and you're back to the same responsiveness problems of the shitty web.
Edit: clarification
Say hello to shaders.
It's not the fault of Firefox that Reddit's new UI is pathetically slow. It's the Reddit's implementation of their UI itself which is total garbage.
And given that people do write fast, complex, real-time games in JavaScript for the browser, gamedev absolutely becomes a valid reference point for the possible performance of any individual page.
Why should Firefox or any other dynamic software have the ability to be slow for what it archives? If compilers should be fast, Web engines should be equally as fast. The Web should have never been designed such that a slow website (relative to the task) could be achieved. In the same way that you can only express memory safe code in rust and type safe code in haskell why not being only able to express "fast for what is interactive"?
That's already the case, your orders of magnitude are just off. Long-running AJAX or page loads are timed out at a pretty consistent point across browsers. Half-open/closed TCP connections are timed out at a pretty consistent point across operating systems. Busy-looping JS gets you a "page is not responding" block in a similar amount of time; same for nonresponsive native applications on many operating systems.
Their definition of "slow" or "stuck" just tends to be "tens of seconds or minutes", not the threshold of perceived responsiveness you want in a website.
Also, your parenthetical is a pretty tall-née-impossible order:
> a slow website (relative to the task)
How could the "task" be classified? Do you mean "task" as in "clicking a button and having a DOM update"? Or as in "this is a TODO application so it should have responsiveness threshold X"?
As a web developer, sending an 8 KB JSON response is no problem. That's nice and light. In a networked action game, that's absurd. First, (hypothetical network programmer talking here) we're going to use UDP and write our own network layer on top of it to provide reliability and ordering for packets when we need it. We're going to define a compact binary format. Your character's position takes 96 bits in memory (float x, y, z); we'll start by reducing that to 18 bits per component, and we'll drop the z if you haven't jumped. Then we'll delta compress them vs the previous frame. Etc.
Really, what's happening is things are getting optimized as much as they need to be. If your game is running at 10 fps, it's going to get optimized. When it's hitting 60+ fps on all target platforms, developers stop optimizing, even if it could potentially be faster. Same for Reddit; it's fast enough for most users.
It depends on what that 8 KB is doing. If that 8 KB is a chat message, that's way too big. On the other hand, I've never seen an 8 KB game patch.
When I talk about sending 8 KB in a "networked action game", I'm referring to the update packets sent to and from clients in something like Fortnite or Counter-Strike, not a game patch. I'm not trying to make a competition for who uses the least bandwidth (which a 60 GB game would lose just on the initial download). I'm trying to illustrate that games don't run faster than some website because it's inherently easier to make games run fast, but rather that developers, by necessity, optimize games until they run fast (or in this example, until they reduce network lag enough).
I'm not sure why a chat app would tack on something like 7.5 KB of overhead on a chat message, but I wouldn't be surprised if there's a chat app out there that does so. Users won't notice the extra couple milliseconds (especially so because they don't know exactly when the other person hit send). A 3 character Discord message is close to 1 KB including the headers. The same message over UDP in a game might be under 20 bytes, including the UDP header (games could also use TCP for chat - text chat isn't going to strain anything). So I'd say the overhead of a Discord message is still an order of magnitude or two bigger than it could be. Which is perfectly fine; we can afford 1 KB of overhead on a modern connection. It's optimized as much as it needs to be.
Think about all the gas that is saved because people don’t have to drive to the library, all the plane trips saved by video conferencing, all the photo film, all the sheets of paper in file cabinets, all the letters being sent as emails, all the mail order catalogues, ... you get the idea.
Does anybody know of a comprehensive study on this?
You're right that computers have saved huge amounts of energy compared with the things you mention. My point here was that even more could be saved with a bit of thought about efficiency in programming.
I'm quite disenchanted with software myself. It takes way too long to open any program, for this JIRA ticket to properly display.
One thing that has improved was boot times, I seem to remember that Windows 7 was quite a bit faster than XP. Maybe someone in upper management wanted it to be as fast as MacOS? So speed IS possible, if it is prioritized.
I seem to remember boot times being a frequent topic of discussion the early 2000s, because people turned off their computers.
In a way, this is a great little microcosm of the problem. Just fix habits instead of fixing the software.
In my - perhaps limited - work experience, there's enough slack in the process of software development that I don't buy the "time to market" argument all that much.
(Do individuals really bother mining bitcoin these days anyway?)
if you haven’t seen it already, you’d probably be interested in the below talk by chuck moore, inventor of forth.
Fascinating talk. Thanks for the link.
But the vast majority of software is one-off stuff. It makes no sense to optimize it for performance instead of features, development time, correctness, ease of use, etc.
Is it now? I can't think of any example, except a few tools we use internally in a project. Everything else I use, or I see anyone else using, has an userbase of many thousands to millions, and a lot of that is used in context of work - which means a good chunk of the userbase is sitting in front of that software day in, day out.
I wouldn't be at all surprised if it's the majority. For every big-ticket software offering there's going to at least be things the user never interacts with, like a build system and a test suite and a bug tracking system and whatever else. And there's just so much software everywhere, most of which we never see. Every small business has its little web site. Who knows how much software there is powering this or that device or process at random factories, laboratories, and offices.
The Debian popularity contest looks like it has a big long tail of relatively unpopular packages [1]. It looks like the app store has 2 million apps, only 2857 (.14%) have more than a million dollars of annual revenue. These are of course incomplete and flawed and do not really directly address the question. I don't really know how to research this in a thorough way.
[1] https://popcon.debian.org/by_inst [2] https://expandedramblings.com/index.php/itunes-app-store-sta...
The things I'm doing on my phone today are not fundamentally different than what I was doing ten years ago. And yet, I had to buy a new phone.
1. The environmental cost of ineffective software is negligible, when compared to Bitcoin mining or other forms of hardware planned obsolescence.
2. By using more efficient software, surely, you can save a lot of CPU cycles, and it can improve the energy efficiency of some specific workloads under some particular scenarios. However, on a general-purpose PC, the desire for performance is unlimited, the CPU cycles saved in one way will only be consumed in other ways, and in the end, the total CPU cycles used remain a constant.
Running programs on a PC is like buying things, when you have a fixed budget but everything is cheaper, often people will just buy more. For example, I only start closing webpages when my browser becomes unacceptably slow, but if you make every webpage use 50% less system resource, I'll simply open 2x more webpages simultaneously. LED lightning is another example, while I think the overall effect is a reduction of energy uses, but in some cases, I think it actually makes people to install more lightnings, such as those outdoor billboards.
This is called the Jevons paradox [0].
For PCs, certainly, as I previously stated, in specific workloads under some particular scenarios, I totally agree that there are cases that energy use can be reduced (e.g. faster system update), but I don't think it helps much in the grand scheme of things.
"An Android system with no apps takes up almost 6 GB. Just think for a second about how obscenely HUGE that number is. What’s in there, HD movies? I guess it’s basically code: kernel, drivers. Some string and resources too, sure, but those can’t be big. So, how many drivers do you need for a phone?
Windows 95 was 30MB. Today we have web pages heavier than that!
Windows 10 is 4GB, which is 133 times as big. But is it 133 times as superior? I mean, functionally they are basically the same. Yes, we have Cortana, but I doubt it takes 3970 MB. But whatever Windows 10 is, is Android really 150% of that?"
My favorite line: "Windows 95 was 30MB. Today we have web pages heavier than that!"
If there's a new saying for 2020, it shouldn't be that "hindsight is 2020"... <g>
Also... each web page should come with a non-closable pop-up box that says "Would you like to download a free entire OS with your web page?", and offers the following "choices":
"[Yes] [Yes] [Cancel (Yes, Do It Anyway!)]". <g>
...maybe?
I don't know if that's 150 MB' worth of data... but it's certainly a lot.
So what is in these huge downloads? Layers upon layers of virtual machines?
The analogy holds as long as we don’t cross layers, which is quite often true.
Therefore counting total number of layers is performance-wise quite similar to running this many layers of virtual machines. Of course, you need code for all of these translation layers.
I remember once reading that IBM was going to implement an XML parser in assembler and people were like "Why? If speed is needed then you shouldn't use XML anyway." I thought that concern was invalid because these days XML ( or JSON ) is really non-negotiable in many scenarios.
One idea that I've been thinking about lately is some kind of neural network enabled compiler and/or optimizer. I have heard that in the javascript world they have something called the "tree shaking" algorithm where they run the test suite, remove dependencies that don't seem to be necessary and repeat until they are getting test failures. I'm thinking why not train a LSTM to take in http requests and generate the http response? Of course sometime the request would lead to some sql, which you could then execute and feed the results back into the LSTM until it output a http response. Then try using a smaller network until something like your registration flow, or a simple content management system was just a bunch of floating point numbers in some matrices saved off to disk.
Why? With responses generated according to what? Are you really just suggesting using neural networks in the compiler's optimiser?
> Then try using a smaller network until something like your registration flow, or a simple content management system was just a bunch of floating point numbers in some matrices saved off to disk.
Why? What's the advantage over just building software?
Why would I want to do this? I'm not 100% sure ... I think it would be super fast once you got it working. I think it would avoid many security bugs. You wouldn't have to read that "oh drupal 3.x has 20 new security bugs" better go patch our code. I think when I had this idea I was thinking about it terms of a parallel system that could catch hacking by noting when actual http responses diverged too much from the predicted response. The main idea being that for a given input the output really is 100% predictable, assuming your app doesn't use random numbers like in a game or something.
To link this idea to the article, I think things like XML parsers could be written this way .... I can't prove it but I suspect that they would be very fast and not come with all the baggage that the article complains about.
I started thinking along these lines after reading stuff like this https://medium.com/@karpathy/software-2-0-a64152b37c35
Also I'll bet you that your neural net is > 100x slower than straight line code.
For session based variables? Not sure, either it all becomes stateless and the code has to read everything from storage for each reqeuest .... or maybe the lstm is able to model something like an entire user session and remember the stuff that the original app would have put in the session.
That Andrej Karpathy article that I linked to two comments above ... he pointed out, in a different blog post, that regular neural networks can approximate any pure function. Recurrent neural networks like the LSTM can approximate any computer program. It is because they can propagate state from step to step that allows them to do this.
As far as it being 100X slower, well at a certain point I will be willing to take your money :)
I imagine it would be fast, then you realise you've made a static content caching layer out of a neural network and replace it with Varnish cache and it would be hyper fast.
I just don't know how you can achieve that with static cache ... only if somebody else requested that exact mortgage calculation before and it is still in the cache.
Also, my idea of the "given input" from the earlier comment would have to include results of sql queries that would form the entire input to the LSTM.
But honestly I think over trained auto encoders can be used as hash maps. That would be an application more in line with what I think you are saying.
And then you're hoping that the NN would act like a JIT compiler/optimiser and run the same code faster. But if it was possible to process (compile? transpile? JIT compile?) the Python code to run faster, then writing a tool to do that sounds easier than writing an AI which contains such a tool within it.
So there's a handwave step where the AI develops its own innate Python-subset optimiser, without anyone having to know how to write such a thing, which would be awesome indeed .. is that possible?
I'm not sure I'd like a http webserver to silently fail, or be undebuggable when it comes to security vulnerabilities when given strange inputs.
If you consider yourself a world-leading expert on neural networks and have some secret sauce in mind, by all means, good luck... otherwise it sounds like a fools errand.
I do want to point out that I'm thinking of doing this on a very limited website, not a general purpose thing that replicates any possible website. When I imagine the complexity of a modest CMS or an online mortgage calculator I think that it is much less complex than translating human languages. The fact that web code has to be so much more precise than human language actually makes the task easier. But to be fair, I'm all talk at this point with no code to show for it. So I will keep these comments in mind, this thread has been helpful for helping me think through some of this stuff.
Imagine how bad it would be if not!
In the crypto world, this is called "bitslicing":
But I believe that's not the problem here, or at least not so much as the business being very impatient and just never respecting objections from the programmers that the quality of the software suffers due to far too tight deadlines.
I can’t remember who it was, but the idea always stuck with me.
Anyway, I agree that we should test our applications with the same hardware (and internet speed) of our average user. Very few people use a computer as good as a software engineer’s. :)
When I make music I have to do the opposite; take my earbuds out and listen to the music over my car stereo or via my MacBook Pro speakers, to make sure it doesn’t sound good only through earbuds.
I like them, but I've found that they make a bit too conscious about the sound, so I have to be extra careful. The isolation emphasizes noise and frequencies that the monitors don't, and that will be masked in the full mix. It used to lead me to a rabbit hole of noise-gating and EQs.
That's true! This happens all the time with super expensive recordings too, and I love it. IMO discovering new sounds in the music you're used to only adds to the experience.
I think it's very sensible (if a little unrealistic) to test applications on as many pieces of intended hardware as possible, and this somewhat reminds me about this story of snow clearance[0], where the solution was not only the intrudction of a new perspective but also empirical measurements.
[0]: http://info.gritit.com/blog/what-is-gender-balanced-snow-cle...
That being said, there's a lot of crap software out there that's slow even on 2020 hardware.
I've listened to quite a few people on YouTube and some musicians who clearly did their editing with a certain pair of headphones/earbuds. It comes through immediately on my speakers because high pitched whining noises made it through the edit (making it nearly unbearable to listen to) and/or they boosted the bass to infinity. (Usually indicates they never listened to their project on speakers because the bass heavy music between cuts becomes disgustingly loud whereas on headphones it's not that bad)
It makes listening to their content impossible on decent speakers but fine on okay headphones.
I still manage to crank some terrible solutions, though. :P
Since then, I've never worked in a team that would be so dedicated to perfomance. May be because that was a team that worked on a product with over 100 million installs that had all major features already developed - everybody else is too busy trying to figure out a market fit.
These kind of specs are incredible for a high-end desktop machine from 15 years ago that would be good for almost anything — gaming, browsing, etc... What happened to software (Linux & Friends, Firefox, etc) that renders hardware obsolete so quickly? Is it just purposeful optimization that uses more RAM to benefit performance elsewhere, or is it truly this disenchantment?
[1] "In any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control and those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely"
[1] https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle%27s_...
For instance, I started noticing that a lot of the code I've written or worked with in many projects have a particular flavor to it. Pieces that take some data, repackage it, and pass it on to different code that does essentially the same - all arranged in a structure that's supposed to reflect some shared, abstract understanding of the problem. I've started to call this type of code "bureaucracy", and I see it as something to be kept in check.
Try planning a route in an unfamiliar area with this slow an UI when you are standing outside and there's no place to sit and rest, and you need to click around on a bunch of stops just to see what busses are going through a stop and where they are heading.
So yes, optimization is still important.
We replaced the glorious and easily iterated and expanded google maps app with a photograph of a public transport map, and we could get an answer of how to get from any A to any B within seconds of looking at the map without typing or searching or waiting for anything.
Which also shows that, sometimes, slow software is less than useless.
Sites like Hacker News or https://i.reddit.com are still perfectly usable. In contrast to the 'modern' Reddit UI that that takes a couple of seconds to load even on my home WiFi.
Bonus points if it also lists the before and after streets at direction changes so people know if they missed a turn.
This could be implemented as a proxy to either of the maps apps (Google, Apple, OpenStreetMap)
On my phone it takes about 4-5 seconds to load even on Gbit WiFi. It's so bad I have a conspiracy theory that the pulsing Reddit orb is just on a timer and it's not actually loading anything. More likely it's just horrifically unoptimized I suppose.
You can contrast that with mapbox which seems to have been designed upfront with offline mode in mind.
If you mean UX, then fair enough.
Even my e-book reader Linux port boots to UI in ~2s.
It really is just bloat and lack of care.
I shut it down and repeated that, looking at my wristwratch.
FOUR SECONDS!
I couldn't believe that and tested maybe five times again. It took never more than 4 seconds from rocking the switch to WfW 3.11 Desktop with Word 6 and Excel 5 in autostart ready to use! It even got a DHCP address while being at it.
The thing is, at that time i had some Sun SparcStations, some HP PA-Risc Workstations, and assorted X86 PCs, one of them an AMD Athlon XP PR1800 overclcocked to PR2100 with 1.5 GB virtual channel memory SD-Ram, running NetBSD (because it just worked, don't ask). I felt very ahead of 'the curve'.
And then this trash came along and burst my bubble...
I sat there and only thought: 'what for?!'
Now some old Windows running atop of DOS isn't something to really envy, but in this combination of components, BIOS, drivers it got the job done without any hassle, FAST!
That was eye opening for me.
But on the other hand, a lot of web content does need to be faster. Gmail has somehow gotten so much slower to load over time. And every time I visit a newspaper/magazine website I am aghast at how bloated they are. Does that mean nodejs is inherently bad, no, but it does mean people should try to optimize noticeably terrible performance that actually degrades UX.
Much of it on the web also has to do with how much browsers can do. The number of CSS properties that can be applied, the ways different elements' sizes can automatically influence the layout of other elements, etc. These traits are what make the web such a powerful and attractive platform for user interfaces, but the complexity of the platform is definitely becoming a real issue that deserves attention.
A couple of points:
- NodeJS is server-only and usually has nothing to do with perceived performance of web apps
- The biggest offender of web performance is ads. They dump piles and piles of crappy JavaScript from dozens of different sources that all include their own copies of common libraries and have no incentive not to slow down the page.
- Beyond ads, the bottleneck is usually not even JavaScript, but layout (as in the paragraph above). Web layout is incredibly flexible and incredibly complex. Computing and rendering it all is slow, but it does serve a purpose. Not that it couldn't be improved.
Bad ads are a tragedy of the commons and I don't know what can be done about them unless Google or Facebook decided to throw their weight around to force them to be better.
I do wonder if a new web standard could be developed for using some constrained subset of the layout vocabulary, that would be cheaper and more straightforward to compute. The current version has to remain for backwards compatibility reasons, but it's trying to serve a bunch of different types of cases at once, and therefore doesn't do a great job at any one of them.
And as it's surrounded by numerous such "slightly inefficient but efficient from some other perspective" interactions overall efficiency dies a death of a thousand cuts.
When you actively start paying attention to it and comparing it to little examples of what could be you start noticing how utterly garbage everything is. Even at the base level in things that have a massive userbase and that are used not just once in a while but constantly it's disgusting trash. I look around at the company i work at. They're all using it and probably not noticing at all but the fucking file explorer in Windows is slow as fuck as is countless other elements and interactions of it. The companies website so simple in it's content and functionality and is made by a webdev agency but is a bloated mess that takes a while to get to....a logo that shows whilst it continues to load. The software my coworker wrote is small scale and he said i was wasting a lot of my time making some small action faster not recognising that it's been used many thousand times a day every day for more than 10 years now.
There's way too little moral panic
In this day of "Agile" development, as long as something's working during UAT, that's all that's needed for sales and consumers.
Webdev, IME, is an example where the ecosystem has facilitated bloated websites. I've worked with developers who throw any library they can just for basic things because they don't have a need to try to optimise. The meme of using jQuery for everything when it came out has just been replaced by other frameworks. I find it often depends on developers who really want to work on something and take pride in it vs those who just need something on their CV or got hired by following a few tutorials on the web but not understanding what they wrote (which, to me, signifies a hiring problem in the company). During code reviews, I encourage leads to keep calling up hacky code to a point where the developer will just start writing it properly the first time round. As developers, I feel we should be aware of not creating selfish software which hogs memory from other software or requires huge data downloads for mobile users (whenever doable). Possibly a naive ideal but if it's a byproduct of developing fast software for my end users, I think that's a win-win.
The guy is most definitely at least a genius.
They are great examples for his overall point though. It probably would've been better just to leave out the genius bit and talk about them as folks proving it can be done.
For those who don't want to dig, one example was Ember.js, which has a dependency called "glimmer" which makes up ~95% of the code size. The author looked into glimmer and found that it had the entirety of the Encyclopedia of Britannica's "G" section to include a definition of "glimmer" in their help menu.
And that wasn't even the most ridiculous example.
It's shameful that it's gotten this bad; but when you look at what people are expected of in the current climate it makes sense that this would happen.
* Horrendously short deadlines for enterprise CRUD (and the "frameworks" that support it)
* REUSE REUSE REUSE THIS REFUSE (few seem to know how to read source code before installing the dependency)
* "Not paying me enough for that shit"
* "We can't rewrite, we put 20 years into this codebase"
* Even our languages are shit, JS (despite it's usefulness) has undefined behavior as a feature.
* [among many others I'm sure you could think of]
Civility has a lesser place in efficiency then what we have now; you can't make a decent product without bashing a few skulls (figuratively ofc)
Lastly, don't be afraid to reinvent the wheel if your wheel is better than mine.
That is, you can still write small and light software. It takes the same amount of time as it always did, not much changed.
The troubling part is that the proliferation of bloated software is steadily establishing a new status quo - the software is now _expected_ to be big and heavy in order to be proper. Doubly so for the enterprise-y software. Bloat is becoming a sign of maturity and robustness.
If you "simply blocked all ads" the people making the pages wouldn't have the income which they maintain the pages with.
How smart do you have to be to understand that?
>We haven’t seen new OS kernels in what, 25 years? It’s just too complex to simply rewrite by now. Browsers are so full of edge cases and historical precedents by now that nobody dares to write layout engine from scratch.
Well, there's Fucsia, speaking of new kernels. And Mozilla is doing exactly that, written a new layout engine from scratch (plus a language to write it in).
(I agree with the general sentiment of the post, but the examples are often shoddy)
Well, in most cases this is not the case. Subscriptions don't work but for outlets with passionate niche audiences, or for upper income targeted "high class" outlets. Which leaves most of mainstream media outlets outside. Besides those outlets where ads are peripheral to the main income streams (niche fan supported, paywalled) usually have less ads to begin with.
>What if the site's actually a part of the OS that I've already paid for, but which inexplicably has to be connected to the Internet and serving me ads for Candy Crush?
Well, then it doesn't really need ads.
One upon a time, adverts were just cross-linked GIF images. No iframes, no Flash, no javascript, no cookies, just images. Easy for rendering engines to show, no need to call script interpreters or to add boatloads of rubbish into the DOM. No real performance hit beyond downloading the image initially.
I would quite happily return to that world.
That's how we get deliberately slow pages, articles split over multiple page loads, image slideshows and deliberately slow pages (because time spent on your site is time not spent on the competing websites).
On the other hand we have pages like HN, paid for(?) using other means and built to be perfectly usable and fast. Or some CSEs where revenue comes from affiliates and CPC fees instead of ads, so they try to keep things fast too. Then we have news sites with paywalls and hopefully some day better engineered UI to read those news.
People should just vote with their pockets for a better user experience, IMO.
How? How can I vote with my pocket for old.reddit.com instead of new reddit UI? How can I vote for another site which is different, when the main draw of Reddit is the number of people on there?
How can I vote for an instant messenger which works the way I want, when the people I talk to aren't using it because the people they talk to aren't using it?
How can I pay for a faster Windows 10 where the start menu works every time I press the start button and updates install in the time it takes to copy the data to SSD and no longer? That option isn't on the market.
I agree with regards to ads specifically; that's why I don't use an ad blocker.
But the problem is so much greater than ads. Ads aren't what's slowing down the new gmail, or Slack.
All systems - biological, physical, meta-physical are built on layers that once deep enough are pretty well cemented in. Hindsight is 20/20 and though we know if the foundations were different things would be better - they won't actually change until the energy gain exceeds the energy cost of uprooting everything to make the change.
Just saying this problem isn't exclusive to software, but also laryngeal nerves in giraffes, x86, and Esperanto.
But they don't waste much material.
An increasing number of providers (for example databases) charge per server, per core, or another hardware-usage metric. The more hardware, the more revenue for them as they make about 90% margin for every new machine. There is a high incentive to get users to need more machines on their "managed cloud".
Vendors could try to improve their software so it requires twice as little CPU. But why bother since this would result in 2x cost in revenue? It makes more sense to focus on horizontal scalability than on core efficiency so users can keep on adding machines to their cluster over time.
If software is running at 1% of the maximum performance as suggested in the article, a 1% improvement could reduce costs by 50%. But I think none of the existing vendors will ever make the move as it conflicts against their own interest.
> I can comfortably play games, watch 4K videos, but not scroll web pages? How is that ok?
IMO the comparison should be buildings <-> fine-tuned libraries (i.e. video decoding algorithms), modern applications <-> cities.
Go to any city center in Europe. Urban planning a century ago was much more elegant and elaborate taking into consideration the city as a whole. Nowadays developers and investors often ignore important aspects, such as surrounding buildings, infrastructure, making cities inefficient for the people who actually live there.
Any system which has plenty of resources has to become inefficient. It's just that the Moore's law allows for pretty damn inefficiency.
Eh, no. Just a castle/wall, shops around in the main St's, a market in the middle, and expanding chaotically from that.
The problem of modern urban design is that of unrestricted freedom for developers. They build mostly whatever the like, however they like, the city be damned as long as they make a profit off their construction. What's lacking here is care - personal care about the city, and centralized care in form of a city that can tell them they can either submit to the constraints of a more holistic planning, or take their business elsewhere.
Since the current practices are so inefficient, users are having to pay for this from pocket as hardware expenses. Buy a $1000 smartphone, to get the same experience as last-year's.
A different stack (don't need to reinvent the universe), can be branded as brutally efficient, uses slim hardware, and strict engineering practices to provide a much better experience at a fraction of the price (1/5 possible.)
I believe nobody is doing that yet since there are two main barriers:
1- Risk of no-market (I think this might be proven unfounded, given the current trend in price hikes)
2- Capital investment necessary to get started (But this also can be solved given the obvious appetite for the next-big-thing money being poured left and right, without anything catching on yet)
The argument is incomplete.
The correct question (to maintain the analogy) is:
"Would you buy a car if it eats 1000 liters per 100 kilometers, but that doesn't affect you at all (you still get to where you want to be fast enough), and the time to manufacture and cost to buy it is much lower than would be possible with more efficient car that used 10 liters per 100 km?"
The answer to which would be yes. A software that does a task we do once a day or so at 1 second vs 0.2 seconds doesn't cost us money (and even the environmental impact is small).
I can deal with a 1 second delay to run a script. I can deal with a 20 second delay to launch a large program. I resent dealing with hundreds of 1 second lags every day for tasks which didn't used to have any lags ten years ago.
In the same way that recycling was meant to fix the environment (and garbage now sits in warehouses), millions of objects get mass produced and chucked onto the memory heap, in the hope that garbage collection will fix it.
The culture of "developer time is most important", makes overall system performance someone else's problem, because "my program is fast enough when I measure it off the wall clock". But who's responsible to fix overall system performance, and how can they fix it? I think a lot of people would just upgrade their RAM, CPU or IO to solve the issue (create more petrol stations), rather than asking vendors to change the programming language, or to be more conservative on RAM.
And because there's costs to switching language stacks, people will stick to writing in the language they are comfortable in, so critical business systems get written in slow languages.
That's to maintain parity with the example TFA gives. I'm not saying it's never worth it... Of course you'll optimize often repeated tasks, loops, etc.
Here's what TFA quotes as bad reasoning, because it makes fun of optimization effort in some cases:
"@tveastman: I have a Python program I run every day, it takes 1.5 seconds. I spent six hours re-writing it in rust, now it takes 0.06 seconds. That efficiency improvement means I’ll make my time back in 41 years, 24 days :-)"
Unlike TFA I agree with the twitting guy above that the efficiency improvement here wasn't worth it.
But as many languages support implicit static typing, the folklore that dynamic languages are "more expressive" is not actually correct.
such as:
let x = "string literal" //compiled as string
let y = 42 //compiled as int
let z : i32 = 100 //compiled with type declared as 32 bit int.
Using a real example: in a certain company that wife worked in, there was a task that - every couple of days - would be repeatedly done by multiple office workers for several hours. I was visiting that place once, and since I had to wait for my wife, I was asked to give them a hand with that task. Growing frustrated, I very quickly located a hidden option for batch processing they didn't know anything about. It solved the entire task in couple of minutes. By finding that option, I freed several man-days a month for that company. Time that can be used for other tasks (or even goofing off).
This example sticks with me because these days, a lot of work is done in front of computers, and every inefficiency there removes productivity from the economy. The way I see it, if you're developing software, and there is a possibility that that software will end up being used by someone in their job, you owe it to them to make it efficient; by ignoring efficiency, you'll be robbing such people of their life and mental health, and their employers of potential profit.
>That is not engineering. That’s just lazy programming
I don't believe that developers/programmers are all lazy. There are a lot that want to do a good job optimizing their code and making sure it performs well and is future proof as much as possible. I believe that budget limits and pressure from deadlines set by non-technical people forces even the good programmers to cut corners in order to deliver.
Somebody needs to write this software. Efficient, maintainable, and debugged are not the low-energy state.
That means it either needs to come from business, or from open-source hobbyists. Business doesn't see a competitive advantage in it -- even Apple, which has traditionally cared more about UX than anybody, just has to be better than their #2 competitor (and price of entry to "desktop OS" or "smartphone OS" is high so that list is short, and not changing on any relevant timescale). And the open-source world has never delivered well on the end-user experience side of things.
The sad truth is that users would rather have new software for $0 (paid by ads, or media subscriptions, or whatever) than pay what it truly costs to develop software.
My main hopes today are that the end of Moore's Law will force companies' hands, or that government will step in to regulate minimal quality, or that workers will organize so they can stand for quality behind a CBA. These all seem rather unlikely at this junction. The number of programmers in it for the paycheck far outweighs the number of people who care about simplicity.
Software is going to get much worse before it gets better.
It seems he does not realise that this is a satire piece, and seem to completely buy in to his view of the world instead of seeing things in a more nuanced way.
The software in nutshell is programmable transistors. Each CPU instruction is in effect just a convenient way to design a specific electronic circuitry. Even a trivial act of printing Hello World takes an astonishing amount of complexity when you take into account all kind of protoicals, APIs, driver codes, kernal code, fonts, rendering, graphics that gets executed in between. If you showed a computer printing Hello World on screen to someone from 1920s who knows how to build electronic circuits and primitive "display", they can estimate the amount of work that would be required to do that. Nothing has changed from 1920s to 2020s in terms of complexity to enable simple Hello World. A relatively simple program will easily exceed 30,000 low level components working together to achieve a goal in an intricate dance. Now think of large code bases with million lines of code... This is why software is hard, software is complex, software is messy and software is magic.
The exception would be games and embedded software, but even there, there's certain degrees of lazyness. For instance, games are very cpu/memory/gpu efficient, but they're almost always ridiculous when it comes to disk space usage. There's no reason that your average AAA game needs to take up 50GB other than that things which could address that aren't worth the fuss. (I'm thinking common demo-scene tricks like procedural textures/data/everything, aggressive compression schemes, reusing assets, etc.)
For that matter, one could for example take the Windows NT 4.0 source code, add in drivers for the necessary hardware, fix boot code, linking, etc to be compatible with late model computers, spruce up the UI with better font rendering, antialiasing, 24-bit color wallpaper, OpenGL rendering even--and in the end, you'd have something just as functional as Win 7/10 but at 1/4 of the bloat.
This sort of thing would be technically very easy to do. It's much easier than the status quo of continually reinventing the wheel. So why, oh why, is there this overpowering desire to continually throw out good code and replace it with heavier, more bloated junk, which doesn't really offer any real increase in functionality?
You've almost described ReactOS!
One second times five million users is almost two months. One second times five thousand users is less than 90 minutes.
Besides, you could essentially do what you’re describing by piecing together a Linux desktop from a lean distro. It’s not hard to find efficient software, it’s just not really mainstream
But something crucial is missing from this manifesto. As the author says, the problems don't exist because we can't solve them, but because no one ever takes an interest in solving them. We're all engineers, so why don't we just do some engineering and fix this bullshit? Well, because it wouldn't make any money.
Perplexingly and contradictorily, the profit motive drives both innovation and stasis, both growth and sprawl, and both efficiency and inefficiency. The problem is both technical and social, and probably so will be the solution.
This works great for my personal machines, but not for my company, which wants to do fancy networking things without going over the open internet.
I do fully agree that it’s an order of magnitude cheaper if you know what you are doing though. I’m sure our $3000/month AWS stack would run a lot more efficiently on my $20/month Robot server.
But you don't have to go to those extremes to have good software performance.
Whenever I read posts like Software Disenchantment, I find myself agreeing with that philosophy. In other words, it’s probably by design. Of course this doesn’t account for the enormous waste of time and money that occurs during software development, but that doesn’t really affect my feelings on the matter.
Software today is in the snake oil era. "Secure", "privacy-friendly", "robust"...
We need the government to create the FDA of software, to protect the consumers against potentially harmful software, marketed using false advertisement.
We also need job applicants to be protected against predatory companies that hire people that want to do right by the customer, but are forced to produce rushed, poor quality stuff.
What frustrates me above all else is the trend of compile-to-javascript languages. IMO interpreted languages are great, of course some performance was sacrificed to get there - I think that was a fair trade-off because saving the developer from having to build the project is a HUGE advantage when developing (at least for my particular development flow)... So when I see people throwing away that massive advantage by adding an unnecessary compile step in order to get slightly better syntax or static typing (e.g. CoffeeScript or TypeScript), I find it deeply disturbing. Static typing can be a useful feature, but is it worth adding a compile step? Not by a long shot.
And the idea of transpiling a language into an interpreted language is just ridiculous in principle. We had an army of very smart people who invested a huge amount of time and effort into making efficient interpreters for certain languages but all that work is thrown away as soon as you add a build step.
And the stunning thing is that it's actually possible (easy even) to create excellent software with clean code without a build step (I've done it many times) but these simple, clean approaches are never popular. People want to use the complex approach that introduces a ton of new problems, delays and compatibility issues.
The problem is that when your software is built on top of a framework and/or uses X different web APIs etc then you often run into issues where a part of the system that you don't have control over causes performance issues and you don't have the expertise/time to profile it in order to fix it. So I think what's causing problems is that software has become a lot more about putting together frameworks, libraries and reusable components and when faced with such a complex system a programmer will often give up and say "there! it's as fast as I can make it without rewriting everything from scratch".
Therefore, the issue seems to be that programmers are building on top of other systems that they don't know enough about to use efficiently. The author does mention this issue in his article as well, but in a slightly derogatory fashion blaming programmers for bringing in dependencies they don't need.
I think if everybody had the time and ability to write everything from scratch like Jonathan Blow is doing with Jai then yes, things would be more efficient. It is far easier to profile and debug code you've written yourself. However, seeing how this isn't feasible for most projects, I think more focus should be put on better documentation of frameworks and libraries.
I think we need a guild. We need licensed software engineers.
Not every programmer needs to be one, just like not every engineer needs to be licensed, but there needs to be a licensed engineer on every team. And, of course, sometimes there doesn't need to be. But I sure wish there was the option.
And hell, bring back apprenticeships and mentoring with the guild. There is so much we could learn from the physical science engineering disciplines
However, the idea of working closely with Senior Engineers and learning from them is certainly something that I vehemently agree with. I've been fortunate to have had that opportunity.
Mastery of frameworks is NOT mastery of our craft. They are useful tools that can come and go. But the underlying principles are what should concern is.
To that point, none of your examples of stable disciplines are static. New surgical tools, techniques, and technology are constantly produced and surgeons must learn to lay down their old ones and adapt. Metalworking, carpentry, plumbing all need to learn about new materials developed and code changes.
All of those things are like frameworks.
As self driving cars enable us to fit far more cars on the road before causing the same level of congestion, people will start taking increasingly longer car rides for decreasingly valuable reasons, up until the roads are equally intolerable as they were before the innovation had occured.
Replace "self driving cars" with "faster hardware / more memory".
http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...
Honestly things were way more sucky years ago. Your Windows computer crashing was just normal in the 90's. Rebooting and reinstalling par for the course. Getting Linux to install with drivers working seemed impossible unless you carefully chose hardware.
Or fewer features. Talk to an actual professional who uses spreadsheets all day long about switching from Excel to Google sheets. It's infuriating the infantilization of UIs and the "oh they'll never miss it" attitude.
but:
- economic incentives do not align with common sense
- landscape is fragmented, continually evolving and unstable
- we are all posers. we all have opinions and morals, look at others and criticise, but when it comes to our own work we are just like anyone else. we need money to live, so we just go with the flow
The whole point of Babel was to allow us to use the latest JavaScript syntax so that we wouldn't have to update our own source code when the new syntax finally became broadly supported by browsers and other engines.
IMO, the Babel project is a failure because:
- Babel itself is always being upgraded to support newer ECMAScript syntax, so people still need to upgrade their own code whether they use Babel or not. The only benefit is that using Babel allows you to use these features before other people.
- Instead of just worrying about how JavaScript syntax changes affect your code, with Babel, you also need to worry about how Babel upgrades will affect your code. The babel plugin dependencies often change and break over time (even if you don't change your own code) and you always have to support both ecosystems.
So when you consider the big picture, Babel doesn't save you from having to upgrade your own code (as per its original promise). When you evaluate the pros and cons, the cons greatly outnumber the pros:
Pros:
- You get to use the newest language features before other people.
Cons:
- You need to maintain your code for compatibility with two ecosystems instead of just one. Keeping up with both ECMAScript + Babel is a lot of work. I would even argue that staying up to date with Babel is more work because dependencies keep changing underneath your project.
- It forces you to use a build step so you lose the benefits/iteration speed that an interpreted language brings to your development flow.
- Adds a lot of bloat and unnecessary, hard-to-describe dependencies to your project which can open up security vulnerabilities and make your code more opaque and brittle.
An explanation of "why" does not explain "why this is acceptable".
I support the idealism of the article, but this quote is very accurate. Nobody wants to pay for quality software. Not your users, not your stakeholders, not even you!
And that's because the price of quality isn't just a little more expensive. It's not 1$ a year. It's exponentially more expensive. And some of those paid efforts will still end up slow and crappy in outcome, because of something else the article doesn't acknowledge:
Writing fast, efficient, simple, correct, full featured software is really really hard.
So not only is it expensive, it's also just plain difficult. Meaning it's not just about money and resources, but also time, time to experiment and fail.
If I think back to my engineering school days, the definition of "engineering" for my classmates in civil and electrical engineering was to look up well defined procedures and calculations from a book and apply them. No deep understanding required to design a bridge that didn't fall down or a circuit that didn't overheat.
What's the equivalent for software? Design patterns were a bust. SICP is for cultists. It's a huge void. There is hardly any such discipline as "software engineering" yet.
Well, where's your word processor buddy? Try to write one to achieve those goals -- and offer what people want today, including syntax highlighting, linting, auto-completions, etc, and come back to us...
Emacs could do that since 25 years ago, if not more.
And, Lisps aside, the kind of linting, auto-completion, understanding of syntax/AST, etc emacs could do 25 years ago is much easier compared to what devs expect today.
Also you're conflating syntax highlighting/auto-competition with time to render an update to an input. You don't need the popout letting you know that 17 source files and 50k lines over it found "TextComplete()" is a valid auto-complete and should be turned blue in order to draw an "x" to the screen in response to a keystroke.
There's no reference for this in the article, and it caught my attention - anyone have any idea what the author is talking about here? Never heard of this before
> Customizing victim selection: impact on the badness of processes via matching their names, cgroups, exe realpathes, environs, cmdlines and euids with specified regular expressions;
But the best explanation for why this problem persists even in teams of proficient engineers that I have seen comes from the 2005 GDC Keynote from John Carmack [1].
> I can remember clearly in previous projects thinking [...] I've done a good job on all of this but wouldn't it be nice if I could sit back and really clean up the code, perfect the interfaces, and you know, just do a wonderful nice craftsman job [...] interestingly this project I've had the time to basically do that and I've come to the conclusion that it sort of sucks [...] there's a level of craftsman satisfaction that you get from trying to do what you do at an extreme level of quality and one thing that I found is that's not really my primary motivation [...] that's not what's really providing the value to your end-users [...] you can sit back and I am do a really nice polishing job it's kind of nice but it's not the point of maximum leverage and I found.
So, as others have mentioned here, there is a threshold where extra performance provides less value to the project than, say, an extra feature.
It seems that in some cases we have crossed that threshold and some software has become comically bloated. I attribute the reason why these are not solved to the same reasoning. Refactoring an existing project to reduce the bloat would take too much time and effort from one single developer, that could be used somewhere else, although in the end everyone would benefit from it. So you are better off adding stuff to the dumpster fire and moving on.
It's sort of a tragedy of the commons [2] of software performance.
Modern text editors have to deal with proportional font, right to left writing, anti-aliasing, and a big bag of Unicode related issues (Arabic is going to break all assumptions you have on language and text editing)
Fantastic read on this subject from few months ago:
You and me both buddy! 11 years here, you are not alone in this!
What I found helped me take my mind of this nagging feeling is to bring in a tool that is like a razer sharp brand new scalpel. That was Nim. It's readable, small, fast, and spits out tiny statically compiled binaries.
Find your scalpel and slice and dice. Back to basics. Purity.
https://docs.gitlab.com/ee/administration/operations/unicorn...
because I've had to deal with that a bit. N.B. the last sentence here:
One other thing that stands out in the log snippet above, taken from GitLab.com, is that ‘worker 4’ was serving requests for only 23 seconds. This is a normal value for our current GitLab.com setup and traffic.
For example, how many authors on medium proclaim themselves "Senior Software Engineers" but when you dig find they've got maybe 5 years experience doing web development, with no CS or engineering education. Maybe stuff like a 36hr "Web Development Bootcamp". Do people really not understand the definition of an engineer anymore?
From there they progress into the deeper parts of software. And create the atrocities to be found in the npm registry, which become dependencies of dependencies of dependencies that results in nightmares everytime one needs to navigate to a website.
If it would've been possible to see the background and education of the numerous critics here, what may we find? If I (the developer as described by the OP) am surrounded by people like me, and the world filled with people who think like me, and create things at a similar cognitive level as my peers, would I not misjudge the collective level of quality that I perceive to be acceptable? Smells like confirmation bias to me? Maybe a few others. Dunning-Kruger anyone?
In their defense, the marketing campaigns created by large corporations to turn the supply-demand (cost of employment) of employees in their favor, has I think been a big part of the problem. First the programmers, then the "Data Scientists", etc - the amount of disappointment and student debt being created as these people eventually realise they've been sold something that they're not suited for!
If we cannot critically look at our industry and admit our flaws, we cannot move forward as a collective.
I jumped straight from Android 2 to Android 8 development and was surprised myself, so did some investigation and have half an answer. The author is actually wrong here, on both looking different and additional functionality. However, the bloat is still far larger than it needs to be.
All the bloat comes from the AppCompat modules, which all the docs recommend to the point of it apparently being required if you don't know better.
AppCompat is for both supporting differing APIs and creating a consistent look and feel across different Android versions. Each Android version has its own visual design, which Google decided was a bad thing, opting to use AppCompat so the most recent designs (and in some regards design functionality like coloring the selection handles) were used in older versions of Android.
To do this however, it includes a crap-ton of images. The build scripts are supposed to remove the unused ones, but even with maximum trimming enabled it can only remove somewhere around 10%. There are hardcoded inclusion rules for some of the AppCompat java code, that no one's found a way to override, which in turn reference the images - so they get kept a well, even if your app never uses them.
As for differing APIs, notifications have changed massively over the years. The interface is so different you do actually need the AppCompat subset for notifications to target different Android versions (and that can be used separately from the rest of AppCompat), but there also have been a huge number of new features added to notifications - such as delay settings, shortcuts, icons, even full-on fancy designs, that didn't exist early on.
I'm calling this only half an answer because there's no apparent reason for some of the notification API changes, and the build scripts/AppCompat can certainly be significantly improved to remove more cruft. I have a sneaking suspicion that it's not done because this is low-hanging fruit for handing over signing keys to Google for their "optimized" builds...
I am a dependency skeptic. I think that you need them to do big stuff, but should probably avoid them for small stuff.
High-quality dependencies can have a drastic impact on the quality of your software, but so can low-quality dependencies.
I think we are at the tail-end of a "wild west" of dependencies.
When the dust settles, there will be a few really good, usable and stable dependencies, and a charnel pit, filled with the corpses of all the crap dependencies, and, unfortunately, the software that depended on them.
The only way out of this is to rethink the web. Which is a hard one to tackle.
Boot used to be a small number of seconds. Now (on the rare occasions I'm forced to actually boot/reboot) I start the machine and bugger off to make coffee while it does its thing. I don't know what takes so fucking long, but it's in the range of 'several minutes'.
Starting apps likewise. I just started up an infrequently used picture-editing app a little while ago,... up towards a full minute of 'loading this crap', 'loading this other crap', etc.
And let's not mention Atom (an Electron app unless I've misunderstood something) -- so laggy for some things that should be near-instant that I'm developing an active hate.
Alright: get the hell off my lawn now!
I also see the problem. I hate bloated websites. I hate all those little unnecessary fancy CSS animations, I use uMatrix and regularly have to figure out which of those blahblahcdn.com domains (subdomains are whitelisted) needs to be enabled to make it work.
But even those webapps can be made to be "fast enough". Developers just have to be very careful on the design from the beginning and try to use as little deppndencies as possible. Prefer simplicity and speed over fancy effects and unnecessary features no one asked for.
Thats....not what google play services is, like at all. https://developers.google.com/android/guides/overview
because we are going to make editors using JAVASCRIPT, which was never meant to be used this way
Also - resources that are essentially free to use (customer/users CPU cycles, storage and bandwidth) will be consumed.
We as consumers are paying the bills for it all in different ways(electricity, new gadgets, cloud costs etc).
It depresses me to no end too, but i am surprised in the least.
There were tiny groups trying to go frugal and solid. Remember suckless ? I forgot other names, there's also alan kay vpri project with ometa.
Maybe we should make a frugalconf. Everything 25fps on a rpi zero
Of course they’re going to accept that delay, even if adding a few million numbers together should really take less than a millisecond.
Not everything needs to be super efficient. Most things are tuned for production cost and time. Efficient code isn't going anywhere. Relax guys.
We shouldn't have to put up with all this shit. That's the point.
Are you willing to pay $20 for a phone app, instead of fishing for the free version? Are you willing to pay for websites to get rid of ads? Are you willing to pay $300 for a new OS?
Alternatively, are you willing to ditch capitalism for a system that prioritizes the commons?
It's us. We are creating the incentives for a world that produces this. Unless we change that world, this is what we'll get.
I don't really know what this is supposed to mean, it sounds rather poetic and abstract, but I'm quite certain few if any people ever asked for a world where it takes 10+ seconds to open a simple email.
> Modern text editors have higher latency than 42-year-old Emacs. Text editors! What can be simpler? On each keystroke, all you have to do is update a tiny rectangular region and modern text editors can’t do that in 16ms. It’s a lot of time. A LOT. A 3D game can fill the whole screen with hundreds of thousands (!!!) of polygons in the same 16ms and also process input, recalculate the world and dynamically load/unload resources. How come?
Text is very complicated. Does your 42-year-old Emacs support Unicode? And not just accents, but whole different scripts?
See https://news.ycombinator.com/item?id=21105625 for some discussion and a good link about the complexities of rendering text.
emacs has also always been (in)famous for its text update render algorithms, that probably still work well over the old slow terminal lines it was originally used for.
Emacs had MULE in late 90's. It was THE best Unicode editor because it supported lot of encodings. You could switch from iso8859 to UTF8 on the fly.
No one's forcing you to use a library. But if you do they come with tradeoffs.
OK, things are slow and buggy. But we've got lots more things thanks to all the productivity we've gained from using libraries, etc. That means we collectively solve more problems for more people.
Purism is a nice idea, but ultimately probably not worth the effort until things become so bad that they are, and at that point become a differentiator. I mean, I don't care if a web page is twice the size of Windows 95 because my computer is way faster than a 486.
a) Lack of fundamental understanding of how computers work
b) Abstraction away from the bits, flops, shifts and pops
c) Quick sort
d) Electron.js
e) Ruby hipsters
f) Magical cloud computing
g) Software companies that know the cost of everything and the value of nothing
h) All of the above
All module folders should be at root level.
Need version 15 and version 16? Yep different folders under the root. Modulename_version, eg somelib_0.1.15 alongside somelib_0.1.16
It would allow clear identification of old versions and much less duplication. Less bloat. No way to have duplicate copies of the same library.
I feel like this is a feature, not a bug of software dev. The fact that you can push updates out so instantaneously allows you to work incredibly incrementally. You can make barely functional, inefficient things (shortcut hacks) just to make sure that your product is something that people actually use before focusing on optimization. If users are willing to put in the extra work to "refresh" the page, then certainly there's some real problem you are solving.
Gas car technology matured, gas got more expensive, and cars got more efficient. Moore's Law isn't going to go on forever. We are hitting limits in battery life. Software will tend to get more efficient over time.
> The demand upon a resource tends to expand to match the supply of the resource (If the price is zero).
Have cheap hardware, software will expand to use more of it.
Conway's Law
In fact, I'd go so far as to suggest there are more good programmers about today then there were in the past. Though having said that, percentage-wise I'd say the profession is definitely being deskilled (good thing if you're a manager, bad thing if you're a good programmer).
FTFY
> I hope I’m not alone at this. I hope there are people out there who want to do the same. I’d appreciate if we at least start talking about how absurdly bad our current situation in the software industry is. And then we maybe figure out how to get out.
Okay, I'll respond, especially on the last part "how to get out".
For the problems and struggles in the OP, I've seen not all of them but too many and sympathize. Mostly though, I don't have those problems, and the main reason is in the simple, old advice in the KISS Principle where KISS abbreviates Keep it Simple Silly although the last S does not always abbreviate silly.
In particular my startup is a Web site and seems to avoid all of the problems in the OP. Some details:
(1) Bloated Unreliable Infrastructure?
My Web site is based on Microsoft's .NET with ASP.NET for Web pages and ADO.NET for access to SQL Server (relational database). The version of .NET I'm using is some 4.x. So far I've seen essentially no significant revisions or bugs.
For the software for my Web pages I just used one of the languages that comes with .NET. I wanted to select between two, the .NET version of C# and the .NET version of Visual Basic. As far as I can tell, both languages are plenty good ways to get to the .NET classes and make use of Microsoft's managed code, e.g., garbage collection, and their CLR (common language run time) code. And IIRC there is a source code translator that will convert either language to the other one, a point which suggests that really the two language are deeply equivalent.
I've written some C code off and on for 20+ years; mostly I remember the remark in the Kernighan and Ritchie book on C that the language has an "idiosyncratic syntax" or some such -- I agree. I never could understand the full generality of a declaration of a function -- IIRC there is some code in the book that helps parsing such a statement. I do remember that
i = ++j+++++k++
is legal; I don't want any such code in my startup; also my old tests showed that two compilers gave different results.
I find Visual Basic to have a less "idiosyncratic syntax" and a more traditional syntax closer to the original Basic and then Fortran, Algol, PL/I, Pascal, etc. So, my 100,000 lines of typing are in the .NET version of Visual Basic (VB).
For my Web site, part of the code is for a server process for some of the applied math computing. The file of the VB source code is 478,396 bytes long (the source code is awash in comments), and the EXE version is 94,720 bytes long. As far as I can tell, the code loads and runs right away. Looks nicely small and fast and not bloated or slow to me.
(2) A bloated IDE (integrated development environment).
I have no problems at all with IDEs. The reason is simple: I don't use one.
Instead of an IDE, I typed all my code, all 100,000 lines, into my favorite text editor, KEdit. It has a macro language, KEXX, a version of REXX, and in that language I've typed about 200 little macros. Some of those macros let KEdit be plenty good enough for typing in software.
E.g., I have about 4000 Web pages of .NET documentation from Microsoft's MSDN site. Many of the comments in my VB source code refer to some one of those pages by having the tree name on my computer of the HTML file; then a simple command displays the Web page. When reading code and checking the relevant documentation, that little tool works fine.
After all, VB source code and HTML code are just simple text; so are my Web site log files, the KEXX code, Rexx language scripting code, all the documentation I write either in the code or in external files (just simple text or TeX language input), etc. So, a good general purpose text editor can do well. And, "Look, Ma: I get to use the same spell checker for all such text!" The spell checker? ASPELL with the TeX distribution I use. It's terrific, really smart, blindingly fast, runs in just a console window.
For KEdit, it seems to load and run right away. I just looked and saw that what appears to be the main EXE file KEDITW32.exe is
1,074,456
bytes long -- not so bloated.
(3) Windows 10 Home Edition Reliability.
For a move, I got an HP laptop; it came with Windows 10 Home Edition. I leave it running 24 x 7. It hasn't quit in months. It appears that now the Microsoft updates get applied without stopping the programs I usually have running, e.g., KEdit, the video player VLC, Firefox, etc.
Using carefully selected options for ROBOCOPY, I do full and incremental backups of my files. I keep the ROBOCOPY log output; that output shows the data rate of the backup, and I have not seen that to grow slower over time. The disk in that laptop is rotating, and I've never done a de-fragmentation. So, I can't complain about performance growing slower from bloat, disk fragmentation, etc.
(4) Windows 7 64 bit Professional Server.
For a first Web server, I plugged together a mid-tower case with an AMD FX-8350 processor, 64 bit addressing, 8 cores, 4.0 GHz standard clock speed and installed from a legal CD and authentication code Windows 7 64 bit Professional SP1. As I left that pair running 24 x 7, occasionally it would stop with a memory error of some kind. I installed an update and never again saw any reliability problem in months of 24 x 7 operation.
Since then I looked into Windows 7 64 bit updates and concluded that (i) there was a big roll-up of about 2016 or some such; (ii) since then there have been updates and fixes monthly and cumulative since the big roll-up, and (iii) the updates for Windows 7 64 bits and Windows Server 2008 are the same.
I can believe that Windows Server 2008 long ran and still runs some of the most important computing in the world. So if my Windows 7 64 bit Professional has the same updates as Windows Server 2008, maybe for use as a server my Windows 7 installation will be about the most reliable major operating system in computing so far. Fine with me.
So, I am not screaming bloody murder about operating system or software reliability.
(5) Smart Phone Bloat and Reliability.
I have no problems with smartphones if only because I have no smartphone and don't want one: When smartphones first came out, I saw the display as way too small and the keyboard as just absurd. Heck, in my desktop computing I have a quite good keyboard but would like a better one and, of course, would like a larger screen -- no way do I want to retreat to an absurd keyboard and a tiny screen.
Next I guessed that there would be problems in security, bloat, reliability, system management, documentation, and cost. Maybe history has confirmed some of these guesses!
For a recent move, I got a $15 cell phone and did use it a few times. Then I junked it.
My phone is just what I want -- a land line touch tone desk set with a phone message function from my provider. Works fine. So, I have a phone with lower cost and no problems with keyboard, screen, security, bloat, or documentation. Ah, old Bell Tel built some solid hardware!
(6) Web Site Speed, Reliability, and Bloat.
My Web site apparently has few or no problems with speed, ....
Why? The site is simple, just some very standard HTML code sent from now old and apparently rock solid ASP.NET.
Fast? The largest Web page sends for just 400,000 bits.
The HTML used is so old that it should look fine on any device anywhere in the world with a Web browser up to date as of, say, 10 years ago.
The key to all of this? The KISS Principle.
YMMV!
Hmmm. Let's think about this.
I piled standout quotes below.
I think a big takeaway from the intersection of Bret Victor, Alan Kay, Jim Hollan and the ink&switch folks and your work is that the right dynamic interface can be the "place we live in" on the computer.
Victor shows a history of interactive direct manipulation interfaces, live environments where explorations of models or the creation of art go hand in hand with everything else related to that task: data input, explicit (programmatic) requirements and the visual output.
Hollan and ink&switch show the environment (ZUIs, canvas) can contain everything for doing work, the code alongside any manipulation of the viewport that can be conceived. Tools infinitely more advanced than Microsoft OneNote and designed 40 years ago.
From what I know about your work, I see another take on the environment I want to live in on the computer. I dont understand why I would want to lose power by stepping away from my language/interpreter/compiler/repl into a GUI or some portal when I can bring whatever it is which is nice about GUIs or portals into my dynamic computing environment. I very much want a personal DSL or set of DSLs for what I do on the computer, and I want to be able to hook into anything ala middle mouse button in plan9.
The superior alternative to walled gardens and this absurd world of bloat and 'feature loss' (for lack of a better term for software engineering's enthusiastic rejection of history) seems to be known, and facets of it advocated by you and these others. It seems clear that "using the computer" needs to return to "programming the computer" and that to achieve that we need to fundamentally change "programming the computer" to be a more communicative activity, to foster a better relationship between the computer and the user.
Where is this work being done now? VPRI shut down 2 years ago, Dynamicland seems to be on hiatus? I am inspired most these days by indie developers who write their own tools and build wild looking knowledge engines or what they sometimes call "trackers."[1] And of course the histories and papers put forward by the above and their predecessors. And I play with my own, building an environment where I can write, draw, code, execute and interact with it all. I see no existing product which approaches what I want.
> Everyone is busy building stuff for right now, today, rarely for tomorrow.
> Even when efficient solutions have been known for ages, we still struggle with the same problems: package management, build systems, compilers, language design, IDEs.
> You need to occasionally throw stuff away and replace it with better stuff.
> Business won’t care Neither will users. They are only learned to expect what we can provide.
> There’s no competition either. Everybody is building the same slow, bloated, unreliable products.
> The only thing required is not building on top of a huge pile of crap that modern toolchain is.
> I want something to believe in, a worthy end goal, a future better than what we have today, and I want a community of engineers who share that vision.
I agree with the post, though as others have pointed out, it doesn't really dive into the fact that this problem is systemic and would require a shift in incentive structure.
I think the last quote you have is one of the most important missing pieces for making a meaningful change in this space. A lot of people want something better, but right now, as a community, I don't think we really know what that is. What is the complete story for an ideal version of software development? And by that I don't mean idealized examples, I mean the ideal version of the real process we have to go through. What does perfect look like in the world of changing requirements, shifting teams, legacy systems, crappy APIs, and insufficient budgets? If we could show that - not the simple examples we had for Eve, but something that addresses the raw reality of engineering - I think it would just be a matter of beating the drum.
I don't think there is anything mentioned in the article that is not realistic, including a 1 second boot time on a phone that you bring up. The only reason we can't get a phone with a static hardware configuration to boot in a second is because that's not what we've been optimizing phones for. LinuxBoot is a great example of how the time it takes to boot is purely about how much time we assign people to optimizing it not some laws of physics that require servers to take 7 minutes to boot.
All of this bloated 'shitware' today is the result of it having been written by people who a) have no deeper understanding of what the computer is actually doing; also known as typical Python/Java/etc/etc/etc programmers, and/or b) simply not giving a damn about conservation of resources--as further evidenced by all of the other extremely wasteful and destructive habits they hold in their personal lives, and in their societies in general.
After all, this is the same civilization that's burning through increasingly vast quantities of oil at an astounding rate, despite the fact that previously existing abundant and cheap oil is nearly depleted, with no possibility of replenishment or replacement. So is it any surprise that foolish developers also burn through CPU and memory with reckless abandon?
Really, the problems we face aren't just in software; they're more about the foundations of our entire Western 'civilization.' Such problems generally tend to be rather intractable, in the historical view.
I'm working to construct, in my own computing life, something of a 'personal oasis', which is increasingly removed and estranged from all of the horrible things I see Other People out there having to suffer in their personal computing lives, thanks to talentless 'developers' who Just Don't Fucking Care. Some of these pricks actually have the audacity to call themselves 'engineers', even.
My "oasis" is just text files in git, or photos in directories for the most part. The challenge is integrating with others and my god damn iPhone.
Much of the blame lies with bloated frameworks, such as Electron, rather than with the developer who uses the framework. You've covered this under Option B.
This topic cropped up recently on a Show HN. Someone built a Slack client for Windows 3.1. [0] It uses a tiny fraction of the memory used by the official Electron-based client, of course.
Not just software, see houses, furniture, consumer goods, etc.
I see no point in getting angry, you are in control of what you use.
That statement contradicts itself. Is this satire?
> All of this bloated 'shitware' today is the result of it having been written by people who a) have no deeper understanding of what the computer is actually doing; also known as typical Python/Java/etc/etc/etc programmers, and/or b) simply not giving a damn about conservation of resources--as further evidenced by all of the other extremely wasteful and destructive habits they hold in their personal lives, and in their societies in general.
Everything has a cost / benefit associated with it. Pretending otherwise show how little you know about engineering.
> I'm working to construct, in my own computing life, something of a 'personal oasis', which is increasingly removed and estranged from all of the horrible things I see Other People out there having to suffer in their personal computing lives, thanks to talentless 'developers' who Just Don't Fucking Care. Some of these pricks actually have the audacity to call themselves 'engineers', even.
I dunno how far your head has to be up your own behind to actually believe this? It not like most developers these days work on large software projects that are normally poorly costed, estimated, planned usually with stifling restrictions because certain "enterprise" technology is mandated by some architect who hasn't written a line of code in a couple of decades. Most people like a consistent regular income which allows them to support themselves and their family. I suspect these concerns are more important then incoherent ramblings of some guy on HN.
> they're more about the foundations of our entire Western 'civilization.'
> talentless 'developers'
It's not talentless developers, it's the foundations of civilization. Capitalism, chasing the $ not the writing of beautiful software. If an Electron app makes me more money (e.g. by getting to market quicker) than a well crafted native app, then the Electron app will be built. See also building construction and why we end up with 60m^2 poorly constructed but funky looking apartments - who to blame. The concrete pourers? The architect? I don't think it is that simple.
The twelve steps program may be controversial, but I think the first step, 'admitting you have a problem', is generally the first step of any productive approach to problem solving.
Effiency is a selling point that most users don't care much about in most markets. There are efficient browsers out there but everyone uses Chrome because those browsers are inferior to Chrome in many other ways. Ways that are more important to the average browser user.
If there's a market for a more efficient software solution, go make it and get rich. Otherwise, I'm getting sick of the complaining.
The least competent programmers are the once writing slow code. The least competent programmers are the ones working at the top of the stack.