Modern software performance is firmly in its “beyond parody” stage

For example, speaking completely independently from the problem here but putting on my performance hat, I would expect that if we take all things as a given (“no major architectural changes allowed”) Teams should not take 10 seconds to start up to show UI. We know that the tech stack they’re using isn’t fundamentally that bad. This suggests that there is probably a handful of things they are doing that are low-hanging fruit but haven’t been identified yet. Once they find those and bring down the time to something more reasonable like a second things will get much harder for them to fix.

It's not just that Teams is failing at solved problems, it's also that they aren't using Microsoft's own SDKs and APIs themselves anymore. Windows has good fast UI frameworks and all that cool stuff like SuperBoost which can make app startup appear instantly. Why is nobody using it?

And there is no useful integration to speak of. Why do I need to re-upload an OneDrive file into Teams? The exact same file is already stored on the exact same server. Does nobody there know about the magic of SHA512?

And how come if the internet has a hiccup while I upload a file, I have to re-upload the whole damn thing? In 1980 we already had resumable uploads via FTP.

Oh and who's the vendor creating one of the best performance tracking and optimization tools? Microsoft themselves. Did nobody ever dare to run this in a proper Visual Studio?

> it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.

When a company grows and add features that customers need, the data model has to grow with it. Queries necessarily become more complex.

I used to work at a big video sharing site. I wasn't there in the beginning, but I imagine the initial profile pages had code like

    SELECT * FROM user_videos WHERE user_id = ?

So then the simple query became

    SELECT * FROM user_videos WHERE user_id = ? AND privacy = ?

And then the query suddenly became much more convoluted. We started loading all of a users' videos in memory, then in-memory filtering, returning only the videos the viewer had access to (slow for users with thousands of videos).

Then some users came along with 30,000 videos and we had to abandon the in-memory filtering because we were getting OOMs, so we developed a hybrid system that could run queries either in-memory or in the DB depending on volume.

This is a trivial example (there are much more painful examples of data model bloat) but it illustrates the point: if you think it should be easy, you probably have an innacurate mental model of what needs to happen behind the scenes.

Users don't care because Teams still basically functions and the performance problems are at most a minor inconvenience. As a result, organizations don't care because Teams is cheap and does everything they need it to. As a result, Microsoft doesn't care because dedicating dev resources to something which neither saves them money nor attracts more customers is a waste.

Microsoft employs many smart people and I'm sure Teams' performance could be improved by orders of magnitude if they wanted to. But why would they want to?

(And to reinforce my point, when the performance _did_ get so bad that people apparently actually started caring, Microsoft not only improved it but made a video advertising it. It's just that there's no incentive to optimize any further than is necessary.)

When you work in orgs with hundreds or thousands of developers working at a breakneck pace churning out feature after feature for years, on a perpetual "sprint", your architecture tends to resemble a multi-layered bowl of spaghetti. A single button click may result in hundreds of database calls (maybe concurrent, maybe cached), or at a minimum invoke dozens of javascript packages in a Rube Goldberg-esque fashion to perform an action which should take a modern computer a single digit number of milliseconds, but instead results in 10 second load times. Sometimes people just completely forget about entire code paths, and then it becomes dangerous to delete since the persons(s) who wrote it either no longer work there, or got reorged and moved on.

There are absolutely people who care about software performance, but it isn't a priority because it usually doesn't make you more money, won't get you or your boss promoted (unless it saves money), and is difficult to maintain in the long-term given the pace of change in large codebases. When you do tackle performance problems, it's because you were absolutely forced to due to necessity. In certain domains it is also more important e.g. games, embedded systems, ML applications, etc.

One other thought, is that many developers today cannot build software from first principals, as they never learned how. Many only know how to build within the context of a browser environment, and think a "minimal" app is https://create-react-app.dev/ . That's not just me grumbling, it's how our industry works. You forget the layers below you, and build new ones, each generation forgetting most of what the prior knew. The result is giant bloated applications, and a normalization of slowness in software.

I still have hope for small teams of dedicated devs. Fast, coherent software gets written by tiny teams or single individuals.

Probably a few thousand features you have never heard of and will never see or use. And the vast majority of their customers will never see or use. But each one has one or two clients for whom someone decided it was vital at the time to implement that feature to please that client.

You can probably imagine one small feature that only one or two clients use that adds a very tiny, almost imperceptible amount of loading time. Now just multiply that by several thousand.

Sounds like they just rewrote the whole thing from scratch, down to replacing Electron.

Making software is not challenging. A CS graduate from 2002 could build Teams. And it would be faster! You can hire dozens of engineers based on whiteboard problems, throw them into a disjointed cacophonous open-plan work environment, give them arbitrary deadlines and 10 layers of middle management, and somehow, somehow, software will still be built!

There's no reason to streamline the architecture, or the workflow of the team. If the software is going to be made regardless, why bother? Just do whatever feels normal.

Contrast this with the engineers who built the Saturn V. I'm sure a lot of brainpower went into thinking about the overall architecture of the rocket, and how the teams would be structured, and how they would communicate. Tons of process.

Anyway, as a result of this lack of organization, software teams usually self-structure into a network of arbitrary mini-siloes. The way this works at FAANGs is you'll have the Foobar Core team, the Foobar Presence team, the Foobar Video team, the Foobar Audio team, the Foobar Notifications team, etc. etc. It's like microservices for teams. And they barely communicate (unless two of these teams are lucky enough to share the same corner of the same open-plan floorspace!)

Anyway, these mini-siloes ensure that there are lots of seams between modules, and no one can optimize across boundaries easily, without stepping on the toes of someone else's team. And unless customers are actually switching to other products, there's no political willpower to do that toe-stepping in the name of saving the product.

Maybe modern software is like that. Just a hundred small inefficiencies that don't look like much when benchmarked on an individual part of the stack but combine into a monster.

...I can't write CSS and nobody moans so why should web devs be expected to engineer software?

Architecture suffers, code health, and performance suffers. Silly things are done, but it's hard to see they're silly because it's hard to see the entire picture

Could you build it better from scratch rather then organically? Of course. And often that's part of the solution to these situations.

tl;dr Building software is hard, and its difficulty does not scale linearly with complexity. (And yes, Teams/Slack are complex, even if they "shouldn't be")

One way to test this...MSFT could open the Teams API and some 16 year old could try building a client.

Even React isn't all that slow. If you look at benchmarks you may be impressed by how quickly it can do some operations. It's consistently a bit slower than other frameworks, but not that far off. There are (big) React apps that are quite fast, too.

So writing performant applications with these technologies isn't impossible.

Yet a lot of apps built with them are slow. The fast ones are rare.

I don't know what it is yet (happy to be pointed somewhere), but I'm fairly convinced that these technologies have common footguns that encourage programming things in slow ways. I can imagine it being related to React's coarse-grained and easy mismanage reactivity and wonder how much of an improvement Solid is in comparison (it is also functional, but is built to always recompute as little as possible rather than saying "whatever, it's cheap", which React does).

Correct me if I'm wrong, but on a 3ghz CPU (3bln cycles per second), there are 27 billion cycles per character for 1kb of text. Take into account that on any non-stone-aged CPU you've got probably 4-16 cores, and can do ~4-8 SIMD ops per cycle (at full clock rate), so at the limit we're actually talking .. something like _3.4 TOPS PER BYTE_ ..

Sounds about right and, actually, is quite a feat.

I just wonder how they went from "build a chat app with some filesharing built in" to "include everything and the kitchen sink and oh, in your own universe" kinda thing...

There's all sorts of room for optimization. Those familiar with 3D programming probably know the "use distance², not distance" to avoid needing to sqrt(), and similar tricks apply (you don't need to work in miles: e.g., you can convert the target distance to arc distance, and use that, which is what haversine speaks natively). We were also trying to get Go to inline some of the wrapper calls (so that we could work in more sane units, like lat/lng, and have a wrapper convert to, IIRC, radians) … but alas, Go only inline leaf functions, and the trig calls haversine requires basically meant we were never a leaf.

Also hit the uniquely human problem of: if you compute the query, you can spend infinitely long doing it: the right circle is fine if centered over Wyoming, but quite a problem if centered between NYC & Philly. (So as to overlay both.) If you don't detect the case, you'll time out, and users are unhappy. If you detect it and abort, users don't get data … and are unhappy. There's no way to win.

… then the "higher level" stuff knocks you down three pegs for trying to eek a few milliseconds out of haversine when you find out the protobuf library encodes the response slower than JSON, despite being binary. (But protobuf was written in Python, vs. the JSON library is in C. There's a C protobuf lib … but its own unittest segfaulted at the time. I hope it has been since fixed, but ugh.)

Also looked at our point type in Python. As you can imagine we alloc'd a few million of those, and the simple definition in Python of,

  class Point:
    __slots__ = ('x', 'y')

    def __init__(self, x, y):
      self.x = x
      self y = y

(And then we'd done some rather dumb things … which I see repeated a lot, not just at this job, like SQL databases with an ENUM column declared as VARCHAR, and the same string repeated forever and ever through the DB. All you needed was a SMALLINT, an ENUM if its supported…

And so much CSV.)

Is it? I’ve been having an inner rant about how modern web ignores the back button.

Reddit and YouTube completely breaks the back button. Click a video, click back and a completely new list of videos are shown. Same for Reddit.

On top of that almost every news site inject their front page behind the article you clicked so back keeps you on their property.

Clicking back on Facebook gives you a new feed.

Cynicism about modern software aside, I’m glad to see Teams, an app that many of us are forced to use by our employers, improve performance. It was desperately needed.

I don’t think this type of performance is some kind of horrid modern software epidemic, it’s just Teams being a garbage application that was thrown together to try and answer to Slack.

The real problem is just the general practice of how software is developed now. If we lived in a world where software development was sane, it would be possible to prioritize speed of development and cut some corners to ship faster, and get a result that is somewhat slower but not ten whole seconds to start up slow.

Sometimes good old tools beat the new shiny turd.

I had to do this once for a large piece of JSON that ballooned quite large when loaded into RAM. But in my case I could wait until after json.loads to do the interning; but I think you can do it on the fly with object_hook or object_pairs_hook.

Alternatively, Rust+Serde & doing as bare-bones a computation to get it working, again with an eye out to whether the decoded form will fit. (Hopefully, since Serde'ing into a struct should make the keys 0 bytes, essentially, but there would be some memory lost for pointers and the like.)

Alternatively, a 20 GiB swap file.

Naturally I was curious what such a shell script consisted of, but most tools choked on it. IIRC, even vim and less couldn't open it, but emacs could.

I keep a couple old laptops kicking around for perf testing on and, can confirm, I routinely find perf hiccups that when added together would create a huge amount of debugging work.

While I like doing dev with a threadripper & beefy GPU, I certainly support the idea of having crappy, old test machines around.