The 0.5 MB of nothing in all Apple Music files (2020)
ctrl.blog
ctrl.blog
When I was ripping my CD collection, I religiously tagged the MP3 files with ID3v1. Later on, I had some tracks whose titles were just a tad longer than 30 chars so I had to use ID3v2 for those and I noticed the file size grew by _a lot_.
Frustrated by this, I opened them in a hex editor and learnt that ID3v1 was a fixed format of 128 bytes, but v2 was variable. I also found out that the software had added a 4KB zero-byte padding to the v2 tag, which was "necessary" because the tag is now at the front of the file, and this padding allows more tag data to be added easily later on.
I tried various ID3 tagging software at that time and all of them added a padding. So I learned about the tag format and wrote a tagger myself that didn't add any padding. It was a great learning experience, and I managed to shave those useless zero bytes from my MP3s.
Must have been pretty fun tracking this down, but I'm curious as to why you'd go through all this trouble. Storage is cheap, my time is not haha.
And also, I love to optimize things, so it made no sense to me why squeezing 1-2 more words in the track title would inflate the file size by 4KB. As a teenager, I probably had more time on my hands back then.
Ain't this the reason why most of us ended up here :)
Think about it, if transparency guarantees applied for lossy transcodes, you could encode mp3 -> mp3 thousands for times with no generational loss. This clearly isn't possible.
Of course, you could rip all your files again in a better codec, but the time required to do that is pretty intense.
That's not really a counterexample for the idea that one pass of lossy -> lossy reencoding doesn't necessarily mean certain doom for audio quality.
Having a problem organically emerge in front of us that affects us personally is just a really easy thread to pull on for learning things.
Always has been..
ParseError? You recognise fun but not that it might be a reason to do something?
OP answered, and he was a teen back then, which may have played in the "too much time on my hands" part.
> PADDING: This block allows for an arbitrary amount of padding. The contents of a PADDING block have no meaning. This block is useful when it is known that metadata will be edited after encoding; the user can instruct the encoder to reserve a PADDING block of sufficient size so that when metadata is added, it will simply overwrite the padding (which is relatively quick) instead of having to insert it into the right place in the existing file (which would normally require rewriting the entire file).
> which can easily exceed 0.5MB
If that happens then you were better off never reserving space at all.
It's absurd to spend the same 0.5MB or more on the same piece of art over and over on every tiny lossy-encoded track. Even with FLAC it's pretty questionable. Better to include a tiny 100x100 jpg or nothing at all.
I actually recently encountered a bug with a music encoding tool where it failed to resize my album art when creating Opus files. I ended up with an Opus-encoded album that was bigger than the FLAC version because every single song had a 30 MB high res scan of the vinyl album art embedded in it.
Doesn't it? Unless you're embedding print-quality artwork or more than just the front cover, I would have thought that 0.5 MB was plenty…
Then again I question who in the first place is keeping very large libraries of apple music files......
My wife and I sync music from the same Apple Music library.
On her iPhone, the album artwork is completely random.
On my iPhone, it's always correct.
No amount of wiping the phone, or even upgrading to new phones, will change this. Apple Music just hates her.
I found the same problem with my phone a long time ago for a non US english account. Once i had that configured differently than what my setting was on the app store it would do all kinds of stupid things with the artwork.
That might be confusing, though. Too bad they don't have a way to show a directory as a single "package" in the Finder. I can imagine that would be useful for a lot of things.
But with downloadad music with all the metadata (hopefully) correctly set, even 5kB is a waste of space.
Sometimes the album or track names have extra info that I don't care about and will delete.
And TV show metadata is usually a mess and needs quite a bit of editing.
I'm not talking about less-known artists too, two examples of this in my collection are Steven Wilson and Linkin Park.
Oh BTW talking about less-known artists, I've even seen the whole song being incorrect (duplicate of another song on same album) on Apple Music after switching part of my old MP3 collection to Apple Music native.
They are probably bulk importing with little human in the loop, which I get as there are millions of songs to import.
I don’t expect normal people to do this, but for me the ability to edit metadata and mix streaming songs with ripped songs is a huge advantage of Apple Music over other streaming services. Unfortunately Apple Music has a lot of other issues, especially with syncing, and I still can’t understand why there’s no dedicated “featuring” metadata field (everyone pollutes the song title: Spotify solved this), but for me there’s no other possible alternative.
So it's cheaper than buying the albums individually, but then you have to spend some time unpicking them again and restoring the original albums. At least the track/disc numbering is still correct, so that helps a little.
Judging from some reviews left on some box sets, this wasn't always the case, i.e originally even box sets had the proper album tagging…
I have to change the genre on all of the Christmas songs my wife buys from Apple Music from "Holiday" to "Christmas" because there are holidays other than Christmas.
Otherwise, a "Holiday" smart playlist transitions from Silent Night to Monster Mash.
The article said that the default is 5KB but Apple uses 500KB, hence the poster is saying that even 5KB is overkill.
> If you rip a CD with Apple Music using the Apple AAC Encoder (the default option), it will reserve approximately 5 kB of free space inside each file for this purpose.
I was saying that even this is too much, and 500kB is 'way way too much'.
Split the file after the metadata block; change the data inside the metadata block (possibly changing its size); add the difference between the old and new size to all the offsets in it; recat the two pieces together.
You could even do this "streamingly", to inject album art etc., since you know what size the added content will be. Simply add the diff-offsets to the right fields as they get streamed out.
The above solutions didn't take much thought to come up with. I wonder why no one at Apple thought of that (especially the latter)?
The quintessential HN comment!
Or, if the article's speculation is correct, Apple designed their formats assuming only 5kb of empty space would be allocated, which was insignificant even for the 1st Gen iPod's 5 GB hard drive. No reason to do more complicated stuff if a simple solution works. It only becomes a problem when people don't bother to keep track of how much empty space is being added automatically.
Even the argument of "Well, they need to download the whole file before they can see the metadata" seems really weak. If someone is streaming the music, why not stream the metadata separately from the data? Why do those two things HAVE to be together?
Heck, why not have a metadata "db" (ala plex) instead of insisting that stuff be bundled on the media itself?
Just seems like a bad solution all around.
Because that isn't portable and then a different faction will be screaming "vendor lock-in". The metadata inside the media file itself is in a standard format that anybody can read, and storing it inside the file also ensures that it cannot become separated from the file it's belonging to.
Besides, any reasonable kind of software involving any kind of media library will already be keeping its own metadata database (and that includes iTunes/Apple Music), because rescanning the whole library on each program startup would be ridiculously inefficient (all the more so historically when hard disks were more common). But it can't be the sole source of truth because see above – people who still keep a local library of music around might want their media file collection to be interoperable with other software, too.
On the very rare occasion you adjust the artists name in a music file, the user expects the whole file will be rewritten. So just rewrite the whole file.
Whats next? Photoshop only writing out a quarter of an image when I edit my ex out of a nice photo? MS Word only touching a few bytes of a 100 page document when I make a heading bold?
> In 2022, we shouldn't be reserving any area of a file as 'spare space' like this.
I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again.
Alternatively, you could just switch to a text-based format like rtf, as long as you don’t need any specific features from odt or docx.
(EDIT: someone else mentions a promising sounding “flat xml” format, which I’ve never encountered before.)
I have a current project where one of the other developers has tried this. It might work for a solo project, but with multiple developers on Windows (Excel) and Linux (Open Office), I find that Open Office feels a need to continually fuck with unrelated fields. Editing a single field in an xslx file results in changes in a dozen files in the unpacked version. Making any more complicated changes results in a diff that's impossible to manage.
Still, interesting to hear some feedback on how well (or not) that idea has worked for some.
The Flat XML option seems to be the best approach if you need to edit documents more complex than what can be represented in either markdown or rtf (rich text format).
$ ls -hl todo.fodt
-rw-rw-r-- 1 dotancohen dotancohen 150K Apr 7 16:41 todo.fodt
$ ls -hl todo.odt
-rwxrwxr-x 1 dotancohen dotancohen 117K Apr 6 12:01 todo.odt
$ wc -l todo.fodt
1983 todo.fodt
$ cp todo.fodt todo-edit-one-line.fodt
$ libreoffice todo-edit-one-line.fodt
$ diff todo.fodt todo-2.fodt | grep "+ " | wc -l
19However SQLite stores multiple files on disk for crash persistence, complicating matters. But then again so does Audacity 2, and Microsoft Office to an extent (file locking rather than data integrity).
Microsoft Word used to only append changes to the document on save rather than rewriting the whole file, but this was disabled since if you deleted text and saved the file, it remained in the document. https://www.cnet.com/culture/microsoft-disabling-word-2003s-...
Those 0.5 MB presumably also need to be written to disk, making this a trading away of 392 TB in disk writes in exchange for 17 PB over the wire + 17 PB in disk writes? I'm sure this is overly simple and overlooks a lot of technical details, but I can't see how this can be worth it.
[1]: https://www.businessofapps.com/data/apple-music-statistics/
I completely understand your point, but maybe it would be a breaking change.
I have a rising suspicion that some game companies with fluid ethics do this - avoiding to optimize and strip unused content, in order to inflate game size.
"Wow, it's 102 GB, the game must be huge with boatloads of content!" (ie huge == higher chance of I'm getting my moneys worth). Which in turn becomes a problem with the latest generation consoles that have quite little drive space (Playstation ca 625 GB).
Occam's razor says that tooling, or developers, just aren't great at shaking unreferenced content from production builds.
I'm not a game developer, but its common to still ship unused code or css in websites, just because someone forgets to remove it, or its hard to tell if it's still used. I could only imagine this is more of a problem in more complicated game dev, especially when time to focus on non-functional requirements like that can be hard to come by.
The Cutting Room Floor[0] is an empirical proof of this assertion.
https://www.pcgamer.com/call-of-duty-warzone-cant-have-more-...
At the same time I saw some comments, reviews about my apps, that they are probably would not be good just because the app sizes are too small.
As much as people like to hate on consoles gamers, it's predominantly console "TRCs" which put limits on the size of games, that's the only time the company really cares about the sizes of games.
But even if you do notice that and file bugs what team in their right mind would spend time trying to reduce their own component's size by 2% when they could spend that time on features or bug fixing instead? And if you asked the majority of their users would agree - "F'k the 20MB, fix bug XYZ/finally implement feature ABC".
Despite each individual decision along the way being arguably the correct decision the product still adds up to +30% every year.
Usually there's some ultra low-hanging fruit that everyone knows about, and if download size has become an issue, a bit of time should absolutely be allocated to basic housekeeping tasks as long as the product is supported.
> 625 GB
When did I get so old?
Often the developers also forget to remove unused files and cut content, but I don't think this has a large effect on the game size.
Does anyone actually think this way? SNES games were some megabytes and often have more content than I can get through (I still haven't beaten the SNES zelda... maybe I should pick it back up.) Once you're in the Gigabyte range anything beyond is likely waste.
DRM iTunes Store files were 128kbs, whilst non-DRM ones are 256kbs, as beyond removing DRM one of the selling points of what was called iTunes Plus at the time was better quality. So it's possibly back in circa 2009 someone made a fuckup with the encoding settings and no-one has noticed since. Or its some extremely weird conspiracy to sell larger iPods.
Or perhaps they wanted to make the filesizes just a bit bigger so that iTunes Plus files felt like quality, like the way expensive products have metal weights added just to make them feel heavier and more solid.[1]
[1] For avoidance of doubt, this last suggestion may be what is known as "humour".
This became a thing when distributing QuickTime Movies on the internet became a thing (it's not an issue with random-access media), because one needed Movie metadata at the front of the file in order to support progressive playback. Because the MPEG-4 file format is effectively the QuickTime Movie file format, the need to put metadata at the front of the file continues if you want viewers/listeners to be able to play files as they're downloading.
That Apple isn't performing this extremely trivial pre-distribution process is extremely curious. It makes me wonder if this "common knowledge" was lost along the way, or if the people who would know this kind of super-obvious production step just aren't the same people in charge of Apple Music standards.
For anyone curious about what fast-start implementations look like:
https://github.com/FFmpeg/FFmpeg/blob/master/tools/qt-fastst...
The author used this flag in FFMPEG because.. well, despite the main goal is to shrink the padding, you still want to have metadata at the beginning.
You're correct in the sense that having metadata at the start of the file means that these files are ready for progressive download/playback, albeit with a chunk of unnecessary data transfer.
But the other important part of the "faststart" process is that you get what used to be called a "flattened" file, where the data is contiguous — metadata is followed immediately by compressed media data. (Tools for QuickTime Movies also compressed the 'moov' header, but I'm not sure if that's supported in MPEG-4.)
The author don't actually need it to be"contiguous"; it's just that FFMPEG stream copy is the easiest way to so.
In this sense, Apple don't need to "flatten" their files, they just need to have smaller padding just as what iTunes is already doing when ripping user-provided CDs.
You're talking about the space reserved to allow for in-place metadata updates on songs that you've ripped. As you've noted, it's not much and doesn't "need" to be smaller.
I and the TFA are talking about music files purchased from and delivered by the Apple Music Store, which include 500KB of wasted space. The solution the article suggests (and that I added a bit more background to in my post) is to do what any production workflow should do before distributing MPEG-4/Movie files — fast-start those suckers.
0.5 MB is not much, but multiplied by the billions of tracks Apple sells and streams in a year, pretty soon we're talking about a significant amount of wasted storage and transfer.
> In this sense, Apple don't need to "flatten" their files, they just need to have smaller padding just as what iTunes is already doing when ripping user-provided CDs.
What's the benefit to shipping any empty KBs with every song purchased or streamed?
For (quicker) in-place metadata updates? You just said it. You can do so with the purchased music in iTunes.
Not sure about the streaming services, but the article isn't about it at all (nor did it talk about if they actually have these 500 KB padding to begin with).
I just tested this, and it takes about ~140ms for ffmpeg to faststart/remux one song-length .m4a file (M1 Air, internal storage).
All iTunes (Apple Music) content is served via CDN. By leaving some empty space in the file the edge server can just write the owner's account ID fingerprint without having to remux the whole file. Apple can just send a virgin copy of a song to the edge servers and they can do the customization inline with serving it. That will be computationally cheaper than remuxing it and require no extra storage since the file is tweaked in flight. Leaving a lot of space provides enough room for some worst case size of watermark data without producing invalid files.
But that's only my guess. I haven't looked at any files in question.
Most of time wasted on remuxing the file comes from re-writing the whole thing on your hard disk, which is more noticable if you save your songs on HDD.
Is it really that bad? No. But it's nice to have some padding.
Same goes to MP3s and especially FLACs (which is even more noticeable because file size is much larger). Some of the digital distributors for hi-res (hi resolution) music do use "flatten" files, and it's quite annoying to edit them.
According to who?
Do any media players care if there's a couple kilobytes empty there?
Funny side note: I recently made it's frontend crash. Turns out it _is_ a JS frontend after all. I suspected as much because of how slow & slugish it is, but I got an actual JS error & it fell back to the old table interface
Surely you are talking about different clients than I have on i/macOS? With Apple Music running on my MacBook, I couldn’t even control the player on the respective iOS app from the couch. It was insultingly trashy UX, and I’m deep, deep into apples ecosystem. My reaction was “what a joke, guess I have to use Spotify”.
I switched to Apple Music as well. It's not perfect but at least it's not trying to shove Podcasts into my music.
Spotify also likes to complain about Apple being anti-competitive by locking them out of features, then when Apple does let 3rd party apps do things like Apple Watch offline background music playback, Spotify takes two and a half years to actually use it.
https://appleinsider.com/articles/19/01/05/pandora-releases-...
https://newsroom.spotify.com/2021-05-21/enjoy-more-ways-than...
Edit to add: AirPlay 2 from iOS 11.4 (May 2018) still not supported, so you can't do multi-room audio, only a single speaker at a time
Entirely Apples fault, by the time they enabled the possibility of the feature it had become apparent that the Watch as a platform wasn't really worth focusing on so it's on the backburner now.
If that feature was possible day one, it would have happened. But instead Apple has this attitude of locking things out from 3rd parties and only moving when it starts to make their products look bad.
Isn’t the entire concept of the device about fitness users? Seen as it failed to carve out any other niche.
But I sorta wonder if their main motivation was that it was a bad look to continue complaining about Apple not letting them make a watch app on their "Apple is cheating" timeline, while also choosing to not make a watch app. If they actually cared about doing it, why take so long?
Apple doesn’t twiddle with its UI nearly as much as Spotify does which is nice too. It has its shortcomings, but the consistency means that workarounds stick.
Finally, Apple is surprisingly more permissive with Apple Music’s SDK than Spotify is, even allowing full streaming/playback capabilities (a capability Spotify removed from its SDKs several years ago), which means that there are now several alternative front ends for Apple Music available that can play music themselves — alternative Spotify front ends can only ever control the official Spotify app and Spotify Connect devices which is a real shame.
I haven’t used Spotify for a long time since switching from Spotify Premium to Apple Music, but I know of a third party open source library that might be of interest to people who have Spotify Premium and would like to explore alternative frontends:
> librespot is an open source client library for Spotify. It enables applications to use Spotify's service to control and play music via various backends, and to act as a Spotify Connect receiver.
https://github.com/librespot-org/librespot
More details from same GitHub page:
> A sample program implementing a headless Spotify Connect receiver is provided. Once you've built librespot, run it using :
> target/release/librespot --name DEVICENAME
> The above is a minimal example. Here is a more fully fledged one:
> target/release/librespot -n "Librespot" -b 320 -c ./cache --enable-volume-normalisation --initial-volume 75 --device-type avr
> The above command will create a receiver named Librespot, with bitrate set to 320 kbps, initial volume at 75%, with volume normalisation enabled, and the device displayed in the app as an Audio/Video Receiver. A folder named cache will be created/used in the current directory, and be used to cache audio data and credentials.
Then, I love the Essentials playlists, basically, if you search for any large enough artist they'll have a curated playlist always called Essentials which makes it easy to find. The curation is usually spot on although there's been one or two that felt like misses out of maybe fifteen or twenty that I've tried.
Finally, the Apple premium plan for 30 bucks a month, five members, 1TB online storage, Apple Music and TV (couldn't really care less about sharing my workouts or Arcade, but those are there too) is a pretty sweet deal for my family and I.
Dunno if Spotify or Tidal have things similar. And there is some funkiness around sharing occasionally as I'm not sure if my dad ever got access to Music, his account may have been in a weird state since he already had a subscription, unsure, will ask later.
My dad likes that for his own music that Apple doesn't have, if he imports it to his Music app on his computer, Apple will auto upload to the cloud so he can stream it from other devices. I haven't used this much but it sounds pretty neat.
[0]: https://apps.apple.com/us/app/itunes-remote/id284417350
(That, and all my friends use spotify, too, so sharing/group sessions etc. work out of the box)
And this definitely isn't the case here. If you run out of space it usually means an entirely new device.
So when did this get added?
One is that it stores a "play count", the number of times you've played a song, directly in that song file. So it's always re-writing the source file of the track in the library.
That would be only mildly irritating -- radically blowing up the size of system backups, for instance -- but for the fact that it routinely corrupts the new song file stream.
I see so much bit rot in my iTunes audio files -- and nowhere else -- that the stupidity of not simply keeping some small database of play count veers over into uselessness.
-
The second complaint is that they decided that any file I had ripped from my CDs were pirated, and therefore would require an annual fee for me to keep. I understand that's a concession they agreed to from music industry in order to roll out their "iTunes Cloud" stuff.
But yes, I bought more than 400 CDs new and yes, I still have all of them, and indeed, I never loaned them out to anyone else (no one asked).
-
After a while, I just set my iTunes audio files to read-only, and a to-do item from ten years ago is to go through and re-encode my physical CDs. Still haven't done that, yet.
Stopped using iTunes, instead. Still need to rip them discs...
It writes ratings to the file, but not play count. I don't believe any player does that.
Ratings (5 stars!!), not play count.
Thanks!
(And a cursory search does indeed bring up various people searching for workarounds to that in order to transfer the rating from iTunes to some different software…)
I (from Denmark) find it "disturbing" to see `.´ as the decimal point, AM/PM for time, as well as writing the DAY before the MONTH (MM/DD-YYYY).
https://www.ctrl.blog/about/ "Ctrl blog (“Control blog”) is the developer-focused blog of Daniel Aleksandersen based in Oslo, Norway."
Isn't this a really, really infrequent operation? And aren't hard drives fast enough these days that reassembling the entire file doesn't even take that long?
And local music apps already build their own library so they don't have to scan all files for metadata, I don't see how the location in the file changes anything.
Besides that’s not the only place they eat storage. Actually it’s so opaque you don’t really know what’s “really” happening.
I'm still using the charger I got with my 2008 (pre-unibody) MacBook Pro for my 2014 MacBook Pro, which is perfectly operational in 2021.
I'm using a Spigen 48W USB-C + USB-A charger (and a 60W, 2m Baseus USB-C to USB-C cable) to charge my Macbook Air M1 2020.
I'm using a Belkin wireless charging mat for iPhone X.
So, no.
> earphones (ensuring its wire gets dirty faster than any other)
Making rugged plastics requires a lot of nasty chemicals. Either be nasty and durable, or be less nasty and be less durable.
Also none of my Apple devices' cables have frayed (incl. a 30 pin connector cable for my now lost iPod Nano 2G). How come?
> extra storage
I think their 200G plan has good bang for the buck, and even I can't fill it up with normal usage.
> Actually it’s so opaque you don’t really know what’s “really” happening.
System preferences -> iCloud provides pretty good rundown of which application is saving what, including the ones you can't see in your drives (i.e. game saves, application stuff, whatnot).
So while Apple is not the best company out there, it's not that worse either.
P.S.: Repairability is a problem. I have nothing to say about that, but at least battery change program is very good. They've even found and changed some damaged parts in my phone, for free, when I sent it in just for a battery change.
> Either be nasty and durable, or be less nasty and be less durable.
In my experience apple cables are both nasty and less durable than most other cables I own. I have multiple old 30 pin cables that are frayed at the connector, my old ipod headphones had electrical tape on the cable where the rubber was breaking, and all my apple cables are a gross yellow color, while similarly old non-apple cables I own are all still perfectly white.
> I think their 200G plan has good bang for the buck
It's average at best. It's the same price as google for monthly payments, but google is cheaper if you pay yearly.
I'd argue, though, in my experience working at a large tech company, the impact of a particular decision on the company's overall bottom line rarely comes into play for a decision like this. Instead, it's a mid-level manager who has been given a goal to increase a particular number, and will be rewarded based on quarterly performance against said arbitrary number.
Which manager put in their OKRs for the year to grow usage of Apple's Music Library using a primary metric of total storage consumed? Now, they didn't necessarily cause the insertion of blank space into the files, but they'll be damned if they let some engineer make a change that makes it significantly harder to hit their growth goal for the year.
Organizations don't cause things to be the way what they are: the dynamics between people inside of organizations cause things to be the way that they are.
"When you do things right, people won't be sure you've done anything at all."
Personally, I think it is likely the padding was created for future use. Maybe tagging or some other meta data.
Y'know, iPod (and later iPhone) sync, music browsing and purchasing, (payment handling), intelligent media organization (or some approximation thereof), working with giant media libraries...
I get the idea that everyone just feature-piled on top of the SoundJam code without fully scaling the design to the point of being able to coherently load-bear the additional functionality. So basically failure to adequately PM where it counts. That requires PM that understands engineering, which is sadly a tall order out of the gate and generally the most crucial before projects get big and the need for support becomes obvious.
*IFF* that happened in this case (I have absolutely no concrete idea), my question would be a) where "Steve really wants this - put a good PM on it" and/or b) where "this code explosion is killing everyone, we need PM help", ended up.
iTunes history anybody? (Genuinely curious, wondered for a while)
iTunes had to move fast. There was not time, from what I understand, to step back and refactor it in any meaningful way.
I'd love to see those discussions - "The OKR was to sell more iCloud storage so the Music team added half a meg of junk to every song file"
"Great!"
... I mean... weird shit has happened, but... I dunno, I've worked at quite a few companies and I just don't get the sense that that's Apple's MO. The bad publicity if it became public that they actually did this on purpose would be terrible, it seems pretty conspiratorial in this thread.
Also, Music is a real shit, buggy app that constantly annoys me.
Apple is unique in that their product marketing has a big role in the product design process. If you step back and look at what they do objectively, there are a few journey mapped “happy paths” that Apple designs to. They actively remove features that distract from the true paths.
It’s noticeable with them as Apple has no sympathy for legacy. But you can see it with other companies. An easy one is Google Drive/Docs - they make it easy to find a file, but didn’t consider that you may want to see the folder context of where the file is.
They wrecked the UI completely. Which makes sense - I imagine that the program management of iTunes Match is probably a low priority. Basically, i use search history as a sort of playlist.
I could see it playing out like this, incompetence causes the problem, incompetence fails to identify the problem, incompetence fails to fix the problem.
I think we are on an unproductive tangent, but these are quite important questions. An institution doesn't have its own free will, so I don't think I can claim it can have "intent to cause harm", but it can certainly cause harm.
An institution is not just a collection of people, it is also the business processes and policies. While these are created by people within the institution, they stick around over time. As the world changes around the organization, well intentioned policy can begin to harm.
Individual employees following policy and normal business practice (i.e. just doing their jobs) can behave maliciously without their own intent to do so (sorry, my hands are tied).
An institution needs not only non-malicious members, it needs to constantly re-evaluate its practices. Importantly, when harmful outcomes are observed the institution must re-evaluate and adjust, otherwise the harm will continue.
It's not enough for members to conduct their own behavior without malice, they must also observe outcomes and actively fight against the inertia of existing policy.
This absolutely seems like the most reasonable explanation.