Someone should calculate how many petabytes of Internet transfer capacity are wasted annually solely due to bad algorithms used in performing software updates. It'd be a surprisingly large figure.
Firefox has historically shied away from complicated patch algorithms for this reason among others. (To be fair, of course, we're also dealing with several orders of magnitude less data than AAA games are...)
Patching small to moderate sized resources transactional ly might be a solved problem, but I imagine doing so on large resources in an acceptable time frame may not be for this use case.
That's where bundling all your data files into a single large archive bites you in the butt. Individual files the game uses ain't gonna be that large.
Perhaps they could break stuff apart prior to shipping, but I imagine that could cause a lot of QA headaches as sure it should all work perfectly fine if a solution is used that transparently supports both combined and split formats, but who wants to bet the ship date of a multi-hundred million dollar project on that?
I sometimes wonder if games were distributed on cartridge again if they would be more thoroughly tested before being released.
Plus, they'd load a brazillion times faster.
This includes a game I work on, because it's built in a version of Unreal where we do not want to drastically change the packfile format and modify the build process to handle this case.
Bethesda is really bad at tech. They use a ancient engine that binds physics calculation to frames and can't go over 60 FPS. They warned Fallout 76 fans during the Beta that they should not click the "Update" button (big download, game screwed). They trust the game client 100%. You can hack/cheat in this multiplayer game by changing INI files with a text editor.
If the file is smaller than 10x the base patch unit size: don't bother just add the whole file as a single unit / send it all for each update.
Else: Create a manifest of the file: Size on Disk, Checksum of file, Name of file mapping to (one or more) list of segments within file by segment offset, segment size, segment checksum. A custom list might be provided by a resource packer that is aware of assets / code segments within the file.
A website will provide an interface that has a list of file names to target size/hashes.
There will also be a list by size/hash that enumerates the segment lists within the file.
To update, the file size/checksum would be computed and the manifests obtained from the server; these would be compared to the files on disk and invalid segments/files discarded.
The target version would then be selected and the target manifests collected. First limited by filename and then optionally project wide, segments would be matched first checking size then checksum. There are a couple different methods that might then be optimal depending on the desired outcome, which depend on disk space, operating system support for hard-linking/de-duplicating subsegments of a file, if the media is known to be an SSD or spinning rust, etc.
However, in any event, that infrastructure allows for easily checking and upgrading / repairing to any version from any version at the expense of storing a small quantity of metadata and having full copies via the actual delivery service for segments on file. The backend might also have de-duplication of the stored data as a bonus.
This is probably a very naive implementation of compression via de-duplication and should be obvious to anyone skilled in this form of art.
It would be a rather large figure, but I'd wager that superfluous video streaming (eg, video advertising, or youtube/netflix autoplaying to no audience) overwhelmingly dwarfs any bandwidth wasted on suboptimal update delivery.
When downloading from a fiber connection to a PS4 HDD the bottleneck is likely the HDD's speed. To merge a file you would need to at least read it and write it again, which is at least twice as slow as simply downloading it. Alternatively you can pay the cost by seeking constantly instead (like a texture per file), which may be even slower and probably sucks at runtime.
(This has sort of happened. Valve managed to roll out a version of the launcher for Linux a while back which used instructions not available on older CPUs, and crashed before it reached the point where it could check for updates. Was a pain. Normally, if Steam exits uncleanly they force an update check immediately on next restart, but if it never gets that far you're screwed.)
Well, one pirate group compressed all the .wav files into MP3s and wrote a quick batch script to decompress them all back into the .wav's. Turned a 700 megabyte download into a little under 100 megabytes.
Similarly with all the other resources that would be used on its journey between the data center and the end user. You’re wasting time if you could send an efficiently-generated 2 gig patch instead of a 60 gig file, especially for users in remote places. You’re taking up more space and cycles on your CDN. CPUs switch from idle power consumption to active. Every stage you can think of takes some amount of power and other resources, and if you’re delivering an unnecessarily huge file, you’re wasting everything it took to run all the steps along the way.