Faster LZ is not the answer to 150-250 GB video game downloads
richg42.blogspot.com
richg42.blogspot.com
There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall)
Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think this is the smart way to do it.
Examples of games shrinking I are hard to come by but special credit to IO interactive and Hitman 3.
The install size with all the content of the previous games included (If you own them) is less than the install size of the first game. That is impressive and commendable.
Games are weird and some people will look at the file size and think it means the game is better, just the way some people think games with more hours are better. The psychology of people who play video games is weird.
Duplication of assets on disk was a very weird byproduct of HDD file access conventions, but we shouldn’t need that anymore. There is still a shrinking, but valid crossover window between devs being able to assume all storage is solid state. I think things will get better but the debate of file size vs average/expects storage capacity vs Network speed will never end but there is certainly work to be done.
https://en.wikipedia.org/wiki/.kkrieger
https://www.youtube.com/watch?v=ya_MUKc343U
It is dated, but looks pretty good by 2004 standards.
[1]: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
For me, the net result is that Titanfall was one of the first games to go when I needed space. Low hanging fruit.
Even if it was deliberate, I have to question the logic and wonder how much usage they enabled vs. uninstalls from file size over time.
I was thinking about buying Mass Effect yesterday, the recent update, but it requires 150GB and I don't have a Windows partition that big! (I'm Dual booting Linux) So I think money is being lost in people who aren't often gamers and haven't bought big disks for gaming.
They may have some of my money, but not much (discount!), and they’re not getting money for DLCs, arguably the reason the base game was so cheap, and I’m not in a rush to buy more stuff that I can’t fit on my PC.
Prior to the era of the initial game purchase being just the entry point I’d have agreed with you, but many modern games have monetization strategies that make uninstallation a problem.
And if they have a strong enough incentive to care about CPU usage, they’re clearly putting effort into optimizing for some audience.
That wouldn’t help people whose issue with large game size is the amount of space used on their disk, but it would help people whose issue is the long download times.
My guess is Valve doesn't offer a 'decompress your ogg files to wav at install time' feature though.
Even when you do want lossless assets, as long as you compress in reasonable chunks there is no downside.
Another problem is chunking the data properly so that incremental updates are not hugs.
The launcher is bugged and downloads either empty or 137mb patches (maybe bad CRC at some server? Different CRCs at some?). Every time.
Starcraft remastered was bugged at some point and couldnt be played for a week.
If you statt Heroes of the storm, it takes few minutes to download some 600 byte ghost patch, then does some (antihack?) processing... then you can start the queue after like 10 minutes.
It's probably like the GTA5 that also took ages to start due to poorly done code and devs not caring at all.
I waited hours to play Witcher 3, Cuberpunk, Assassins Creed Odyssey. With WoW I was playing within minutes.
I worked on Guild Wars 2, which has this feature. I made a first prototype of it that streamed all content on-the-fly. It's pretty easy to implement - you have an abstraction that asynchronously loads a file off of the disk, and you can just make that download from the network instead.
The tricky part is when you want to ensure all the assets are there for a specific area before you load in, or simply knowing what order to download things in. For example, there was a starter area of Guild Wars 2 that spawned monsters from many other areas, this meant that the manifest of what was needed was enormous for that area.
So the 'playability' threshold becomes a trade-off between game experience (assets popping in as you play) and quick entry.
If you include download time, even that doesn't hold up to your claim.
This is not an uncommon internet download speed even today (let alone 5 years ago when it launched) according to the steam download stats (it varies by country, check australia for example, US is a little over 100 average): https://store.steampowered.com/stats/content/
Even at 100Mb, 45GB is 1 hour. Witcher 3 looks to be about 30GB but Cyberpunk is apparently 70-100GB.
Do the math next time :)
I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing.
All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want to rewrite OpenJPEG in Rust, go for it. You can download part of a JPEG 2000 file and get a lower resolution texture, and that's what's done for distant objects. The general idea is to download textures at a size that gets one texel per screen pixel. (Yes, subpixel textures would be better.) Textures are locally cached to SSD. Background threads are frantically loading and unloading textures in the GPU at different resolutions as the viewpoint moves. The highest resolution is compressed losslessly; the others are lossy.
In metaverse systems with user generated content, there's not much instancing. So the texture load is much higher. Wander around Cyberpunk 2077 and you'll see the same railing show up in completely different contexts. This doesn't happen in worlds with user-created content. There's no producer or art director to insist on reusing content.
If your game downloads content on the fly, you need substantial content delivery capacity. Second Life front-ends Amazon AWS with Akamai. Open Simulator, on the other hand, often just hosts the content on a single game server, along with the the simulator itself. My own viewer pulls about 200Mb/s from the content servers for about 15 seconds when you first connect. Then it's done until the viewpoint moves.
HTJ2K has support in OpenJPEG (master branch at least), might need to compile with a specific flag or something I can't remember.
I think JPEG-XL is ideal for these types of applications in my opinion, even if you ship the decoder with your client. Their progressive encoding/decoding is much better than J2K in my opinion.
A metaverse is not a game. Games can be built within it by users. This has been done in Second Life, but the games are 1) sluggish, and 2) space-constrained, because land is expensive. Those are scaling problems which can be solved.
This requires solving the scaling problems that led to the original article here. You can't just download everything in advance. There's too much stuff in the larger games.
Open Simulator is an open source re-implementation of Second Life servers, written in C#. It's been around for a while, and now it's getting a bit more developer attention. There are multiple federated grids of Open Simulator servers, and content stores where you can buy items, all under different management. Land is much cheaper than in Second Life, but servers tend to be under-resourced and slow. There are some people working quietly on trying to improve the Open Simulator technology to work better.
Other attempts to solve this problem include Improbable's system. Improbable managed to blow through $400 million on the scaling problem, producing a system that's too expensive to run.[3] (They're funded by SoftBank.) Some good indy games tried to use their system, but the server bill was too high. Their approach is a general-purpose distributed object manager, which seems to be the wrong tool for the job. Otherside uses Improbable, and they only turn on their world maybe twice a year for a few hours for special events.
[1] https://web3isgoinggreat.com/
[2] https://www.wsj.com/articles/meta-metaverse-horizon-worlds-z...
[3] https://www.improbable.io/
[4] https://www.ft.com/content/3508bec7-a2f8-414e-8059-7b96b2700...
I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene.
Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things.
Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a lot of the work involved in making things pretty. You don't need to generate or even understand displacement maps if you can just add more polygons to the model. You don't need to billboard the mountain way in the distance if Unreal does it on the fly. Who cares if that means including a 250mb mesh in your assets, versus a 5mb HDRI.
So yes, producers have either become lazy or are simply unable to meet the demands of the hardware by coming up with better algorithms. That's unlikely to change until either hardware stops advancing and either A) can't run the latest games or B) games become too large that they're impossible to download.
[0]: https://www.gamedeveloper.com/programming/sponsored-feature-...
[1]: https://docs.unrealengine.com/5.0/en-US/nanite-virtualized-g...
[2]: https://twitter.com/mariobrothblog/status/163604089376436224...
Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't work with open-world games, especially if they have different times of day. And it doesn't work in highly dynamic scenes. We basically hit the end of the road for scaling graphics without RT. And now we have hardware that can do realtime path-traced rendering of major AAA titles, and it looks miles better than anything smoke and mirrors could've ever delivered.
The real problem in the industry is that GPUs are not getting cheaper. It cost about $250 for an 6GB NVidia 1060 in 2016, and it costs about $250 for a basic NVidia card today that can do slightly better. Meanwhile, people are buying $100 WalMart laptops and trying to play games on them.
There’s still a gap between integrated and 250$ cards, but that’s mostly filled by used last generation cards.
[1] https://www.newegg.com/gigabyte-radeon-rx-6600-gv-r66eagle-8...
> and it costs about $250 for a basic NVidia card today that can do slightly better
Eh, the 3050 is like 30% more efficient in every single benchmark versus a 1060, with 30% more VRAM, and costs like $280. I don't think 30% is "slightly better". The Arc A750 also is around the same price point and is nearly 80% better, but I admit Nvidia has the best software stack on the market at the moment so the 3050 is a more fair comparison.
Lately I've become hyper-aware of games doing screen-space reflections and suddenly finding it distracting when I point my view down and the reflection of the scene at the horizon vanishes off the water/floor.
I upgraded from a 1070 to a 3080 right when the 3080 came out, and I actually have yet to play an RTX game.
> bloom
I hate that this ever became a thing.
Bloom just makes the scene blurry and bright parts end up blown out.
Unreal Engine 5 has a new lighting system called "Lumen". It seems to be visually almost on the level of normal brute force ray tracing, while not requiring any special ray traycing hardware and still running with decent performance on mid range hardware. It's a smarter approach which tries to accumulate lighting information on textures over time instead of computing everything again for each frame.
Despite being a graphics researcher, I’ve been a little skeptical in the past about ray tracing in games. Recently I’ve started hearing for the first time from some hard-core gamers that turning on ray tracing makes a “massive” difference in the quality of the experience, and that they’ll use it as long as the frame rate can stay above 30 fps most of the time.
That said, with all due respect, your speculation about clever non-ray tracing techniques seems a little off-target from my perspective. There has been, and still is, a huge amount of research into these, and the entire field of techniques is hitting limits and failing to progress as fast as ray tracing. There are multiple reasons for this. One problem is that many of these one-off raster effects require their own memory & compute. Another is that they each require their own implementation, each require their own tuning, they impose constraints on the art and the artists, and they only work in narrow ranges of curated input. Shadow maps are a good example - needs a buffer of it’s own, and get too close or too far away from the shadows, or if the scene is too big, and they look terrible. You have to spend precious art+dev time just to make them work, you have to spend precious memory for this one effect, it eats into your rendering budget, it only takes care of the shadows, it’s quality has a hard ceiling limit, and you still have 13 other effects to work on that have the same story. With ray tracing, once you setup the memory for it, it can render all the effects at the same time, it’s the unified theory of rendering, and doesn’t require per-effect memory or much per-effect tuning, not like the grab bag of raster and screen-space effects does. Ray tracing has it’s downsides, and it’s certainly cycle-hungry, but there is a good reason that game devs are gravitating toward using ray tracing, it’s because ray tracing is fundamentally easier and fundamentally looks better than the one-off raster effects.
Regarding ray tracing being ‘brute force’, keep in mind that if you’re rendering many instances, or have lots more geometry than pixels visible to the camera, ray tracing can be more efficient than raster. Is usually isn’t in today’s games, because games have hard budgets that limit the amount of geometry - mostly due to raster algorithms! - but the trend of increasing geometric complexity is clear, and that ray tracing is the likely path forward. Nanite is very interesting and has made waves - it’s being adopted and used for the same reason that use of ray tracing is growing, primarily because it makes game development easier. Maybe we’ll see these two things merge in the future, they are not mutually exclusive. Nanite is carefully designed to make sure you’re always rendering polygons that are about 1 pixel in size, plus or minus, and Nanite has pushed scene complexity, but there are certainly open questions about whether this approach will continue to work as complexity grows.
Modern ray traced games are much less "brute force" than what they were doing back in 2008. The use clever techniques and algorithms like temporal denoising and radiance caching to achieve high quality ray tracing effects with a tiny number of secondary rays per pixel (The budget is often under one ray per pixel).
It's not really fair to call it "brute force" at all, just a different set of clever algorithms.
The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets.
The reality is a game has increasingly more content nowadays. Remember, there's a lot of "games as a service". So you have many "seasons" of additional content. You may not play all the content, but someone is. And when people want 4K and 8K textures, different variations so effects don't look the same twice, on top of all the other sizes, plus all the other effects ... it adds up. Never mind huge open worlds, where the asset footprint for just the map might be 100GB.
Having been in a few AAA studios myself (doing database stuff) and sitting next to artists and even art directors, the sheer amount of effort, headcount, etc. put into the art pipeline is genuine madness. In a large studio, art is over HALF the production cost, sometimes more towards 60-65%. Heck, one of the things sometimes brought up about Red Dead Redemption II is that there were three full-time employees in which all they were responsible for was horses. 6000 man-hours a year, on just horses.
To sum up what Chris said towards the end, "The only thing keeping these numbers down is the compression algorithms. If we didn't have these newer compression algorithms, we'd probably legitimately be at a terabyte. I can actually tell you for a fact, that there are over a terabyte of assets in the COD engine."
I think what this shows is there should be smart delivery of assets. Before installing perhaps you should be able to choose your graphical fidelity and only download assets related to that fidelity level and below.
This makes manifest building and delivery more complicated, and probably UX too, but I'm sure it's a problem that the infra/marketplace provider (Valve, Epic, Microsoft, Sony, etc.) could solve and make easy for devs.
https://www.charlieintel.com/warzone-on-demand-texture-strea...
I've never tested how much bandwidth it uses. It seems a little crazy but also neat.
War Thunder does this - in the game launcher you can choose whether you want the highest resolution textures and most detailed models or not. You can even get a minimal client without vehicle interiors as well (only 3rd person camera available).
It makes a lot of sense to me, to the point where it's surprising that all games aren't embracing this approach. I mean, I don't have a lot of storage, nor good hardware - why should I be forced to download high detail assets that I'll never even see because instead all of the models will be at lower LODs and all the textures will be of a lower resolution.
It feels like the game engine developers don't really care about bundling assets by fidelity and each game acts like it just owns your entire disk.
That's precisely why I don't really play AAA titles anymore and mostly go for pixel graphics and indie titles. There are actually surprisingly few well optimized games out there.
The Diablo 4 beta had this option, drastically reducing the download size if you didn't need 4K
All inference of Stable Diffusion is done in FP16 at best, with multi-step models, and models themselves are highly inefficient. I would be surprised if in May 2024 4090 isn't capable of generating 8k(8192x8192) Stable Diffusion quality image in less than 1 second. And also you wouldn't realistically need to
> also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
Because it allows for much more variance in content. I'm a bit sick of seeing tiled textures everywhere TBH. Also virtually all textures would be AI generated in less than 5 years.
You make it sound like my electricity bill would be lower by just downloading the bigger files...
no CPU required
it's marketed by Microsoft as "direct storage"
GPU decompression is also a thing though, so compression is still a way to save I/O.
For HDDs, yes, compression is generally a win. HDDs are dead as far as game development is concerned, all modern targets have flash storage.
Some compression codecs that produce smaller files are also MUCH slower to decompress, which swamps any benefits from the reduced i/o. This is part of why sometimes people pick less efficient formats for use in games.
Heck, just the geometry is probably far larger than folks anticipate.
People often repack video games, making programs to compress and decompress them so they are easier to pirate. It's not uncommon to see a 2-3x reduction in size, for bit-identical results.
This allows it to be ran automatically after download without having the user need to launch the game.
I'd expect that they use a compression that is good for transport, at transport time. I'm not too shocked that they have the assets in a way that is easy to use at runtime.
The crackers who remove copy protection often repackage (or someone else does) to greatly reduce size; since they’re in there poking around the code they can see what isn’t used.
This isn’t new, either, ancient console games have unused assets and even entire levels in the code but disabled.
I can see some benefits to having uncompressed assets in that it would help with asset loading. I'd still be surprised to not have any form of compression at all happening. I'd also be curious to know just how much the assets could be compressed.
So it’s exactly speculation. A lot of games have done a lot of things for a lot of reasons over the decades.
Why are new games so large? Good question. This entire thread is speculative.
The thread might be speculating about _why_ they do that, or whether this particular game does that, but that's a non sequitur, that isn't what I was talking about.
Has any game in the past 5 years that has an install size >100gb used uncompressed assets? I would be very very surprised! The likelihood of this being the case on PS5 or XSX hardware is approximately zero.
Yes I’m familiar with the infamous uncompressed audio in Titanfall.
Fortnite didn’t publish what they did. But I promise it’s not “bEnableCompression=true”. Compression is used up and down the stack with the utmost of care. Even the older games that didn’t use compression did so only after great consideration of the trade offs.
“Games as a Service” games like Fortnite, Destiny, and Call of Duty have often bloated and then been optimized. The fix is never bEnableCompression. It’s more about how the content is sliced and packaged. How to efficiently serve new content is a really tricky problem. And a somewhat new problem that didn’t exist 10 years ago. At least for consoles.
Can you capture a trace in RenderDoc to prove this? Or show raw bmp/uncompressed dds files on disk?
8k texture with mipmaps takes about 100mb with BC7. Which is huge. Should be around 30-40mb with kraken and oodle texture.
> I do wonder how much of it is that a Kraken license costs money and it takes time to integrate, though...
Kraken license doesn't cost anything for Unreal developers. And Sony licensed it for all PS5 games.
Check this out, you can sort by the number of submissions to filter out indie games https://docs.google.com/spreadsheets/d/14CVXd6PTIYE9XlNpRsxJ...
I am kind of curious on why they wouldn't use compression in transport, as it were. I'm still mostly assuming that they do.
Encoding is the hard part, but only need be done once when building the final release.
It's a shame that IFS hamstrung the adoption of fractal compression with their expensive license fees - wouldn't necessarily have been The Future Of Images but at least we'd have had the option.
[1] https://www.pcgamer.com/biggest-game-install-sizes/
[2] https://www.pcgamer.com/how-hitman-3s-devs-shrank-the-entire...
How does that work?
There is no excuse for them to be shipping a 150GB game on the PS5 when so many cross-gen games are 40-60% smaller on PS5 than PS4
Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio.
I do think they could have figured something out, like compressing the right subset of audio assets, but I just want to give the context for why they made that choice.
Maybe 50 hours is a lot for a non story based game but even then once you factor in all the sounds and all their variations to reduce repetition you might be up there.
https://www.escapistmagazine.com/titanfall-dev-explains-the-...
I can attest that MP3 and OGG decompression (the two audio codecs that would have likely been used at the time) are both pretty slow. It adds up.
MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4 in 1996. Certainly a system in 2014 can handle decompressing MP3s on the fly.
Even if you didn't want to do it on the fly during game play, they easily could have been decompressed at load time and it would have taken only a couple seconds. It's not like ENTIRE 35 GB of audio needed to be loaded all at once, nobody had that RAM back then, and very few people (especially gamers) have that much now.
"It's not like ENTIRE 35 GB of audio needed to be loaded all at once" misunderstands the nature of games fundamentally. Some games do potentially need the ability to access any of thousands of sound effects at a moment's notice with the lowest possible latency.
It's like trying to use PNG for texture compression instead of better suited formats.
The minimum requirements to play this game were a 200MHz CPU w/ MMX and 64MB RAM - I was pretty close to this baseline. So anyway, I discovered that the game played at much better FPS (like 30 instead of 5) if you turned the music off - but only when playing MP3 - CD Audio had no hit. Now perhaps the game used a sub-par audio codec, but that single MP3 decode stream was enough to make the game unplayable.
Anyway that's not to say that I would expect MP3 decoding to be a problem in 2014, in fact you can likely play audio with no noticable increase on CPU usage, but when you have multi-stream audio (think voices, background music, sound effects from various channels - guns, explosions, etc.) I can see it starting to add up - especially when the CPU is already constrained for the graphics, game logic and perhaps of course everyone's favourite anti-piracy/anti-cheat logic.
[^1] For younger readers, yes, CD Drives used to come with built-in DACs and a special cable you could hook directly into the audio card, allowing you to listen to CD Audio on PC for "basically" free in terms of CPU cycles.
In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it.
Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively performant on the hardware they’re running.
I can guess what execs will pick when faced with “delay the game or ship a massive bundle.”
However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep.
If I were to dive into the JIRA for any one of these "missing compression" incidents, I wouldn't expect to find an easy win, I would expect to find "missing compression" beneath a pile of much worse bugs like "level 2 crashes on AMD cards." So it goes.
Yeah, I've often heard that the advice for people who really want to make games is to get some friends together and try to put out something together as a side gig while you work somewhere boring to pay the bills.
SWE at a game studio, especially any AAA game studio, is an awful experience. The expectations are high and the salaries are low because there's a seemingly unlimited bunch of young adults that want to make games.
And yes, it takes extraordinary talent. Your code needs to be able to generate a frame in under 16 milliseconds in order to maintain 60 fps. If you want to appease the hardcore players with these crazy 360 hz monitors, you need to get your game loop down to a mere 3 ms.
The amount of shortcuts and optimizations needed to pull that off is insane. At that level, you're trying to figure out how to minimize cache misses and branch mispredictions. How many Node devs are thinking about those?
Just as a counter point - I work at one of the largest AAA game companies and we have great work life balance(been here 9 years). Don't remember when was the last time I worked more than the required 7.5h/day, never worked on the weekend, the people I work with are all passionate about their work and want to deliver something great. The pay is "ok" - it's not exceptional but it allows for good quality of life. In comparison all my friends who went to work in traditional software development are paid somewhat more than I am but they all 100% hate their jobs, so......I don't think it's a bad trade off.
As for the infinite supply of people wanting to do this - it's only really true at junior level, as soon as you talk anyone above junior it's extremely hard to recruit and you might be getting a handful of candidates a year. We've been trying to hire a senior rendering programmer for over a year now and we've had like 5 candidates total.
If most commercial software is a feature factory, games are "megafactories" because they are not really competing on the robustness of the implementation, just on delivering a certain kind of experience that delivers some interactivity and shows off some assets, and that tends to lead towards approaches that hack one feature on top of another and then lean on manual testing to catch "enough" errors. Many games also take characteristic shortcuts that eliminate some options for game design in order to simplify the tech.
The larger companies will often keep their tech teams separated from their game teams and treat the tech coders more generously. This comes with all the pluses and minuses of siloing the work - sometimes they solve the wrong problem, but do it well.
When I talk to people in the game industry, it sounds a lot like my experiences outside it. Plenty of programming skill to go around, but the skill is often allocated inefficiently, and everyone's under pressure to just ship stuff and move on to the next deliverable.
I'm not saying that you are doing this, I am saying that this is done.
watch some GDC developer videos on YouTube. these people work HARD for performance.
The issue at hand around install size comes down to the engines, their ability to decompress asset bundles, the fidelity of the assets (8k vs 8k,4k,2k,1024) if no runtime optimization is done. WAV sounds vs OGG vs MP3 vs flac, there’s a lot of shit in those asset packs. You could compress it all like they do, into a folder structure you can index and load from, or you can write that game code to make boss fights cause camera shake, the decisions you make making a game are really all about making the game and little to do with how best to deliver your content.
Some texture formats like DXT are already compressed. Raw PNG’s could have compression. All these things are possible if the engine supports it.
Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it.
I don't know why you're trotting out this misinformed BS.
You can also simply watch the statistics when Steam is installing a game - the amount written to disk is larger than the amount downloaded over the network, because they also compress assets in flight to reduce bandwidth usage (the most important thing, since raw assets often load faster from an SSD as long as you have available storage.)
You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
Yeah, I could've indicated better that I'm talking about on-disk compression of files.
>Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it.
Not really if we look at 3rd party games. Jedi Survivor is likely not using that since it takes 147 gigs on PS5 and only 139 on Series X. Warzone doesn't use it, Fortnite doesn't use it(despite Epic's acquisition of RAD), Hogwarts Legacy doesn't use it, Elden Ring doesn't use it, FIFA doesn't use it. These are some of the most popular games on the platform. My point was that he is talking about something beyond LZ compression when most games aren't using tools that are available off-the-shelf.
>You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
We have hardware-accelerated decompression in all current gen consoles and GPU decompression on PC. We would be in much better world if everyone used that.
I find this extremely unlikely. The packaging tools by default will compress files when building a package. Decompression is transparent and done by the hardware; the game doesn't have to implement anything or link any libraries. You would have to explicitly select "no compression" during packaging. Warzone definitely uses it - I worked on Cold War, Vanguard, and MW2 specifically on streaming and decompression across our supported consoles.
You're talking nonsense. I worked on Fortnite and have direct experience with it and the HW decompression on PS5. I can even point you to the patch [0] where Fortnite enabled oodle compression and shrunk the game by 60GB.
[0] https://twitter.com/FortniteStatus/status/131867784125345382...
>You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.
Is this really true? I feel like you'd need an absolutely blisteringly fast SSD to match something like an modern, optimized lz decompressor's speed.
he's saying that game asset paradigms are headed in the wrong direction rapidly.
I think the right definition is this one:
Lempel-Ziv (dictionary-based lossless data compression)
Because of that optimization for download time and startup time end up optimizing for different things.
With loading speeds so different in these two cases, I wonder if the best solution would be to recompress the data with a different compression level during download and installation. For example, download textures compressed with a good but slow and non-random-access image compression format, and then convert them to a lighter GPU-friendly compression format locally.
I do think in this case he misses a bit of the point, goes too far:
> They are designed for fast loading and streaming off SSD's. Who cares about saving off a few hundred ms (or a second) when it takes hours or days to download the product onto the SSD?
Most games will have only one two or three downloads over the user's lifetime, and it'll never be instant, will always require some wait.
But we load textures many times if we keep playing a game.
Rich's overall point remains, but I think some balance of optimizing load speed is also very appreciated & very necessary.
Are there any statistics on that? Because I have deleted and redownloaded games a whole bunch of times because you can't fit more than two or three games on a single SSD anymore.
I've played large games off of hard drives because I simply didn't want to delete 240GB of other files off of my main drive. The game experience was clearly much worse than the developers intended, but their space demands are unreasonable. Especially considering I'm only playing at 1080p!
Give me a 1080p installer that shaves off 75% of the texture size and you'll win more than a few milliseconds because the stupid thing can actually fit on my drive and doesn't need to come off the internet again.
Original Size: 93.1 GB -> Repack Size: from 35.5 GB [Selective Download]
Original Size: 3.1 GB -> Repack Size: 844 MB
And these are lossless, bit for bit reproductions. In many cases assets (audio, video, images/textures) could be using lossy compression for even greater reduction in size with no significant loss in perceived quality (but maybe some loss of advertising/bragging rights).
The downside is, some effort is required packing it: https://fitgirl-repacks.site/new-compression-algorithm/
If you're on a metered connection (common in many of the more forgotten parts of the world), the bloated download costs you real money or is simply not possible (or you have to spread it over several periods of data allowance).
Or at the very least just offer two different downloads of the game, one with lossless textures and one with lossy textures...
Enlisted, which uses the same engine, is fine though.
I've seen many games create a vast file system mess, and they're forever plagued by "corruption" or something, complete with "fix" tools, "cache" clearing push-ups and players reinstalling after big updates that don't install correctly for whatever reason. Somehow ArenaNet surrounded this whole problem and solved it optimally.
This happens to touch upon a pain I keep getting with Steam. Games like squad keep getting tiny patches (compared to the whole game size), which is great, until you see that steam needs to read and write the whole file again to finish the patch. It's not as simple as their "dumb allocation wearing out SSD" problem either -- there's no API to do this anywhere unlike the common posix_fallocate(). Linux FALLOC_FL_INSERT_RANGE comes close, but no studio will tend their files so they make exact filesystem block-sized units.
You wouldn't even need Linux extensions for that. You just do it like ZIP (or multisession CD-R, whatever) and put a new directory at the end. Just don't ask about streaming.
Asset production is already the bulk of the cost of AAA video game production.
On the opposite end of the spectrum, you have Inigo Quilez ( https://iquilezles.org/ ). He literally draws with maths, and he is famous for his 4k demoscene productions. The problem is that about no one else is capable of doing what he does.
Because of this, its very suited to math and procedural generation. You just change some parameters, render and show. Games instead have to do this in real time, with less latency and CPU/GPU usage. And procedural/math generated content tends to be CPU heavy (more usage+latency), if not also on the GPU (cant be cached, etc).
Shaders are more on the middle of this, because you want to apply certains transformations to and from rendering data.
Just my armchair observation.
Anyone who thought this was incredibly naive.
I look around my room and try to think about what textures could be generated procedurally. My walls, carpet, and ceiling? Yeah, sure. The stickers on my computer? No way.
But the characters are flat shaded so those textures are tiny.
You need to look at the data, animation weights compresses 95% with a simple lz.
One big dissappointment was half-float support having low performance on the some drivers, that would have saved 50% off the models.
One advantage of not compressing is that your disk shows the (V)RAM usage you'll have.
I'm aiming for 1GB total (to support 2GB Raspberry 4) and my biggest concern is uncompressed 120Hz animations! (to save CPU)
But so many of those can be hot-loaded off disk.
To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene.
- Result : a) File size < 1 mega Byte ( really). b) It downloads in 4 seconds.
Check it here : https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug)
Question : - why video games STUDIOS do not use this strategy ?
I am pretty sure most modern video games use more mesh points than that for, like, a hat your character can wear.
The time to : download the globel zip file + open and reading the points and all textures => 5 to 7 seconds. This is super quick !
Note that I'm not storing assets at rest in a ZIP file because there's no reason to. If I was doing that it might close the gap some.
- Step 1: download zip file containing my XML point file and my 90 PNG texture files.
- Step 2: open zip file and dezip in 91 filed
- Step 3: run 3D games with all the 91 Assets. The (small) game engine reads each 90 PNG files and - of xccourse- transform each one in a GPU compatible texture.
All zip takes about 6 seconds. Not so bad
* Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually make this worse because you can't delta compress big archives as well as individual files.
* Smaller on disk doesn't mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See https://ph3at.github.io/posts/Asset-Compression/ (from elsewhere in this thread) for one example.
* For image and geometry data that is headed for the GPU, it's best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I'm considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2.
* For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it's less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency.
* In some cases the install you get depends on the hardware you have. It's not uncommon for older platforms to see 'high res texture packs' that could be installed optionally. The Xbox platform has something called "smart delivery" where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere.
* Ultimately, unless your game is too big to fit on people's hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam/Itch/Xbox/Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn't make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game's framerate instead.
* Lots of people decide a game "isn't compressed" because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it.
An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there's no point doing it if you can just use the browser's built in brotli decoder instead.
It is either the author is missing the point, or am I missing something?
What's most important about game textures is that they can load fast at a quality good enough not to disturb gameplay. Absolutely everything else is a secondary concern.
Remedy's Quantum Break had an unusual spin on letting users choose the install size, it made heavy use of live action FMVs which could either be downloaded ahead of time or streamed on the fly as they played. Downloading the FMVs more than doubled the games footprint.
It's not like we don't have the money, I bought a 10900k and 2080 super for my brother back then, it's just that the speed is unavailable. A "git gud" simply don't do that justice.
BC1-5 have been standard practice since before the original XBox. I think when people here are talking about "uncompressed assets" they mean "not applying variable-rate zip style compression on top of BCn compression."
* Every additional configuration i.e. 'texture pack vs no texture pack' is more QA workload to verify that everything works as expected. So adding this option is more work for your already-busy testers.
* Adding DLCs to deployment systems like Steam or PSN is additional work and then when patching the game you would also need to patch the DLC. I can imagine the coordination overhead here being a pain.
* Some users will fail to install the DLC and then complain that the game is ugly or that the screenshots were misleading. This is ultimately a user education problem but it's also the case that the flow for installing DLC on most platforms is awkward.
* The DLC install process adds additional points of failure, so you need to handle all of that now too.
* Users installing a new graphics card will probably be annoyed to have to suddenly wait for a HD texture pack to download before they can test it out.
All pretty minor individually, but it probably adds up to explain why most studios don't do this.
I wonder how widespread developer support for Xbox Smart Delivery (which is basically what you describe) actually is at this point?
And as someone who just got 1.5Gbps internet installed, my tolerance for file size is continually increasing.
But I understand where fiber isn’t available these are large downloads.