Solving the 100GB Game Download
venusoft.net
venusoft.net
My one question is, is it supposed to be solving for disk space constraints or connection speed constraints?
I'm guessing both, but the former is pretty cheaply solved these days with a new drive (or by finally admitting you only need to have 2-3 games installed at once and probably aren't playing a dozen games in parallel), and I'm not sure it seems to be solving the latter.
The site says "stuck with 120mbps", and the demo on YouTube artificially caps download speeds to 200mbps, presumably as if to simulate "poor" connections. I'm in Australia and get 90-100 mbps, which is half of the demo speed, and I would be an upper outlier in this country. I can only imagine the people really suffering from 100GB downloads are on circa 5, 10, 20, 40 mbps. I get 10MBPs (megabytes) down reliably from Steam, which means even a 100GB game would "only" take ~ 2 and a half hours to download. The speed used in the demo would take ~ 75 minutes to download a 100GB game! Is that really an outrageously inconvenient wait?
I sound more negative than I am. I guess what I'm saying is I'd love to know how slow your speed can get before this stops working well.
That said, truly an impressive piece of tech.
I noticed Battle.net from BLizzard tracks couple of thresholds in the progress bar - when the game will be fully downloaded, but also when the game is playable. For many updates, those two thresholds are very close together; but occasionally, the "playable" is half the distance or less of "completed", which makes me think somebody there is still putting in the effort :)
I know iOS has something like that, though o don’t think I’ve ever seen it used.
And I’d expect Xbox does as well, but I’m not in that ecosystem so I don’t know as much about their features.
I think you're also over-estimating how hard it is to flag something as not-necessary for the majority of users. Especially when downloading a fresh copy of a new expansion. If you know most players aren't going to be in the final boss lair for 3 weeks but still ship the game with that content it's not unimaginable to just allow the game to download those files in the background.
It starts with placeholder content to get you running, and anything immediately necessary for your area. For example, if you’re in a capital city, it doesn’t wait to download the entire city, just where you are. It can rough out the other areas and add details almost in real time as you start making your way through it.
Things get more difficult if you mount up and start flying more world to load, but even still it doesn’t load it all.
There’s certainly a cost, there can be FPS loss during busy areas, but by the time you get to something important, it should all be loaded.
Contrived example is you’re in a raid, some reason you feel you need to reinstall, you could do that, fire up the game, it’ll load the raid and by the time you hit the first boss, it’ll all be there. This will take minutes.
Since I leave the client running, it updates in the background. During new releases it starts to download the entirety of the new release a couple of weeks before it’s due, to get the bulk of the content down. When they cut the release, it’s typically a much smaller update to turn the new content on. I rarely have to wait to play the game.
Experiencing a few other games makes me really appreciate the work Blizzard had done in this regard. It’s top drawer.
Firstly, your entire asset pipeline which combines and compresses resources needs to be aware of it.
Secondly, it's work that can't be reused cross platform. (The effort for Steam will not apply to PS5&etc).
Thirdly, it's another configuration to test, and as the user will forget about the feature after they get started playing, it's natural to just say it's too much effort.
Ahhh, that's interesting. Personally, I've relocated to Brisbane recently and the NBN connections up here go to 1Gb/s (!). I'm on a 400Mbps plan, with downloads of .isos from mirrors (etc) routinely being 40-50MB/s.
But the fastest downloads from Steam (using Brisbane download server) are 5-6MB/s, which really sucks. :( :(
I didn't realise some places were genuinely getting 1Gbps. Amazing!
Ahhh, yeah. I'm with Launtel (https://launtel.net.au), who seem to be extremely technically clueful and the opposite of the shady places that are around (Dodo, etc).
So I've been with them in Melb, Syd, and now Brisbane. Melb speed was 100/40, Syd was 50/20, and here is Bris currently on 400/50.
At all locations, the download speed has been "as advertised" or better. Technically the connection here can do 1000/50, and when I tested it the download rates were in the 80-90MB/s range for various things (isos, etc). But that's $15 (AUD) extra per month while 40-50MB/s is totally fine for my purposes. :)
Really wish Steam downloads were faster than 5-6MB/s though. :/
But testing again now - just in case I was misremembering - and Civ 6 is downloading from the Melb server at ~15MB/s currently.
Thanks for prompting me! :)
---
Testing the Sydney server now too, and that's ~20MB/s. Looks like a solid win there as well. :)
That's in line with "line speed" for my current NBN plan through Launtel. Might try bumping that up to the 1000/50 plan at some point and see if the download rates go that high too. :)
---
After bumping the NBN connection speed up to 1000/50, and confirming that was effective (eg through speedtest.net), I've retried the download of Control Ultimate Edition.
Downloads from the Brisbane server are still mostly around 50MB/s, though the peak download rate was 72.7MB/s (very briefly).
Downloads from the Sydney server were mostly around 20MB/s, with a brief peak up to 43.7MB/s.
Downloads from the Melbourne server were mostly around 10MB/s, with a peak of 17.1MB/s. Seems like some kind of artificial restriction is in place now, though it could turn out to be something else (tcp buffer config of some sort maybe?).
Anyway, it seems like the local server (Brisbane) is now easily the fastest download for me, so I'll be using that from here on. Might as well drop the NBN connection back down to 400/50 as well, as there doesn't currently seem to be an advantage using the higher 1000/50 plan.
This incorrect, although commonly repeated as we love as a country to collectively moan. Shadow leader at the time Albo was fact checked on this 2 years ago[1], and we clocked in at 82.58mbps back then. We now sit 16th at 96.05mbps[2].
1. https://www.abc.net.au/news/2021-12-02/fact-check-albanese-b... 2. https://www.speedtest.net/global-index
https://learn.microsoft.com/en-us/office/troubleshoot/office...
The tech/product behind it has its roots in game distribution: https://en.wikipedia.org/wiki/Microsoft_App-V
Demo: https://www.youtube.com/watch?v=e5wAn-4e5hQ
TL;DW: Mounts a virtual filesystem that on-demands downloads game content to your local drive in the form of a cache. Start playing games instantly without downloading the entire game.
This video shows a good demo of it with Halo on Steam.
Halo presumably has a solid streaming foundation, so it's a good fit for the service.
I worked on a title where this was particularly problematic from a game design perspective(limited ram, large assets), we ended up designing in "slow" sequences to mask the disk load.
Not sure how well it works.
They might even be able to integrated a bit of brains into it and keep a record of which files tend to be historically requested next for a game. In the background you're slowly grabbing those, and those threads pause for direct requests. I guess this is simpler but it feels like using via is costing you by inducing longer load times for the whole game.
All that data represents visual fidelity. I don’t want the of type games that boast amazing graphics and endless content to get smaller and more crude.
I think this product is actually a big dead end because all you need is fast Internet to make this whole thing obsolete.
Storage is also stupidly cheap even for blazing fast SSDs, for the price of a typical AAA title you can get a 2TB pcie nvme tlc SSD.
I also see cable companies like Xfinity offering speeds above 1Gbps over plain copper. They’re selling speeds that consumer WiFi can’t even handle. Even the laziest monopoly cable companies have rolled out new docsis versions at minimal cost that deliver really impressive speeds on their ancient copper networks.
If you saturate WiFi 6 at 600mbps a 100GB download is only going to take around 30 minutes if not less.
Leftover assets that weren't properly cleaned before shipping the build?
Uncompressed media that could be? (I recall that even well into the 2000s games often came with basically uncompressed CD audio tracks)
(I think the uncompressed audio was usually because of CPU decoding restraints, and the fact that a typical PlayStation game had such a crazy amount of “free” storage to work with on a disc compared to the amount of data needed to create the game and its world - might as well toss in voice acting and FMV video)
As someone who bought such an SSD, I'm surprised that AAA titles cost more than 100€ nowadays.
Also stupidly difficult when you have a preconfigured device such as a laptop. I think anything that would require you to modify your equipment's hardware configuration is not much of an argument.
I am not condoning piracy, but when looking at game sizes, go check out fitgirl repacks. Right now there is a Steam game there reduced from 20GB to 10GB. Another is reduced from 3.5GB to 1.1GB. The downloads from there are consistently and significantly reduced.
If a pirate can do that, why not a game company? or Steam?
Why should I have to download gigs of languages I will never use? Just give me the language option at the time of download?
That of course makes for more work for the game companies.
I tend to bounce between 2 or 3 AAA games at most before I "finish" and delete them. Maybe I will come back some day, but I certainly don't switch every day to warrant keeping dozens at once or insta installing them, and if its an intricate mod setup or something I just archive it.
Check failed: g.steam_receive_prefix_bytes[1] == 0x31305456
File: client_steam.cpp
Line: 2620
Last Error: 0x00000000. The operation completed successfully.
Primo Cache [2] has one disadvantage over storage spaces - the SSD cache space is dedicated as Cache, and does not add to the total space of the drive. 1TB SSD + 4TB hdd = 4TB available, vs storage spaces which will give you 5TB. This minor (to me) downside is outweighed by the fact it works for everything. It shows up as a normal drive, not a virtual disk, and it just works. I recommend it highly for anyone who has limited SSD space.
1: https://github.com/freemansoft/win10-storage-spaces 2: https://www.romexsoftware.com/en-us/primo-cache/
Whoah. I know 3Mbps isn't blazing fast anymore, but it works fine for streaming video and downloading a 100Gb game from Steam overnight, back when I used to play games.
Via seems like a cool idea, but the energy cost is high if I'm playing Vermintide 2 with only 10Gb available- that's a lot of continuous downloading, depending on how long I play.
Not wanting to bother with downloading so much data contributed to to my preference for small games like Valheim and Deep Rock Galactic, and ultimately to being done with videogames.
It downloads the database on demand and optionally caches on the client side: https://presentations.clickhouse.com/meetup73/new_features/#...
But it is not a new idea; for example, it was already implemented for SQLite a long time ago.
Turns out that's just a visual side affect of the app itself - everything appears as if it's already installed. So when he opens Destiny in the demo it's not actually installed yet (even though it looks like it is).
I feel like that could have been made clearer.
0. Write a custom storage driver that presents an read-only overlay volume where the app should be.
1. If the OS doesn't support it, redirect conflicting writes, deletes, and moves to another volume or cache locally.
2. Cache the app gradually on-demand from the network.
3. Profit!
This is exactly how I think tools like Git should work. I hadn’t actually consider it for game downloads! Neat.
But this doesn't hook into the game itself, causing issues with VAC like the AMD AntiLag+ recently did?
They can probably monitor file access to figure out what they should cache locally without actually touching any game files.
Honestly if he gets this working really well (especially on linux / steamdeck) this could be an excellent aquire + hire by valve.
That’s been solved already. Play older games. Better story, less fluff. As an extra credit, they are smaller, don’t require top of the line hardware, do not need to insert any kernel anti-cheat rootkits, and do not spy on you.
My apartment contract forces me to pay $99/mo. for roughly 120mbit, while most residences on my block can get 1gbit symmetric from Google Fiber for $70/mo.
It's also quite slow, depending on what you compare it to.
The nanobots I can live with
Over that same span of time, I've seen my local storage throughout go from ~100mb/sec to ~6000mb/sec.
The improvement of internet bandwidth for the average person is moving at a glacial pace. I'm happy to live near the middle of that spectrum; and I'm frustrated to be so close, yet so far away, from the bleeding edge.
I'm stuck with 100, which is actually closer to 80 most of the time.
I could have 250 if I wanted to more than double my monthly bill, but paying 200$ a month for internet isn't my Jam.
I grew up with dialup and my brain simply automatically makes me do several things in parallel so some can finish loading while I'm working on others. If things load too quickly it throws me off, quite honestly. A large download is for tomorrow, or after dinner at the very least.
On the flipside, I do expect local desktop apps to open instantly, and when Discord or Teams takes 25 seconds to start up and render an interface, I'm profoundly annoyed.
Why, yes, I am cranky all the time, thanks for asking.
Downloading 20GB at about 5mbit was painfully retro, but I got the idea to Ethernet it to my MacBook and in a few clicks was borrowing its quite decent wifi connection. But my MacBook doesn’t have an Ethernet port or a normal usb port. So I got a dongle with Ethernet and plugged it into a USB to mini and then a mini to C (didn’t have a USB to USB C cable).
Probably the most “LAN party experience” I’ve had in 15 years.
The excitement when you revive an old device by using a USB stick to pass a file to Bluetooth and then send the file via infrared can't be beat.