For the VMs, the absolut minimum memory requirement is 16GB. The more, the better.

Otherwise it takes around 40 GB of disk space for every day of data. For an history of 100 days, 4000 GB are required. With compression I could save 50%, but have to invest a couple of days development time to make is work. You could calculate the AWS bill now ;-)

Data on cold storage is an option, but it also super slow....

Currently, I did not yet integrate all the high-resolution models that I want to. Coverage for Europe is great. In North America I will add high resolution NOAA models next.

Most likely I will keep only a limited subset of data as history, but on fast storage to make is accessible quickly