For the search database, I did have to manually download each and every company, all 856,000 of them. To cut down on size though, I included all data points except for filings. This is because they're large, but also because they're updated often. By excluding them the database stays more up-to-date.

Other than the search database which needs to be available on demand, the rest of the filers are queried and analyzed on demand. This is required because some filers get really, really big. Blackrock Inc alone is about a 30MB file.