Hacker News Reader

Comment by ethicalsmacker | Hacker News Reader

[flagged]

Really? Then you lack imagination!

Google Drive is a backup solution. I can think of a lot of reasons to have a lot of small files, whether they be for logging purposes, analysis, training data, etc.

In fact, I can think of more legit uses than nefarious ones! Especially since an illegal use would be easy to trace back to you personally!

5 million files is a lot of files. You could back up windows several times over and that’s saying a lot.

What normal user has millions of typical data laying around? Not many, if any.

"What normal user has millions of typical data laying around?"

Enough that this policy is impacting some of them.

Again, realize that this isn't just talking about free accounts. It's paid accounts, too, regardless of the plan size. 2TB... 5 million file limit. 20TB... 5 million file limit. 1 plan with 10 users... still a 5 million file limit.

If you can't see that this could be a problem for normal customers (or normal accounts that may comprise multiple users), then you're not thinking creatively.

I wouldn't doubt that I have millions of files backed up on various drives, simply because I back up every system, and I've owned computers since the 90's. And I'm not intentionally trying to generate data!

There are billions of internet users. The "abnormal" group may be hundreds of thousands or millions, and still be < 1%.

Can you think of a good reason to design a retention policy, as well? (I can!)

Where do they state it's a backup solution?

Edit: why the down votes? It was an honest question.

> Where do they state it’s a backup solution?

Backup is a built-in function of the Drive desktop app (previously, “Backup and Sync” was one of two separate Drive desktop apps.)

Backup to drive is a built-in core feature of Android.

The moment they are okay with wsp backing up things there?

Why would you care about reasons, if you offer a certain amounts of space? There is a size limit. Why would you implement an item limit?

I have about 15M files in my Dropbox. The limit of my box is the size of my files combined AND their own size not their number. "Files uploaded to dropbox.com must be 50 GB or smaller. All files uploaded to your Dropbox must be smaller than your storage space. For example, if your account has a storage quota of 2 GB, you can upload one 2 GB file or many files that add up to 2 GB. If you are over your storage quota, Dropbox will stop syncing."

Can I ask what the bulk of the files are?

Only things I can think of are mapping tiles. Because while logs can take up space they don't generally create that many separate files...

Of course you an ask, but the answer is "no". It does not concern you, unless you are my host.

I'm not asking for private data, I'm just trying to understand the use case...

I mean I know why they want to restrict the amount. It adds computing expenses to their bill. But wow.

I certainly can, but I can't imagine why anyone would have 5M in cloud storage meant for manipulation by end users.

Locally you could process those pretty quick. Or in cloud block storage. But the amount of latency involved in each cloud API operation to create a new file... yikes.

Lots of software has export to and import from gdrive features.

This is meant and advertised for backup.

hearing the argument "This is for data, not for backups." strikes me as so strange.

I can understand why a company would care about the access policies; if you are storing some kind of database that is accessed constantly and routinely then it's nice to know -- but when did we become okay with categorizing bits beyond that?

I don't really care for the future of needing a subscription to backup cat photos, specifically. I'd rather bits be bits except in limited instances where the data is priced based upon access frequency/retention/guarantees. (yeah, it stands to reason that cold storage that has limited access should be a fair bit cheaper than hot storage where the data is ready to go ASAP.)

This kind of thing just accelerates the stenography arms-race wherein people try to store their terabytes of data as cat pictures.

Where is the difference though seriously? If it is for "data" I might be accessing it every other minute. If it is for backups I might be accessing it every other minute as well. I do look at my cat pictures every other second though and yes I do encrypt them before and after I do so. The whole spiel of cold/hot storage ..

But not for backing up 5M files. Again, the latency per-file will absolutely kill you.

Just for performance, if you insist on backing up the content of 5M files to Google Drive, you really need to write a custom script to tarball and compress whole directories on-the-fly or something.

I mean, 5M files is like multiple sets of map tiles for the entire world. Sometimes people definitely need to work with those, but using cloud backup through the Google Drive API interface is just not the way to go there...

This is not for you or Google to decide. And no, I can sync changes without having to tarball. I might only touch one of those 5M files a week actually, depending on my structure.

Well of course it is for Google to decide. They're the ones running the service. ;)

But again, if you ever want to retrieve all 5M files, I hope you're willing to wait a few weeks-to-months, because that's what it's going to take...

Which is why it's a terrible idea not to tarball them first to something more manageable in terms of latency per file operation.

It was not for them to decide up until now. And yes of course users were willing to wait $duration for it until now. If you are suggesting a work around that is okay. It is not okay for google to tell their users to tarball "now" "because".

Honest question, but where do they state it's for backups?

In all their docs concerning Android devices for example.

You are right about the problem, but they did also sell it as a backup product.

Why should anyone have to think about number of items at all? For most of computing history, the only thing people really care about is storage space, not items stored...

Noticeable limits on item count were fairly common until around the mid 1990s.

FAT16 has a limit of 65460 total files. FAT32 limits 65535 files in a directory.

The RISC OS ADFS filesystem had a limit of 77 files in a directory; I used this until about 1994.

That isn't really true. Not only do most filesystems give you free byte counts, they also give you free inode counts, which are like number of files.

There are also limits to number of files in a directory.

That is the part that baffles me. There might be an argument against when this becomes unsustainable (e.g. the number of bytes inside a file is actually smaller than the number of bytes the metadata associated with that file takes) but that can be solved quite intuitively by making user pay for it (essentially they should be counted towards the same usage limits, if they aren't already?).

Because even a zero byte file consumes space and if you give people the option someone will create several petabytes of zero byte files.

Frequent backups that include node_modules.

You got me, node modules would. But who puts node modules in google drive? I wish google ill but even I wouldn’t wish that on them.

I was thinking similarly. Any kind of programming language environment can leave all sorts of real + cached files lying around.

In the Python word, I just created a new virtual environment on Linux, and that had 1339 files. I then loaded it up with a few standard players (numpy, pandas, scipy, scikit-learn, matplotlib, dask, and django) - this ballooned the file count to 16,945.

People use these cloud agnostic backup systems that just dump everything into the cloud account. Add version history to it and you can rack it up.

a brand new minimal install of Ubuntu MATE hovers around 700,000 files.

If you can't imagine how that might inflate, I don't know what to tell you.

Wait, really? Coming from Windows and macOS that boggles my mind.

What on earth are the main contributors to that file count? I can't even begin to imagine, but I'm incredibly curious now.

Also, that seems like an enormous waste of disk space. Assuming a 4 KB block size means roughly an average of 2 KB of space is wasted per file, which for 700K files would total well over a gigabyte of wasted space. Yikes.

Many filesystems, ext4 included, support inlining of small files directly in filesystem metadata to avoid such waste.

well, it's not like windows is too far off - on my w10 install, c:/windows/ comes up to about 280k files. (and in total, that small-ish system drive, 40 gb, has about 380k files on it. i'd imagine macos has numbers that go something like that as well, since it is a *nix system.

What about a business with 5M customers? Storing one file per customer seems reasonable.

Use sqlite and store it in one file. Or use s3.

If you have 5 million users, you should be using a real system

If you're managing this in a free Google Drive account, shame on you (seriously)

Paid accounts are limited too.

https://issuetracker.google.com/issues/268606830?pli=1

> What it should say is that no matter how much you pay you have to adhere to the 5m file limit.

These are paid, large plans. 5 million files on a 20tb plan seems unreasonable.

Why would it seem unreasonable?

That's 4MB/file, which is not exactly large. That's also a small use case. 20TB is 4 people on Google Workspace. Our company is 40 people, so our limit is 200TB (which we pay for), but 5 million files.

We pay $18/user/month for Google Workspace and hit this limit.

The issue is common to all account types, ie also affects paid accounts.

This is not about the free tier.

But a business isn't "anyone".

This is about a single user. No user should be storing separate files for 5M customers in their individual Google Drive!

That's what databases and storage solutions like S3 are for. Use the right tool for the job.

Reply on news.ycombinator.com