Ever wondered how much data web giants generate? Grab says 1.7MB/user/day | Hacker News Reader

Ever wondered how much data web giants generate? Grab says 1.7MB/user/day

theregister.com

62 pointspseudolus26 comments

A useful reference for magnitude of "1MB" is that a typical (english) novel contains around 0.5 to 1 MB of ASCII text. Neuromancer is just under 500kB; Snow Crash is ~900kB. So 1.7MB/user/day is roughly equivalent to 2-3 novels written about each user every day.

It is more likely to be a tiny fraction of that, stored in a really suboptimal way.

Raw ndjson files for tabular data? Don't mind if I do.

I don’t doubt this one bit.

Here in Thailand I use grab at least once every day. Sometimes up to 4 times a day.

Taxi to office, from office, meals, groceries, courier service. Grab would not work without the army of motor bikes.

Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data.

Transactions are probably the smallest amounts. Another big one is probably all the inventory records from supermarkets, convenience stores and restaurants in probably the number one foodie city in the world - Bangkok. Then there is taking photos to prove delivery pickups and drop offs. Damn…

Then travel to the Philippines, Vietnam, Indonesia - grab also just works. It’s actually quite an impressive operation and they’ve improved the app quite a bit in the last year alone.

Now with the lockdowns at the moment they are super busy more than usual - they probably up that amount of daily data by amount 50%

Seeing how prevalent grab & foodpanda has become here in Singapore, I'd say much more than 50%.

> Just logging GPS every few seconds on a delivery to your home must be the vast majority of the data.

Logging gps once per second every second of the day using two doubles will get you 691Kib of data. Unless you have two deliveries in flight 24/7 the math does not check out.

I believe precise GPS coordinates is 12 bytes per sample, so that’s ~1MiB/day. However, it compresses extremely well even with lossless content-unaware compression techniques (though I don’t have any real numbers), and if you’re serious about compression you can do even better still, with http://cs.joensuu.fi/~mchen/GPSTrajComp.htm showing a technique that gets you down to ~3–6KiB per day if you’re happy with up to 10m of error.

You would only need to store the relative differences between the coordinates, so the actual precision is much lower than 12 bytes, even for a naive scheme.

For correctness, I must note that if you’re using the usual static-size data types like floating- or fixed-point numbers, deltas take up roughly the same amount of space unless you impose speed limits, which allow you to get a bit cleverer about the space consumed, though it’s still complicated by the world being a sphere rather than flat.

Notwithstanding that, when you get to compression, delta encoding does make it losslessly compress a bit better, though not as much as you might think, especially with the better compression algorithms. I simulated some stuff for this last year, but didn’t keep the answers I came up with. My extremely vague notion is that zip or gzip got something like a 10% improvement, but LZMA only saved another 1–2%. Don’t trust my numbers, though. And yeah, if you’re caring about this stuff you can do way better by a more deliberate encoding.

If you have a list of coordinates, then you know a priori what the maximum precision of the deltas is. If your delivery driver doesn't teleport at any point in the journey it's likely that you can shave quite a few bits off the top of each delta.

For example, if your coordinates are precise to 1 angular meter, and the maximum delta in the list is 200 angular meters, then you only need 2x1 bytes for each coordinate.

It's not just your own GPS, they would log the driver car GPS too, so it's, on average, 4 coordinates per ride.

> All that data is clearly valuable: Grab has also announced record profits.

What do they say about correlation and causation?

that there often is one?

Obviously you can't say clearly all the data is valuable, but you can say clearly some of the data is valuable because without any data they would not have any business.

Now, imagine storing that on 3.5" floppy disks...

With some basic compression, this daily 1.7 MB will fit on a 1.44 MB 3.5" floppy... Our Father, who art in The Cloud, give us this day our daily floppy ! Don't forget to swap your floppy every morning. https://i.imgur.com/ntrjkV7.jpg !

So, how much of this data is speculative hoarding, and how much energy is being wasted here?

Number of values would be much more interesting. An image is big, a bloated json somewhat big, and compressed or not compressed? (actually the compressed size would be more telling)

So that is 3400 Terrabytes of data in a year for 2 billion users.

I used the number 2 billion because it's approximately the number of people on Facebook. Not vetted, just for a quick calculation.

It is 3400 Terrabytes of Data per day for 2 Billion users. Or 1241 Petabyte, 1.241 Exabyte per year.

No wonder why HDD maker are happy.

Oh, yeah! You're right. In between trying to convert from MBs to TBs, I forgot that I was looking at the day scale, not year.

1.2 Exabytes a year is a lot.

It’s 12 AWS Snowmobiles, which are container trucks full of hard drives. A corporation that generates that much data probably has more container trucks of furniture.

I don't think they factor in compression here. It's likely the 1.7MB they're talking about is raw data. Google's location history for example can be compressed down to very manageable single digit percentages even with fast compression.

Edit. I'm talking about Grab, the "transportation, e-commerce, payments, and financial services" company in the article, which sounds like it collects little if any multimedia from users. Also thinking more of data the company generates from the user (usage history, metadata and collection), not what the user generates and uploads.

The 1.7MB figure can't really be extrapolated for a Facebook-like data set (or worse, YouTube). Social networks have a lot of mostly multimedia content created and uploaded by the user, and lower volume of company generated content like location tracking and lists of interactions.

Isn't it also likely that a lot of that "raw" data is comprised of JPEGs (which are already compressed)?

If it’s raw data and not simply actual disk use divided by users, I would guess the savings from compression likely pale in comparison to metadata such as indices, backups and other overhead.

Is there any reason to believe that Facebook is collecting the same amount of data as Grab?

It's reasonable to assume they're collecting a comparable amount. Even if Facebook collected half as much, 1700 terabytes per day is still a lot.

Reply on news.ycombinator.com