Hacker News Reader

Comment by ConcernedCoder | Hacker News Reader

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours."

TLDR; Horrible architecture decisions like this can be very costly.

View on HN

I don't think they seem aware of the architecture problem, besides they seem proud of having 40 billion request.

I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.

A bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.

I agree with you. There are so many ways to make this infinitely more efficient. For starters, why are they re-calculating and re-rendering the value every time they get a new donation? Also, they could store those values in a more cold&cached-storage and just make the reads to update from Firestore. Don't use a freaking database that is charging you for every single read and write, to deal with mundane client-side renderings.

Changing the currency will re-trigger all those conversion calls ...

It's more like a naive database design which doesn't consider the access pattern and therefore requires an inefficient query, which is definitely an architectural issue.

It's only 460k QPS, with 16k documents everything would be cached really well. Or a single instance of redis can serve that load of reads fairly easily.

Good luck with that. A single Redis would not be able to serve that workload. Maybe a single machine but you're really pushing the limits there just with concurrent TCP connections.

At that qps Redis has 2 microseconds per request.

I agree it caches well but your proposed architecture is definitely not production quality.

There is a difference between queries and connections. With pipelining, a redis machine can easily serve that. If they stored things in a better structure, like a list, they'd easily be able to get that.

On my machine 5 concurrent requests, at 100 items, it can do ~6 500 000 items per second, at 300, 11 000 000 per second, and it kind of caps out at that. Even with 1 concurrent connection, at 600 items, you get 6M per second.

Legit question here; what would be a good architecture for this case?

SQL with no ORM. Harder to be unaware of what you’re querying when you have to write the queries.

Not using attempted magic like Firebase would also fix the problem where the home page transfers 9 MB of data from Firebase on top of the 1 MB JavaScript, which appears to be… their entire database or something?? Accessible to the frontend??? Censored excerpt from that response:

  "email":{"stringValue":"soXXXXXXXsu@gmail.com"},"fechaDisponible":{"integerValue":"1527483600000"},"identification":{"nullValue":null},"key":{"stringValue":"1527560309061"},"listaNotificaciones":{"arrayValue":{"values":[{"stringValue":"alXXXXXXXXi@yahoo.com"},{"stringValue":"soXXXXXXXsu@gmail.com"},{"stringValue":"pXXXXX2@hotmail.com"},{"stringValue":"BeXXXXXXXXXXXXXXez@hotmail.com"},{"stringValue":"pXXXXXX0@gmail.com"},{"stringValue":"trXXXXXXXXXXro@gmail.com"},{"stringValue":"joXXXXXXXXXXXXie@hotmail.com"},{"stringValue":"ivXXXXXXXXxd@gmail.

Also appears to expose, for each campaign, the poster’s bank name and date of birth.

And wastes a bunch of various resources making separate requests to a currency conversion service for each amount, as others have noted. And requests /null and /undefined. This might be the most irresponsible development I’ve ever seen.

Exposing contents of DB without even any SQL injections... I think they have no idea about network monitoring. They _have_ an idea about console, since they're logging stuff there extensively.

Reply on news.ycombinator.com