Hacker News Reader

Comment by monochromatic | Hacker News Reader

Is there a fast way to detect duplicates when you first generate the records? If so, could just keep a continuously updated counter for each client, incrementing it every time you add a record, and decrementing on duplicates to avoid double counting.

View on HN

Agreed.

The analysis and proposed solutions all seem to ignore the one important aspect of this data: time.

Why would you wait until the end of the month to count the recorded activities on the first of the month? And, what happens to failed activities that are repeated on the first of the next month? Are customers double-billed? If I'm interpreting the scenario correctly, a large network outage on April 30th would see a large number of duplicate charges as they are re-applied in May.

Counting during the send process makes much more sense. Duplicates can be handled in-line, and even detected over month boundaries. If its not "worth it" in terms of engineering time, then I'd be concerned about the leadership; the author indicates 'this was the only way we made money'

This answer is both boring, and entirely correct. Good work!

Ah, thanks? :)

I have no problem with 'boring' solutions; actually, I prefer them. Processes that are simple, self-explanatory and/or easy to follow are usually superior IMHO.

Billing was by month. This was a business decision.

It makes sense too. Billing requires a lot of man hours to pull off. Invoices, auditing, excel spreadsheets, etc. It's a people problem not an engineering problem.

I'm not sure of the disconnect here, but de-duplication is not trivial. If you do it every day nothing is easier than if you do it every month.

Doing it for all time is completely infeasible.

There was not a database of every piece of social media data sent out the door. That's what you would need to make sure not to record the entry again. All we had were flat files in s3.

Big flat files. It took hours to download and merge them all.

Once we had a database (Cassandra) it was updated continually (by a kafka consumer) and we could query it in a few minutes.

Fair enough... but somehow your system knew of failed attempts and the need to retry, correct? I'm sure I'm trivializing a rather complicated workflow, but if there's a way to detect failed attempts, there should be a way to remove them from the billing. (perhaps a separate S3 file that can be used to reduce the number of charged attempts, or split the ones you have into 'success' and 'fail' versions?)

Oh, and Cassandra is great... used it for years. Just beware of over-sized results. I've seen run-away queries crash the instance(s) running the query; ugly stuff.

Glad you found a solution that works!

I’m not sure I follow your argument here. You switched to a system where the data was downloaded continuously. You could have, with the shell script, easily:

* downloaded the files daily

* sorted and removed the intraday duplication in each new file.

* sorted and removed intraweek duplication each week and on the first of the month.

* You could give your customers a weekly estimate of their bill.

All of this would spread the work out and tell you when the system had glitched out weeks before the results were due.

Bloom filters could do it. https://en.wikipedia.org/wiki/Bloom_filter

Are you pulling that joke every time someone asks something you think you know the answer to? It's kind of contradictory.

I've edited my response, but I think you misunderstand the comic.

Was it the XKCD Ten Thousand comic?

Indeed.

It's possible with hash tables too...

Also, the Unix approach can do something like this by spitting out days at a time and sorting them, and then doing a

    sort -m "$month/*" | uniq -c

on the individual files.

`sort -m` takes basically no memory, and should chew through data fast enough.

Reply on news.ycombinator.com