Hacker News Reader

Comment by philshem | Hacker News Reader

They were downloading at 50 Gbps for a while

https://twitter.com/donk_enby/status/1348497204940595201

https://twitter.com/donk_enby/status/1348440720504401921

Also, auth provider (twilio?) removed Parler as a client so for a short while it was possible to create accounts without a phone number (2FA).

https://twitter.com/donk_enby/status/1348298836930867204

edit: okta, free trial, thanks: https://news.ycombinator.com/item?id=25774943

View on HN

Just a note here, it wasn't Twilio.

It was a free trial of Okta that they were using for their entire userbase.

https://twitter.com/okta/status/1348191370528256002?ref_src=...

How cheap can you be to run your whole site on a trial and fail open if it doesn't work anymore?

I read that they chose to fail open when Okta dropped them in order to stay online for awhile longer.

She is probably liable for that data egress bill.

Also, how does some randomer have ten of terabytes of disk lying around?

Have you ever been to r/Datahoarder or r/Homelab?

10TB fits on one desktop drive, it's completely pedestrian

I have not. I have one 4TB HD in my machine and I've never come close to filling it!

Many people have <1TiB on their machines and are content with it.

Others, like me, have home NAS's which have 10-20TiB, and they're usually close to full.

Fewer people, but a non-zero amount, hoard data, they have 20-100TiB or even more, full on homelabs.

If you're a person collecting datadumps or running rainbow tables, you probably have such a unit. It's not even that expensive really, you can get a 24TiB Pegasus Thunderbolt raid array for <1,000 USD

It's really a matter of multimedia: video, raw format photos, music, ripped movies. That's the vast bulk of the probably 8TB or so I have.

The raw files off my D5600 are roughly 30MB each. If I took 100 photos a day, I would need +1TB every year to keep all of them. The battery is good for about 600 shots. Someone who does photography professionally can easily blow right past that on one job before accounting for their second shooter's files.

Wasn't this an ArchiveTeam Warrior project?

https://archiveteam.org/index.php?title=ArchiveTeam_Warrior

If so it's distributed among many volunteers.

But the data still has to end up somewhere... Archive.org?

the data is being processed by archive team https://www.archiveteam.org/index.php?title=Parler

Grab status: https://tracker.archiveteam.org/parler/

and will be hosted by archive.org

> some randomer

They're not a randomer, they're a person interested in data dumps.

donk_enby wants to be called she/her according to Twitter, unless you're speaking of someone else

I'm not a huge data fiend but I've got maybe ~3-4 2TB drives in my house: a Synology NAS + a spare drive.

Including old desktops and a couple of random external HDDs, I could probably hit 8-10TB easily.

And it's not like it's hard to get more. If I hit a montherlode and I need keep it, it's a 20 minute drive to Target / Best Buy / Walmart for a drive or three. Not as cheap as bulk orders off of Newegg but cost-effective enough to store these dumps.

Given the speed I’m guessing it was towards S3 so there’s plenty of terabytes.

Didn't Parler just get booted from AWS though? Seems odd it'd go right back to Amazon.

Yes probably. Wherever it is, that provider is in an interesting legal position by holding this data.

Also, I assume there is content from EU citizens in there, and so GDPR violations galore.

So people post it on Parlor themselves, it is publicly available, but it would be illegal to download for me? That does not make much sense.

If you were to scape someone's private messages from Facebook - or their private posts - and then post them online en masse ... that may or may not be illegal in Europe, but it ain't ethically white. Grey at best.

Nothing being discussed was private. It was all globally visible.

It might be illegal for you to possess such data after you were asked to destroy it, yes, at least in the EU.

The GDPR talks about "data controllers", and citizens have the right to demand such controller remove their personal data. A "data controller" in this context means you knowingly possess the data and are in the position to make decisions about it. You're not a data controller, tho, if e.g. you run some service that allows users to upload data, without your involvement and direction and also do not decide how to use such data. E.g. amazon would not be liable if people put a data dumb in their cloud (unless amazon used the data themselves, instead of just storing/hosting the data at the behest of their customers).

Even before the GDPR there have been related laws and court cases, like the case that culminated in the "right to be forgotten" based on a decision by the European Court of Justice, which may well come into play here. I also remember a case in Germany, where a women allowed her partner to take intimate pictures of her, then after the relationship ended had a court order him to destroy the material (not a revenge porn case, there was no allegation he ever shared any of those pictures), meaning it's not always about what's public.

I don't know how California's mini-GDPR compares.

Then of course there is still the avenue of copyright law if the stuff is put online. Just because a parler user gave parler the permission to distribute a certain piece of content doesn't mean that everybody else has the same permission. I'm pretty sure Parler didn't make people assign them the copyright (which isn't even possible in some jurisdictions), therefore the people who posted on parler still retain the rights to their content. They can therefore use the DMCA or other similar jurisdictions around the world to demand takedowns.

My understanding is that not all of it was public posts.

As an EU citizen, I can request that a company deletes my data. Unless this data dump is being treated as a crime scene or something, then the holders of this data will need some way to comply with these requests.

Also, now AWS (or whatever cloud provider it is) is holding content that contains racist and/or illegal content. Are they not effectively now just another Parler?

No, they're not. Holding a static, private archive is not the same thing as hosting a live, public site.

How so? Aren't people freely perusing all this data, extracting GPS metadata and reading both public and private conversations?

You cannot be more wrong with this claim, you might not had to deal with this privately or professionaly I presume in which case it is understandable. Parler is liable for taking absolutely no precaution with their users data with whom they had a terms & conditions agreement (in short: users agreed to upload their data for a specific purpose)

Now, ripping that _user_ generated data from the website this way without justified (justified = platform user agreement + legitimate interest) purpose or intent or even agreement and storing / distributing / processing it is the epitome of a GDPR transgression and borderline criminal at least in the EU (saying this as a EU citizen). They are liable. I wouldn't touch that dataset with a 10 foot pole. And I would even less brag about it on Twitter, things we do for clout I guess... :)

I have no stake in this thing, it's just to emphasize that statements like this are what get people and businesses in big trouble. Stay safe! Archive only your own data or data you gathered legitimately. Take the rest up with a lawyer or ... read the laws.

The claim I'm making is "they are not, in fact, 'just another Parler.'" You are pointing out many correct things about their potential liability and problems with holding that data! However, those things are, at most, a subset of the problems with Parler and at worst disjoint problems from Parler's problems. I stand by the argument of "a static, private data archive is not the same as a live, public web service." I did not make the argument of "there are no problems with the static, private data archive."

Agree with the legal implications. I would not go anywhere near this data.

I don’t think this relates to GDPR though. There are some exemptions for personal use which I think there are arguments this could fall into (IANAL). But my opinion is this isn’t in the spirit of GDPR.

There are many other laws that are broad enough in most countries to cover gray area scraping sadly. CFAA in the US for example. This sounds similar to the AT&T weev case.

Maybe the data compresses well and/or a lot of it is redundant (so instead of storing it raw you store it in a database and use relationships to link related pieces of data)?

That's not how you usually measure a database size.

Reply on news.ycombinator.com