Announcing the data.gov archive
lil.law.harvard.edu
lil.law.harvard.edu
This is the same thing that's been bothering me with archive.org lately, by the way. I haven't found a good way to simply (for some reasonable definition definition of "simple") contribute 10 TiB or so of redundant storage on my (european) home server either. That kind of thing might (have to) serve to ensure tamper-resistance for that data, given the current political climate on both sides of the pond. Any pointers welcome.
For anyone who still thinks the existing laws, constitutions and policies mean anything to this current regime, prepare to get some whiplash. They are proving that none of that matters if they simply ignore it and do what they want to do anyway.
[0] https://www.businessinsider.com/elon-musk-calling-for-harvar...
Muskolites are taking on the SSN system without any Congressional oversight as we speak. The President is attacking ius soli which is a Constitutional right. If they decide that sending their sleuths to Cambridge MA to physically destroy this data is in their best interest, they will do so and handle the courts later. Just stop pretending they will play by the book.
Exactly. And while it might not be possible to protect against all the new vectors day one, no one should be surprised.
for start, he thinks FDR was a dictator but the only reason FDR had such power was because the support of labor and some parts of capital. he would know that if he was a historian instead of LARPing as one.
so organizing and supporting the labor movement would be a good place to start, since a organized labor could become a powerful poltical force. otherwise everybody is fragmented on both sides which further enables their ideology.
a recent blog of his gave me this idea: https://graymirror.substack.com/p/the-pleasure-of-error
he legit thinks trump is about to be FDR 2.0 but his mistake is ignorance about how the power worked during that time. you can even use his "differential" idea in that essay against him. the differential now is labor against wealth. always has been, especially when FDR wielded all the power yarvin thinks was a dictatorship since a lot of that power was given to him by the support of the massive labor movement at the time.
more practically, this blog has some good insight into the "what do i dooooo?????" question https://wagingnonviolence.org/2024/11/10-things-to-do-if-tru...
The Senate is 2 Senators per state regardless of the population and isn’t as susceptible to gerrymandering.
That being said, I don’t feel bad for anyone who voted for Trump and is having buyers remorse.
We are already seeing it with Arabs in Minnesota who thought Trump would be better for Arabs in Gaza and are now having regrets seeing how Trump wants to use US troops to “clear out Gaza” and Latin Americans who are appalled that he tried to remove birthright citizenship.
Next up will be all of the “rural American” voters who are going to bare the brunt of inflationary tariffs.
The crap that Michelle Obama and the rest of the DNC spew about “going high when they go low” and “this is not who we are” is just that.
This has always been who we are.
When national guard troops shot and killed student at Kent state university protesting Vietnam, 58% of Americans were in support of it.
Yes this is who we are.
60% of the people in the south supported them and 60% of people nationwide were opposed to interracial marriages.
To the extent that he has a role in this (certainly has publically claimed to!), he will in no way seek to make America a powerful allied country to Russia. There's no reason to assume he intends Trump or even Musk to actually be powerful in their own right.
Especially Musk. Musk must be mad to put himself in the position he's in. The only reason Putin would allow Musk to do what he's doing, is because it's patently obvious that Musk is undermining his own potential political base in every way, with every action. A widely hated man who's gone full Bond villain is no threat to Putin, politically. He's setting himself up to be demonized (Musk is) and doesn't even seem to register it.
- 20% likelihood of no election at all
- 70% "managed democracy" with outcome decided in advance
- 10% normal transfer of power if Trump or his successor loses
I think they're racing against the mid-terms next year. If they can lock it in by then then they're home free.
(disclosure: Brit not USian)
This is the race. Historically the party in power loses the mid-terms, and I think we're already seeing huge portions of the population having buyers remorse. But, Trumps team knows this. I was waiting for them to attack the FEC, and here we are.
https://www.nytimes.com/2025/02/06/us/politics/federal-elect...
Sensitive information just can't be hosted on a centralized server anymore, it has to be distributed for the good of the project.
Yup, I was about to ask whether Trump could still force them to delete what he doesn't like. Time will tell, I guess.
With that said, the current administration appears to have a war on education in general so who knows what ideas they could cook up.
> Furthermore, I will direct the Department of Justice to pursue federal civil rights cases against schools that continue to engage in racial discrimination. And schools that persist in explicit unlawful discrimination under the guise of equity will not only have their endowment taxed, but through budget reconciliation, I will advance a measure to have them fined up to the entire amount of their endowment.
[1]: https://www.forbes.com/sites/michaeltnietzel/2024/10/17/harv...
IIUC, The controversy was that some folks wanted to draw down the principal to do capital improvements (new buildings, etc) and other folks wanted to be as conservative as possible (potentially not even using the interest for operations).
It's kinda like passing laws that require certain behaviors from businesses and such, except also taxpayers get their money funneled into private orgs for the privilege.
https://www.forbes.com/sites/richardvedder/2018/04/08/there-...
Harvard like other liberal institutions has little to no political capital in a Republican white house in the first place. Why would it cost them any to host data that is in the public domain?
This sort of silly overreaction is part of why people voted for Trump: it is genuinely funny to see people overreact to Trump because theyve been told by the legacy media that he is a "threat to democracy" over and over.
Last time he was elected his government also removed a bunch of climate data from govt websites which was quickly mirrored by third parties. Nobody was taken away by the Gestapo then and there is no reason to think things will be any different this time.
Someone forgot the "kids in cages" episodes in 2018 when Trump was only testing the waters...
EDIT: https://en.wikipedia.org/wiki/Trump_administration_family_se...
For the record, my previous comments being upvoted, then largely downvoted, for a matter of facts.
Nothing to do with Trump. Also Trump's policy was completely sensible, and no different to what happens when adults are sent to prison anywhere else: you cant put their kids in prison with them obviously.
You’re denying evidence. You’re deflecting. It’s documented in the press and in the legal proceedings that it was Trump policy.
You’re deflecting too by stating a different matter. Here kids were grouped and put in cages, from all ages, without relatives, without adults other than the guards.
We know deflection is part of the strategy… you’re not fooling anyone.
I'm not sure pointing out 'last time' is such a great idea considering it ended with a violent insurrection attempt which I watch in real time. I don't need the news or anyone else to tell me that a threat to democracy is exactly what he is.
https://www.vanityfair.com/news/2020/07/trump-administration...
Is there any risk of the government ordering them to take it down? That seems unlikely to me. The US has strong free speech protection, stronger than European free speech protection.
>keeping this data online against the express will of the government is gonna cost (political) capital.
Costing them political capital (aka the government is unhappy) is different from the government ordering them to take it down. Also, when you say "express will", are you saying the government has explicitly publicly stated that they don't like that Harvard is hosting this data?
The US literally just told (e.g., [0]) all scientists working for it they are not permitted to publish papers or speak at conferences or travel. What the US "has" is irrelevant when laws are being ignored.
[0] https://www.science.org/content/blog-post/revised-and-extend...
Is it a normal thing for the US government to do? Why or why not?
Do other government do this? Which ones and for what reasons?
As for public good, you'd probably find it depressing what the majority of Americans view as good.
In other words, you don't think we have to worry about Congress bringing in university presidents to grill them over political activities against the government's current policies occurring on their campuses? Certainly we wouldn't see any of those whose didn't give testimony that the government liked would end up being forced to resign in the aftermath; we're not in the dark ages anymore like we were in 2023...
The comment I replied to said "What I'm missing from this announcement though is any mention of how they intend to secure this "vault" against the current government.", which I believe is talking about a case where the government orders Harvard to take the info down. If Harvard will delete the data because a few government officials say they should (but don't order them to do so), then I don't see what can be done to secure the vault against the government. E.g. hosting it in Europe won't be any help, because Harvard could just delete it from the European hosting.
Citation needed. "European free speech protection" doesn't exist, each country has its own rules and freedoms. Hungary is drastically less free than e.g. France... but overall, as a rule of thumb, unless for you freedom of speech includes freedom to be a Nazi, European countries are pretty free. And don't take stances such as "spending money is free speech which is sacred so no campaign financing rules". You might get arrested for a Nazi salute though... which, if you think is a bad thing, you haven't been paying attention in history classes nor modern news.
>Opinions on the right to be forgotten differ greatly between the United States and EU countries. In the United States, accessibility, the right of free speech according to the First Amendment, and the "right to know" are typically favored over removing or increasing difficulty to access truthfully published information regarding individuals and corporations. Although the term "right to be forgotten" is a relatively new idea, the European Court of Justice legally solidified that the "right to be forgotten" is a human right when they ruled against Google in the Costeja case on May 13, 2014.[20]
>The European Union has been advocating for the delinkings requested by EU citizens to be implemented by Google not just in European versions of Google (as in google.co.uk, google.fr, etc.), but on google.com and other international subdomains. Regulators want delinkings to be implemented so that the law cannot be circumvented in any way.
https://en.wikipedia.org/wiki/Right_to_be_forgotten#European...
Free speech and "right to know" are different things. The second one is about transparency, and is very important... but as you say, the EU draws the border with "your rights stop where others' begin". We have a similar thing in many EU countries where people arrested/on trial still have a right to privacy until conviction, to avoid their names being forever plastered everywhere, with mugshots, based on nothing proven.
The first paragraph I quoted earlier.
>Those who oppose the right worry about its effect on the right to freedom of expression and whether creating a right to be forgotten would result in a decreased quality of the Internet, censorship, and the rewriting of history.[5]
>Wikimedia is based in the United States, where the First Amendment protects freedom of speech and freedom of the press. In Germany, the law seeks to protect the name and likenesses of private persons from unwanted publicity.[63] On January 18, 2008, a court in Hamburg supported the personality rights of Werlé, which by German law includes removing his name from archive coverage of the case.[64]
>There is opposition to further recognition of the right to be forgotten in the United States as commentators argue[who?] that it will contravene the right to freedom of speech and freedom of expression, or will constitute censorship, thus potentially breaching peoples' constitutionally protected right to freedom of expression in the United States Constitution.[96] These criticisms are consistent with the proposal that the only information that can be removed by user's request is content that they themselves uploaded.[clarification needed][96][97]
>Major criticisms stem from the idea that the right to be forgotten would restrict the right to freedom of speech.[111][112][113] Many nations, and the United States in particular (with the First Amendment to the United States Constitution), have very strong domestic freedom of speech laws, which would be challenging to reconcile with the right to be forgotten.[114] Some academics see that only a limited form of the right to be forgotten would be reconcilable with US constitutional law; the right of an individual to delete data that he or she has personally submitted.[96][97][115] In this limited form of the right individuals could not have material removed that has been uploaded by others, as demanding the removal of information could constitute censorship and a reduction in the freedom of expression in many countries.[116] Sandra Coliver of the Open Society Justice Initiative argues that not all rights must be compatible and this conflict between the two rights is not detrimental to the survival of either.[117]
>There were concerns that the proposed General Data Protection Regulation would result in Google and other Internet search engines not producing neutral search results, but rather producing biased and patchy results, and compromising the integrity of Internet-based information.[119] To balance this criticism, the proposed General Data Protection Regulation included an exception "for the processing of personal data carried out solely for journalistic purposes or the purpose of artistic or literary expression in order to reconcile the right to the protection of personal data with the rules governing freedom of expression."[70] Article 80 upheld freedom of speech, and while not lessening obligations on data providers and social media sites, nevertheless due to the wide meaning of "journalistic purposes" allows more autonomy and reduces the amount of information that is necessary to be removed.[2]: 9 When Google agreed to implement the ruling, European Commission Vice-President Viviane Reding said, "The Court also made clear that journalistic work must not be touched; it is to be protected."[125] However, Google was criticized for taking down (by the Costeja precedent) a BBC News weblog post about Stan O'Neal by economics editor Robert Peston (eventually, Peston reported that his weblog post has remained findable in Google after all).[126][127] Despite these criticisms and Google's action, the company's CEO, Larry Page worries that the ruling will be "used by other governments that aren't as forward and progressive as Europe to do bad things", though has since distanced himself from that statement.[128] For example, pianist Dejan Lazic cited the "Right To Be Forgotten" in trying to remove a negative review about his performance from The Washington Post. He claimed that the critique was "defamatory, mean-spirited, opinionated, offensive and simply irrelevant for the arts".[129][130] and the St. Lawrence parish of the Roman Catholic church in Kutno, Poland asked Google to remove the Polish Wikipedia page about it,[131] (in Polish) without any allegations mentioned therein as of that date.
>Index on Censorship claimed that the Costeja ruling "allows individuals to complain to search engines about information they do not like with no legal oversight. This is akin to marching into a library and forcing it to pulp books. Although the ruling is intended for private individuals it opens the door to anyone who wants to whitewash their personal history… The Court's decision is a retrograde move that misunderstands the role and responsibility of search engines and the wider Internet. It should send chills down the spine of everyone in the European Union who believes in the crucial importance of free expression and freedom of information."[132]
>In 2014, the Gerry Hutch page on the English Wikipedia was among the first Wikipedia pages to be removed by several search engines' query results in the European Union.[133][134] The Daily Telegraph said, on 6 August 2014, that Wikipedia co-founder Jimmy Wales "described the EU's Right to be Forgotten as deeply immoral, as the organisation that operates the online encyclopedia warned the ruling will result in an Internet riddled with memory holes".[135]
> Volokh, Eugene (2000). "Freedom of Speech, Information Privacy, and the Troubling Implications of a Right to Stop People from Speaking about You".
>Portals: Freedom of speech
>Categories: Freedom of expression
For this to be said in the current circumstances shows either a political viewpoint or a lack of knowledge.
Additionally:
* In some European countries newspapers cannot publish the names of people on trial. In the US they can.
* In some European countries you cannot take pictures of people in public and publish them. In the US you can.
Maybe this?
> In addition to the data collection, we are releasing open source software and documentation for replicating our work and creating similar repositories. With these tools, we aim not only to preserve knowledge ourselves but also to empower others to save and access the data that matters to them.
https://github.com/harvard-lil/data-vault
And since the data lives here: https://source.coop/repositories/harvard-lil/gov-data/descri...
Combined with this:
> To download an individual dataset by name you can construct its URL, such as:
> https://source.coop/harvard-lil/gov-data/collections/data_go...
> https://source.coop/harvard-lil/gov-data/metadata/data_gov/f...
> To download large numbers of files, we recommend the aws or rclone command line tools:
> aws s3 cp s3://us-west-2.opendata.source.coop/harvard-lil/gov-data/collections/data_gov/<name>/v1.zip --no-sign-request
So one could "easily" mirror the whole thing, making it distributed.
Definitely a concern, if they want to harass Harvard and the other universities they could, but I don't think they'll bother. They know the data will be backed up, that's not the point.
Taking it off of data.gov accomplishes two things:
1) Makes it look like they're doing something, playing to the base. Easy to do
2) Delegitmize any insights the data might have. "Sure you have 'data', is it official data? I don't see it on data.gov. How do we know its not fraudulent?" It makes it harder to use it to justify policy changes. It adds one more tool to the denial crowd.
Theres something that about this that just rings second amendment. Personally I think the concept of civilians having weapons to be a check on a nation state is absurd, but in this case it feels pretty empowering.
This is a topic that came up at work today as we rely on this data and are considering backing up most of the Lidar data from there ourselves (100s of TB probably)
EDIT: no, looks like it is only the footprints
I haven't had much time to look at this yet and see what all is there, but whether currently included or not, a couple of things I really hope get archived are the contents of the DTIC (Defense Technical Information Center) document repository (lots of really interesting older scientific publications) and the NASA TRS (Technical Report Server).
I'm working on my own archive of at least some portion of the DTIC stuff just to be on the safe side. So far everything I've tried to access is still there, but who knows how long that will last.
"where they burn books, they will ultimately burn people as well."
Those who delete research will ultimately delete people as well.