Shirky.com is gone
web.archive.org
web.archive.org
I'm not sure when this happened but I see that Wikipedia has adjusted to this and now links to the archive.org link:
https://en.wikipedia.org/wiki/Clay_Shirky
This is yet another example of the history of the early Web disappearing. Shirky's early essays were fundamental to way we understood the potential of the Web back in 1999-2008.
I actually discovered this while looking for another old Shirky essay which, as far as I can tell, is now entirely gone from the Web.
I'd been writing about what we came to call social media since the early 90s (alt.culture.usenet and alt.folklore.urban ftw), but by the middle of last decade, all anyone wanted to talk to me about was marketing on Facebook, which was the boringest possible topic.
At the same time, my wordpress host had lousy security, and my site was getting frequently disabled because of some malicious javascript uploaded through some hole they hand't patched. I wasn't writing there anymore, so it was pure cost at that point, and cost of my time, not just dollars.
Then I moved to Shanghai for several years, working on other stuff, and fixed the site a couple of times again, and one time, my host was like "We disabled your site!" because of their own security flaws had let it get hacked again, which, the whole thing had entered 'ugh field'territory.
I never decided to let the site lapse, I was just tired of dealing with it, and the political circumstances in both China and the U.S. seemed much more urgent than rescuing some historical essays, so one day at a time of not dealing with it became years.
And here we are, me reading my own eulogy. Which is incredibly flattering and touching, I have to say.
I'm not even sure what of it can be resuscitated -- maybe if I want it back, I'll have to copy it from Wayback (and will say "Thank you Brewster", not for the first time), but if anyone here has advice about competent and secure hosting for an old Wordpress blog, hmu at cshirky@gmail.com, because reading this, it makes me embarassed not to have just fucking fixed this a year or two ago.
And thanks, all, for this thread. -clay
Just want to thank you for your great work.
I used to work on a lot of US Department of Defense projects, mostly stuff I can't talk about. One very notable project I CAN talk about was an initiative (pushed by utterly clueless, insular, and frankly corrupt academics) to spend billions of dollars in 2008-2010 timeframe on implementing Semantic Web technologies in various military business systems across the DoD.
As an actual technologist who knew how to build things, I was perpetually in the awful position of having to explain to leadership that these highly credentialed academics were selling garbage. I had tried to implement systems according to their design. The graph databases they pushed (they hated Neo4J, for reasons of purity because it didn't actually use RDF/OWL in the database...... i get a headache just talking about this...) were slow piles of dogshit that couldn't scale. No amount of reality could dissuade the academics. They had their theories, and any collision with reality was merely an implementation detail that I and my team were simply too incompetent to overcome in their eyes. Almost none of them had actual technical experience. A smattering of Comp Sci folks, and a ton of "Library Science" idiots.
Your essays on why the SemWeb was utter bullshit were a potent weapon I used with the generals the academics were pushing, and I eventually got the generals funding the project to see the light. Got them cancelled, and sent the idiot egg-heads packing. I still see them on LinkedIn to this day. They desperately continue trying to push that rock up the hill, and only recently warmed to more practical graph database solutions.
They HATED YOU. It was hilarious, watching them try to refute your obvious points and clear writing with jargon and hand-waving. Utterly unconvincing to the generals.
Thanks for your essays saving my ass back then!
Most of my writing was about social media, back when the web was young, but "Ontology is Overrated" is actually my favorite thing I ever wrote, and it makes me happy beyond measure to know that it helped someone manage an actual argument over whether to buy into the semantic web!
I have never been talented enough to write production code, but I often thought of myself as trying to provide ammunition to people like you who are, when talking to bosses who didn't understand that the phrase "Now it's just a simple matter of programming!" was a bitter, sardonic joke, not an upbeat assessment of possibility.
Thank you for telling this story! This whole thread has been like hearing my own eulogy, but this in particular is just :chefs_kiss:
The response of the academics to your writings was actually a master class for me in the nature of academic corruption and groupthink. What I learned from that experience was that (contrary to my prior beliefs) high IQ individuals are actually far more susceptible to cognitive dissonance than others are, not less so. They are far more adept at mentally constructing rationalizations and false realities that bolster and protect their existing belief systems from new information than most people are. Add to that the fact that they are extremely economically vulnerable to reputational damage, and you have a really toxic recipe.
You see the same with CompSci/tech people treating data like there's no bias in its collection.
100% concure with this view. The semantic web was one of the biggest wastes if time ever and set back the open web fatally.
You'd have to take all of the text from the Wordpress blog and format it into Markdown but that shouldn't take a huge amount of time unless there is a lot of weird formatting or different media types.
No you wouldn't. Just dump it in as-is.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/Websit...
I'm in favor of static HTML myself where possible, but it's not hard to maintain a secure Wordpress install. Keep automatic updates enabled and don't install any third party plugins.
It's that second part that most people screw themselves with.
When the maintenance is "ensure auto updates are on, and don't do anything that would not get updated automatically" it's not like it requires regular effort.
> Whereas if you just have a collection of articles that you want to keep around as an archive, if you convert them to a static site, you can basically forget about them afterward...
Your web server, your operating system, etc. still require at bare minimum the same level of maintenance.
You can outsource that maintenance to someone else of course, but you can do the same with WP as well.
--
My point is that WP alone doesn't massively increase the maintenance burden, it's what people tend to do with (to?) WP that increases the burden and eventually leads to unmaintained sites.
no dog in the fight here but I felt impelled to point out that ensuring auto updates are on solves almost all security holes except for the security hole it opens up.
In almost any computing context, but especially in the context of a personal blog, the vast majority of exploits are against known security holes for which patches have already been released and those with automatic updates enabled are already safe from.
Yes, hypothetically updates can deliver new flaws of their own and even potentially intentional malicious code, but from a practical sense it's not worth worrying about if you're using mainstream software packages on a major OS.
More effort that you'll be able to exert when you're dead.
Although I highly doubt learning a jekyll config would be harder than managing a PHP daemon, web proxy and mysql database.
Thank you for all of it!
Good luck with restoring your site.
Amusingly, nobody here has stated the obvious: wordpress.com.
Do you remember the title or subject or any key words/phrases?
https://www.wired.com/2006/11/meganiche/
"Now that more than a billion people have access to the Web, there is no longer a trade-off between size and specificity. The basic math is simple: A tiny piece of an immense pie is huge. A decade ago, reaching one-tenth of 1 percent of Web users amounted to 36,000 people, a number that compared favorably with the circulation of, say, the daily newspaper in Bridgewater, New Jersey. Back then, reaching a million users required a decidedly mainstream offering (Amazon.com and MSN come to mind). Now, getting niche can be the path to getting big; one-tenth of 1 percent of today's Web audience is a million people."
Whereas in the past, my store might have 1 or even 0 options on peanut butter, but travel 10/50 miles away and there'd be an entirely different option.
We have a little bit left of this with beer, as most places have a "local" beer available.
And it's the same all over the US. The only variation tends to be that some flavors Flamin' Hot Dill Pickle or Wasabi Ginger, e.g., appear more often in some markets than others.
https://web.archive.org/web/20060210230250/http://www.shirky... "Help, The Price of Information Has Fallen and It Can't Get Up"
http://web.archive.org/web/20140117014453/http://www.shirky....
> People who work on social software are closer in spirit to economists and political scientists than they are to people making compilers. They both look like programming, but when you're dealing with groups of people as one of your run-time phenomena, that is an incredibly different practice. In the political realm, we would call these kinds of crises a constitutional crisis. It's what happens when the tension between the individual and the group, and the rights and responsibilities of individuals and groups, gets so serious that something has to be done.
> And the worst crisis is the first crisis, because it's not just "We need to have some rules." It's also "We need to have some rules for making some rules." And this is what we see over and over again in large and long-lived social software systems. Constitutions are a necessary component of large, long-lived, heterogenous groups.
I was a big Shirky reader back in the day.
But the truth is that all kinds of things disappear all the time in all aspects of life. The web is no different at all.
Take my dad for instance - a quite profilic and famous mid level artist. It’s coming to the point where not much people remember him. And when my siblings pass on that will be that.
Let’s not get too nostalgic. If someone is interested they should try to preserve the writings and keep them going.
That’s why certain groups have taken it upon themselves to preserve old important films. And as we know they are always fighting for more donations to keep things going.
Because in the end… no one really cares.
50 years from now children will be asking who was Van Gogh.
Why not say his name here, to give him a bit more memory?
> 50 years from now children will be asking who was Van Gogh.
This seems a strange cut-off. Van Gogh is an artist who died 130 years ago; why should the next 50 years be the ones that forget him? There are plenty of artists today whom we remember from earlier than 200 years ago.
Indeed. I think he will more likely become even better known as people use style transfer and such to generate many new pictures in his style.
No one really cares.
We simply haven't had enough time pass. Eventually, some day, people will forget who Julius Caesar was. It may take 50,000 years. It may take 500,000. They'll forget.
[1] https://knowyourmeme.com/memes/complaint-tablet-to-ea-nasir
what does it matter if you're forgotten after 5 years or 50,000
It isn't as if you would notice anyway
We do have more insight into ancient Egyptian pharaohs and architects, however, as these details were more carefully preserved.
That being said, I'm sure 500K years from now, these details will all be buried on some thumb drive in an underground archive and our descendents will lack the drivers to decode them.
Unless the humans figure out reliable ways for them and technology to survive multiple nuclear world wars
Which doesn't seem impossible, not at all
Glad we have the Wayback Machine then. But if you don't want your blog mirrored by Wayback you can declare that in your `robots.txt` file. Do this:
User-agent: *
Disallow: /
> Some sites are not available because of robots.txt or other exclusions. What does that mean? Such sites may have been excluded from the Wayback Machine due to a robots.txt file on the site or at a site owner’s direct request.
If you exclude them in your robots.txt file they will also absolutely retroactively remove your site from the index.
Bach has been dead for hundreds of years and his works are being rediscovered by people every day.
In the early 2000s I had a manager who evangelized "the semantic web" to our team as being just years from taking over the world. He was convinced that we needed to integrate RDF into every product to remain relevant. Shirky's essay persuasively articulated why this would have been a waste of effort for us.
Years on, these ideas still influence my analysis of elevator pitches, business plans and requested features.
edit:
Thank you, Clay!
[0]: https://web.archive.org/web/20150323162650/http://www.shirky...
Clearly a lot of them have been looking back at that period with mixed feelings over the last few years (and have said as much), but even still it's shocking that something like Clay Shirky's blog would disappear when he's very much still alive. The fact that we apparently now have to worry about whether or not something like Danah Boyd's MySpace vs Facebook essay could disappear though is ridiculous; even with all the known technical and social problems with the web it's hard to imagine that it's come to this.
I am wondering, are there any organizations that actively scan and archive books, even if they don't share them because of copyright laws? Amazon is almost a monopoly when it comes to books, and we cannot rely on it to preserve the books for the next 50+ years, and not every purchased copy is guaranteed to be around by then.
I have every ROM released for pre-2000 consoles + a ton of old software, OSes, and PC games on hard drives that I keep backed up. I have friends who prefer to specialize in keeping/archiving zines. Someone else does fanmade ROMs. Etc. Most of us have a good understanding of copyright law even where we disagree with it, and it's common to archive 'grey' material off the record and then fight for the right to make it official.
Books are even easier than digital assets since the laws around book archiving and preservation are much kinder to archivists. So yes, there are definitely DRM-free, digital copies of MOST books floating around and will continue to be for quite some time. The main issue is whether or not we'll be prosecuted if we open up our personal archives or distribute them.
The University of Michigan and Google Books have something like this, at least for the UM library: https://publicaffairs.vpcomm.umich.edu/key-issues/google-set... . I can't find much about whether it stopped completely, or just went quiet, after the lawsuits.
The Internet Archive definitely does this. It's what powers their controversial book borrowing feature.
We've had plenty of things that had that general appeal, but they've always been owned, run, and eventually shut down, but companies of some sort. I'm not opposed to companies having websites, but I'd love to see a public space as well. The idea that we could post our content to that space and expect it to live far longer than us would be a huge deal.
Usenet still exists, and it is definitely filled with trash and spam. There are various projects working on similar sorts of public (or sometimes private but open) spaces without those downsides, it remains to be seen what will end up filling this niche.
Unless you are specifically referring to someone providing general purpose hosting that you don't need to think about administering — well that isn't a feature of our present-day landscape simply because it wouldn't be a profitable venture given the risks and liabilities from the messed up things people could stash there, along with the inherent costs of admin and hosting.
But if you are willing to set up your own box and procure sysadmin for it, what you suggest exists.
Especially in america, this country has been running as fast as societally possible in the opposite direction from providing such services as a public good -- so if anything even those libraries and sidewalks are mortal, and at the whims of a political machine idealists may not get to steer.
It sure would be nice if such a thing could exist in stability ad infinatum, but at the same time, there is a small turdy nugget of truth to the american principles of not trusting government with acting in the interests of the people's freedom indefinitely, as has been demonstrated by nations around the world.
We need more grassroots organizations like archive.org supported and run by the people who truly believe in it.
Governments won't necessarily get it right, but I would still love to see public efforts toward the goal. Despite recent adjustments, the US postal service has been an impressive slice of government service for a very long time.
What's interesting about this particular goal is that it can easily cross our physical borders. A couple savvy countries can easily team up and get things rolling (or at least throw significant support behind existing efforts)
(Of course, that’s unless you’re talking about sharing the type of content that would make interpol want to track you down)
Good luck having a VPS 'box' that has an uptime record of 10 years. I know through personal experience that your VPS instance will go down, no matter how much you try to mitigate that. There will be bots and bad actors either trying to DDOS it, or trying to brute force `/wp-admin`.
You could go for some obscure CMS to try and thwart that, but you run the risk of having vulns in that software because it doesn't have the eyeballs of vanilla Wordpress. You could always go for the shared hosting approach but the caveat being: there is no guarantee the shared provider will provide 100% uptime either. (And it will go down at the worst possible moment, like during a HN hug of death)
Your best bet is to have your content distributed and mirrored across multiple services such that any attempt to take it down is impossible. I would go into details about that, but due to op-sec reasons I won't. Tip: Plaster your content all over the web such that a removal of one piece of content does not affect the others.
A $5/month server with debian + nginx serving static content can survive way more than a HN hug of death.
The ephemerality of the thing is the issue I'm speaking to. We've lost something here.
The requirement for books to last is physical space, and those shelves and boxes continue to exist far longer than the publishers, authors, illustrators, etc. We don't have that with this medium (except, of course, archive.org which is excellent and not nearly enough). We've built something that's lighter than books and easier to store in smaller spaces, but we've [collectively] given no thought to maintaining a proper archive.
The freedom to publish to the world in an instant is as magical as it is fleeting. On a longer scale of time - and not a very long one - it's practically worthless.
I still see your point, though. Maybe mixing in IPFS with some public solution, like a guest book, might be something?
https://web.archive.org/web/20191130082649/http://shirky.com...
On the one hand, I like the idea of the internet being organic where things rot and die, truly forgotten.
On the other hand, it's much easier for future historians to look back a hundred years and see the actual content that's posted at any point in time.
In both instances I think we overestimate how important any one piece of content is. If an idea dies when a site goes down, it probably wasn't a very good idea.
So are we talking about accurate attribution of ideas, or the loss of good ideas per se?
Clay Shirky’s stuff was a regular “go-to” for me.
RIP shirky.org, your insights then are just as good today
http://web.archive.org/web/20140117014453/http://www.shirky....
And noticed that the site was down.
I think that's a good question. It's just a bunch of articles and perhaps there were some videos linked to youtube? I find it hard to believe that Shirky just abandoned it given how media savvy he is, but stranger things have happened.
There's some weird stuff on it if you look at Aug-3 2019: https://web.archive.org/web/20191102024012/http://www.shirky...
Looks as though it's been defaced with cialis garbage copy?
Vulins found every other day in some obscure package you thought you were one and done with. You get to spend a couple of hours fixing it. Oh new update on the java core you are using few more hours. Oh that update breaks 2-3 things. More time. You float the idea that someone else takes it. But those who step up have 'other ideas' what they want the site to be. Oh and your base OS is 2 releases back better get on that.
Then the actual cost. While you can get a cheap site up and going for not much. If you get even slightly popular you are now looking at a decent amount of money for many people. You may not see a couple hundred a month as 'no big deal' but many people do. You can pay a provider to take some of that patching work out of your hands but you pay for that.
The programming world is very ephemeral. We get bored easy. We move on quickly. Sometimes we are just cheapos. Things that cost time and money get left to rot or turned off.
What I find interesting in my 'internet' life. Is I always seem to find out about the really interesting places as they are being closed out :(
Agree with the whole message and tone of your comment but wondering about this bit. We run a bunch of Wordpress sites with decent traffic and a bunch of badly optimised front-end, heaps of old plugins from decades passed: we can hit 20k uniques and a million requests per day, with nightly backups for $30/mo. It could be less if we didn't care about completely surviving every traffic spike and bot crawl.
Edit: Re-reading what I wrote and how much I quoted I realise now that wasn't clear :)
No maintenance upkeep, minimal server costs.
It is just setting it up. Also sometimes people just lose interest in it. Even a couple of bucks a month would be not worth it. Then if you stand it up and 'forget about it'. What happens when your CC expires? It goes away. You do not care anymore so it is probably not something you care to fix.
For me 50-100 bucks a year is not something that is that big of deal. But if I have totally lost interest in it. It would be on the list of expenses to get rid of. It is one of those things a lot of clean up your financial problems people talk about. Look at all of those little charges. They add up to decent money sometimes. Not saying that happened here. But it probably does happen?
A static website hosted on AWS S3 and CloudFront would need to serve a LOT of traffic to generate a $100 bill per year.
But if you pre-pay for the next 50 years, will AWS exist until there?
Will the internet exist?
Would Putin have already f** humanity up before that?
Hard to guarantee...
[1] https://aws.amazon.com/premiumsupport/knowledge-center/prepa...
It would be nice for something like the Internet Archive to offer "perpetual hosting" where you pay upfront for enough to fund hosting "forever". $100 would generate $1 a year in interest which would be enough to host small data.
> Q: Do I get interest on my deposit? A: No[, but...] We periodically reevaluate this situation, because we think a web account that runs forever purely off of its own interest is a pretty cool idea.
NFSNet also has an interesting part in their FAQ in response to the question If I think services you host are currently unavailable due to lack of funds; is there anything I can do?, they outline a process whereby third-parties can fund a hosted service by creating an account themselves, depositing funds into their own account (NFSNet services are prepaid instead of billed after the fact), and then submitting a manual (but free) request to transfer those funds to the original accountholder based on the service's domain name. I've always thought this was interesting because in theory someone could set up a community, disappear, and then the community could step up to keep it funded long enough for the person to get out of the hospital/be rescued at sea/etc, so long as the infrastructure is solid enough to remain operational without being attended to (not vulnerable to exploits, etc.)
They've also got a policy where if the member who operates the service is a willing participant, they can publish their NearlyFreeSpeech.Net account ID and have donors add funds to cover 100% of service costs via automated transfers. <https://www.nearlyfreespeech.net/about/faq#Lifeboat>
https://www.wired.com/2008/06/service-lets-yo/
>Website Lets You Send a Post-Rapture E-Mail to Friends 'Left Behind'
>If millions of Christians suddenly disappear from the face of the Earth as the opening act for Armageddon, Threat Level thinks most nonbelievers will be too busy freaking the hell out to check their e-mail. But if they do log in, now they can be treated to some post-Rapture needling from their missing friends and loved ones, courtesy of web startup YouveBeenLeftBehind.com.
[...]
Good thing the sysadmins are loving trustworthy Christians:
>Users can also upload up to 150 megabytes of documents, which will be protected by an unidentified encryption algorithm until the Rapture, then released to up to 12 nonbelievers of your choice. The site recommends that you use that storage to house sensitive financial information.
>"In the encrypted portion of your account you can give them access to your banking, brokerage, hidden valuables, and powers of attorneys," the site says. "There won't be any bodies, so probate court will take seven years to clear your assets to your next of kin. Seven years, of course, is all the time that will be left. So, basically the Government of the Antichrist gets your stuff, unless you make it available in another way."
There was a pretty good Law and Order episode where one of those sites accidentally triggered, sent an email confessing to somebody's crimes prematurely, which led to an unfortunate chain of events and salty remarks.
https://www.imdb.com/title/tt1343619/
>The owner of a Rapture website is killed by a man working to return Soviet Jews to Israel to fulfill Biblical prophecy. However, the killer seeks shelter at the Iranian embassy, leaving the DA's office in an unenviable position.
https://tvtropes.org/pmwiki/pmwiki.php/Recap/LawAndOrderS19E...
>Van Buren wonders why the emails were sent at all.
"Yeah, but the Rapture didn't occur."
"As far as we can tell."
"I'm still here."
"You mentioned."
—Anita Van Buren, Cyrus Lupo, and Kevin BernardThe provider of your choice probably has a similar billing feature.
[1] https://aws.amazon.com/premiumsupport/knowledge-center/prepa...
From the article:
> Now, thanks to a series of breakthroughs in network theory by researchers like Albert-Laszlo Barabasi, Duncan Watts, and Bernardo Huberman among others, breakthroughs being described in books like Linked, Six Degrees, and The Laws of the Web, we know that power law distributions tend to arise in social systems where many people express their preferences among many options. We also know that as the number of options rise, the curve becomes more extreme. This is a counter-intuitive finding - most of us would expect a rising number of choices to flatten the curve, but in fact, increasing the size of the system increases the gap between the #1 spot and the median spot.
Unless your real tech equates to censorship or control or something, I am not sure how it would help reduce power law effects?
crikey
clay shirkeyed on his duty
to keep up a webshitey