Marion Stokes: A woman who taped 30 years of TV news | Hacker News Reader

Marion Stokes: A woman who taped 30 years of TV news

bbc.com

107 pointsonemoresoop61 comments

Transdiffusion, a small UK based group of TV presentation enthusiasts, have been recording and archiving material related to TV logos, idents and presentation since the 1950s.

https://www.transdiffusion.org/

Also recently seen on HN: This gentleman recorded 55,000 boxing matches over 40 years:

https://news.ycombinator.com/item?id=19471312

In the US there is the Vanderbilt Television News Archive, dating to 1968:

The core collection of the Archive consists of regularly scheduled newscasts from ABC, CBS, NBC, CNN and Fox News. Special news reports are recorded in addition to these newscasts, including material from other networks. We record broadcasts as they are televised, provide the widest access allowable within copyright for scholarship and research, and preserve the content for future generations. The database currently includes 1,116,752 records, including abstracts at the story level of regular evening newscasts and catalog records for each special news report.

https://tvnews.vanderbilt.edu

https://tvnews.vanderbilt.edu/about

The archive itself is not publicly viewable online, but the rundowns are (including national advertising spots running with the programmes), and are text searchable. Trends in programme length, topics, and to some extent quality and bias, are evident. It's a hugely useful, little-known, treasure.

Example; every nightly news story on "privacy" since 1968:

https://tvnews.vanderbilt.edu/search?utf8=&query=privacy&but...

Archives, including (again, text-searchable) close captioning, here at the Internet Archive (https://archive.org/details/tv), are hugely useful to researchers and even investigative reporters. Here at WashPo:

https://www.washingtonpost.com/politics/2019/04/08/televisio...

Andrew Tyndall, independent news analyst, has monitored TV news since 1988, publishing review and data at The Tyndall Report:

http://tyndallreport.com

Would be interesting to know if those tapes have Teletext info which can also be recovered, as well as the news video:

  https://github.com/ali1234/vhs-teletext/

(Not sure whether this works in the US, as I'm in the UK)

The US never used Teletext, but we do have closed captioning, which should be recoverable and would provide some machine-processable date. Much less than Teletext would, though, for sure.

It's funny you should mention this - it's exactly what is on my mind at the moment too. I'm trying to resist the temptation to trawl ebay for peoples old VHS collections (search "used VHS" gives good results) and start building an archive.

Pity there doesn't yet seem to be any centralized effort to this though. I can imagine a wayback machine-like archive being an amazing resource.

It's gonna be awesome to learn history in the next 100, 200 or 500 years. Video records for all the major and minor events, archived since the second half of 20th century.

Although they may complain that it's "just a video", not a virtual reality or something. Also, we need to last that long.

Now that video forgery is easy [1], I think the mountain of video footage will make learning history in a few hundred years painful and difficult.

How would you have any idea what happened and what didn't, from the 21st century on?

Yes, historians worked without video or photography for centuries, but most people aren't as rigorous as a historian, and it's going to take a long time for our cultural norms to adjust to video being "default untrusted".

Furthermore, the written word and eyewitness testimony aren't nearly as important in our culture as they used to be, precisely because of the "pics/video or it didn't happen" mindset. Again, that's what historians have relied on for centuries, but it's considered unacceptable now in favor of technology that's just become completely untrustable. There won't be trustable records for people studying our near-future.

Finally, since we've all given up on "primitive" technology like dead trees, most information is stored digitally now. That's great for convenience but terrible for long-term archives [2]. Digital information will vanish if not tended actively and carefully, and historians have long relied on physical artifacts like letters that largely don't exist any more.

Our culture is a nightmare for future historians.

Edit: expanded on why it will be hard to study history in a few centuries

1: https://en.wikipedia.org/wiki/Deepfake 2: http://howicode.nateeag.com/data-preservation.html

I am not a blockchain enthusiast but proof of authenticity is a use case that I've heard. It could be used to prove (for example), that a specific video existed in it's current form at a certain point in time.

That doesn't mean the video is real though!

At least you can track how the propaganda changes focus over time.

True we gotta look on the bright side

I wouldn't be surprised if camera makers were embedding invisible watermarks in their images/videos already.

I would. The market has not demanded this yet, so why on earth would they waste the resources on building it?

I might wish they did, but my experience of life says that product won't exist it until it looks profitable, and maybe not even then.

That aside, watermarking video from cameras buys you nothing in terms of trust.

Not all cameras will have the feature, so videos without it will not be known to be untrustworthy. For cameras that have it, the owners will be able to forge videos using them one way or another, so videos with watermarks will not be known trustworthy.

Chains of trust are fundamentally about humans, and now that forging videos is trivial, we will never be able to safely assume in a vacuum that one is not forged.

It's back to credible eyewitnesses, like we had to rely on for most of history.

With a low definition video that's probably challenging. Maybe it's possible to assert correct physics happening. To my knowledge at least 5-10 years ago Quantum Chromodynamics simulations were so prohibitively expensive that they were simulated on 20x20x20 ("pixel") grids. There aren't that many studies about (boring) Newtonian mechanics but even that provides its complexity challenges. Same holds probably true for most liquids.

I wonder how many people are doing this now and what media they are storing to.

Internet Archive has been recording TV news since 2001:

https://archive.org/details/tvnews

They have projects based on this as well. One of the prominent ones was the Sep 11 Archive. https://archive.org/details/911

Kind of tangentially related but this compilation of TV footage in the morning before the 9/11 attacks [0] is always slightly unsettling to me.

[0] https://www.youtube.com/watch?v=IMVTB2aVUg0

I would bet there are many people with very extensive porn collections. The more specialized, the more likely they would create a library.

I remember the first time I was in awe of storage capabilities was roughly 2003 when someone sent me a screenshot of his pirated porn collection, it was in the hundreds of TBs, at that point already, mostly stored on Disc spindles. I often wonder if he's stopped accumulating at some point.

In /r/datahoarders people are abusing Google Drive Quotas for the same purpose, but the capital required doesn't remotely compare to what it seems some people are willing to invest.

I think private torrent trackers are your best bet when looking for this kind of obscure content.

I think / hope those are archived as well; e.g. what.cd and waffles had a huge archive of all kinds of rare music, bootlegs, obscure bands, limited releases, etcetera. I hope there's an organization that made sure those were archived.

Are museums exempt from copyright laws for archival purposes?

A number of sites filled the void left in place of What.cd and preserve even more content than before. The community is very dedicated.

When I was living in Australia (2007-2012) I had a tv card in my computer so I could watch tv in my room, and I could record another channel while watching. So I used to record shows to mpg to watch later.

I happen to have been doing this since Feb 10 this year, as I've been using a USB DVB card to capture/watch the news. Up to 370 GiB so far, just storing the MPEG-TS streams on hard drives.

Out of curiosity, why?

Why do I watch the news? I don't like the idea of relying on the news sites here (New Zealand) in getting a brief overview of the current stories.

If you meant why I'm keeping them .. I just don't really like deleting things. If the space usage ends up being impractical, I'll probably just end up reencoding them.

Edit: to clarify, by "the news", I mean the hour-long show (though I skip the sports segments) that's on at 18:00 every day; we don't have the "24-hour" news cycle that seems to be common in the US.

Thanks!

Would you consider donating a copy of the TV news programs Internet Archive?

Reddit's /r/datahoarder is full of people archiving all sorts of things that I wouldn't expect to be archived. Recently there was a community effort to archive YouTube's annotations before they were removed. Link here:

https://old.reddit.com/r/DataHoarder/comments/b7imx9/youtube...

I only watch TV through downloads (get_iplayer & youtube-dl), and I have kept it all [edit: only things I downloaded intending to watch, not all TV]. This is since about 2008:

    fs:/volume1/media   14T  8.0T  5.6T  59% /mnt/media

The reason? Why not - disks are cheap. The hard thing is deleting something since I don't know if I might want to watch it again. And at almost no real cost I don't have to make that decision.

Edit: Some stats: 20,269 TV programmes; 11,830 radio programmes; 925 movies

I wish I started doing similar thing way earlier, because now I can't refer to a lot of content that vanished from the net for a number of reasons (creators getting bored, flagged out from youtube by bots that random haters set up, bogus copyright claims from impostors that upload all sorts of stolen things into contentid in order to create a quick buck, UMG taking down videos from music theory channels despite it obviously falling under the Fair Use and so on)

For music on youtube, this trivial ytdl wrapper is essential:

    $ cat bin/youtube-mp3 
    #!/bin/bash -
    exec $HOME/d/youtube-dl/youtube-dl -t -x --audio-format=mp3 "$@"

Saved to my Squeezebox library, it's available to play forever.

Doesn't the audio quality take a hit when you transcode to MP3 though?

Possibly, but on the flip side neither my car nor my first-gen SLIMP3 can play other formats natively.

YouTube audio is usually aac. Sometimes I have saved audio as an .aac file without re-encoding. (Not using the same scripts, what I have done typically is get the mp4 and do: ffmpeg -i foo.mp4 -acodec copy foo.aac)

Any audio player worth its salt should play .aac files. I think for the most common types [low-complexity] the patents are expired. (Curious if somebody has one that doesn't.)

Probably, but it's already down to ~150kb by the time you download it.

Do you maintain any kind of database or additional metadata alongside all of those video files? I only ask because the default filenames generated by youtube-dl don't give you anything more than the title and ID for a particular video and I imagine it's difficult to search among tens of thousands of media files without contextual data about the files.

No, just filenames and whatever metadata is stored inside the file. For get_iplayer this isn't very much unless you have "AtomicParsely" installed too, but I uninstalled AP after it segfaulted once too often. Note for BBC TV programmes you can search on the "PID" (which is part of the filename), eg: try searching for "m000427z".

However I also download subtitles where available so I guess I could search those as they are just text.

yt-dl has a full-fledged output template system. It's just the default output template that is only the title+id.

https://github.com/ytdl-org/youtube-dl/blob/master/README.md...

Also, it has the --write-* switches, in particular --write-info-json, which allow to store metadata data. in files, and --add-metadata to embed (some) of the metadata in the file itself.

Of course, the metadata is only as good as the site in question provides + the yt-dl plugin handing a particular site supports.

I was familiar with the -o template system for filenames, but wasn't aware of the --write-info-json flag which is very useful to me. It makes a lot more sense than overloading the filenames with usernames and everything else only to parse it out after download.

Nice man!

Would you consider donating a copy of the TV news programs and radio to the internet archive?

Stephen Wolfram (the man behind Mathematica) had this odd but fascinating blog entry:

https://blog.stephenwolfram.com/2012/03/the-personal-analyti...

He captured every keystroke (over 100 million) on his computer.

This is a great display of Mathematica's power in data science and plotting. Somehow I'm feeling compelled to reach for my wallet....

> Somehow I'm feeling compelled to reach for my wallet....

Which is a great display of Mathematica's power over people's wallets

Oh. If I didn't know any better I would suspect Stephen Wolfram made a blogpost for shamelessly advertising the data science capabilities of Mathematica for the purpose of selling it to data scientists.

I think the blogposts are being generated by a collection of cellular automata that are being created by a learning algorithm with access to that dataset. In other words, Stephen Wolfram.

Anyone know if this has been digitized yet or when it will be available? I am so impressed by this woman’s work. So much of our history is being lost, forgotten, or is inaccessible by the general public.

It was mentioned in the article that the Internet Archive is in the process of digitizing it. No mentioned of a time frame.

Closest I ever came was recording a few weeks of MTV's 120 minutes.

I still have a bunch of VHS tapes of 120 minutes

Send it to the Internet Archive.

I've actually been searching for tapes of 120 minutes, since all you can usually find online are snippets.

Did you ever digitize them?

> all you can usually find online are snippets

I haven't looked too closely, but what's this then:

https://120minutes.tylerc.com/

This is basically an index of the music videos played, not an actual recording of the broadcasts

There's another comment at the same level as yours that you might be interested in: https://news.ycombinator.com/item?id=19855812

I recorded much of AMP.

Content that includes other content is usually a licensing mess so at high risk of unavailability.

I always thought the original draw of Beavis and Butthead was the MST3K approach to music videos, but those are often stripped from current versions.

The State, WKRP, and The Wonder Years all lost great original soundtracks too:

https://www.techdirt.com/articles/20110801/11062015346/new-b...

Yes, I would so love to see Beavis and Butthead again as originally broadcast!

Reply on news.ycombinator.com