Show HN: MovieChat.org – Archive and Replacement for IMDb Message Boards
moviechat.org
moviechat.org
My name’s Jim, and I created MovieChat.org as an archive and replacement for IMDB’s message boards which are shutting down this week. For those of you not familiar, the IMDb message boards allowed you to discuss any single movie or tv show with others (there was a separate forum for each movie/show). IMDb recently announced they were shutting down the message boards and its users were furious (there's a petition with close to 10k signatures here: https://www.ipetitions.com/petition/petition-to-keep-the-imd...). I ventured out to create an archive (of all the existing posts) and replacement and hence MovieChat.org was born.
Key Features of MovieChat.org:
1. Any movie/show on IMDB is also on MovieChat.org (over 4 million and counting) - we have separate boards for each movie/show, just like IMDB
2. I backed up most of the posts for IMDB’s top 10,000 movies/shows - most existing conversations on IMDB should also appear on MovieChat.org - we have over 3 million posts already (and I'm working non-stop to back up even more from IMDB)!
Please visit http://MovieChat.org, join or start a discussion, and let me know what you think. If you like it, please spread the word. If there’s anything I can improve, just email me (jim@moviechat.org) and I’ll get on it.
Jim jim@moviechat.org
Just throwing it out there, but would you consider making a dump of the data you scraped that could be used by data scientists? Maybe as a torrent or something like that? Data about movies and what people say about them could form the basis of a lot of NLP projects.
What other big datasets are there for forum post text data? The reddit dataset most immediately comes to mind, and I've also seen a similar one for HN comments. Any others?
Thank you!
Edit: I see the other comment about archive team already collecting and releasing this data, for free in an open format. I think that will be a good first source as well.
One actually hopes that IMDB doesn't in fact have any claim on posts written by users that happened to be hosted on their site.
"...disable our message boards on February 3, 2017 but will leave them open for two additional weeks so that users will have ample time to archive any message board content they'd like to keep for personal use."
Note the "personal use" above. Is a website like MovieChat.org considered personal use?
From http://www.imdb.com/conditions , section Copyright: "The compilation of all content included in or made available through any IMDb Service is the exclusive property of IMDb...".
I did my homework before starting this project and I'm confident we're in a good position :)
The users did not give you permission to use their data, and you're not associated with IMDb, and therefore do not inherit the rights.
Lastly, IMDb specifically states that scraping or extracting data is forbidden without written consent.
I'm sure there's a story there since I would have thought the usual direction, even for a small aggregate number of eventual engineer-patent attorney pairings, would be reversed. Hopefully you can post an "About Us" on MovieChat.org at some point when you feel comfortable. Thanks!
Who owns this post data? IMDb? The users? Do you have any plans or ideas for existing IMDb users to somehow claim their username on MovieChat, linking to their archived posts?
Totally agree. I often also went there before watching a TV show or movie to get an idea what it was about or if I'd like it (though wary of spoiler potential), whereas I almost never read the reviews there, which I find are often far more useless than the message board comments.
After they're gone IMDB will be just a soulless movie listing site, with no reason to visit other than to check a mark out of ten, and no reason to stick around afterwards. :/
one item i didnt like is that if i search for a show, i get hits for each episode.
would be good to have one entry per show, and then you can drill down into individual episodes.
also would be good feature on ability to rate shows (as a registered user, but i guess IMDB is still offering that functionality)
For it to work I would need for Jim to enable Access-Control-Allow-Origin from imdb.com (and ideally make a json/xml version of the thread list since I'm regexp'ing the html at this point).
Maybe it would help persuade him if we showed that there was some interest for this.
I'm surprised there isn't enough interest in people to recognize tMDB as relevant. Especially since iMDB is ignoring users by taking down the forums.
I just heard about the IMDb message board deletion and was feeling deeply sad about it. I love cinema, and their message board (despite its flaws) was a valuable resource. It was the only place where you could go to participate in in-depth discussions of even the most obscure films. I used it for 14 years and learned so much from other users.
As far as I'm concerned you are a SAINT for preserving this bit of film and internet history. Major gratitude to you for your caring and putting the new site together.
I honestly want to cry, I am so relieved.
it's nice you created searchable archive of IMDb discussions though guys at Archive are doing same anyway, not posing as alternative and I am not feeling very happy sometime took 14 years of my IMDb posts and put them on his website without my agreement
However, surely rule #1 of the internet is, once you've posted it, it's out there forever. Are you going to go around to every cache server, and everyone's browser cache and demand that your comments be deleted from there as well ?
as much as i hate IMDb shutting down boards i hope they will close this illegal website
OTOH i have no problem with recognized Archive.org instead of this solo thief
However, I'm curious about exactly how you're going to promote this site. The people currently on IMDB likely don't know much about it, and it's unlikely the administration there will redirect people over when they view the pages for a movie or TV show on their site.
What's the plan to get the users whose data you scraped to come to this new domain and participate again?
- How about linking back to the IMDB page for each movie?
- It would be great if you displayed various ratings from IMDB, Rotten Tomatoes and elsewhere kind of like JustWatch.com does (but with links to those ratings pages).
- Actually, it would be really cool if you linked to a JustWatch search too so the user can easily see if the movie is streaming anywhere that they subscribe to.
Another crazy thought - a merger with JustWatch would be awesome too (but I really wish JustWatch would also link back to IMDB, Rotten Tomatoes, etc).
Do you know the founders at JustWatch?
But rest assured we're working heavily behind the scenes to make our site & UX better and better every day with free and open movie data sources - but always trying to stay safe on the legal side.
To my knowledge this is the only instance someone has managed to scrape the Imdb boards, very curious how you managed that.
I really hope you get a sponsor/donations. This needs to be promoted to other sites/social media and I will do my part.
I do have one request - can we do this for actors as well?
You can even easily join in if you want.
---
[1] To populate my Kodi .nfo files in a way that I was happy with. I know scraping is probably against IMDB t&cs, but as this was for my own use, I took the risk.
I am not comfortable running a daily, task scheduler based, third party script with local admin rights, on my Windows Server, that checks my Lets Encrypt certs and auto renews them.
However, now that Chrome v56+ is revoking StartSSL.com certs, I've probably got no choice. Either that, or actually PAY for a multi-domain cert (Comodo do a fairly cheap one).
Why? Just have it drop the certs in some directory and then reload the services. Just give the account running the script permissions to only reload services.
I assume such is possible on Windows, but I don't know for sure because I only use Linux servers. It is trivial there, so I assume you can do it on Windows as well.
Despite your comment and the (well-intentioned) plans from browser vendors to do what they can to squash unencrypted HTTP, failure to use HTTPS, even with Let's Encrypt, is still a totally forgivable sin today.
With nginx:
- `certbot certonly`
- press `2`
- type in your domain name
- press return, done
Add a few lines to your nginx config, done.
``` ssl_certificate /etc/letsencrypt/live/<yourdomain>/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/<yourdomain>/privkey.pem; ssl on; ```
> `certbot certonly`
You make an assumption that this is going to work with no hiccups, across all environments. Spoiler: it doesn't. I specifically alluded to certbot and certbot-auto being (surprisingly) rough around the edges.
Now, it is possible that you don't agree with that specific demand, and that something that involves switching the backend is merely one viable option—it's sufficient and not necessary to achieve that goal. And that's fine. But as someone who showed up to throw in their support for the claim that it's inexcusable not to be using TLS today, then the burden reverts back to you to justify the claim.
So, I ask you, as someone on record as disagreeing that it's still forgivable to be running an HTTP-only site in 2017: what are, as you see it, the minimum reasonable demands to be placed on someone operating a website?
But to answer your question: I think if you are running a message board / forum for people to discuss various topics in general you should try to keep your users as safe as possible within your means. That means https, no plain text passwords in the database - basic stuff really.
PS: The DNS on your website as linked in the profile isn't set. Only the www. subdomain works.
Are we talking about the suitability of operating any unencrypted endpoint without TLS, or are we limiting ourselves to message boards? If the latter, does that mean there are forgivable reasons to run HTTP-only applications that aren't message boards?
> The DNS on your website as linked in the profile isn't set
DNS is set up, it just doesn't have an A record or CNAME for the bare (second level) domain. That's intentional.
> Only the www. subdomain works
The other subdomains are certainly working.
> I'm not sure why you even want to discuss the details of my simple suggestion or "demand" at length like that
Because in response to this:
> failure to use HTTPS[...] is still a totally forgivable sin today
You said this:
> I disagree
In any case, I like the project but I think that in 2017 it's irresponsible to do logins and user registration over http - even if it's just a weekend project.