imdb-sql.com
That said, it would be amazing to have (anonymized) data about raw ratings per person, and be able to build your own personal recommenders based on that... One can dream!
Is it or are they just trying to preempt a more open alternative.
Although is cannot be queried in SQL, but SPARQL instead https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
You will get a better result.
SELECT * EXCLUDE (titleType, primaryTitle, language) FROM 'imdb01-11-2024.parquet' WHERE (region is null and titleType = 'movie') ORDER BY ( (numVotes * averageRating)+(1000007))/ (numVotes+1000000) desc LIMIT 100
P.S. Query sharing is supported:
https://www.imdb-sql.com/?query=SELECT+*+EXCLUDE+%28titleTyp...
SELECT * EXCLUDE (titleType, primaryTitle, language) FROM 'imdb01-11-2024.parquet' WHERE (region is null and titleType = 'movie') ORDER BY ( (numVotes * averageRating)+(100000 * 7))/ (numVotes+100000) desc LIMIT 100
https://www.imdb-sql.com/?query=SELECT+*+EXCLUDE+%28titleTyp...
I feel like I'd pay money for Netflix to add that option.
"I want to watch a science fiction movie that is not a super hero movie. Movies I love are Dark City and Daybreakers. Prioritize movies of short length, nothing more than 2h30. Filter out movies I've already rated, here's my Watchlist"
The movie has countless things we know from much later productions and countless things not seen elsewhere.
It takes place in the distant future, 1980, a place where everyone has a flying car, people have numbers in stead of names, the government decides who you can marry and babies come out of vending machines.
One IMDB comment described it as: I sat on the tip of my chair wondering if it could get any stranger and then it did!
Currently there are only few columns, can there be more e.g. I was looking for movies based on age rating for example (for my kid or myself)
It has these columns: https://www.imdb-sql.com/?query=SELECT+*+FROM+%27imdb01-11-2...
It is produced using https://github.com/afiodorov/imdb-sql/blob/main/notebooks/im...
From this data: https://developer.imdb.com/non-commercial-datasets/
The age rating is not part of the imdb non-commercial dataset.
I've also been working on something recently to quickly search in the IMDB datasets. No hosted version yet though.
https://github.com/jeroenpelgrims/imdbsearch
Mainly also to learn some rust/htmx.
P.S. We can share links too https://www.imdb-sql.com/?query=select+1
The SQL flavor is https://duckdb.org/docs/sql/query_syntax/select
I'm not quite sure what you mean by this.
If you gave the SQL a light gray border and perhaps overflow-y:scroll, it would make it much more obvious that it can be edited directly without needing any other adjustments.
If anyone with actual css experience wants to open a PR in the meantime I'll be very happy!
I just run npm run dev locally. You'll need the parquet file of the data in public/ though, can get it from http://www.imdb-sql.com/imdb01-11-2024.parquet
That being said it's an impressive demo of what the browser can do without breaking a sweat! Will try and discover some new stuff to watch now!
However titles are repeated often due to the region/language variations.
Although "Untitled Pixar Animation Project" is basically garbage data, but that's a harder problem to solve...
would be even better if modifications to the actual select part would somehow persist when building the query
Not sure what you mean by scraping in parallel.
Maybe I'm just an old fart, and blowing 10GB of data is cheap nowadays?
I'm surprised some users think it's a lot on an individual scale. Consider an ideal persona for this submission: query some movies, then go stream several GBs for one. The 100 MB payload isn't much in comparison. I admit it is kind of bad form for mobile users who might be on metered data plans, and a warning and trigger for manual action for those devices would be kinder.
One thing that might help reset your notion of what a lot of bandwidth is would be to browse around with your network tab in the developer console open for a day. nytimes home page is 14 MB and they get a ton of traffic. Even a corporate blog on the HN front page right now that could just be the tiny compressed text is 2 MB. Single image loads on many pages can be 1 MB or more. Glancing at the submission, the response headers indicate it seems they're serving from S3 Cloudfront, which is free for the first TB per month, though after that it gets back to absurd pricing. AWS is not price competitive.
Exclude movies with very low number of rating or potentially very low scores too.
The long tail reduction would be significant
That’s an irresponsible waste of both bandwidth and storage. You should have really made that clear in the website.
It is on hosted on http://www.imdb-sql.com/imdb01-11-2024.parquet
In fact the main reason this project exists is
a) I wrote a jupyter notebook ages ago that'd join the raw data into a queryable form.
b) I committed it and forgot about it after the initial viewing of top movies/series.
c) For the Halloween I wanted to find a good-rated horror movie, a genre I don't watch much.
d) I found my notebook but it was a drag to get it running again, first pandas would keep throwing OOM errors so I had to migrate to polars. Secondly I had to find a spare working laptop since iPad is my primary off-work computing device as of late. Finally the schema is not so intuitive - I need to look up things like region/language codes.
I want something I can just leave unattended but still be able to use a few months later without going through a dev-env set-up. A static website with all the data is one potential way of avoiding maintaining it as it should keep running, just like an exe would keep running on Windows in the past.
I avoid maintaining personal projects as my software job requires me to maintain software and I don't find it enjoyable to do the same in my spare time.
For example, I found "Paris, Texas" to be pretty disappointing, yet most reviews were overwhelmingly positive. So, I sought out others who also didn't enjoy it and explored their review history to find movies we both agreed on.
Occasionally, you'll hit the jackpot and find an avid reviewer whose taste aligns perfectly with yours, providing a treasure trove of excellent movie recommendations.
I like to call these users my "IMDb doppelgangers."
Another plus was you didn’t even need to create a profile. Maybe that was their downfall