Hacker News Reader

Looks like they offer a full data dump: https://dl.opensubtitles.org/addons/export/

I haven't actually tried to use it. So long as they offer a dump you could use I think it's entirely reasonable to require payment for the API.

I think it's reasonable to consider them morally open even if the web UI is closed source if the subtitles are open. It feels to me like a project selling a shiny hosted app around an open source project. I think in this sort of project the core "source" is the subtitles, not the web UI.

If OpenAI let you download their models but didn't provide the tooling you'd need to run them I think their name would be much more reasonable.

That's not a full dump of subtitles, but rather of list of which subtitles they have, with download links. Here are the first lines of the `subtitles_all.txt` file:

    IDSubtitle MovieName MovieYear LanguageName ISO639 SubAddDate ImdbID SubFormat SubSumCD MovieReleaseName MovieFPS SeriesSeason SeriesEpisode SeriesIMDBParent MovieKind URL
    1 Alien3 1992 English en 2004-10-31 23:54:23 103644 sub 2  Alien.3 11.000 0 0 0 movie http://www.opensubtitles.org/subtitles/1/alien3-en
    2 Identity 2003 Slovenian sl 2004-10-31 23:54:23 309698 sub 1  Identity.DVDRiP.XViD 0.000 0 0 0 movie http://www.opensubtitles.org/subtitles/2/identity-sl
    3 Ghost in the Shell 2: Innocence 2004 English en 2004-10-31 23:54:23 347246 srt 1  Innocence.2004.DVDRip.XviD 23.980 0 0 0 movie http://www.opensubtitles.org/subtitles/3/ghost-in-the-shell-2-innocence-en

Yeah, I was wondering if that could be the full dump. I know how far compression algorithms have come, but 318MB for all the subtitles of the world? Would be interesting to know though how large a full dump would be.

My guess is not by much. Text doesn't require a lot of storage. After looking at a few random files my estimate is that the size of the full compressed dump would be about 30x (~10 GB)

EleutherAI cleaned an OpenSubtitles dataset that was 12.98GiB (13.9GB).

https://arxiv.org/abs/2101.00027

https://lifearchitect.ai/whats-in-my-ai/

When I see "open" I read it as "I can do what I want with it". I don't read it as any obligation on the part of the provider/creator to incur ongoing costs on my behalf.

"Open" as in "open beer", so to speak.

As in, “Going stale pretty soon“, you mean?

I feel like every time something has started with the name “Open”, the promise that lured people in (sometimes implicit, sometimes explicit) has been broken.

Open Exchange Rates: the fact that you didn’t need an API key was an important part of their gaining popularity. Of course, this fell down at some point, at least partly due to stupidly-coded clients, and so they started requiring an API key, and completely hobbled the free plan for many previously-recommended uses (only hourly updates and 1,000 requests per month; a month is up to 744 hours, so they really are just saying “if you want to use this broadly, either pay us $97/month, or cache it yourself hourly, proxying it if you have multiple access points”). Not very open.

OpenAI: eh, you probably already know the story. They broke their word and premise thoroughly.

"Open" means if you already have a copy, you are free to share it as you like. It doesn't mean someone is obligated to spend money giving it to you. The FSF always charged a profit rate on Emacs tapes.

Like some open source projects moving to a "source available" or "fair code" licenses, it is going to be bumpy finding equilibrium as to what is reasonable. Tragedy of the commons for sure. There are costs to cover, and usage profiles can vary wildly.

And when they do that people just fork and go along with their own. Tofu being a great example, opensearch somewhat less.

By “people” you mean organizations with budgets and stakeholders.

Very few people, even few programmers, would be able to take on maintaining complicated software like tofu alone, for example.

More like Reddit. The contributors will just leave and you'll be left with a corpse of content to rip going forward. Tofu is different because of the broad contributor base around the Terraform ecosystem. Lots of people's jobs (or companies even, Spacelift for example) depend on Tofu not only existing, but improving into the future.

If you have a benevolent dictator with deep pockets, throttling usage and retrieval is less of a concern. The Internet Archive comes to mind, as does Wikipedia. Reddit's outcome happened because they were a discussion board utility masquerading as a 10-100x VC funded business, and their actions drove away the concentration of mods and contributors who were putting in all the free work (broadly speaking).

Did anybody (outside of the tech/HN open-source-die-hard bubble) actually leave Reddit?

From what I saw, there were protests which most Redit users barely noticed (the front-page just served content from the subreddits that didn't go private), and then there was nothing. The long-term impact seems to be small to nonexistent.

Maybe if the users organized better and started posting "brand unsafe" content everywhere instead of just going dark, the effects (and especially the effects on advertisers and Reddit profitability) would have been different. One can dream, though.

Hard to say without Reddit showing their traffic cards, just anecdotal evidence. Admittedly, if you're consuming big subs similar to what you'd consume on Tiktok (r/funny, r/worldnews, etc), likely little impact to traffic. I know the mods from some of the subs I follow fled (admittedly smaller readership, r/econmonitor and r/machinelearning for example). Reddit will have to show their hand when they attempt to IPO, as they then tread into securities fraud territory if the metrics are not accurate.

https://news.ycombinator.com/item?id=36289290 (Ask HN: Anyone at Reddit can confirm a traffic dip?)

(n=1, I consume Reddit now with a Lemmy mirror of the subs I'm interested in)

Is it easy to find Lemmy mirrors? I left reddit but haven't really replaced it.

https://lemmyverse.net/communities

https://github.com/maltfield/awesome-lemmy-instances

https://github.com/mushroomlabs/fediverser

https://www.google.com/search?q=reddit+lemmy+mirrors+site%3A...

I'm addicted to the logged-out Reddit front page and I can tell you that besides a few subreddits committing suicide and being immediately replaced by similarly named subreddits that didn't care about the boycott, and a few days where more interesting content was getting surfaced when some big subs went dark, there has been absolutely zero noticeable effect on the average user experience.

I would be -shocked- if Reddit had data showing that the protest was more than a fart in the wind

Yes. The long-term impact of the quality of posts in interest subreddits is immediately noticeable and staggering. This is not even to speak of the moderators who are no longer participating.

For my feed, a ton of the NSFW subreddits went away, became so low quality that following them was pointless as the mods left or engagement dropped to ridiculous levels, even the OG big tankers.

r/gonewild for example has nominal 4.6 million subscribers - but among the default "hot" feed on the first page there's just four posts with > 500 karma, and on the 2nd page there's tons of posts with 80-120 karma. That's insanely low, back in ye olde days a simple boobie could easily hit 2k karma in a matter of hours, and there was pages and pages of highly upvoted content.

Now, I can't say whether this is a consequence of users leaving, mods leaving, Reddit manually downranking NSFW sub engagements, or the massive increase of ragebait redirecting attention and engagement - r/europe is filled with utterly vile comments on everything regarding immigration and Islam, r/therewasanattempt even managed to get geobanned in Germany because they put up the (banned) Hamas slogan on their banner [1].

(In the German Reddit sphere, it seems that additionally Reddit has created a ton of outright clones of popular English subs like AitA, DIY, legaladvice, ... that further remove engagement from the large subs and redirect it to the small new ones - this "pushing" of new subs likely also drives NSFW subs with low engagement down the feed)

[1] https://old.reddit.com/r/Israel/comments/17vqch7/rtherewasan...

I follow r/europe and I have also noticed a huge change. What once seemed to be a pretty open subreddit appears to have turned into exactly what you mentioned, with vile comments on immigration and Islam on almost every thread.

I did start to wonder if people's viewpoint on immigration has changed as the narrative that Europe is "saturated" with immigrants seems to be being pushed a lot at them moment, but I have a feeling that the audience of Reddit and r/europe seems to be changing. I wish there was some way to know for sure.

It's due to the extreme right wing and other opportunists fear mongering and inflaming people's grievances with an easy scapegoat that cannot defend itself.

Social media of all kinds are effective tools in this. Vilification of minorities works, it always has, but those who would exploit it haven't had access to the public forums required to promote it. Now they have.

Nuclear weapons won't destroy the earth, social media will.

I think a lot of Reddit's earlier userbase was more lefty types and that also overlaps with the group most annoyed by the API changes since they'd gotten used to what Reddit used to be. So more of the left wing users left, resulting in the attitude shift you're seeing. Even the right wing users on early reddit were more right-libertarian than right-conservative or alt-right initially - Ron Paul was one of the first political figures to have a big following there.

That said, there is also a shift to the right in real life, as seen by the rise of groups like AfD, Brothers of Italy or Sweden Democrats.

r/europe also has a bit of a weird history with this too. It started relatively left wing in the early days of reddit, drifting to the right until the mods started clamping down on the more overtly racist stuff. At this point, a lot of the posters involved moved to r/european, which sort of reset r/europe back leftward a bit again. But then r/european was banned around the same time as r/the_donald for much the same reasons, and those users drifted back into r/europe which shifted it to the right a second time.

The fact that a banner with the slogan "from the river to the sea" is banned in Germany speaks to the massive self-defeating overreaction by some parts of the West. This sort of thing is only promoting a spiral into a wider war.

It should also be noted that it's not a "Hamas slogan" but has a much wider use:

https://en.wikipedia.org/wiki/From_the_river_to_the_sea

> The fact that a banner with the slogan "from the river to the sea" is banned in Germany speaks to the massive self-defeating overreaction by some parts of the West.

Germany, and Europe in general, has the cultural understanding (as a consequence of 1933-1945) that no, not all speech has to be allowed - particularly not when it calls for or supports terrorism.

And that is precisely the context in where this slogan is currently being used.

Do you mean because of use by the Israeli right wing (Likud) or by Hammas?

"Kill Jews" is clearly evil and should be banned, but banning this phrase ends up smearing ordinary Palestinians who use it to describe an non-violent aspiration, as did the Israelis.

As I said, an over-reaction.

I haven't left reddit, per se, but I browse it less and comment less. I doubt I'm alone. Unfortunately, there's not really a replacement for much of what it offered.

This is why I love technologies like BitTorrent and IPFS, where it's not a single server bearing the brunt and costs of being popular, everyone is contributing. The more popular it is, the more performant it is.

> It's a question I've never considered before -- should "open" mean "you can API-download everything all the time anonymously no limits", or "you're free to API-download a reasonable amount for consumer purposes with an account, and the more you upload the more you can download"?

Neither, Open should mean that the resources are publicly licensed such that they can be freely rehosted or redistributed. It should mean that the resources are owned by the community rather than the website operator, and that the website should be selling coordination, not access.

Whether or not there's an API limit is irrelevant provided that:

A) the website does not claim ownership over resources it hosts

B) the website is not going to go after people hosting or distributing those resources outside of the site -- ie, users have explicit permission to distribute any and all of the subtitle content on the site

C) there is a way to get the resources off of the site, ie it's not impossible for someone to set up a rehost.

If OpenSubtitles still allows a bulk download or has a mirror, then honestly I don't see a problem with them making this change -- I'm not a fan of it, but if the subtitles are still downloadable in bulk and if OpenSubtitles isn't claiming ownership and would have no issue with someone setting up a separate service or rehosting those subtitles and if (to the degree they can be licensed openly, given that they're subtitles of copyrighted films) they are licensed openly using an OSI-compatible or comparable license in the case of the GPL, then I don't have a problem with them using the word "Open" and I think it still applies.

Having said that, I can't immediately find any information on their website about what copyrights they do or don't try to claim, and it's not clear to me how I would go about building a full dump of the content (a sibling mentions a dump but that it doesn't contain the full subtitles). But that would be my bigger beef. I'm less concerned about whether the website is Open and I'm more concerned about whether the subtitles are Open (again, as open as movie subtitles can be).

Exactly. Open doesn't mean "we will pay all of the bills so that you can enjoy these resources for free".

Open means "we will not take any legal action against sharing and/or modifying these files".

Open projects can have all the ratelimits and paid APIs that they want, as long as they are okay with people setting up mirrors and re-hosting content for free.

And notably there are some "open" services like Doesthedogdie that should not be considered Open even though as far as I know they don't have rate limits on their APIs or force users to log in. They still claim ownership of all of the content, you don't have permission to take that info and mirror it to another site.

Goodreads also springs to mind (although I guess Amazon has started monetizing that now, so it might not be as great of an example). The immediate access matters less to me than Amazon looking at all those user reviews and recommendations and saying, "we own them."

In an age of companies looking to cash in wherever they can on AI plays based on free data, tiers start to make sense to keep the services viable.

For consumers with normal usage, on-demand downloading of subtitles for 5 movies in a day seems pretty good. For TV, not so much. Even rate limits would be fine. A movie can be downloaded every hour, a TV show every 15 minutes.

For a media server that wants a local cache of the subtitles, it could still run through with those rate limits in the background while you’re not watching. It could be a slight annoyance when first setting it up, but would be a one time thing and not much of an issue for one-off adds over time.

I was messing around to mass download stuff from a wallpaper site with an API about a year ago, just because I’m a data hoarder and was worried about it going away at some point. The API required auth and had a rate limit, but as a consumer, it was fine. I just added in a 2 second wait between downloads and had no issues. It took longer, but what do I care? After a week of screwing around with it I saw I downloaded about 50gb of wallpapers, which was more than I wanted. I felt kind of bad about it and stopped. I decided to be a bit more intentional on my downloads after that. The tool I made let me put it a tag and download all the results. That was a bit too greedy.