Hacker News Reader

Comment by Pigalowda | Hacker News Reader

I had never used youtube-dl until the story happened. I downloaded for windows and its speeds were throttled to around 50kB/s. Posters on Stack recommended the ytdlp fork which i tried and it was 5-10 MB/s. Just fyi

View on HN

Yeah youtube-dl is missing workarounds for throttling implemented by google. youtube-dl is pretty much unmaintained compared to yt-dlp:

https://github.com/ytdl-org/youtube-dl/graphs/commit-activit...

https://github.com/yt-dlp/yt-dlp/graphs/commit-activity

And yt-dlp uses an 'Android API' to stop throttling

I expect the ability to use APIs like this will be hindered once remote attestation becomes the norm.

This is reason why we must diversify these techs. If same company builds everything like mobile, os, search engine, Youtube, it would have debacle in future. If there is diversification, it simply takes toil and sweat to convince everyone.

One has to execute functions from base.js^1 to modify the "n" URL parameter and the "sig" parameter to get the fastest download speeds. (One can still download videos with the original n parameter, or without the n parameter, but download speeds will be slower.)

By itself, using the ANDROID API instead of the WEB API^2 does nothing to affect download speeds. I can block yt-dlp's POST request indicating the client and API name and this has no effect on download speed.

A website that forces users to run Javascript in order to get faster download speeds. This is not a new idea.

1. https://www.youtube.com/s/player/{player_version}/player_ias...

2. From the YouTube video page JSON:

ANDROID API KEY

AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w

WEB API KEY

AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8

yt-dl and yt-dlp use Python to interpet the Javascript functions that modify "n" and "sig", but a faster language could be used. For example, V8 is written in C++.

One of the reasons yt-dl and yt-dlp are so slow is because they do (too) many do other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved, but waiting for Python to start up and do this is slow. A YouTube video page can be retrieved much faster using netcat, outside of Python. YouTube video can be downloaded much faster using an HTTP client like tnftp, directly, outside of Python. YouTube video can be converted much faster using ffmpeg, directly, outside of Python. And so on. These programs start instantaniously when compared to the slow start up time of Python. The Python startup latency is unbearable.

At the very least yt-dl and yt-dlp should accept an already downloaded YouTube video page as input instead of forcing the user to use Python (or Python calling another program) to download the page. The recalculation of "n" and "sig" do not have to occur within seconds of retrieving the video page. There is no need to require the user to use Python for downloading webpages. The values in the YouTube video page are good for a substantial amount of time.

Both scripts have an option to output the new n and sig values in a download URL or JSON containing the download URLs, so the user can use an HTTP client directly, outside of Python. But the user still has to use Python to download each video page. Using an HTTP client directly would be faster.

Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill.

If you find yt-dlp particularly slow, you can use aria2c to perform the download itself. It doesn't solve any startup latency problems, but it generally can do the HTTP requests faster.

This is complete nonsense. The overwhelming majority of video download time is spent transferring the actual video data, limited by throttling or network speed. The time it takes for these tools to start up and deal with the API/JS stuff is inconsequential for any video longer than a minute or so. And even then most of the time is network latency for the API stuff, not local processing. Netcat isn't going to go any faster.

Absolutely nobody thinks optimizing the meta/API processing of yt-dlp & co is worth it. This is exactly why we have high level programming languages that make all of this much easier, instead of trying to write HTML and JS parsing in plain C. Keep in mind these tools support dozens or hundreds or websites, not just YouTube.

If you think rewriting yt-dlp in C is worth it, go right ahead, but you're not going to make it significantly faster; you're just going to make maintenance a much bigger pain for yourself. Pick the right tool for the job. Python is absolutely (one of) the right tools for this job.

(For the record: I use C and Python on a daily basis, and will use whatever language is appropriate for each situation.)

"Python is absolutely (one of) the right tools for this job."

This opinion assumes it is a single job. I see multiple jobs. The number "options" provided by yt-dl(p) gives us a clue.

There is nothing wrong with preferring to use larger, more complicated, "multi-purpose" utilities. There will always be plenty to choose from.

However the idea of using smaller, less complicated, single purpose utilities is not "nonsense". It makes sense in many cases and some users may prefer it.

The statements I make about speed are from day-to-day experience not conjecture.

I look forward to seeing your implementation of this solution.

What is especially slow on some hardware like the original PinePhone is that youtube-dl and yt-dlp are actually executable that extract themselves at runtime. Python itself is quite fast to start.

The startup time for netcat is less than for yt-dl/yt-dlp.

Plus I can request multiple video pages over a single TCP connection with netcat.

For example, in a single TCP connection, with a list of 30 videoIds, I can get initial sig and n values for 413 videoIds. No wait time for netcat to decompress or startup. Using netcat is quite fast. Then I have utilties written in C to extract URLs from stdin. As such, all I need is a utility to update the sig and n values in the download URLs to make them fast ones instead of throttled.

How long would it take to get sig and n values for 30 videoIds let alone 413 with yt-dl or yt-dlp over a single TCP connection. The startup time for yt-dl/yt-dlp plus the time waiting for the downloading of each video page makes it much, much slower. These are "do-everything" scripts that are as a result quite inflexible.

Fixed typos/misspellings, edited for more clarity:

yt-dl and yt-dlp use Python to run the Javascript functions that modify the initial "n" and "sig" values to make non-throttled download URLs, but a faster language could be used to interpret and run the snippet of Javascript. For example, V8 is written in C++, not Python.

One of the reasons yt-dl and yt-dlp are so slow, IMO, is because they do (too) many other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved. I have self-created utilities for downloading webpages wth TCP clients that are sgnificantly faster and more flexible than yt-dl/yt-dlp. I use these small dedicated programs every day. I am used to the speed. Waiting for yt-dl/yt-dlp to start up in order to download web pages is annoying. The latency is unbearable.

Both yt-dl and yt-dlp have an option to output the new n and sig values in a download URL or as JSON containing the download URLs, so the user can use an HTTP client directly to download video, wthout need to launch yt-dl/yt-dlp. But the user still has to use yt-dl/yt-dlp to download each video page. Using a TCP client directly would be faster.

Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill. It would be nice to have a program that just focuses on running the inecessary JS in base.js in order to output optimal download URLs. Then the user can use whatever programs she wants, directly, for downloading HTML/JSON, extracting URLs, downloading video files, converting video files, etc. Downloading video from most websites is generally easy. I never need a program like yt-dl/yt-dlp. It is only YouTube that plays games with users trying to get them to enable Javascript and be tracked. One need only look at the size of the extractors in yt-dl/yt-dlp as evidence. The extractor for YouTube is 3x the size of the next largest, and over 10-20x the size of most of the others. I want a small utility that just focuses on YouTube. A simpler solution instead of a massive project.

There is a pure js version here: https://www.npmjs.com/package/ytdl-core

This is why I come on HN, I always learn something useful.

The workaround for vanilla youtube-dl is to use it with aria2, with options like:

  --external-downloader aria2c                                                                                         
  --external-downloader-args "--continue --max-concurrent-downloads=3 --max-connection-per-server=3 --split 3 --min-split-size 1M"

(possibly in your config file)

Also, neither --format best nor --format bestvideo chooses the best encoding in all cases; they use bitrate as a heuristic for quality, and a less efficient codec can have higher bitrate but worse quality, resolution, or framerate. The workaround for this is specifying --format with an enumeration of every combination of codec, resolution, and framerate in preferred order, which goes like this:

  --format "(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320] ...

Here's a full example (hmm... they're using it with yt-dlp, which I thought had fixed this?):

https://github.com/TheFrenchGhosty/TheFrenchGhostys-Ultimate...

I think there's a bit of variation in the exact order among the config files found online. If you're goals are archival, consider also retrieving metadata, thumbnail, and subtitles in all languages; I also have in my config the options:

  --verbose
  --download-archive ./ytdl-archive.txt                                                                                                                       
  --cookies ./ytdl-cookies.txt   
  --merge-output-format mkv
  --add-metadata --all-subs --embed-subs                                                                                                          
  --write-info-json --write-thumbnail                                                                                                             
  --no-overwrites --continue                                                                                                                      
  --force-ipv4

(the only remaining workaround for age-restricted videos is to give it cookies extracted from a browser with a real Google account logged in)

Am I the only one totally chill with the throttling? Look, I'm not sure if it's the best idea for their own resource utilitization (there are probably peak times when more throttline is better, and quiet internet times when throttling doesn't make sense) As long as they are being gracious hosts and allowing downloads, I'm good with being throttled I guess?

50 kB/s means you'll be downloading a 10min video like 10 hours.

> 50 kB/s means you'll be downloading a 10min video like 10 hours

Very misleading phrasing: you would download in 10 hours a ~10 hours long video, which (of course) could have been downloaded in a fraction of the time.

The throttling has the user download at a speed similar to that required to viewing the video.

I played a random 1080p YT video and the player-reported data rate is at least 2.5 MB/s, peaking to 20 MB/s (cca 10 MB/s average). 4K is of course much more than that.

I just took a video as a parameter - a recent engineering animation - and got the following:

  format  18 -  640x360  AVC1, V+A :  ~50kB/s
  format  22 - 1280x720  AVC1, V+A :  ~60kB/s
  format 137 - 1920x1080 AVC1, V   : ~128kB/s
  format 400 - 2560x1440 av01, V   : ~450kB/s
  format 401 - 3840x2160 av01, V   : ~850kB/s

Are you sure you are not reporting mega/bits/? When you mention the «player reported data rate», are you sure that is not the "connection speed"? 10MB/s means downloading 36GB/h (one CD per minute, dozens gigabytes per hour)...

You're right it's somehow weird. Perhaps the player shows decompressed figures? Check for yourself - right-click on YT video player, show developer/debug info, and there's a chart.

I checked that interface (if you mean the "Stats for nerds"), hence I asked you if you were not by chance reading the "connection speed", which could be the closest to the values you reported and may be misleading to a glance.

No, the throttling is much more aggressive than real time. I suspect it's Google's way of sneakily breaking ancient YouTube clients that won't run arbitrary JS as a countermeasure for downloaders, without really breaking them all the way, so users of e.g. ancient smart TVs just think their network is (unusably) slow instead seeing an outright error, so they don't complain en masse that their TV no longer works properly.

Typical video bitrates for bog standard 1080p30 will be in the 1MB/s range, so the throttling is around 20x slower than real time.

Right: I forgot about the higher end encodings. 50kB/s is a speed similar to that of the "base" 360p ("format 18") - ~100MB every half hour.

Sorry, one detail (cannot edit the former reply): 1080p30 will be in the 1Mbit/s range (not «1MB/s»). That's maybe ~2.5x slower (not 20x).

For me the throttling without yt-dlp was often slower than the video bitrate, makeing it unusable for streaming.

I noticed this recently, too. I frequently scrape concerts from YouTube by adding them to a private playlist and then running a script on my HTPC to pull down everything from that playlist. As recently as six months ago, pulling down a 2-3GB playlist file (maybe a 30-40 minute concert at 720P) took ten to fifteen minutes. The last time I tried it, it estimated several hours.

If you're downloading a playlist you could run it in parallel to achieve your max network bandwidth speed:

  function ytp() { youtube-dl --get-id "$1" | xargs -I '{}' -P 200 youtube-dl -i --embed-thumbnail --add-metadata -f 'bestaudio[ext=m4a]' -o '%(title)s.%(ext)s' 'https://youtube.com/watch?v={}'; }

FWIW, just recently wrote a go program that watches a Youtube playlist and downloads new videos to a configurable path: https://github.com/raine/ytdlwatch

I use it to download videos into a Plex library.

FireDM (https://pypi.org/project/FireDM) is an awesome front end for ytdlp for those interested.

GitHub Links from the project page all get a 404.

https://github.com/firedm/ shows no public repos and of course https://github.com/firedm/FireDM gets a 404

Yeah, trying to delete `youtube-dl` from the GitHub triggered a huge Streisand Effect[1].

[1]: https://en.wikipedia.org/wiki/Streisand_effect

I always find jwz's youtubedown works flawlessly for me. https://www.jwz.org/hacks/youtubedown

Do not direct link to jwz.org from HN. Here's a click-safe link:

https://dereferer.me/?https%3A//www.jwz.org/hacks/youtubedow...

Does not work either because apparently bit.ly's 301 preserves the Referer.

Weird, I had tested it. I’ve updated to a de-referrer site instead.

Might be a browser difference (firefox here). Thanks for the update, it works better.

> Do not direct link to jwz.org from HN

Why?

Because the admin detects HN referer explicitly and presents a joke page.

Perhaps I’m in the minority, but I find this kind of subversion amusing. Like all good jokes, it’s pretty close to the mark.

I agree, so it’s a safe bet you are in the minority.

I've since noticed...

Just copy the link to not send any referrer information.

It genuinely shocks me that there are still people who don't disable sending the referer header cross-origin in the browser: I have not encountered a single website that breaks when setting `network.http.referer.XOriginPolicy` to 1 in Firefox, and only 2 or 3 sites that break when setting it all the way to 2.

It not only completely prevents stuff like this, it profoundly increases your privacy on the web by preventing sites from tracking which domain you came from. There is no good reason any site needs to know that. I am surprised that Mozilla hasn't simply made this the default setting for all users.

I think people don’t know about it. I’m going to do this now that I do. Thank you!

> I am surprised that Mozilla hasn't simply made this the default setting for all users.

I was under the impression it was, I doubt I'm the only one, so thanks for drawing attention to it.

The Wayback Machine interface breaks.

Do you have an example of this? What exactly breaks? I use Wayback all the time and have had no obvious problems.

Has been the case for a bunch of people with them turned off for a while now:

https://old.reddit.com/r/WaybackMachine/comments/kzvzxl/fail...

Unless I'm mistaken, you've misunderstood the issue. From your link:

> it happened to me and I figured out it started when I disabled website.referrersEnabled

So if you completely disable sending the referer header, it breaks. This would probably also happen if you set `network.http.sendRefererHeader` to 0 or 1.

But that's not what I suggested! I only suggested disabling sending the header externally, to other sites, when the host domain (*.example.org) changes. In fact, someone in the thread you linked says that doing this instead of disabling it completely fixed the issue for them:

> ok I left network.http.sendRefererHeader on 2 (had it on 1), and changing network.http.referer.XOriginPolicy to 2 it works

There is also `network.http.referer.spoofSource`, which I have used instead of that.

This breaks in the case that the endpoint actually does use the referer for something. I have actually encountered plenty of cases where a site will use the referer internally, i.e. on the same origin, to check something.

Yeah that's fair, but I haven't really noticed it for myself, but then again, I disable javascript by default and have compiled a lot of other stuff into the netwerk module that probably breaks sites even more to the degree if its noticeable, i just don't use the site.

>> `network.http.referer.XOriginPolicy` to 1 in Firefox

Thanks! Just checked and it is at zero for the default. Reviewed https://wiki.mozilla.org/Security/Referrer and learned something new.

oooohhhhhhh. The "egg". My bad.

How often do you update it ? Because it seems to have seen quite a few update from the beginning of 2022.

Every time I go to use it I grab a fresh copy.

I miss when youtube-dl wasn't getting so much attention and just worked. Ah well.

I started getting systematic

> Connection reset by peer

in a script that I have that downloads podcasts and immediately transcodes to low bitrate opus using ffmpeg.

It's a real bummer...

Fast alternative:

https://github.com/89z/mech

Reply on news.ycombinator.com