Comment by 1vuio0pswjnm7 | Hacker News Reader

Comment by 1vuio0pswjnm7 | Hacker News Reader

Parent Full thread

ytsearch does not support continuation searches. (It only retrieves the first 15 results.)

Continuation searches occur when a user scrolls to the bottom of a search results page.

YouTube (or NewPipe, etc.) sends automated search requests to continue the search, triggered automatically via Javascript when the user scrolls to the bottom of the page.

YouTube continuation searches leak a large amount of unnecessary user data to Google.

1.sh[1] sends continuation searches manually and sends only the minimum ($MIN) data needed to fulfill the request.

It can be used to continue a search indefinitely over an extended period of time. Using an app like NewPipe, often recommended on HN, one could try to do continuation searches by repeatedly scrolling to the bottom of the search results page. However if one goes forward to view a result and then presses a back button, she will be returned to the first page the results, losing all the continuation search results. And if the browser tab or app is closed or crashes, the continuation search results are lost. It is easy to lose one's place when doing continuation searches using a Javascript-enabled browser or app. Using this script, this will never happen.

1.sh only uses TLS1.3 so no server certificate is sent in plaintext over the wire. Nor does it send SNI, i.e., plaintext hostnames on the wire.

ytsearch requires sed, column, awk, jq, fzf, and curl.

1.sh requires sed, netcat, stunnel and flex.

1.sh could be adapted to use curl but how does one disable SNI with curl.

Not to mention curl has a TLS fingerprint that sites may use to try to block requests or treat them with prejudice.

Usage for 1.sh is

   1.sh query; #<-- saves line-oriented JSON to query.yts

   1.sh << query.yts >> query.yts; #<-- appends more JSON to query.yts

To convert our line-oriented JSON to TSV, we can use line-oriented UNIX utilities. In general, this will be faster than jq and more robust for large files.

1. 1.sh is a 25 line, 1000 character shell script. https://news.ycombinator.com/item?id=30408770

oh wow, that's pretty cool and impressive! thanks for sharing.

> ytsearch does not support continuation searches.... Continuation searches occur when a user scrolls to the bottom of a search results page.

true, I never bothered adding "continuation searches" feature to ytsearch because I very rarely scroll down to the second page of results. usually if what I'm looking for is not in the first 4-5 results, I just "refine" the search query.

ytsearch is a quick-and-dirty script that avoids me to open the browser for simple YouTube searches (I stole the idea from the initial version of ytfzf).

> It only retrieves the first 15 results.

that's not true, I always get more than 20-25 results on average and it's usually enough for my use:

    $ ytsearch foo | sort | uniq | wc -l
    37

> 1.sh only uses TLS1.3 so no server certificate is sent in plaintext over the wire. Nor does it send SNI, i.e., plaintext hostnames on the wire.

> 1.sh could be adapted to use curl but how does one disable SNI with curl.

that's pretty cool if you care about this kind of stuff!

> Not to mention curl has a TLS fingerprint that sites may use to try to block requests or treat them with prejudice.

yes I see how this could become a problem.

> ytsearch requires sed, column, awk, jq, fzf, and curl.

> 1.sh requires sed, netcat, stunnel and flex.

what are you implying here? do you think those 6 dependencies are "too much" for ytsearch compared to those of 1.sh?

anyway, it seems that the 2 tools offer a slightly different user experience out-of-the-box:

- 1.sh dumps json and can be used to continue searches without leaking too much data.

- ytsearch allows to quickly select a few urls and it's meant to be used interactively.

> To convert our line-oriented JSON to TSV, we can use line-oriented UNIX utilities. In general, this will be faster than jq and more robust for large files.

faster yes, but how it will be "more robust for large files" than jq? I always thought jq is the right tool for the job when dealing with json.

The number of results, as well as the schema of the JSON, might vary based on YouTube's "guess" as to the form factor of the computer being used to send the request. For example, when no User-Agent header is sent I have noticed sometimes web developers just assume the form factor is "mobile".

Providing a list of requirements is a courtesy for the reader.

s/indefinitely/up to 500 results/

Reply on news.ycombinator.com