Online streaming sites and URL obfuscation | Hacker News Reader

Online streaming sites and URL obfuscation

thesquareplanet.com

67 pointsJonhoo16 comments

Is a shorter solution possible?

Is decompiling SWF and/or de-obfuscating Javascript really necessary?

Years ago, I approached these sites like the author in the blog post. I spent hours reading about Flash and figuring out the schemes websites used. I decompiled swf. I also tried rtmpdump, livestreamer and other all-in-one solutions. I used early youtube downloaders (clive, etc) and later quvi. I also used /dev/bpf to capture HTTP requests. I tried it all.

I can relate to what the author is trying to do. I never settle for "streaming". Download only.

However I found over the years either the websites have made things easier or I was simply trying too hard. Or maybe I just am not interested in the type of video that is served in this way. For some reason it takes much less effort now.

To make sure I am not imagining this, I decided to try one of the author's examples.

I chose projectfreetv.so

1. I followed a link on the main page for some TV show I have never seen.

2. I read the HTML.

3. I noticed /watch/?aff_id= URL's.

4. I chose the first one, 493165, and followed it.

5. I read the HTML.

6. I noticed a URL pointing to a website that serves TV shows and followed it.

7. I read the HTML.

8. I noticed a URL pointing to an MP4 file.

9. I downloaded the file.

I assume I got the TV episode on offer. The screen size is a little small but the file was about 89MB and appeared to be the full episode.

Total time: less than 10min

What did I use? sed, netcat, less and ftp for the download. The entire process is a one-liner.

If anyone has some more examples to try, please list them. I would welcome the challenge.

Author here: for some sites, that approach works well, and this is what I originally did (and why I decided to write streamsh in bash). Have a look in the "tricks" directory and you'll see some of the much simpler solutions that are tried first. However, an increasing number of sites try to hide the URL, and through increasingly complicated means. The article is about vidbull.com, but there are others. They often also layer multiple obfuscators (vidbull uses both JS obfuscation and URL encryption for example). While it's true you can get pretty far just by scanning for mp4s on some mirrors, they are becoming fewer and farther between.

vidbull looks more like the old "file storage locker" sites that were popular before the megaupload crackdown. Isn't this the same content you can find using torrent sites? Is it likely that vidbull would have something for which no torrent would ever exist? Sorry for my ignorance.

No no, there are torrents for all of these videos (at least usually). However, torrents are slightly less convenient in that you can't play the file until the download completes. For me personally it's also an upside that HTTP is easier to proxy, and there's no need to seed. That said, torrents will probably be a better solution for many people. If you're interested, I wrote a tool a while back that scanned torrent sites for the newest episode of a particular show and automatically downloaded and started the torrent: https://github.com/jonhoo/smore. No idea if it still works, but shouldn't be too hard to get it up and running again.

rtmpdump [1] and livestreamer [2] are other good tools for this type of thing.

I've always had trouble trying to rip JWPlayer videos that sit behind a login. I did manage it but I had to stream (second-by-second) the entire video.

[1] - https://rtmpdump.mplayerhq.hu/

[2] - http://docs.livestreamer.io/

Youtube-dl [1] is my go to tool for pulling video content out of a page, and its source [2] is a nice lesson in de-obfuscating the sites it supports

[1] = https://rg3.github.io/youtube-dl/ [2] - https://github.com/rg3/youtube-dl/tree/master/youtube_dl/ext...

"streaming freedom" is a really clever euphemism.

At the end he says something like "We need Encrypted Media Extensions to get ad free streaming...". Hmm, never thought about that this way. Maybe he is right, but couldn't it be that with EME the ads will not be sitting next to the player, instead the streamed video will contain the ads? Or does anybody believe (enough) people will pay for streaming on youtube or similar pages?

> the ads will not be sitting next to the player, instead the streamed video will contain the ads?

This tech already exists, and it's called 'Server Side Ad Insertion', with Brightcove, The Platform, Adobe and Google all offering solutions.

It's actually pretty cool - it works by instead of streaming a baked file, the client gets a 'live stream' URL and ads are inserted into that stream, dynamically, just like regular video. It's like having `badge.svg` image that's actually a PHP script that dynamically creates an SVG.

I'm not overly familiar with video tech, so I've definitely butchered the exclamation.

And then ad blockers will snip out the ad portion of the stream and the cat and mouse game will continue.

VCRs did that long before streaming became popular, as in those days the only way ads could appear were as directly in the video signal.

https://en.wikipedia.org/wiki/Commercial_skipping

Not surprisingly there was some legal action, but I think it's rather unenforceable in practice, and saying it's illegal to do so is nearly equivalent to saying that you do not have the freedom to change the channel, look away, or even close your eyes - a rather scary situation.

For a long time I've used the by hand method of URL extraction either inspecting source code and searching for .flv or .mp4 filetypes (or simply "file:*"). The other method I use is the built in developer tools "network" tab of chrome under "inspect element". So far, I've only encountered one host (sorry I cannot remember the name) which I wasn't able to extract the source file via one of these two methods.

Yes, the second method I used to use quite frequently, but it means you have to open your browser, click play to get the video to load (which will spawn a torrent of ad pop-ups and captcha prompts and whatnot). This way I can get the download straight from the command line.

How do you use streamsh?

https://github.com/jonhoo/streamsh/blob/master/tricks/jwplay...

stream.sh <video URL (preferably the embed URL)> <filename to download to (without extension)>

Many of these sites have started supporting iOS. Wouldn't it be easier to spoof iOS Safari and then only have to dig around in JavaScript?

Reply on news.ycombinator.com