Show HN: A website that converts any IPFS file to HLS and reups it to IPFS | Hacker News Reader

Show HN: A website that converts any IPFS file to HLS and reups it to IPFS

ipfs-converter.com

126 pointstimdaub45 comments

Hi this is Tim. I made this website. It's currently running on a $5 droplet on DigitalOcean so conversions will take a while. I'm testing out the waters here on HN to see if there's anyone interested in it. Please let me know if you have any questions regarding it.

Github: https://github.com/TimDaub/ipfs-hls-converter

Edit: I haven't chosen I license for this yet, sorry.

I think you're experiencing the Slashdot effect.

What did you end up using for queueing conversion jobs?

Fantastic question. I'm not even doing that yet. How embarrassing!

Right now I'm just spawning sub processes like crazy and hope they don't overwhelm my machine. In the future, I plan to use a queue and also to monetize on API queries.

I haven't looked into queues in Javascript yet. If you have recommendations, that be appreciated :)

This is really just an MVP to see how the internet crowd reacts to it. I'm glad it does in a positive way :)

Haven't used it, but "bull" shows transcoding as their example, so I assume they've run through your use case: https://github.com/OptimalBits/bull

Don't use bull, use NATS[1] - happy with it so far with the Node SDK.

[1] https://nats.io

That seems more messaging oriented than job oriented. Does it manage job concurrency limits, time outs, etc?

Why do you say not to use bull? I've been quite happy with bull in our production environments so far.

RabbitMQ is what we use at work or sns for messages to lambdas.

Why are you transcoding all the time? You could probably save your little droplet a bit of heartache by using ffprobe to check if the video is already h264 (or h265)+aac and just do -c copy in that case.

That's a great suggestion. I'll add that to my list of todos!

Did you look into using something like ffmpeg.js to do the heavy lifting client-side? Note I have no idea if ffmpeg.js supports all the features you need (ie HLS).

I didn't even know that existed, but now I will definitely take a look! Thanks for the feedback!

Question from an IPFS noob:

When I run the demo, the video plays as expected. However, I wanted to check the network tab to see what it's doing under the hood.

Based on the description, I expected to see a bunch of HTTP GET requests to ipfs.io, fetching the different segments of the movie. However, instead, I see a varying number of websocket connections to ipfs.team, soundclart.com and libp2p.io.

Could someone explain what is going on here? Is this just another IPFS-to-web proxy (based on websockets instead of http) or is this already part of the HLS streaming mechanism?

The ipfs.js library requests the files through WebSocket, and those various domains you see are different IPFS gateways. (for redundancy/latency I'd assume)

Ah, that makes sense. Thanks for the info.

What's the utility of this? The site is down and github doesn't really explain why you would do this.

IPFS is a distributed p2p filesystem. The idea, I believe, is that you can put your big media files on it, to share bandwidth and (iiuc) reduce censorship risks; but it's better to do it with a stream-friendly codec, so this site will turn your file into that first.

The acronym is for Inter-Planetary File System, with the main idea being getting a file from the closest person who already has it, so you don't have to request it from a central server that is potentially very far away.

The main benefit on earth, as far as I understand, is reducing bandwidth required of a single person to share media by distributing hosting, similarly to torrents.

I just came back from a run and the website was down. Sorry. It's up again.

Utility: https://github.com/ipfs/js-ipfs/tree/master/examples/browser...

Anyone know how js-ipfs is able to load files without it showing up on the chrome dev tools' network tab?

All the data comes over the websocket connections. If you click on any of the 'pending' connections, you can inspect the individual frames.

Is this one of the first examples of an IPFS-based API? This is very cool.

I like that you can take something that is effectively on your disk, ping some endpoint, and it will get "pulled" by the service as a peer and process it.

Next step could be to make the "ping" step as an IPFS pubsub thing.

No, I don't think it's the first example. https://www.eternum.io/ was one that was before, but I'm sure there are others.

We keep a list of some applications and other things over here too: https://awesome.ipfs.io/

"One of" :)

I'd argue eternum is not quite at the same level since it just takes an IPFS object as an input (but no IPFS output). Op made a thing that takes an IPFS object as an input and produces a different IPFS object as an output.

There are certainly many IPFS-based apps.

I would agree with this, Eternum isn't so much an IPFS-based API as an API for helping with IPFS pinning.

I'd say IPFessay (https://gitlab.com/stavros/IPFessay) is much closer to an IPFS-based API, since it can run entirely on IPFS with no outside servers.

The thing that doesn't make sense to me is that you say any file but then seem to be doing video-specific stuff to it. So did you mean to say any _video_ file?

Also, what is HLS?

HLS is HTTP Live Streaming, an open, pure-http approach to live streaming video with support for adaptive bitrate delivery. first popularized on iOS circa 2008 as an alternative to Flash and other proprietary streaming systems.

He's referring to the title which says it "converts _any_ IPFS file".

The OP also asked 'Also, what is HLS?'

Oh, completely missed that. My apologies.

> So did you mean to say any _video_ file?

Yes you're absolutely right. I just changed the description on the website.

> Also, what is HLS?

Someone else answered that already here: https://news.ycombinator.com/item?id=18132277

Cool project! Love IPFS usage :)

So this is going into the land of RPC, have anyone work together on the load this would demand on a bigger scale than one man's $5 box.

Already a seed of thought in place for this: https://github.com/ipfs/notes/issues/116

Does this require resources on your server? (Apologies because I don't understand what this is.) What's to stop me from uploading 10TB of video files and using all your resources?

Yes, it requires resources. I currently run FFMPEG for every video I get.

Currently nothing is stopping you from uploading a 10TB video except that maybe my DigitalOcean droplet would run out of space and crash.

Why HLS rather than DASH? Is there some way that HLS is a better fit for IPFS? Or is HLS easier to use, or more well supported? (Doesn't YouTube use DASH?)

I don't understand in which situation I should be using that.

I built this mainly because I was feeling the pain while implementing a website for a customer that is allowing their users to upload and stream files in the browser.

As the file upload should accept any kind of input, but the streaming only allows for certain files to be played back in the browser I researched different codecs that are available on the web for streaming and came across: https://github.com/ipfs/js-ipfs/tree/master/examples/browser...

I though, why not make this a SaaS.

So your customer is using IPFS on his site? How do you manage that? Do you run the IPFS node for him and put the uploaded videos on your node?

Is your SaaS fetching from IPFS, then reuploading because your customer's visitors upload using js-ipfs, then you fetch from your node, totally unrelated to the website's backend?

Just wondering.

My customer is using ipfs-api-js on their website and it uploads to an ipfs node (in our case Infura). In the future we might run a node if the customer requires it. For now Infura is very convenient.

My Saas is fetching from IPFS, then reuploading so that the upload and conversion of the file both could potentially be triggered on the client side. Many Dapp architectures I've seen so far have no backend component.

Oh, so Infura let's you pin stuff for free? How does that work?

As far as I know Infura let's you pin for free yes.

Awesome work, I'd try to check it out once it's up again.

Blank page with JS disabled.

Reply on news.ycombinator.com