Hacker News Reader

creekorfulon Show HN: I have written a cloud native dark web crawler in Go

Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.

creekorfulon Show HN: I have written a cloud native dark web crawler in Go

I'll certainly publish an index of interesting (legal) hostnames on my GitHub account.

creekorfulon Show HN: I have written a cloud native dark web crawler in Go

I have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces.

There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.

creekorfulon Show HN: I have written a cloud native dark web crawler in Go

Release 1.0.0 will be out very soon. Check it out guys!

creekorfulon Making a product is not even half of the path to success

Let me know what you think !

creekorfulon Show HN: Urbex Community – Discover unique urbex locations

Let me know what do you think of it !

creekorfulon Your private Docker Registry

Feedback is welcome !

creekorfulon Show HN: RWP – An Android App to keep your wallpaper fresh

Thank you!

Unfortunately the application will be only available on Android. It is impossible to change wallpaper programatically on iOS.

creekorfulon Show HN: RWP – An Android App to keep your wallpaper fresh

Hello guys !

I have built an Android app with a friend since 2 months and everythings's finally live. (you can check the announcement post here: https://creekorful.me/pimp-your-phone-like-never-before/) The application allows users to customize their phone with HD wallpapers grouped by categories. You don't have to choose the wallpaper anymore, you'll just need to select the things you like, set the change interval and let the app do the magic!

The technical stack is composed of 2 big parts:

- The native android application (written in Java&Kotlin because we needed access to low-level API).

- The API (written in Go) that communicates with a MongoDB. The API is used to abstract the different wallpaper providers that we use.

Authentication, analytics and user management are done using Google Firebase.

The whole stack is deployed on a Docker Swarm and routing stuff is done using Traefik.

Feedback is welcome ! And It would be a pleasure to answer any question (technical or not)

creekorfulon Traefik 2.x setup guide on Docker Swarm

What do you think?

creekorfulon Building a Dark Web Crawler in Go

Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?

creekorfulon Building a Dark Web Crawler in Go

the proxy which I refer is the local SOCKS proxy used by tor browser / your applications to be routed through the network.

creekorfulon Building a Dark Web Crawler in Go

I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code.

Why should I face legal problems?

creekorfulon Building a Dark Web Crawler in Go

Thank you sir. Actually my other comments are invisible too. That's weird.

creekorfulon Building a Dark Web Crawler in Go

Author here. I'm fairly new to Golang too and it's my first project.

Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available.

Not taken but available.

I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!

creekorfulon Building a Dark Web Crawler in Go

You are right. That's why it's an educational project and not a public search engine

creekorfulon Building a Dark Web Crawler in Go

Thank you!

There's no practical purpose for the crawler. It's more an educational project than anything.

creekorfulon Building a Dark Web Crawler in Go

Author here.

The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy.

The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy.

They are a type of dark web services, but not the entirety