Hacker News Reader

Well, that didn't take long – our API endpoint keeled over. Luckily, you can fetch all the data directly from firebase:

  # fetch raw tag data
  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/.json > tags.json
  $ du -hs tags.json
  14M tags.json

  # fetch tag metadata (colors, remapping label names, possibly other stuff in the future)
  $ curl -fsSL https://experiments-573d7.firebaseio.com/user_meta/.json > tags_meta.json
  $ du -hs tags_meta.json
  376K tags_meta.json
  $ jq tags_meta.json

Note that's the raw unprocessed data (no yolo). To get info about all experiments, you can use this:

  curl -fsSL https://experiments-573d7.firebaseio.com/meta/.json | jq

I'm a bit worried about the bill. It's up to $50 and rising: https://imgur.com/ZgmXsWU almost entirely egress bandwidth. Be gentle with those `curl` statements. :)

(I think that's due to a poor architectural decision on my part, which is solvable, and not due to egress bandwidth via the API endpoint. But it's always fun to see a J curve in your bill... It's about $1 a day right now. https://imgur.com/4gUTLO7)

Can you set it up so that it's only available via cloud? I'm sure that would bother people, but is a better alternative to losing access or you going broke :)

We're motivated to keep this as open as possible. I really like the idea of an open dataset that continues to grow with time. If it keeps growing, then within a couple years it should have a vast quantity of tags on a variety of diverse datasets, which we hope might prove helpful.

If anyone wants to contribute, I started a patreon a few minutes ago: https://www.patreon.com/shawwn

Perhaps you can mirror it to an s3 bucket or GH repo for people to CURL more easily?

Heh, $1 per day? Try $1k per day. https://imgur.com/duugqHK

We've confirmed that this was someone running `while true; curl ...`, resulting in a $3,700 bill. https://twitter.com/theshawwn/status/1271365062913961984

I guess "be gentle" means "please troll us."

I remember following this as it came out (and learning windshield wipers should be called "swipey bois")

Surprised and happy to hear you're seeing high labeling quality.

We'll re-host with credit on https://public.roboflow.ai What license is this?

Thanks! We've decided to license the data as CC-0. We'll add that to the footer.

We don't host any images directly – we merely serve a list of URLs (e.g. https://battle.shawwn.com/tfdne.txt). But any data served via the API endpoints is CC-0.

I need a dataset and tags for hair, face, neck, arms, left breast, right breast, nipple, torso. Any tips? I'm training a GAN, but I need to specifically segment the parts, as I don't want nipples in the middle of a face. I don't want to have to manually annotate 1,000 images

At the moment, the only experiments with enough data to be useful are e621-portraits (5,407 tags https://www.tagpls.com/exp?n=e621-portraits) and danbooru-e (344 tags https://www.tagpls.com/exp?n=danbooru2019-e) both of which are NSFW.

Those are also drawings/anime, not photos. We have an /r/pics experiment (SFW, 99 tags https://www.tagpls.com/exp?n=r-pics) and /r/gonewild (NSFW, 57 tags https://www.tagpls.com/exp?n=r-gonewild) but currently I haven't gathered enough urls to be very useful -- it only scrapes about 100 or so images every half hour. So there is a lack of tags right now on human photos. We also have a pps experiment (NSFW, exactly what you think it is, 306 tags https://www.tagpls.com/exp?n=pps) but I assume that's not quite what you were looking for.

If you have an idea for a dataset, you can create a list of image URLs like https://battle.shawwn.com/r/pics.txt and we can add them to the site. You can request an addition by joining our ML discord (https://discordapp.com/invite/x52Xz3y) and posting in the #tagging channel.

Also, if anyone's curious, here's how I'm measuring the tag count:

  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/danbooru2019-e/.json | jq '.' | grep points | wc -l
       344
  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/e621-portraits/.json | jq '.' | grep points | wc -l
      5407
  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/r-gonewild/.json | jq '.' | grep points | wc -l
        57
  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/r-pics/.json | jq '.' | grep points | wc -l
        99
  $ curl -fsSL https://experiments-573d7.firebaseio.com/results/pps/.json | jq '.' | grep points | wc -l
       306

Yeah, I mean, the tagging is awesome, but I'm thinking I'll need more image segmentation than object recognition. With a segmentation map, I can make a great image->image translator.

I love that it's porn (and specifically furry/hentai) which pushes the limits of image recognition and creativity within computer vision. Between this and the de-censoring tool "DeepCreamPy" I can't look most data scientists in the face anymore .

that's a great name, turning jagged edges back to smooth and applying reverse Gaussian blur /s

on a serious note, kind of interesting the authenticity/accuracy if it's just filled in... eg. turning black and white pictures back to color eg. was it actually green or blue

This is really cool, thanks for sharing

> I don't want nipples in the middle of a face

There is a market somewhere

You should post this as a Show HN!

Okay! Thank you. We appreciate the encouragement.

It looks like an HN user on an EC2 server decided to fetch data from our firebase as quickly as possible, running up a $3,700 bill. Once (or if) that's sorted out, and once we verify tagpls can handle HN's load without charging thousands of dollars, we'll add an "about" page to tagpls and submit it.

How does this have anything to do with the linked article?

The idea with the site is that you can tag your own datasets, and then get the data suitable for yolo training. We've done that ourselves to train an anime hand detector, and other users have reported similar successes. I could've been a bit clearer about that.