Comment by kgodey | Hacker News Reader

Comment by kgodey | Hacker News Reader

I'm Director of Engineering at Creative Commons and part of the team that is working on CC Search.

We've been working on the product for over a year and we are just out of beta today! One of CC's goals is to encourage the use and remixing of CC-licensed content, and we hope that CC Search will help make that content more discoverable. The current version is very much an MVP and only searches images, but we plan to add more content types in the future and index the ~1.4 billion works out there under a CC license. We would love any feedback you might have.

Also, CC Search, the associated API, and the scripts we use to index data are all open-source and developed completely openly. Our sprints and roadmap are public and we welcome contributions from the community.

Relevant links:

CC Search code: https://github.com/creativecommons/cccatalog-frontend/

CC Catalog API code: https://github.com/creativecommons/cccatalog-api/

CC Catalog code: https://github.com/creativecommons/cccatalog/

2019 Vision: https://creativecommons.org/2019/03/19/cc-search/

Roadmap: https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...

Active sprint: https://github.com/orgs/creativecommons/projects/7

Backlog: https://github.com/orgs/creativecommons/projects/10

How to contribute to CC projects: https://creativecommons.github.io/contributing-code/

If I put images in the internet, how can I tell your search about them/about the license?

https://wiki.creativecommons.org/wiki/Marking_Works_Technica...

That should probably be linked somewhere prominently then (I was aware it exists, but unsure if it is still current, given that there have been little changes to that page in the last decade). And I guess I have to hope I end up in Common Crawl?

Hello, I work on this project! For the most part, we use Common Crawl to discover which websites carry the most CC content, and then integrate the platform through either their API, if available, or put together a bespoke scraper. If you put your content on one of these integrated platforms, eventually your work will appear in our collection.

In my mind, the dream is to have the user embed an asset from our servers on their web page (like an updated version of these old CC license buttons [0]), read the referrer headers from the server logs, and then dispatch crawlers that read ccREL [1] data embedded on the page, which would allow us to instantly index content as soon as it is published. Performing broad web crawls searching the web for ccREL data is also possible but probably not what we're looking to do in the near-term.

We have a ways to go before we are able to do this, since there's no easy way for end users to create and embed ccREL at the moment, and there are of course lots of other unanswered questions about how we would moderate incorrect attribution, how these tools might be abused, etc.

[0] https://licensebuttons.net/l/by-nd/4.0/88x31.png

[1] https://www.w3.org/Submission/ccREL/

We plan to implement an API for users to submit CC-licensed works that they'd like to add to our database later this year. Once we have that, you can call our API when you publish a new CC-licensed work and we'll automatically add it to our catalog. We have to work out some details like how we can verify that the content is actually CC-licensed.

Encouraging use of ccREL embedding like aldenpage mentioned in another response is also in our long-term vision since that would require no additional actions from the creators.

You could have the same (only existing?) process than other content platforms like YouTube, Github, etc. In practice, most people are fair and post the content with the right license, but anyone can send a DMCA request.

An annual transparency report is probably mandatory with that process, like these ones :

https://github.blog/2019-01-23-2018-transparency-report/

https://transparency.facebook.com

Excellent project, this simplifies the process I currently have of searching many different sources and applying a cc filter (mostly Flickr and Google images).

The two image filters I use most often there are image size (larger than) and orientation (portrait or landscape). If these would be included here it would be perfect.

Thank you for the feedback!

Nice project! Thanks for working on it!

It would be great if there were more search filters. Especially size ("larger than") and date ("within x-y", "last 1 day/week/month/year").

And when you have filters anyway, filtering by EXIF data (geoposition, and typical photography details like camera model / aperture / focal length) would be pretty cool.

Thanks for the feedback! I've added those filters to our pipeline of ideas for our roadmap [0].

[0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...

This is nice, I can see this being used for computer vision researchers and practitioners to train CNNs. Have you thought of implementing a reverse image search engine? That would be a great feature or project to work on as it can help users find images they are interested in as tags tend to be noisy.

Thanks! I’ve added this to the pipeline of ideas on our roadmap [0].

[0] https://docs.google.com/document/d/19yH2V5K4nzWgEXaZhkzD1egz...

Do you mind me asking how many man-years the project took? Is there an article on/diary covering the dev decisions?

We've been working on it for a year and we have one data engineer, one backend engineer, and one frontend engineer.

We recently set up a tech blog[0] and we have upcoming posts that will talk about the architecture and decisions we made. Here's a post about the original proof-of-concept (we are no longer using that version): https://hackernoon.com/cc-search-developer-notes-and-reflect...

[0] https://creativecommons.github.io/blog/

Reply on news.ycombinator.com