mekarpeles

https://github.com/internetarchive/openlibrary/issues/4320

Some classifications (LCCN) are better at encodind Author data and we also have a significant amount of author data in Open Library (we'd just need to integrate it more meaningfully in our search index).

I opened an issue for you here: https://github.com/internetarchive/openlibrary/issues/4319

Given our small team size, not sure we'll have the bandwidth to prioritize any time soon, but contributions are also welcome from the community and we (the community) meet every Tuesday @ 11:30am to discuss and unblock together.

https://ianews.wordpress.com/2011/03/03/open-library-opds @ https://bookserver.archive.org.

Do try out the "Settings" cog at the bottom of the UI -- one thing that makes this interface so powerful is you can add custom queries to transform the entire library (such as to only show kids books, text books, or biographies on any subject).

Do you mean this option doesn't work through the search.json API? If so, can you please make sure we have an issue open https://github.com/internetarchive/openlibrary/issues/new/ch...?

You should be able to add has_fulltext:true to your query (which at least returns a set of books for which there is some full-text in existence:

e.g. http://openlibrary.org/search.json?q=has_fulltext:true%20AND...

@rayrag, there are researchers, vendors, libraries, partners like wikipedia, authors, and hundreds of thousands of people who use Open Library to track when they're reading, to promote books and reading, and to create curated lists.

Also, organizations like Internet Archive use such catalogs to manage what books to acquire (based on interest/demand from the community).

Having a home for bookcovers, citations, quotes, ISBNs, publishers, years, subjects, related editions are all valuable even if the book (source) is not yet available.

Hope this helps!

I love to see this, more access for the book community -- especially how quickly it came together. Nice work Jabo and excited to see Typesense in action.

Open Library gladly & thankfully tweeted its support of your hard work:

https://twitter.com/openlibrary/status/1338531822536310789

In every intention of being additive to Jabo's work, thought I'd share a few resources folks may not be aware of:

1. Open Library's Full-text Search

Major hat tip to the Internet Archive's Giovanni Damiola for this one: Folks may also appreciate the ability to full-text search across 4M of the Internet Archive's books on Open Library: https://blog.openlibrary.org/2018/07/14/search-full-text-wit.... You can try it directly here: http://openlibrary.org/search/inside?q=thanks%20for%20all%20...

As per usual, nearly all Open Library urls are themselves APIs, e.g.: http://openlibrary.org/search/inside.json?q=thanks%20for%20a...

2. Open Library's search API

By adding .json to our search endpoint, you can use Open Library's json search endpoint (which also contains bookcover links). You can also use this to fetch books by ISBN:

http://openlibrary.org/search.json?q=isbn:9781400067824

3. Book Cover Service (has issues)

We need to improve our cover service, it's definitely been neglected, but the cover IDs from our data dump, solr search index, and book APIs can be plugged in to our cover store to point to or fetch book covers. We don't yet have a current data bulk dump of covers available (it's on the roadmap) but if you need "all" the covers, please email us openlibrary@archive.org instead of hitting covers 1-by-1. https://openlibrary.org/dev/docs/api/covers

4. The new (unannounced) Open Library Explorer

I'm honestly kind of hoping no one reads this far into the comments because we haven't announced this one yet and I'm sure there are major performance issues (it's beta), but cdrini recently added dewey + library of congress classifications to our search index... Making this masterpiece possible:

https://openlibrary.org/explore

The ability to browse Open Library digitally, with all the glory of a physical library.

Ideally, it should probably bring you to the login page and (once you login) bring you back to the import page.

We'll see about creating an issue for this, thanks!

e.g.

https://openlibrary.org/search.json?q=

https://openlibrary.org/books/OL28018730M.json

This is one of the best implementation decisions that Aaron Swartz, Anand Chitipothu, et al made when starting Open Library.

I haven't shared this yet -- it's more for the community, but I've tried to address various questions from the community and distill answers + resources for Open Library here:

https://blog.openlibrary.org/2020/12/13/importing-your-goodr...

Tried my best to include others players in the space (wikidata, inventaire, bookbrainz, worldcat, bookwyrm) who are doing great work and pay respects to readng, storygraph and other innovative services which are breaking onto the scene.

Books pre ~1973 don't have ISBN. And goodreads covers far more material than just modern material.

The current Goodreads importer makes a big/unfortunate tradeoff of trying to have something which works for the majority of material on people's goodreads lists.

It doesn't preclude further efforts -- Open Library knows about millions of books that don't have ISBN. So if anyone wants to help us improve our importer (or at least register their interest by creating an issue which calls out what other identifiers should be considered -- e.g. LOC, etc) that would be very helpful.

# To open an issue:

https://github.com/internetarchive/openlibrary/issues/new/ch...

# Here's the code for the importer

https://github.com/internetarchive/openlibrary/blob/892adebb...

Great points here

1. The banner happens last month of the year (Wikipedia being the perfect analog). Yes, there are mixed feelings and it's not the world's best experience :P

2. Our entire data set is available to download as in bulk https://openlibrary.org/developers/dumps because we'd love to see a decentralized p2p version

3. https://github.com/mouse-reeve/bookwyrm Mouse who used to work @ Internet Archive has a decentralized version of Open Library (Bookwyrm) and it's worth checking out.

4. For the last 5 or so years the Internet Archive has been cultivating a dweb/dapp community and integrating with IIIF, Dat, IPFS, gun, bittorent, webtorrents, and others and hosting regular summits and meetups https://blog.archive.org/2018/07/21/decentralized-web-faq/

5. The wayback machine is an interesting case study: it turns out, incentive structures (even things like FIL/filecoin) haven't been able to perfectly crack the nut on getting folks interested enough to preserve the whole wayback machine. There's petabytes of material and there's a powerlaw about what people care about today. Internet Archive realized what we care about today may not be the same as tomorrow, and so there's a cost eaten (the incentive comes from economies of scale generated by intrinsic desire rather than $). And in a way, this centralized solution (economies of scale) IS the solution a community came up with. It has flaws and advantages (tradeoffs), such as centralized points of failure, and I think the archive would be (and has been) ecstatic to explore improving these opportunities.

Many of our librarian processes & FAQs are detailed here: https://openlibrary.org/librarians

Most data on Open Library is publicly editable by members. In this case, author merging is a capability that only folks who have been added to our Librarians usergroup can access because the process of reverting an accidental merge is quite time consuming (and so we have some training in place).

If you tweet @ our lead community librarian (http://twitter.com/seabelis) she can likely help you make this fix! Also, if you're interested in helping us make changes like this, we can invite you to our slack channel :)

https://github.com/internetarchive/openlibrary/issues/new/ch...

The importer is a new project (as of August this year) and I think there's a lot of opportunity for improving it (including importing missing records on demand).

Other hat tips: I also want to call out the great work of https://inventaire.io, another great project in the space which uses wikidata, as they also have a goodreads import which we're learning from. Another project I'm excited about (which has goodreads import) is beta.thestorygraph.com. Not sure the status of its source code but I heard an interview from the founder and she and her work so far seems great.

If you use the https://openlibrary.org/volunteer page and click the Librarian link, this will send an email to Lisa, our head community librarian and we can help you access our slack channel and request access to our librarian features. We also have an optional weekly call for folks to raise issues and questions (e.g. "I want book series!")

More about librarianship @ Open Library: https://openlibrary.org/about/lib

If you hit any issues with this process, feel invited to send me an email to mek@archive.org.

https://github.com/internetarchive/openlibrary/issues/new/ch...

If you tag @tabshaikh who helped implement the importer and me @mekarpeles we can make sure it gets triaged and tagged correctly this week :)

As a project of the non-profit Internet Archive, having a trusted catalog is pretty paramount to what we're trying to accomplish and Aaron Swartz's original dream for an Open Library.

I've created an issue for this feature (as it's something we've discussed doing for a while). I'll also be adding this to our upcoming Tuesday 2021 roadmap discussion.

https://github.com/internetarchive/openlibrary/issues/4263

Thanks for raising this

For an invite, please send me an email at mek@archive.org or go to: https://openlibrary.org/volunteer

# APIs & Data Dumps

- https://openlibrary.org/developers/api

- https://openlibrary.org/dev/docs/api/books

- https://openlibrary.org/developers/dumps monthly data dumps for if you need bulk access and the APIs are not enough.

# Spread the word

Also, if you want to help raise awareness of this resource, please help us get the word out on twitter!

1. https://twitter.com/openlibrary/status/1338185940469051392

2. https://twitter.com/openlibrary/status/1338186553915367425

# Issues

Thank you all for helping us discover some issues with our goodreads importer and search (recently migrated to Python3 + thanks @cdrini et al for these fast bug fixes! If you notice an problem, please help open an issue here: https://github.com/internetarchive/openlibrary/issues/new/ch...

# Learn More

- https://archive.org/details/openlibrary-tour-2020/openlibrar... if you want to learn more about Open Library, here's a short intro vid.

- https://github.com/internetarchive/openlibrary if you want to follow on github.

https://github.com/internetarchive/openlibrary-client/tree/m...

https://github.com/internetarchive/openlibrary/wiki/Library-...

This is an active conversation on our slack. Send me an email and I'll be happy to invite you to join the discussion.

Honestly, a lot of our Librarian bottlenecks aren't engineering, they're documentation and organization related.

Since Aaron Swartz started Open Library ~2007, there have been scores of trailblazing Librarians (like Karen Coyle and Jessamyn West) who have nudged us in the right direction and documented their opinions.

I've done by best to organize these learnings here: https://openlibrary.org/librarians

We could really use help here if you or anyone else is interested in doing some digital archaeology and helping us better organize (so we can make good schema decisions to move forward).

Thanks for bringing this up.

Fun fact, by the way: Nearly every page on Open Library is a json API of itself if you add `.json` to the end.

e.g. https://openlibrary.org/books/OL7353617M/Fantastic_Mr._Fox & https://openlibrary.org/books/OL7353617M.json

Librarians (which is a permission role) also have the ability to merge Authors, so data improves over time.

Furthermore, members of our community coordinate with Wikidata, VIAF, etc -- we match up identifiers and sync data (and in this way, bots are able to make meaningful changes in bulk on a rolling basis).

See: http://github.com/internetarchive/openlibrary-bots (most of which are built using the OL client: http://github.com/internetarchive/openlibrary-client)

e.g. https://github.com/cdrini/openlibrary-wikidata-bot

I think many of us (in the Open Library community) want to see many players in the space and as much creativity as possible. We love books and we love giving book lovers options. Inventaire, LibraryThing, Goodreads, and fivebooks are all amazing services which have pioneered different, amazing experiences. And each brings something unique to the table. Goodreads is loved by 90M people.

It's also true that it's challenging for any single application (e.g. Goodreads) to serve 90M diverse users in all the ways they may prefer.

We see opportunities for the community to be better included in helping cultivate a more inclusive and diverse book ecosystem. Will it be better than service X? I don't care as much, so long as it allows researchers, students, enthusiasts, parents, and other patrons (who may be left behind by current solutions) to achieve their goals.

In order to support this mission, our primary objective at Open Library is making sure useful book and author metadata is as accessible as possible via APIs, data dumps so the community can build the right solution for their use cases. And of course, doing the best we can to build basic, useful tools to help connect readers to their reading experience and to each other.

We're a very small team (staff of 2 eng -> 4M patrons) so without the community being excited and driving, what we're able to accomplish is very limited, beyond keeping the lights on and the book data happily flowing.

So I want to echo your sentiments olah_1. As a non-profit, our goal isn't to out-complete or replace any existing service. But it very much is to provide an "alternative".

An alternative where every netizen and book lover has the opportunity to shape the future of their library. Where there are books which represent everyone. Where everyone's use cases may be served (if people are willing to put in the work). And where the limit of what's possible is bound only by one's creativity and a shared vested interest in being a good neighbor.

https://www.lesswrong.com/posts/xg3hXCYQPJkwHyik2/the-best-t...

Recommendations are better when you have enough info to draw a line rather than consider a single point.

There are many websites online which give you detailed reviews. But how does their review compare to other similar books? And has this reader also read those books?

Thebestbookon.com requires a reviewer to have read 3 books and justify choosing one of them as a winner (like a college basketball bracket).

Open Library is an free, online California Library with millions of digital books to read and borrow. It's additionally an open catalog of millions of books which you may track like goodreads.

Best part? If it's not to your liking, the whole project is open source! https://github.com/internetarchive/openlibrary

Open Library has a very strong volunteer tribe of book-loving librarians, developers, and designers working to make the project better for our community.

We meet once a week at 11:30am PT. Ask me for the link: mek+ol@archive.org

RIGOROUS BOOK RECOMMENDATIONS: We're also cultivating a 2nd non-profit open source experiment called TheBestBookOn.com (it leverages Open Library's catalog) which allows book-lovers to ask for or make rigorous book recommendations:

https://github.com/Open-Book-Genome-Project/TheBestBookOn.co...

It's a very early prototype (we discuss development during our weekly Open Library call). It's being organized by Lauren Milliken, Aasif Khan, and myself and we'd love more contributors to join the discussion: mek+bbo@archive.org

Been here for ~5 years.

There's pretty high leverage for impact. It's very fun and satisfying work. Big community of passionate, smart people who care.

This can be true... and at the same time, I'd say there hasn't been a day which has gone by where I haven't felt impostor syndrome as I'm forced to compare my work against the aggressive forward progress achieved by well funded and staffed for-profit organizations.

One of the reasons talent steers clear of non-profits is circumstantial. VCs (who have a vested interest in smart people starting companies or joining their portfolio companies) encourage smart people to do startups. Makes sense, not bad advice.

But this also results in a strange form of dissonance where I'm perfectly certain millions of people could do better work (better design, engineering, and PM'ing) than I'm doing, and yet in the past 5 years, no one has come along and done it.

I think it's in part this self fulfilling prophecy why more, talented people don't try the non-profit path. You likely won't earn stock and may not save up enough to buy an Bay Area house.

1. The impact at non-profits can be extraordinarily high.

For years, I was the only PM / SWE / ops / dir of openlibrary.org responsible for ~3M+ patrons. We were enabling millions of meaningful book borrows for patrons and students. I've been able to independently drive large programs forward, shape partnerships, and am afforded a massive amount of trust and agency w/o having to escalate through a chain of 20 directors.

2. Many non-profit orgs (Wikimedia, Internet Archive, Mozilla, EFF, et al) offer competitive salaries

(I guess SF is a magical place) I'm still able to earn (sans stock) w/ a competitive 6-figure salary.

3. Non-profits can offer opportunities for self-directed personal growth

As someone who considers themselves an early stage "startup person", I've been able to (at my discretion) participate in community organization (50+ contributors), project planning & management, design, devops, software engineering, program creation, and partnerships (I've helped lead a design partnership with the Tradecrafted Group & Rutger's University). I've mentored 3 interns and have had ample opportunities to participate in public speaking.

4. Flexibility

I work remotely with a diverse group of people from all over the world (Canada, New Zealand, Switzerland, Amsterdam, India, Italy, etc) :)

5. The team & mission

The people I work with care so much.

Caveats: Non-profits are like startups in another way; I've found myself perpetually having to move quickly and balance time constraints (knowing even if I build something people love, the "rocket fuel" or funding is seldom available to scale up the team we need). This also means (at least for Internet Archive) it can be hard to specialize and I don't always get to build the sexiest solutions with the latest tech.