Database of all Marvel characters | Hacker News Reader

Database of all Marvel characters

marvelousdb.com

96 pointsuberneo29 comments

Does this adhere to the Marvel API Terms of Service?[0]

Specifically, "Storage. You may not indefinitely cache Content and further agree to follow the content retention rules, as set forth here: http://developer.marvel.com/documentation/attribution."

This looks cool, but it seems to have just scraped the API, and I'd hate to see it taken down for violating their rules.

edit: You also seem to be distributing results of the API in the GitHub repo, which might violate this: "You may not...redistribute the Content or Tools except within your Apps."

[0] http://developer.marvel.com/terms

Here is the Blog about it --https://orchestrate.io/blog/2014/04/08/explore-the-marvel-un... And here is the Source Code -- https://github.com/asamiller/marvelousdb

Very pertinent information. Thanks!

Apparently they can't cope with two characters having the same name (not a rare thing in the Marvel universe):

http://marvelousdb.com/character/1011256

Yeah, comics have enough duplicates of everything (including, in some cases, multiple identical clones of the same character) that the only sane way way to handle it is to assign an arbitrary unique ID to each character.

It's like Django, can't deal with two column primary keys.

Here is the actual JSON for each comics and characters --https://github.com/asamiller/marvelousdb/tree/master/data/

How are the results sorted? When I search "Spider-man", the first result is Boomerang, and none of the results on the first page are for the Peter Parker Spider-man (result #2 is Ben Reilly Spider-man).

Yeah, I searched for "Nimrod" and got nothing but a few other characters who reference encounters with him. He's in Marvel Universe, so clearly they're just picking and choosing who gets in here. This database is sorely lacking, and the search function is meh... Not a knock on the Node.js code, more of a knock on the Marvel API.

I know we're not supposed to be negative on here, but this is a cut and dry Comic Book Guy situation....

clears throat

"Worst..... database..... ever...."

This was a little project I did when Marvel's API first came out. I scraped all the data and put it in a database that allowed better searching/browsing. I probably need to periodically scrape the API again to pull in changes and additions as they add them. Hopefully Marvel will update the data and make it more complete.

That explains it!

I really do like the presentation and the whole site. I just couldn't resist the comic book guy opportunity.

How in the world is this repo compatible with their TOS - http://developer.marvel.com/terms ?

This looks cool and sexy, but I have to say, I still prefer the wikia database

http://marvel.wikia.com/Main_Page

It's much uglier, but much more useful for any actual, ahem ahem, "research".

First tried searching for 'Quake', then 'Daisy Johnson' [0]. Nada :(

[0]: http://marvel.wikia.com/Daisy_Johnson_%28Earth-616%29

I'm disappointed http://marvelousdb.com/character/1017316

I appreciate this being open source but why on earth would you store this information in JSON rather than a relational DB?

Probably because the marvel comics API that this is likely pulled from gives it out that way.

Generally, if you have a relatively large chunks of non-relational data, you can use a document store.

I could imagine the data model for superheroes being a fit for documents - information on one super hero is not tightly coupled to another super hero. Sure you could normalize some of the abilities and what not, but the main chunk that you'll be searching by/storing is a Super Hero (or whatever you want to call that object).

Also, note that document stores support somethings that you would consider relational (ex. RethinkDB supports joins), so the lines are becoming blurred these days.

But what happens when you want to start associating them on when they appeared together or fought each other? What about the different iterations of an individual character in different continuities? (Captain America was like six different dudes right?)

I think a document store is the worst way to store this kind of stuff.

imho, ymmv.

What you've brought up is precisely where relational databases excel -- it's a tradeoff (as anything else, you have to decide when to use one or the other). How many relationships do you have/expect? How intermingled/related is your data? If the answer is "not much" , then maybe you should go with documents, if it's high maybe you should go with RDBMS.

Also, the lines are blurred, like I previously stated: - RethinkDB does joins - Postgres has good support for storing JSON documents

In reality, almost all interesting data has relationships. Even a two dimensional graph is expressing a relationship (and hopefully a correlation) between two sets of data.

I think whether or not there are relationships is the wrong way to consider the question of document vs relational. I think of a relational schema as a canonical way to store your data so that you can easily and efficiently project it in multiple different ways (queries).

A document schema stores your data as one possible projection of that data. Obviously you're right that if you can know for sure that one particular projection is the vast majority of queries then this is fine. The problem is that you have to be right about which projection of your data is most important, and normally before you write any code.

In my experience, for greenfield projects it's surprisingly hard to predict what kinds of projections are most important. Often what is interesting to users changes, or you didn't quite have the right projection or whatever.

To compound this, a lot of document stores often don't have good schema migration tools. Migrating schemas and data is actually a lot easier with SQL than anywhere else. Postgres can migrate the schema and the data transactionally and you can rollback if there is a problem in any of the steps. They are also type checked.

Basically, when starting a new project, you are probably better off doing a relational model of your data than anything else. Later, when you know your data access patterns, you can change to a document store and get the various benefits of them without the tradeoffs

Agree that how you plan to project the data is certainly important (SQL offers immense flexibility, assuming you're heavily normalized). But still, the point stands that you can use some document stores relationally. This makes this point a little duller.

Also, document stores' migration tools are the supported languages themselves... The only migration tool you need is a function from one document to another. RethinkDB actually supports almost arbitrary function input. Also, migrations are a little less important when it comes to document stores because a lot of the schema validation logic gets moved up one level (to the middle layer).

I strongly disagree about which one to start with. One of the biggest widely-noted benefits of a document store is that you don't have to think of your schema up front, you don't have to maintain constantly changing schema declarations, etc. Did you mix up the terms them by accident?

If you didn't mix them up, here's a concrete example of why document stores are often better to start with:

- I'm writing a chat service

- I am thinking about the message model, and decide a message has three fields, a message, sender, and timestamp.

- Three minutes later, while building the service, I realize that I also want to track the origin of the message (through some geolocation service or something), or the locale.

If I had used a relational database, I'd have to edit the schema, re-up/re-migrate the database, etc, when in a document store, I could just go on using it like the locale was there (and maybe add one line of code adding some default/null value for locale if it wasn't on objects I read from the database)

> What about the different iterations of an individual character in different continuities?

This has nothing to do with document vs relational data stores, and everything to do with natural vs surrogate keys.

Not sure if this is quite right... If you decided to add a "continuity" field that denoted which continuity a character was from, you'd probably want to use that data to relate characters -- in SQL land that would usually be a foreign key, in document stores, it's a little less clear how to do it (depending on which you pick) -- RethinkDB does (equi)joins, so you could just use a string. Can't say the same for other relational databases, I think the query would turn into a MapReduce.

I had to pull the data out of Marvel's API which returns JSON. Then the data was stored in Orchestrate.io as the database. The JSON data is included in the repo so other people don't have to scrape Marvel's API.

Did you get special permission to do this? It is against the generic terms:

http://developer.marvel.com/terms

Use of our Content. You may not change or edit the Content (e.g., modify, augment). You agree to follow Marvel's attribution and linking rules, as set forth here: http://developer.marvel.com/documentation/attribution. You also agree to delete any Content upon Marvel's request (or if Marvel disables the Marvel API (or upon termination, see below)). You agree to configure your Apps to request only Content that is needed by such App. You may not (except with Marvel's prior written approval): (i) use any Content, or the Marvel API or Tools for any commercial purpose; or (ii) redistribute the Content or Tools except within your Apps.

Storage. You may not indefinitely cache Content and further agree to follow the content retention rules, as set forth here: http://developer.marvel.com/documentation/attribution.*

Generally, you don't "scrape" an API.. the point is that you use the API. That way the information is always up to date.
You can do all of this and still stay within the terms of service.

There is one character so super he's in the database but you can't find him.

I didn't realize so many of the Marvel characters had UK citizenship.

Reply on news.ycombinator.com