EJDB: Embedded JSON Database for C/C++, .Net, NodeJS, Python, Lua, Java, Ruby | Hacker News Reader

EJDB: Embedded JSON Database for C/C++, .Net, NodeJS, Python, Lua, Java, Ruby

ejdb.org

119 pointsandrewstuart55 comments

The disappointing thing about Tokyo Cabinet is that an empty database is six megabytes in size. Not good for embedded systems where space it short.

I don't think this targets embedded systems. I believe the term "embedded" is used to signal that the db is embedded inside your app, and not an external server daemon that you need to talk to over some protocol.

Presumably that was one of the motives behind this rewrite for embedded?

I love things like this. I generally work with JSON anyway when I'm doing one-off things like cron jobs, and I usually just keep my data in JSON flat files, so to have options like these really makes my life easier. Great job!

I fully agree; databases that store JSON (or something close to it) natively are really useful when your application mostly inputs and outputs JSON.

I feel there are too many JSON-store databases, though. Competition is good but it's been getting kind of ridiculous.

This looks awesome, but I find it odd how similar things on HN seem to pop up in bunches. It's as if once one thing hits HN, a very similar thing is automatically submitted and promoted to front page. It's a fairly strange social dynamic to be sure.

I posted this link because I saw the post on the HN front page to unQlite and I thought I saw an embedded JSON database recently that's more interesting, so I posted a link to it. Whoever decides what goes on the front page of HN likes open source stuff with no commercial agenda with buzzwords around recent and interesting tech, so they bumped it to front page.

A lot of that is one post starting a train of thought in others. "Oh, I saw something like that recently" or "I think I know a better whatever" and so forth - that train of thought either is or leads them to something on the subject that they feel worth posting. Sometimes this happens repeatedly.

Sometimes it is blatant "karma whoring" (check parts of reddit for many examples of this getting out of hand), but usually (here at least) it is genuine people thinking that if you were interested in X, you might find Y informative/useful/amusing too.

People want karma points so if they see something is of interest to the crowd, you get more of it.

If only karma was cash. I'd be a hundredairre.

Why would anyone want karma? Really interesting.

Personally I'd like to have enough to be allowed to downvote. Not exactly my top priority, but it'd be nice.

Some people take point scoring all to seriously, like any +1 in some way validates their decision to get up this morning.

We all crave a little validation in life. Some just look for it in what others would consider the most mundane, not-important-in-life's-grand-scheme, of sources.

I think one of the key points is being able to run the "same" lib in different programming languages. This is a model of thinking that need to be followed as much as possible. If not, we are rounding in circles with zillions of libraries created for Javascript that already are present in other programming languages the energies end up unfocused. At the same time we need tools that help us in this approach, like HaXe or SWIG.

One lib that I love is NetworkX, If you look at the source code it can be easily ported to more programming languages. At the same time it is impressive what Cocos2D achieved with their Javascript bindings and HTML5 native support.

I was just asking the other day on #python if there was a nosql embedded DB like sqlite. I will definitely be checking this out. Might make things like my one-off test web scraping much easier.

http://www.reddit.com/r/Python/comments/1bap6u/embedded_docu...

Lots of good links in there. Thanks!

Exactly my case. I have this web scraping based on scrapy and I was dealing with some big json files. I'll try it for sure.

Couldn't you layer a nosql engine on top of sqlite?

There's Berkelydb, that's a KV store.

I discovered this a few weeks ago and realized it might be a great fit as a data store for a 'local, self-contained' web application. That is, a web app that is meant to run locally & in the browser (like IPython Notebook) but needs a database for local persistence, without the complexity of using the filesystem for storage. It's basically the benefits of SQLite with a simpler, NoSQL interface/design.

It is based on Tokyo Cabinet (which is under LGPL).

On the homepage of Tokyo Cabinet, it says:

"BTW, do you know Kyoto Cabinet? Actually, it is more powerful and convenient library than Tokyo Cabinet. At this distance of time, Kyoto Cabinet surpasses Tokyo Cabinet in every aspects. I strongly recommend you to use Kyoto Cabinet."

I wonder why that isn't used. Maybe because Kyoto Cabinet is under GPL.

Indeed, this is probably the reason. The company that developed ejdb (Softmotions) apparently uses ejdb in their own proprietary software, and they wouldn't be able to if it contained any GPL components.

This is awesome, I'm wondering how it performs for the average web app. Are there benchmarks load tests anywhere?

Just off my basic understanding of how the database works, I don't believe a production web app would be a very appropriate use-case, since considering it's an embeddable database you'll be bound by the slow filesystem. Mysql/postgres or mongodb would be better options. This is more for self-contained applications such as cron jobs or utilities.

Since the underlying storage system is based on a modified version of Tokyo Cabinet [0], I don't see why it wouldn't be appropriate for small apps where a discrete DB would be overkill.

[0] - http://fallabs.com/tokyocabinet/spex-en.html

Really? I'd be interested to hear why it can't be used as a web server database.

Again, I must offer a disclaimer that I only read a fraction of the total source code, but I would assume it would involve similar use-cases to sqlite [1]. For example, IIRC, the Firefox web browser uses (used?) sqlite for storage. However, reading over the list I just referenced, I must correct myself and say that for most websites, you would probably be ok in using it; however, this seems to be an embeddable replacement for mongodb, which is typically used for high-traffic environments anyway. I'm coming more from a background of high-traffic sites with large datasets, so I typically rely upon more tailored solutions, but I should be careful not to see everything through my own narrow perspective!

[1] http://www.sqlite.org/whentouse.html

Very nice. I'm writing a Python app where this would come in very handy. Can't wait to try it out.

(Maybe silly) question: if we're talking about the node implementation, why use this over mongodb?

Embeddable databases are nice and portable. It's like Sqlite versus Postgres - the latter may be more robust, but requiring it as a dependency greatly increases the effort needed to deploy your software. If an embedded database can do the job, it may be worth it as a matter of convenience.

Thanks!

Same reason why one might use SQLite over MySQL. Ease of configuration, no need to have a full blown DB server, lighter weight which makes it great for things like Raspberry Pi type hardware. For hackathons I've found it simpler to just use SQLite so I don't have to deal with MySQL configuration headaches that come up when IP addresses inevitably change.

Outside of NodeJS, I can see this being pretty handy for iOS development. I'm not a huge fan of taking JSON from an API, converting it into an NSDictionary, and then placing it into Core Data and RESTKit doesn't always fit my needs. I think this might work well for local persistence of API data for an offline mode.

Are there issues with concurrency though when used in a web application?

Yeah. But if you're not publishing it to the entire world, you probably won't run into them. And there are many other applications where databases are useful, that aren't network servers of any kind.

So is it possible to use this as a database in a web app? What sort of problems would come up with concurrency?

Since the storage engine is a modified version of Tokyo Cabinet, I am assuming that you can have simultaneous readers, but a writer will lock everyone else out. From the Tokyo Cabinet specs [0]:

'Tokyo Cabinet provides two modes to connect to a database: "reader" and "writer". A reader can perform retrieving but neither storing nor deleting. A writer can perform all access methods. Exclusion control between processes is performed when connecting to a database by file locking. While a writer is connected to a database, neither readers nor writers can be connected. While a reader is connected to a database, other readers can be connect, but writers can not. According to this mechanism, data consistency is guaranteed with simultaneous connections in multitasking environment.'

[0] - http://fallabs.com/tokyocabinet/spex-en.html

I'd switch to a full DB server for a production Web App. If you're using an ORM it should be pretty trivial (the one time I've done it, Perl's DBIx::Class made it as easy as changing the connection info hash). I'm not sure how many people hand code their SQL at hackathons but I am not one of them ;).

I use SQLite in small and medium-traffic web apps (the largest one getting about 80000 hits daily) all the time with no concurrency issues whatsoever. My rule of thumb is that if you don't expect your app to require more than one server, sqlite will work just as fine as postgres or mysql.

Interesting! For pragmatic reasons, or stress testing SQLite?

For convenience mainly.. It saves me from having to install a "serious" RDBMS, import data, etc. With SQLite, deployment is simply copying the db file to the server.

Use coroutine based servers, everytime there's guranteed one thread accessing the db.

This looks interesting.. I have used nedb on a small project, That is another file based database, uses the same API as mongoDB. nedb is just an npm, so its only nodejs supported right now, and the feature list of EJDB looks more impressive anyway.

Would be pretty cool if replication can be achieved

So you've got C/C++, .Net, NodeJS, Python, Lua, Java, and Ruby covered ... but PHP is a no go?

Why not change it to we're on everything but PHP? Seems more succinct.

I'm not sure how to say this without being inflammatory ... there is a universe outside of PHP, and PHP is not at the center of that universe.

> there is a universe outside of PHP, and PHP is not at the center of that universe.

There is only one universe and PHP is a part of it. A big part. That's the reality.

Some of us have to work on multiple platforms with multiple languages and it's truly a disappointment when the authors clearly put in an effort to cater to the community where multiple platform support is prioritized ... but then exclude a platform due to personal bias.

That makes this project a no-go for me.

I would say start treating the authors with the respect that they, as human beings, deserve. I too would like PHP support, but, if it's not there, I can still appreciate the hours and hours of labor that go into the support they do have. There's an expression "if you don't have anything nice/constructive to say, then don't say it," and claiming that it's a "disappointment" that they don't include your feature, and even going so far as to presume it's out of personal bias, is just rude.

Edit: Besides, if you're running php, you're probably developing a web app, in which case you should be using a database like mysql or mongodb anyway, not an embeddable db.

Why don't you show them who's boss and write your own damn driver. Nobody owes you cookies cause you have a glass of milk in your hand.

Like many Open Source projects, I would assume that Pull requests are welcome.

PHP might be in the pipeline.

So write it yourself if it's so important to you.

Can't please everyone.

It's just nice to see a project where they put the effort into driver development. Even without PHP it's an impressive list of drivers.

It seems there is more demand from Lua developers.

And Pike! :)

I really wish this little language had more publicity.

Reply on news.ycombinator.com