Hacker News Reader

Comment by plinkplonk | Hacker News Reader

"We are moving more and more of the core database in C/C++, while still using many of the concurrency and reliability design principles we've proven with the Erlang codebase. And Erlang is still going to be part of the product as well, particularly with cluster management, but most of the performance sensitive portions will be moving to over C code. Erlang is still a great language, but when you need top performance and low level control, C is hard to beat."

This is interesting. If I remember correctly, CouchDB was first written in C++ and then moved to Erlang. Now the project has come full circle (which is fine of course).

View on HN

I don't think it's coming full circle. They're just properly separating the project code, so that the time critical code is in C/C++ while the scale/fault-tolerant code will be in Erlang.

Erlang or C++ is not that important.

The real drug on performance is using javascript for queries & M/R and HTTP-based APIs vs. binary protocols. Moving to UnQL and Memcached protocol may solve some of the performance problems.

There is a reason antirez choose Lua and not JS for a scripting language.

I agree with much of what you've said there. You clearly understand enough that I don't need to go into detail describing why I agree. :) But this isn't quite right:

> There is a reason antirez choose Lua and not JS for a scripting language.

To be fair, I embedded lua in ep-engine well over a year ago, but haven't ever released it. As it turns out, v8 is performing very well lately (certainly faster than plain lua) and people get it more easily. I've done some fun things with lua and it's really fun and easy to embed, but whatever gets shipped has to be supported and lua has some pretty dark areas.

Oof, seems like a lot of inefficiency in retraining, redocumentation, converting, etc. Not to mention the debugging issues between two different technologies.

Why would it be fine?

I think plinkplonk was saying that it's fine (not necessarily bad) to come full circle in general, not that it will work out in this particular case.

"it's fine (not necessarily bad) to come full circle in general, not that it will work out in this particular case."

exactly. I was not saying this step is specifically a bad (or good!) one. I didn't want the thread to devolve into a "which is better Erlang or C++?" flame war. I am sure the devs on the project have good reasons for the change. I just thought it was interesting that the project came full circle.

Yes. Erlang is great for proving ideas and making something work reliably. I have mad respect for Erlang, it's the kind of language where one guy can code a whole a distributed database. But when you need high performance and low level control, and you have the resources and team to make it happen, you really can't beat C. I'll write more about that soon.

Erlang for abstraction and C for performance sounds cool. I'd be interested in how much (or whether) you are using C++ (vs C). I hope you'll address that in your write up.

ep-engine (the core of membase and what handles all of the data moving around between networks, memory and disks) is almost entirely C++.

Are you using any C++ libraries for the networking stuff or plain OS specific socket APIs?

The networking stuff is the memcached engine branch. It's pure C and loads ep-engine as a module for doing all the backend junk.

I wonder if Riak will ever get to this point.

To what point? Already large portions of Riak (the performance-critical ones) are in C, and have been for a long time.

Oh, I didn't realise much of Riak was in C, and the website now mentions Erlang a whole lot less than I remember. Still, in the technology stack http://basho.com/technology/technology-stack/ Riak Core seems to be all Erlang. Bitcask is possibly C--is that what you're referring to?

Your language and line of questioning leads me to assume that you're perhaps not that familiar with Erlang.

Erlang has facilities for running native code within the VM (NIFs and linked-in drivers) and for interacting with non-Erlang processes in an Erlang like way (ports and c-nodes). Extending the VM via these mechanisms is not the default path but it's also not unusual - much of the standard lib is implemented via these mechanisms after all.

To answer your question, bitcask, ebloom, eleveldb, erlang_js and skerl all have c_src directories which suggests they're likely either all or in part implemented via NIFs or linked-in drivers.

EDIT: You might also find http://vimeo.com/17078993 interesting viewing.

Have you considered a language like OCaml or Haskell? You can still get many of the benefits from Erlang while having near-C performance.

You usually get things correct the second time you write them. You are missing critical knowledge on the first-write.

Reply on news.ycombinator.com