Nanomsg – MIT-licensed ZeroMQ alternative
freelists.org
freelists.org
This probably has to do with both the transition from C++ to C, and the fact that Martin (the author) has a better idea of what he wants to make. In other words, he is (painfully) aware of the errors he made in his ZeroMQ and is doing it right from the first time in nanomsg. This leads to clearer and more consise code. The reduction of strange bugs and perhaps better performance adds a lot of icing on the cake as well. I look forward to using nanomsg in my future projects!
EDIT: github link, https://github.com/250bpm/nanomsg
Now I still prefer C++ at work for certain things as I'm simply too lazy to do it correctly in "C" - give me STL, Qt, any UI framework written in there, and it's easier to use. But for core/system libraries (zlib, png, jpeg, tiff, zmq, libpq, etc.) I'd rather have it in "C", at least on the interface part, but I've seen what hiding of C++ or .NET can do with a crazy debugger :)
I would say we are on the path of separating the two. ZeroMQ does away with the centralised server holding all the queues. It still has local queues though.
nanomsg goes even further and gets rid of queues entirely. What's left it networking buffers, typically implemented at L4 of the network stack.
In a friendly, ideal environment (like the Internet of the 1980s), req/rep is "just" TCP/SCTP unicast, pub/sub is "just" UDP multicast, and so on. Issues like "message routing [and] scaling up" are supposed to be handled on the link layer (i.e. using ARP, BGP, etc.); issues like handling partial failure, on the transport layer (if IP stacks weren't stuck in the 70s, we'd have such a thing as multicast TCP, and "send TCP packet to host set, wait for ACK from all hosts, and retransmit to failed hosts" would simply be its obvious send() system-call semantic.)
We've just screwed the "friendly, ideal environment' pooch by putting not-very-intelligent NATs and firewalls everywhere. So we have to, sadly, reimplement all these nice things on top of the few things we've kept working: TCP and UDP unicast from clients to "business-leased" IPs with open+addressable ports. The necessity of things like this, UPnP, and TURN/STUN/ICE sadden me sometimes, they really do; they're 100% examples of an inner-platform[1] created because software developers have gone on for so long only touching OSI layer 7, and thinking about the rest of the infrastructure as a black box, that now the application layer is the only place you can solve problems, because the infrastructure will no longer transport your packets if they do anything special below it.
So I say, please, look past your abstractions, programmers, and learn what makes the Internet tick! It's worth it to understand what your packets are doing--you might want a property that you can already get by asking something different of a lower layer. Or, even if not, you might just become--as Bret Victor says--very unsatisfied with the state of things. :)
Oh, and start here, if anywhere: http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html
---
The main distinction between L4 and scalability layer is that L4 works between 2 nodes, scalability between arbitrary number of nodes.
There's the special case of UDP multicast, but the multicast itself feels more like a hack rather than a real L4 protocol.
were zeromq's queues just in-memory? it was never meant to be reliable in the sense that messages were preserved in a database at each socket, was it?
alternatively, if i am wrong above, and zeromq was reliable / persistent then why did you drop this?
thanks (i've worked on esbs, but not with zmq).
Now it's in kernel-space (TCP rx/tx buffers). That actually makes it a little more reliable as tx buffers are not necessarily dropped when the application fails.
Yes.
> it was never meant to be reliable in the sense that messages were preserved in a database at each socket, was it?
No. It's meant to be a fast inter app messaging protocol, it's not a queue, it simply uses some in memory queues internally. It's not a queue, it's an improved socket library, having MQ in the name was clearly a mistake that he's now corrected.
1. use an intrusive container in C++.
2. Just use a `vector`, faster in most cases, regardless of complexity.
3. Read about and use allocators properly.
Henceforth, I'll be using this quip whenever debating C++ fan boys; thanks!
Never mind that I've seen plenty of C code with wrapping structs for data structures, too. Intrusive vs. non-intrusive is a design choice in both languages.
In his blog post, he complains that std::list<widget* > causes 2 allocations per insertion. Fine, because that's not idiomatic C++. The C++ way would be to use std::list<widget>, since there's no need for the extra indirection. Note that this would only be a good choice if you for some reason really need a list, std::vector is mostly preferred due to its better cache behaviour and requires even fewer allocations.
Secondly, if he needs to remove list items by their address, there's no reason he couldn't use an intrusive list in C++. The article I linked shows an implementation of one, and there's always boost::intrusive::list if you don't want to roll your own (and you shouldn't).
For the use case required by ZeroMQ, it's faster in all cases, because swap-with-last and shrink is O(1). Sure, if CPUs didn't have caches, a list might be faster.
Do you see where this is going? The C way of solving problems consists of adding one more level of indirection (ever heard of 3-star C programmers?), because it becomes more efficient asymptotically that way. Nevermind the hidden factor of 100-1000 due to pipeline stalls and cache misses.
Fortunately not all C programmers think that way.
Removal with intrusive containers: flip two pointers.
Removal with std::vector<>: copy all the subsequent items in the vector one position backwards.
A vector will still give you better cache-locality and also amortize the number of allocations.
I've probably spent more years writing C++ code than C code, but I'm all with him on this - if what you need to do for whatever reason don't need/benefit all that much from C++ when accounting for all constraints, then choosing C as being the lowest common denominator between other languages is a very good choice for a library.
Ownership of internal resources can be transferred, and emplace_back even allows constructing your object in place.
As others said, the O(n) really becomes more like pseudo-O(1) due to cache effects unless you have elements the size of your cache lines (in which case prefetching still helps) or you're only fetching one element at a time and then triggering a context switch (as in a scheduler).
I'm not bashing your library, and if you prefer to use C then that's great, but it's kind of unfair to blame it on the language in the first place.
This is just inertia and only true on UNIX world due to how C is tied to UNIX.
BeOS, Symbian, Genode -> C++
Mac OS X -> drivers are done in C++ (IOKit)
Windows -> C is now official deprecated and C++ is the way to go. (http://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and... && Herb's remarks at BUILD 2012)
Now C++ comes with new/delete, dynamic_cast<>, throw/catch/etc, RAII, always export "C" interface (to avoid mangling) - While all of these are useful, due to the constraint where the language has to be used they can be forbidden. It's no longer functions that you are restricting from being called (as in "C"), you are forbidding to use parts of the language that are considered by many as vital, and it's basically a different language at this point.
Even if that's the only C++ feature that a project were to use, I'd still consider it a net gain. Anything that makes your program less susceptible to manual resource management bugs in "goto cleanup"-style C is worth using. This doesn't just include memory leaks, but more insidious problems like unclear ownership of various system resources (sockets, file handles...). Things not even a GC would help you with in a long-running server process.
There are several popular "C" libraries that use them - jpeglib comes to mind, but there are others. Also certain language's runtime environment uses it. There is a way to handle it, but you need to take care.
To get defined behaviour with regards to destructors, you simply have to make sure that the function calling setjmp doesn't internally use RAII. Call a wrapper function to do the setjmp and you're safe.
Of course it's a whole other question whether using setjmp/longjmp to implement "exception handling in C" is a good idea in the first place.
1) https://developer.apple.com/library/mac/documentation/Device...
Yes, it is true it wasn't well accepted in the community and never really took off.
Still it is a C++ subset, not C, which is what this discussion is all about, regarding the use of C++ in kernel space.
1) He can solve the issue by requiring C++11 (!!)
2) Just rely on cache-effects making it pseudo-O(1), unless (insert various cases where it might blow).
Sure, he can jump through 200 hoops to use C++, special features, rely on CPU cache behavior, finetune to death, etc -- or, you know, he can use C and be done with it...
2) You missed the part where there's nothing preventing anyone from using intrusive lists in C++.
template <typename T>
void remove(std::vector<T>& v, std::size_t i)
{
std::swap(v[i], v.back());
v.resize(v.size() - 1);
}That wasn't an option for ZeroMQ which was started in 2007.
template<typename T>
struct Link
{
T obj;
Link *link;
}
The way you're thinking of would theoretically work, but you'd have to make sure the compiler isn't adding fancy padding that would break your pointer manipulation and type casting.
boost::intrusive::list on the other hand, works in a type-safe manner by having your class either inherit a list hook, or provide it as member and then pass the list implementation a member pointer as template parameter so that it can be accessed.
For a library that is at such a basic level of infrastructure (which, if it's stable enough and popular enough deserves to be kernel-level) C++ is probably more trouble than it's worth.
The higher you go app the stack, the more reasonable it becomes to go farther from C (high enough, and even ruby makes sense - though it wouldn't for 0mq/nanomsg)
In this case, I would simply say that the author problem is more about how he relates to C++ than about C++ itself.
Saying C++ is abrasive is a valid point, but saying it cannot be as fast as C or it cannot do "something" is stratospherically stupid.
The original post only claims that it makes his life easier, and then makes two further claims about the C implementation without stating you can't achieve the same in C++.
He then links to a post where he goes into more detail, including _explicitly pointing out_ you can do the same in C++, but that he believes that the language encourages you to think in certain ways that negatively affects these factors.
It seems to me that he is stating his reasons for why he believes C is the best choice, and particularly for him, without making any claim that it is objectively better.
In other words, unless I've completely missed something, you are setting up strawmen.
The comments you are referring to where he moderates his position are resulting from the comments he received from the C++ community regarding his incorrect statements.
now there's other things i'd criticize, like nanomsg is a trademark of the company.. so if you repackage it i'm suspecting you have to rename all the stuff, for example.
However, even then, if you modify the code you should name it differently to prevent confusion.
I'm not sure the trademark is a big deal, it's probably to prevent misuse like what happened with the Linux trademark (which took a few years to resolve). I doubt he's doing it to make it more annoying to use.
1. Write the whole thing in valid C89
2. Prevent users from having to use Boost or the rest of the C++ nonsense
3. Users can now write trivial language bindings and link at runtime easily.
4. Profit.
~
C++ is a cool bro and all, but honestly the solutions you've listed make it annoying to work with this code outside of C++ on a platform with subgood template support.
Moving a simple utility library like ZMQ to C makes a great deal of sense.
Now, it's a very useful library, but it's also an amazingly tired refrain.
See `man epoll' for an excellent explanation of the difference.
http://funcptr.net/2012/09/10/zeromq---edge-triggered-notifi... http://funcptr.net/2013/04/20/embedding-zeromq-in-the-libev-...
"iMatix CEO Pieter Hintjens registered the zeromq.org domain in May 2007 and started the ØMQ project together with Martin Sustrik, who was its architect and lead developer through March 2012.
[..]
In March 2012, due to disagreements over control of the project, some of the original developers forked ØMQ as Crossroads I/O. Martin Sustrik has started nanomsg, a reboot of the ØMQ core library."
The ØMQ library is licensed under the GNU Lesser General Public License. All add-ons and examples are published under the GNU General Public License.
- Globally constructed objects, called before main(). Yes you might say, don't use them, but once you are in C++ land you can't be safe that everything else you've decided to use does not use them.
- Throwing exceptions. This is mostly debugger pain. If you've used (again) without knowledge underlying code/library that uses exceptions (and note, you don't) then you might be inadvertently throwing them to the user.
- RAII does not work with longjmp/setjmp. If you have callback mechanisms, and your client does longjmp on you, this won't work with any RAII done objects.
- C++ runtime. You would be requiring this to be loaded, which might become nuisance in situations where your main application never used C++, you wrote a plugin/shared lib using C++ code, and now there is the requirement. Not a big deal after all.
- C++ runtime again but for MSVC (although it's for the C runtime too to a point). Now while you can make code that is more independent from the "C" runtime for MSVC, it's harder to maintain this for C++ - you might run into issue where mixing dynamically (shared) libraries would have to use different (and even more different through WinSxS) MSVCPxxx.DLL libs (C++ runtime). In all fairness, this is more of a Microsoft mishap, and not caring about backward binary compatibility, but hey it's there.
- Most of all, no matter what you do - you can't expose the really useful C++ things - like stl or qt5 or something else's objects/vectors/strings/maps/etc. Well that's obvious - off course you can, but then you are basically forcing the person using your library to go that route. If your interface is in "C" not only he longer has this pressure, he can use it easier through FFI in plethora of other language/runtimes without writing wrappers, and minimizing the number of precompiled binaries that he has to provide (no linking statically to C++ runtime would turn out bad at the end).
- Case in point - STL's ITERATOR_DEBUG_LEVEL in MSVC. By default it's set to 2, which means make extensive checks of every map<>, vector<>, etc. once they are released. This severely slows down the debug. It's useful, but it's not feature that can be turned on or off. Most of 3rd party libraries, are playing safe and don't do anything special, e.g. they expect ITERATOR_DEBUG_LEVEL=2. But in your production code you might want even your Debug versions to be somewhat faster. Most of your day won't be spent tracking down map<>, vector<>, but other things. As such you go ahead and compile ITERATOR_DEBUG_LEVEL=0, and then .... well you can not link anymore directly to the said 3rd party libs. You either have to precompile them, which might be fine, until you are given Middleware that is not compiled this way. And in all cases it's very important that all these libs use at the end just one runtime, not several (as it happens when you have tons of MSVCRxxx.dll/MSVCPxxx.dll in your running image).
- And there is more. For example for statically build libraries, global constructors might be lost, if there is no symbol referring to them used by the main application. So you might lose some important initialization step.
- etc., etc., etc.
If you've got code in your library that you need to run after invoking a user callback (like free or mutex unlock) and the user does a longjmp on you, it doesn't really matter if the code you want executed is done automatically from a destructor (as in RAII) or manually. If you want to allow the user to do a longjmp from a callback you have to write the code carefully to handle this, be it C or C++.
So basically what happens, is that if you have a lone variable globally instantiated, but no one refers to it, it might be stripped from the final linking, and as such any constructors/destructors associated with it would not be called.
Now if you are linking directly to the .obj file where the variable is declared it most likely would be put into account, but if the .obj is in a .lib (.a) then it might not.
Again, do some experiments, try it out.
There is another nastiness with globally initialized variables, that one should not expet some kind of order when the things are initialized. We've had code in the past (Dreamcast) where globally initialized variables, which were assigned the values of other globally initialized variables and then some operation on them would not always work: e.g. this at the top level:
float x = 10;
float y = x + 20;
It worked most of the time, and then if someone started moving variables to other object files, it stopped.
Check the http://yosefk.com/c++fqa/ for more gotchas.
Does this mean that threading is still used/required, or could nanomsg use non-blocking techniques?
Also, that kind of thing compromises internal working of the library, if the user doesn't call do_background_step() fast enough. Timers may be delayed that way etc.
Is this true? Do windows 7/8 have any lists of open source software projects that they incorporated into the operative system? It sound as a really interesting read if such was the case.
There might be a reference to it on Windows Internals series.
Does Nanomsg has the same problem?