EventMachine: scalable non-blocking i/o in ruby
timetobleed.com
timetobleed.com
That said, though, what separates the "men from the Erlangs" is that in Tcl, Ruby, Python, Javascript, and company, "never block the reactor" is the name of the game, you can't do a while { 1 } { something } in your event handler or it will wedge the whole thing.
In Erlang, thanks to its built in scheduler, you can.
In Erlang (or a similar multiprocess / actor-based system) you just kill the computation you want to stop. It's up to the killed side (or their supervisor) to know how to handle it - but it doesn't happen during the next computation loop, or when it's convenient - it handles the condition now.
[1] http://aspn.activestate.com/ASPN/Mail/Message/tcl-core/37509...
Essentially what you're saying is that Erlang allows you to make idiotic mistakes?
Event-based libraries are great hacks on top of weak languages/runtimes that allow those weak languages to hobble forward into the present, but they shouldn't be confused for the right approach in the future. I do hope they don't become trendy.
Of course, look for them to become trendy, then for one or more of them to rediscover the pleasure of actually writing code in a straightforward manner, so they will provide you with a preprocessor that tries to do the chopping up for you, but still won't solve the problem of doing too much work in one timeslice. Then they'll rediscover that cooperative multitasking was abandoned years ago for good reason.
You know, there's nothing new under the sun, but sometimes you just wonder... this much hype around cooperative multitasking? Seriously?
I don't want anything else to decide what order my code runs. I want it to run in the order I decide.
Once you move away from the stupid "I want these 100 things to run at the same time", and realize that's impossible unless you have 100 CPUs, then you can start to move forwards and learn a better way.
Event based / async programming is simple once you get used to it, it's hugely liberating, and _extremely_ efficient. It reduces code complexity, and means you _know_ what order your code will execute in. Something that I kinda like knowing.
Event-based async programming is a hack around language weaknesses and is a strict subset of Erlang's capabilities, in that anything you could write in those frameworks you can write in Erlang. Yes, including "knowing when things execute". In fact I'd say you get more control because you have more choices; you don't have to lock things up if you want to actually run a computation of some sort.
I suggest you spend some quality time with http://www.erlang.org/doc/design_principles/users_guide.html . You can stop at the "included applications" chapter. Erlang is a friend to event-based stuff, after all, it just handles it naturally, without you having to do the scheduling unless you want to.
You can be 100% sure that it won't suddenly stop executing here, and jump over to some totally disconnected bit of code for a while, then jump back.
You don't have to worry about a whole class of bugs which take up weeks/months of peoples time.
Not sure what you mean @ timers, I was really talking about async/event driven in general. Timers seem like an implementation specific detail on top of that. I haven't really seen any issues myself there :/
I reiterate, you need to spend some quality time with Erlang. This time I recommend http://erlang.org/download/erlang-book-part1.pdf ; Erlang has some powerful answers to those "whole class of bugs".
(Actually, not being aware of that does rather make sense of the rest of your views. I have first-hand experience that Erlang really does make those problems go away; I like to say that Erlang takes concurrent programming from an exponential problem to a polynomial one. It still isn't "trivial" and I don't know that it ever can be, but it makes it so a human can do it.)
And I guess you also want to manage all your memory yourself?
Throwing your code at a compiler/CPU and saying "Here! you decide what order to execute this in", means I have to write MORE code to deal with that.
eg asyc/event programming => less code, less complexity, more efficient
Managing memory yourself would mean writing more code and more complexity.
So it's comparing apples to inverse apples.
case recv(10, 5) in
timeout -> handle_this_and_throw
data -> do_something(data)
In an event based scenario:
def got_some(data):
buffer += data
switch(current_processing_state):
case state1:
buffer = try_to_handle(buffer)
case state2:
....
I see more code and more complexity in event style. Care to provide some counter-example? (and I'm not saying it's not possible - just haven't really seen one yet)
On event, do some big, long calculation.
Why should you need to write the code differently from that? It's nice that the language/environment handles things for you, rather than manually dividing things up into little chunks.
Show a calculator.
Why should you need to write the code to do each individual part of the calculator?
I disagree, dividing up the work yourself into chunks means you retain complete control of when your code executes. That actually decreases the complexity of your code, since you know when everything will execute.
Ruby and async network programming go together like gin and vermouth. It's one of the reasons I'm so happy with Ruby's block/lambda support, and so irritated with Python's: many of the basic programming idioms for evented network code have direct support in the Ruby language.
I've done a fair bit of Twisted Python programming in the past, and I find EventMachine to be far more intuitive and flexible. On the other hand, EventMachine will crash on me once in a blue moon, and it swallows exceptions, which is extremely frustrating. They're both great packages.
The big problem that still needs solving for EventMachine is evented adapters for an ORM (any ORM). You've always been able to talk to MySQL over an evented socket, but that's clumsy compared to DataMapper.
I am still totally not sold that Ruby threads are tenable, so a lot of the "advanced" stuff in EventMachine does nothing for me. It's great even without it.
FWIW, threads based on an event backend are a solved problem in Perl. The Coro library is what EM is trying to be; they should really take a look. ("Oh but it's PERL!!!!" Yeah, and it actually works.)
The way the Coro author implements Perl threads, by the way, sounds similar to how MacRuby and Rubinius are attempting to remove the GIL in ruby by copying interpreter state so that multiple OS threads can run simultaneously.
edit: after reading further, it turns out these threads are cooperative in that only one can run at a time. Also, it looks like they act more like Ruby fibers in that one must give up control explicitly or make a blocking call that implicitly gives up control. I'd rather the core scheduler be able to preempt threads directly. What if one of the threads takes a long time or ends up in an infinite loop?
"This module collection manages continuations in general, most often in the form of cooperative threads (also called coros, or simply "coro" in the documentation). They are similar to kernel threads but don't (in general) run in parallel at the same time even on SMP machines"
Usually people dismiss Coro because they want to take advantage of multiple cores without running multiple processes. But the reality is that you'll see a maximum 8x speedup if you run 8 OS threads, but if you rewrite your critical section in Haskell, you'll see a 50x speedup. So really , Coros are exactly the right abstraction for "scripting languages". IO performance is great, and expressiveness is great. Speed can be gained in other ways.
foo();
bar();
foo( cb => sub { bar() } );
But you can switch to preemption and explicit locking if you desire. At that point, IMO, it's time to ditch Perl / Python / Ruby and use a language with a sane STM implementation, because locks suck.
The one that a lot of people have gotten excited about recently is Node.js and that uses threads for file I/O (libeio).
Without defining thread, your question is pretty close to meaningless.
It works best in Thin (EventMachine) or Rainbows! (Fibers).
If you are doing I/O you should use NeverBlock (http://www.espace.com.eg/neverblock/) they've even implemented a MySQL driver, the best part is they're 'drop-in' so you don't have to learn a new API, they just hide the async calls from you.
Fibers I think are closer to Erlang style than the event loop, and much more straight forward to program.
You might look into a continuation-based framework. With continuations, you can do purely event-driven IO written in a direct style: the framework can grab the "callback" itself by snagging the continuation of its slow IO functions.
You still run into the same problems with blocking (non-IO, or non-framework) code taking up more than its fair share of time, however.
I was not impressed.
If you try and mix 'EventMachine' and 'other threadsafe code', it seems to fall down pretty hard. In my case, it would run anything executed via next_tick, but would refuse to run any timers... unless I defined a periodic timer in the main event loop, after which timers started working... but only every time the periodic timer executed.
I dug through the code to see if I could fix the problem, and gave up after trying to figure out why they seem to implement timers in two different ways, and it doesn't seem very threadsafe (next_tick is, but none of the timer code looks to be).
EM also expects that your entire application will run inside the EventMachine loop; there's no way to ask EM for a Reactor that you can then hand events.
If you're not running threaded code, then EM will probably work just fine; the Thin webserver looks to be a really nice Rack deployment option if you're on MRI.
If you need to mix EM up with anything else running on the same VM, look elsewhere.