RestCommander: Fast Parallel Async HTTP Client as a Service | Hacker News Reader

RestCommander: Fast Parallel Async HTTP Client as a Service

github.com

35 pointsnikolay9 comments

This looks impressive but I am a little disappointed about the assumption of REST. We have recently turned away from REST and are moving toward a React/GraphQL architecture. We have been studying the possibility of something like Aleph, or even Apache Storm, combined with "bolts" or processes running our GraphQL server, as a way of running a GraphyQL at scale (Node is known for suffering at scale from its lack of backpressure). I would love to see something like RestCommander for GraphQL. The Play framework certainly seems like it could help in that area, though I personally have not worked with Play, so I can't have a strong opinion about it. A devops person might insist that something like Supervisord is enough to keep things going at scale, but of course, monitoring 1,000 servers is difficult, and even Supervisord can die. I appreciate the effort to come up with something that offers us another way to support running multiple instances of our servers, and perhaps bringing the monitoring closer to the app.

I don't have much practical experience with Storm, but I was pretty interested in it a few years back.

After working with Akka more and more the past year, I have to wonder what would inspire someone to choose Storm over it though. Akka scales from millions of messages all the way don't to "I need an end-point for a web-socket", or a single Actor for unique id generation.

It's really an amazing technology. A bit of boiler-plate, but certainly no more than something like Storm and you get to a place where it's 30 seconds of setup before you start implementing your flows.

It's like that old idea that eventually you'd be able to take a Visio diagram, and just translate it to code. It's flow-chart logic diagrams made manifest. Completely unlike anything else I've ever experienced. Started with Imperative programming with BASIC, moved on to OOP and very primitive rules engines with c#, DSLs and code-rewriting with S-expressions with Ruby, introduced to functional programming by Scala. Then I get into Akka and despite working with the same language it feels like a whole other paradigm with just as significant a shift and contribution to make as the others.

So wondering across posts on Storm (or more routinely lately, Spark), make me wonder if anyone else feels the same, or if all of these tools are kind of in that same category.

Or maybe I'm just overselling Akka. But when you hear a Rubyist reverentially refer to message-passing and Smalltalk, I'm pretty sure no one I ever discussed that with ever envisioned anything like Akka. I certainly didn't. `receiver.send(message)` in Ruby is to message passing as Ruby lambdas are to functional programming (despite lacking composition, currying, partial function domains, etc). Or switch statements are to pattern-matching. IOW, not really at all.

Which was really hard to appreciate as a Rubyist before experiencing it myself.

Have you considered Akka? Found it lacking somewhere maybe? Any direct experience with Storm? Do you get the same sort of "previously unknown to me but now obviously equally fundamental pillar of programming structure" sort of of vibe?

Just curious.

Building off the parent's comment, there's a Graphql library for Elixir. If you aren't familiar, just like Akka it uses the Actor concurrency model and has the proven ability to scale since it runs on the Erlang VM.

https://github.com/absinthe-graphql/absinthe/

Actually setting up a Actor come's with a cost. However akka has a great streams library now. Which is way cheaper for basic pipelines.

I'm not sure what you're referring to about cost. Maintenance? Performance?

REST over HTTP is cache-able using CDNs like CloudFront. GraphQL isn't?

If the client queries get too complex, processing GraphQL might take a toll on the server [0]. GraphQL completely makes sense for a client with limited processing power and battery life, but doesn't offer a great deal in terms of server-to-server communication. JSON Graph by Netflix [1] is worth a look-in for a different take on the problem.

Not sure what you mean by node.js "lacking backpressure"? Node.js is a very well suited solution to GraphQL, since GraphQL implies that the workload is going to to be IO bound.

[0] https://news.ycombinator.com/item?id=11300656

[0] https://news.ycombinator.com/item?id=10365555

[1] https://www.youtube.com/watch?v=hOE6nVVr14c

> REST over HTTP is cache-able using CDNs like CloudFront. GraphQL isn't?

Whenever GraphQL uses HTTP, it is just as cache-able as any other bit of data over HTTP. Whether HTTP is the right choice is a decision each developer needs to make, based on the requirements of their project.

> processing GraphQL might take a toll on the server

I can not figure out what you mean. I denormalize my data to MongoDB or Riak or ElasticSearch. Then I send a response for each request, and the response has already been "pre-digested" and tailored to the queries I'm ready to answer. How does it matter whether it is REST or GraphQL? They should have the same performance cost. I'm not allowing anyone to hit my canonical data sources, whether its with REST or GraphQL. Perhaps your question is about real time data and mutations?

> Not sure what you mean by node.js "lacking backpressure"?

You should read "Backpressure and Unbounded Concurrency in Node.js":

http://engineering.voxer.com/2013/09/16/backpressure-in-node...

The core of REST Commander is the actor based concurrency control described at http://www.ebaytechblog.com/2014/03/11/rest-commander-scalab... We recently open sourced the http://www.parallec.io/ to support more protocols and I suppose it is not hard to add new protocols if needed.

Don't change your architecture only cause there are new shiny things.

Reply on news.ycombinator.com