Simplify Service Dependencies with Nodes
blog.twitter.com
blog.twitter.com
https://en.wikipedia.org/wiki/Zero-suppressed_decision_diagr...
I highly recommend reading "Production Matching for Large Learning Systems"[0] as it was accessible enough for me to try to make a basic RETE implementation while learning C++ and playing around with several other technologies I was unfamiliar with[1].
[0]http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-CS-95-11...
[1]https://github.com/cjslep/cpp-rete-prototype (awful, abandoned prototype)
I guess we all can at least agree that there is a way to pronounce it :)
Of course, since now it's a loanword in English, you could also just use English rules for pronouncing it (whichever those are :) ).
Yet another instance of this is the Disruptor patter that use it in an event queue.
serviceA { Promise.All(b,c).then(...) }
serviceB { (d).then(...) }
serviceC { Promise.All(d,e).then(...) }
- It's hard to figure out what the dependency graph looks like just from promises
- That pattern is verbose and results in gross types when you do it in Java
- With this library, you can easily profile parts of the algorithm to see what's going slow, what's taking a bunch of memory, etc.
Snappi introduces some mild requirements for how a service exposes it's functions, but once that is done the rest can be automated at deploy time. Not only can service-based load balancers be created dynamically, but RPC stubs can be created for each service and injected into peer services that are dependent on them. For example, if ServiceA needs to access ServiceB, it can do so by referencing this RPC stub knowing that the location of ServiceB will be injected at deploy time to ensure requests are fulfilled for the environment.
There are a number of benefits to this approach for a single application, but one of the larger benefits of consistent specs for contract creation is that individual services will be far easier to share. We're trying to create a structure that will not only empower us to re-use our own services, but also to consume services published by others by simply specifying them as a dependency.
Your work could be perfectly integrated into k8s
Essentially they construct an AST for whatever services that need to be called. Most of the time they then simply "interpret" the AST with the lower-level Finagle constructs. But, and that's actually the big point, they can use _the same code_ with a different interpreter to generate a GraphViz representation for visual inspection. This super helpful once your call graph becomes more complex .. Another possibility is to write an optimizing interpreter etc.
Anyhow, what I find a bit sad is that they didn't leverage the Scala part if Finagle more .. this would be a prime example for a Free Monad () based DSL. That way they could have avoided writing 10s of classes with less than fortunate names.
() might need a Free Applicative part as well since they inherently want to have parallelism expressed in the language
I can't judge Twitter's design without seeing it, but I don't like the sound of this:
"the simplest web page on twitter.com requires collaboration of dozens of network services speaking many different protocols"
Simplest page? Dozens of collaborating services? Many different protocols?
I'm skeptical this is the most elegant architecture possible that meets the requirements of their simplest web page.
I think the best way to upgrade your java codebase is Kotlin. On the server side, Kotlin +vertx is extremely compelling.. on the Android side, Kotlin pretty much kills it.
could you talk about the framework that you used? im particularly concerned about ORM and database connectivity.
For ORM, Dropwizard and its ecosystem got you covered.
Also dropping out to Java when ever you want can be done in the same project (which I encouraged) seamlessly both in terms of how Java and Kotlin interoperate, and in terms of Intellij, which is first-class in both languages because Kotlin is Jetbrains' baby, and, because Java has been best supported by Intellij for ages (can't really find that kind of thing anywhere else).
plus with the upcoming kotlin 1.1 release and vertx's upcoming first class support for kotlin.. it is beginning to build traction on the server side.
what is really exciting is Kotlin scripting. Gradle is the first project to start using it.
As for typed alternatives, there's no substitute for Scala. Maybe Haskell or OCaml if you leave the JVM, but otherwise it's pretty much a language wasteland for anyone that's taken a dip in the ocean of Scala.
> because I'm tired of their shit
heh, think it goes both ways ;-) As an observer none of the powers that be took a liking to you calling them out, justified or not (agree that community contrib takes a back seat to Lightbend/EPFL).
Good luck wherever you've wound up (and thanks for the biased Either contrib in 2.12, finally!)
The best way to avoid being called out for saying completely different things in public and private is not saying different things in public and private.
If me politely asking for clarification about this sudden change of opinion is "underhanded behavior", or "renouncing from future interactions" means "let's find some flimsy pretext to contact your place of work to tell them what an evil person you are" then Scala might have some issues attracting and retaining contributors in the future.
It's also thriving/taking off, tons of contributions from myriad brilliant minds; for this reason I'm not too worried about Scala, it will continue on with or without Lightbend steering the ship.
They won't be able to prevent the language from being run into the ground by SIP/SLIP/SPP committees.
Anyway, getting told that I'm not qualified to tell people that their new language extension ideas tick all the boxes of "bad ideas that we are regretting and deprecating since years" after I have more or less managed deprecations and removals for four major versions of Scala ... I guess they need to find someone else to do my job now.
But given past experience, the people who are eager to add more and more features are seldomly the people who clean up after themselves.
Most upcoming language proposals have extremely poor quality, ignore years of lessons learned, repeat many mistakes of the past, reinvent things that have been tried without success elsewhere and have no respect for design principles that made Scala great.
So glad that my name won't be associated with this.
DOT may have been proven sound, but compilers are complex beasts; implementing DOT while preserving an upgrade path for Scala will likely be both difficult and limiting (since Dotty will inevitably be shackled by Scala's past).
For now we've got Scala though, in a couple of years maybe Dotty.
> Nonetheless, Finagle still solved a huge problem for us as we now have real data dependencies and a more efficient execution engine. We gradually converted the Blender workflow code from batch style to Future style in 2013. It ran faster, the code became less error-prone but the readability was still not perfect. It was hard to follow, a pain to debug, tricky to test, and there were lots of duplicate function names.
from the git repo for Node...
> However, this library was written in Scala and isn't exactly Java friendly. It naturally involves a lot of callbacks and repeated function signatures. When it comes to waiting on multiple Futures, the code gets ugly very fast. Nodes is a Java library that aims to solve these problems, making the asynchronous code easier to read, to maintain and to test in Java.
what are they suggesting ? use Finagle for Scala and Nodes for java based projects?
But, Google really has no right to whine about this. They make an explicit tradeoff: they manage to stay 3-5 years ahead of the curve of everyone else by not open sourcing their impressive libraries and infrastructure. But the corollary of that is that there's no place to whine "we got there first!" when someone else produces an implementation of X.