Trapperkeeper is a new Clojure framework for long-running applications, services
puppetlabs.com
puppetlabs.com
> a way to cohesively manage the lifecycle of the various components making up an application
I'd need a more specific example to understand exactly what this is talking about, but shouldn't each component with an independent life cycle be a Unix process?
> a common logging mechanism that wouldn't need to be set up and configured by each component
Just log to stderr. Don't even bother including timestamps; an external utility like svlogd can do that (and rotate the logs too).
> a web server flexible enough to load multiple web apps
This one in particular strikes me as an anti-feature, common in JVM environments. In the Heroku 12-factor approach, each app embeds a web server (like Jetty), rather than the web server containing apps. Then you use a front-end proxy like HAProxy or nginx to route requests to multiple apps.
> and to support a rich set of options for configuring SSL
That's the job of the front-end proxy, or maybe even a special-purpose SSL termination daemon such as stud.
Profiling: Analyse one snapshot to find the main bottleneck across a number of services.
Performance, security etc. etc. Basically the ability to use a lot of the things that make the JVM powerful and save you time and money. (Not only that but the things that make clojure powerful like its concurrency mechanisms etc across services)
Besides one process is often simpler than many. Interprocess communication suffers from some of the "Fallacies of Distributed Computing."
If you like to keep your UNIX process mental model just think of JVM threads as processes.
(One other thing: Running services in multiple JVMs uses more heap than multiple services in one)
That sounds like a failure of the JVM. You are also putting multiple independent services in the same fault domain.
IIRC I saw some talks about introducing it in java9.
That is a natural feature of garbage collection. The asymptotic complexity of copying garbage collection is O(complexity of live object graph)/(total available heap size). That is, having more heap available makes it consume less computational resources, up to an including making many real loads have near-zero garbage collection cost if the heap is large enough. Having three individual processes share a heap can make them actually do less work than if they each had a static third.
Modern operating systems are not set up to exploit this. There has actually been some work done on Linux user space reclaim for it but it's not yet usable. Until there is a way to tell the kernel that you want to use every last free byte on the machine if possible but can likely free most of it when any other process needs it, to make best use of the machine you will have to roll your own in userspace.
> You are also putting multiple independent services in the same fault domain.
In practice, unless you are doing a few stupid things (which most places don't let you do), it's extremely uncommon for whole JVM processes to go down. With a bit of work you can make almost all faults only take out individual tasks.
Not that I personally like JVM. I'd much prefer to do work on the actual OS, not wrap it all and pretend it doesn't exist. However, the people who build and use it are not idiots -- there is an actual method to their madness.
The thing to understand is that, much like Smalltalk, the JVM is its own world. The underlying OS is only there to bootstrap the JVM, there is no Unix underneath. So every time you think they would do better to use some feature of the underlying OS you are breaking with the Java way of doing things.
I'd just like to point out that this is not the only Java mindset. It's certainly true that Java has a long and horrible tradition of people thinking that everything and the kitchen sink should go in the app server (which can then be managed with a SOAP interface!), but this is not intrinsic to Java, and it's not the only way to work with Java. Some of us are actually pretty keen on the 12 factor approach.
Trapperkeeper is simply intended to give us the flexibility to decide how to bundle different services together. If we have 2 or 3 tiny web services that can benefit from some shared state and are lightweight enough that they don't really warrant running a separate web server for each one, now we can choose to deploy them that way. But, absolutely, we can still run more than one instance of this framework on the same machine, to reap the benefits of truly isolated OS processes in cases where that is appropriate.
Another minor point to highlight is that, at Puppet Labs, we're building on-premise software that our customers install and administer on their own networks. We're not building SaaS products that we administer ourselves. This means that we need to be able to support a wider range of deployment configurations, so that we can tailor a particular installation to the needs of the particular user who is installing it. For some users, they may have enough traffic to warrant peeling out every individual service to its own process and distributing them across many nodes. For other users, they may have a very tiny network where it is perfectly sufficient (and much simpler) to run everything in a small number of processes on a single node. The new framework is all about giving us the ability to make those choices more dynamically--not about prescribing a particular deployment model to all users.
It looks very quick and easy to spin up a service which has all benefits and performance of running on clojure / jvm. Combining this with a website front end using Clojurescript looks like it could be a sweet spot.
[1]: http://planet.clojure.in/
Clojure gives you massive benefits, but god it's a brutal road to competence. Things that should be simple like setting up a web server, adding user authentication, making JSON API's, trying to find the best libraries, it's all a time-consuming trudge through pain and suffering. And after some months when you get somewhat competent with the language, you realize that in order to get the full power of Clojure you need to not only learn the Clojure language, but the entire Clojure ecosystem. So you need Clojure, Clojurescript, Datomic, Edn, Core.async etc... And each piece of that puzzle is equally horribly documented and has no beginner resources so you have to struggle at each point in the process, praying to god that it's ultimately worth it.
It's frustrating as hell, but it really is rewarding, because you have to learn every little piece at a low-level to ever make anything happen. There's no Rails-like framework that makes things 'just-work' waiting for you. I find that using Clojure is awesome, but learning Clojure is awful. I'm hoping to play a role in fixing that second part pretty soon, but really wish the community would be more active in helping to remedy the problem. The first step is admitting that the problem exists and it matters, and not sure that we're there yet.
E.g. I'm helping someone with a web application and I've found no need whatsoever for Datomic (PostgreSQL + korma is just fine), I gather that means Edn is irrelevant, Core.async appears to be less than a year old, so I'm not surprised it might be lacking in documentation, and I don't see it as critical for a lot of applications, including mine. Clojurescript: we'll see, I do engines, and would prefer to continue to have others do almost all of that side of things.
Granted, it did take me a while to find and get up to speed on a set of libraries to do the web app I'm working on (mostly CRUD with some non-real time backend external communications), but it's been relatively pain free. I started with a recommended set, then as I found I needed to do something new, like JSON, I found a good library in a few minutes with Google and didn't find it at all difficult to fold it in.
Did change after a bit my form validation library to Red Tape, which modulo one detail (limited data available to form level cleaners (functions that act on more than one item in the form)) is superb.
And "superb" in general describes my experience doing all this. And I fully expect documentation and other learning resources to continue to improve as Clojure gets greater acceptance.
(Disclaimer: I'm an old Lisp and Scheme hand, e.g. not someone who had to learn anything conceptually new to use loop and recur. I could see the above plus learning basic functional Lisp concepts to be brutal if you're new to everything.
On the other hand, I can remember the days when learning MACLisp required reading an old David Moon manual and then the archives of a mailing list following it's publication; thank goodness there was a really great Lisp Machine Lisp manual and they were just starting to be somewhat available to use and learn on.)
(This is exactly the behaviour you are point as wrong, and I know it.)
The resource I had (back in the 2011/2012) were just a book, `The joy of Clojure`, my background was some C and mostly python, nothing about FP.
I don't see any of ClojureScript, Edn (???) nor core.whatever so fundamental... Those are just other tools, you can write any webserver using the old JS, JSON and any other lib you may need... (Anyway you should not try to learn clojure building web server, are the less clojure-y piece of software you can build IMHO)
IMHO, the biggest problem is the paradigm shift that most developer need to have in order of learn clojure.
Then again you probably should have listened to all those people telling you to do The Structure and Interpretation of Computer Programs (SICP/6.001) ^_^, I merely lucked out to be at the edge of the community that created it, so it sort of fell into my lap, with strong recommendations that "this is revealed truth" as I like to put it.
In the last couple of weeks I had to write a bunch of those in Clojure for the first time "in anger" (for real), and that part of it was trivial because of the SCIP I'd done in the early '80s. Learning that and pounding in my head for the first time was a bit hard, but well worth it, back then and of course now.
I think there's going to be some efforts in the near future in the Clojure community to tackle the low-hanging fruit of improving beginners' path to mastery.
If you come from OOP school of programming and the Algol/C++/Java style of syntax (true of many popular languages these days), then Clojure does require a re-think of the programming paradigm. But it truly is worth it -- it will make you a better programmer -- and the community is friendly, smart, and always learning and improving. It's certainly a language that I think that I will grow as a programmer by being with.
I base this in part on the very popular Python language's use of white space, which to a seasoned Lisper looks very familiar.
As for the focus on "diversity", I submit it might be misplaced at the moment, especially to the extent AlwaysBCoding's complaints are valid. To the extent Clojure is not making the sale to a target audience that's already 98%" heteronormative eeeevil white male (https://news.ycombinator.com/item?id=7581226), how likely are you to entice those outside that demographic? In fact, if you view Clojure as an experiment that may fail, are you doing them a favor by evangelizing it to them?
(I'd argue yes, since it's probably one of the easiest functional programming (FP) languages to learn, and it really helps to have FP in your toolkit. But I'd argue SICP or something gentler with multipardigm Scheme would be a better approach if you're just trying to accomplish that sort of thing.)
I don't think any serious student should expect to learn only one language. Clojure can teach concepts that any learner will find valuable in applying to other languages. In particular, I think that clear separation of values, identity, and state avoids many kinds of complication in learning how to program that are introduced early with other languages.
I'll start by saying that documentation, examples, getting started, etc could be better and efforts are ongoing to improve all of those to various degrees.
However, there are substantial and rapidly growing resources (http://clojure.org/getting_started) for learning Clojure and other parts of the ecosystem. Some of the more recent beginner resources I'd recommend are "Clojure for the Brave and True" (http://www.braveclojure.com/) and Kyle Kingsbury's "Clojure from the ground up" series (http://aphyr.com/posts/301-clojure-from-the-ground-up-welcom...). There are also many good Clojure books - I'd recommend any of them.
In the web area, the philosophy is different than other languages and it is evolving rapidly (especially on the client side). Luminus (http://www.luminusweb.net/) is a good effort to provide a reasonable set of defaults that tie together the various Clojure web libraries.
So yes, it could be better. However, there are significant resources now and gaps continue to be filled.
You can find a quick installation guide here: http://blog.crudzilla.com/2014/04/hivemind-on-digital-ocean-...
Here's a screencast showing usage of Clojure, it requires no additional setup besides the installation guide above: http://crudzilla.com/assets/img/info-graphics/lang-demo.gif
The other thing is that you cannot learn Clojure without learning Java, despite various claims; the atrocious error messages are just the tip of the iceberg there (and I say this as a professional Java developer).
I personally do not think the error messages are "atrocious". Some error messages are confusing or lack context. There are many tickets in progress to improve those and they continue to get better over time. This is an area we continue to work on within the language.
This is what drove me back to Haskell from Clojure. Haskell is miles ahead in documentation, references, tutorials and tooling. Tools like Hoogle are manna from heaven for beginners and experienced developers alike. Couple that with the self-documenting aspect of types and even undocumented Haskell libraries are grokkable in a reasonable amount of time.
One of the big problems is that it's -really- hard to go back and explain this stuff once you're already an expert, and I metaphorically burts into tears at your comment because if you'd documented all the things you had to learn the hard way as you went, you'd probably've gone a noticeable way towards fixing the problem you describe.
Note: I'm not, at all, trying to criticise you for not doing so. I'm just saying that if you could arrange for this to happen as you mentor somebody else it would be really really awesome :)
I started Clojure for web development. Once I found out about the Compojure template, everything else was just a stone's throw away.
$ lein new compojure myapp
$ cd myapp
$ lein ring server-headless
Server started on port 3000
> There's no Rails-like framework that makes things
> 'just-work' waiting for you.
rails g devise:install
rails g devise user
rails g devise:views
> because you have to learn every little piece at a low-level to
> ever make anything happen.
At least that's how it was at all the Rails projects I worked on.
https://github.com/ring-clojure/ring/wiki was refreshing. It turns out that managing your own session/cookie/authentication lifecycle in your own application code is easy. And it turns out that these "it just works" generators doesn't actually save any time because when you want to make trivial changes to them, you have to credentialize in yet another abstraction.
I knew how clojure felt and behaved, but IMHO you still have to memorize most of the core library, or most code will look confusing.
http://pluralsight.com/training/courses/TableOfContents?cour...
For beginners, getting familiar with namespaces in the REPL quickly is key, if you're trying to load a sample of code froma file.
Also another gotcha is multiple arity functions which I haven't seen before (in a dynamically typed language).
That said I too still can't figure out how to use leiningen and it seems clunky and slow. It's also easy to find examples that don't work. I'm not sure if getting it to work with as little hassle as npm would help it's growth, but seems like it couldn't hurt.
My journey looked ~like this:
1) Managed to get counter-clockwise working, played around with http://tryclj.com/, solved most of the http://clojurekoans.com/, and first ~15 of 99 functional problems that float around the web.
2) what helped, that I have already gone through a course of haskell while undergrad, so functional paradigms weren't that foreign :-) (immutability, laziness, partial application, e.t.c), so if you want to give it a try, you can look at Learn You a Haskell :P
3) we have since moved to python, and when I am sad, I try to write my code without loops, but using (for in) generator for everything, pretending I am still using lisp :P
4) clojuredocs.org/clojure_core/ were quite helpfull but after a year, clojure 1.5 was already out, and lots of functions recieved updated counterparts, so I spent quite a lot of time reading their sources ...
Second, Typesafe is a tiny company, and the most prolific committer to the Scala compiler just quit - not just Typesafe, but Scala altogether. Doesn't seem rock solid to me.
Clojure has several companies behind it.
Paul Phillips did NOT quit Scala altogether: https://github.com/paulp?tab=activity
This is a surprising bit of FUD that has taken hold amongst gleeful Scala haters.
And yes, he's still involved, but far less so than before when he was working full-time on the compiler at TypeSafe.
On the plus side, Dotty, the new "simplified" Scala compiler, is in the works and collections library will likely be overhauled in the process.
as for Clojure, typed Clojure looks interesting, but otherwise the little rascal Haskell is my Scala escape hatch for the time being.
I'm simply pointing out that it doesn't have a rock solid base.
I would like to use scala more, but I found clojure easier to use and equally powerful in practice.
However, it's false that Clojure doesn't have a company behind it: Cognitect is certainly backing up Clojure by developing ClojureScript and Clojure itself, Clojure consulting, the Datomic database and the Pedestal "framework".
I don't like clojure either, but it has nothing to do with its Lispy-ness.
There are many reasons to not be fond of a language. Clojure is in the middle of two worlds: It's not as pure as a real lisp, and yet it still has all of lisp's syntactic sensibilities.
Some people dislike it for one reason, other dislike it for the other. Both reasons are quite valid, but I'd bet that reading like a lisp is the most popular reason.
My personal opinion is that if Lisp's approach to syntax was a winner, it'd have won a decade or two ago. But that doesn't mean other opinions are not valid.
And again, it's not that I don't appreciate Lisp: Many Lisp features have been added into members of the Algol family over the years, and it has made them better.
Sure, "apple" can't be protected everywhere in all contexts, just like "windows" can't be protected everywhere in all contexts... but what about naming your erectile dysfunction company "Microsoft"?
That's a word used as a specific name, not a commonly used word just being used as a name for a company in a different market. It would be pretty obvious that you're leveraging the popularity and consumer brand confusion of Microsoft.
Edit: right now Googling "trapper keeper" brings up the Mead product. That could change if this product gets sufficient Google juice. I can't imagine that Mead would be happy about that.
Mead has annual revenues of $6 billion, so they could hire enough lawyers to make life plenty miserable, even if they didn't wind up winning in the end.
Unfortunately, my Amazon search and Google fu isn't good enough to find the book I learned most of this from, but if you're really interested in this, it requires a book level and length treatment. E.g. did you known Owens Corning has a mark regarding the color pink? (Only, or at least originally in the context of insulation.)
In between, I'm not sure, but try the usual suspects like Wikipedia: https://en.wikipedia.org/wiki/Trademark
When I looked at migration strategies there wasn't a real migration tool available. To interact with PuppetDB I ended up learing a bit of clojure and rollng my own tool, the built in REPL helped wonders with this (granted it was only enough to accomplish this task and haven't had a chance to touch Closure since).
Only kept getting irritated by the startup time which I don't know if it can be attributed to the JVM, Closure or PuppetDB itself.
From an operational perspective there's too much overhead to effectively running any language in a production environment.
From a maintenance perspective you want to have common skill sets between a large intersection of employees so you're not the only person doing bug fixes on a system.
There needs to be a balance for sure.
The development team was very siloed. Nobody could work on anyone else's code, not even a little bit.
When a new developer approached me with a module that he had written in Jython, I almost lost it.
It also talks about how Trapperkeeper works with Clojure, JRuby and Java modules.
You may well be right about single language companies, but I'm not seeing any evidence that Puppet Labs is one of them.
Puppet Labs isn't a single language company.
One problem we've had is that our frontend isn't clojure/clojurescript (its coffeescript+haml+knockout). In theory that means its easier to find frontend developers and designers. In practice it has meant creating a frontend/backend division where the backenders in particular don't like to work on the frontend. We testing out whether a move to clojurescript (and hence being a truly one-language company) will solve that.
I do question their conclusions about OSGi though - it seems like a good fit for their case despite the complexity. Apache Felix is particularly nice to work with.
Given how often 0-day vulnerabilities are discovered in the Java ecosystem I can't see how this is a good choice for a critical piece of infrastructure.
We love using puppet, but one big issue we have is it's lack of consistency, a big part of that is the fact that theres Puppet, Ruby, ERB syntax and ideologies and now you also have thrown Java in the mix?!
When we saw this news several people in our team have already started looking at alternatives.