A Perl toolchain for building micro-services at scale | Hacker News Reader

A Perl toolchain for building micro-services at scale

engineering.semantics3.com

142 pointsramanan52 comments

I've just started tinkering with Mojolicious, after spending some time playing with a variety of other frameworks in other languages (I'm starting a new web project, and wanted to make an informed decision about how to build it). Mojolicious gets so many things right in such a tiny package, I'm surprised more folks aren't using it or talking about it.

The real-time support is awesome; setting up Websockets is stupidly simple; coding non-blocking services is, I think, easier (at least more concise) than in Node.js (though it uses a similar callback model); testing is just perfect (Mojo has a user agent and DOM support, so you can write super concise tests for web services); building multiple outputs (like HTML for humans and JSON for API) for the same routes is excellent. So far, none of the frameworks I've looked at has been as concise or as...neat, I think might be the right word. So many things about it have me saying, "Now, why didn't someone think of that earlier?"

It's pretty tightly focused on just doing a few things really well, so it's not like a Rails, or even Django, experience...you have to make your own decisions about what ORM to use (or not to use an ORM), front end (though because it is agnostic about front end, you can use whatever you like pretty readily...so, React/Redux, Angular, whatever), and even nitty gritty stuff like authentication and accounts and such. I occasionally find myself wishing it had a little more batteries included, but mostly I like that it doesn't take days of doc/code spelunking to grasp the whole system.

Anyway, I'm not a Perl fanatic, though I like the language OK; especially in recent versions. So many little warts have gone away in recent years. But, the web service ecosystem in Perl is surprisingly strong and modern, given how unpopular it seems to have been in recent years. It's been a while since I really dug into what goes into building a new Perl system, and a lot of cool stuff has been completely off my radar. Mojolicious, in particular, is one of those things.

> I occasionally find myself wishing it had a little more batteries included

Well, there's a lot of helper modules[1] on CPAN (somewhat different than the ones linked in the article), so that may alleviate that concern a little bit.

Mojolicious is bar-non one of my favorite Perl technologies. :)

1: https://metacpan.org/search?size=20&q=mojox&search_type=modu...

Yeah, some of the modules are great. But, I mentioned my current paint point, which is user accounts. There's a couple of half solutions on CPAN, but nothing comparable to accounts support available in Rails or Django, or almost any of the other "big" frameworks. Admittedly, Mojolicious isn't really trying to be that kind of framework, and that's great...but, this one is such a common task. It feels weird to have to roll my own, if I want to make a somewhat traditional web application with it.

But, I agree...Mojolicious is a lot of fun.

Then why not put yours on CPAN?

If I end up building with Mojolicious (which seems likely), and I build some sort of generic user module, I will.

> what ORM to use (or not to use an ORM)

do you have something to recommend ?

used Rose::DB in the past, then I discovered SQLAlchemy and it's difficult to look back...

DBIx::Class is the only one I've really looked at. I also recall seeing a talk on Fey and Fey::ORM, given by the author, at YAPC or somewhere, and remember thinking it seemed really nice. But, I have never used any ORM heavily, so I'm still figuring it out.

I'd need someone suggesting anything other than DBIx::Class to present some ironcast arguments for that choice

I agree. I think the main Perl workhorse modules that provide the best benefit for me are, in order:

DBIx::Class

Kavorka / Function::Parameters / Method::Signatures (take your pick)

Moose / Moo (or use Moops and get Kavorka above built in!)

Mojolicious

Notable mention: Path::Tiny (and Try::Tiny, but everyone should know that one).

It's taken me awhile to find the happy medium where I'm not sticking too much in DBIC ResultSet methods, but being able to define complex search methods that chain is awesome:

  my $rs = Schema->resultset("Foo")->unprocessed->rows(100)->order_by([-desc=>'time']);
  $rs = $rs->for_user($user) if $user;
  $rs = $rs->recent_entries; # Limit to the last week

Makes my life much better, and makes it so much easier to change me schema as needed.

1 point by ashimema 0 minutes ago | edit | delete

Love DBIx::Class.. but it's not a good/perfect fit for Mojolicious by a long way.. it's blocking by nature and thus doesn't play too nicely if your aiming to write a non-blocking mojo app. reply

For the few queries where you need async (most should be fast enough in the first place), there's no reason you can't use $rs->as_query to get hold of the SQL and bind values and then feed those into Mojo::Pg.

Is there such a thing as a non-blocking SQL query? Doesn't it always have to be wrapped up in something to make it non-blocking?

I think my question is: Is there an SQL ORM in any language that is non-blocking during queries, without the programmer having to wrap it in some sort of promise/callback/whatever? I really have no idea about ORMs, so I don't know anything about the state of the art. I'm trying to imagine what such a creature would look like...it seems like if your queries are going to potentially make you wait for any amount of time, you'd need to account for that at the caller side, even if things happen on the ORM side.

Nice post! At my most recent job, we used perl daily for processing scripts and many other functions. After initially having similar thoughts about it being "obsolete" and a "dinosaur", I quickly realized how resilient and flexible the language is (I know the flexibility can come with some criticism).

We investigated a lot of other options (go & python specifically) when building new projects, but found that perl was the best solution. It's flexibility allowed us to develop a custom and durable solution that surpassed our expectations.

We ran into some environment/portability issues and resorted to using docker, carton, and gitlab CI to solve these problems. It is incredible how reliable and easy it became to modify and deploy code to a variety of systems (new or old).

"After initially having similar thoughts about it being "obsolete" and a "dinosaur", I quickly realized how resilient and flexible the language is (I know the flexibility can come with some criticism)."

Not obsolete - can affirm. All of the Oh By[1] back end is written in perl, circa 2016. We use apache + mod_perl and are very happy with this environment ... just like we were in 1998.

[1] https://0x.co

Any thoughts about switching to plack? It seems like maybe you could get some low effort performance/developer experience improvements.

Obsolete was the wrong word - Perl definitely still works (very well for certain things). Cool to see new applications being developed in it still!

I fully resonate with this article! For the last few years I've been trying to do the same thing and in the last 2 years I can say that we successfuly managed to create a similar thing using Perl. More than this we are developing all our web services with Mojolicious we are running them on top of Mesos using Docker. We use Module::Build for packaging the app and to be honest I find it more reliable than using any other tools. I'm gald to see that there's somone else who believes in Perl's abilities to deliver a modern application.

In much the same way that PHP has a reputation for being "bad", Perl has a reputation (ill-deservedly!) for being line-noise, or obsolete.

I've continued to be productive by writing applications, microservices, and websites in Perl. Though these days it seems few people care to hear about the details as it isn't the flavour of the day.

Mojolicious is cool, although I've usually stuck with Dancer instead (which is itself a clone of Ruby's sinatra).

I think the fastest way to convince yourself that Perl is obsolete is to try to hire for good Perl programmers. It's a fine language if you treat it well, but it's going the way of COBOL.

Perl programmer here of 17 years. I'm happy to look into new projects.

I don't have nearly justinator's experience, about a year of heavy Perl use, but I would love to get back into it.

It's not _that_ hard to hire good Perl programmers. Maybe it depends on they scale you need to hire for, but I've been involved in the hiring process for a couple of Perl positions and I've seen lots of really strong candidates.

I disagree. It is not the flavor of the month, like other things. There are excellent people available, and not a dwindling number of them at that.

Module growth is strong, and accelerating.

Not quite the hallmarks of something going the way of COBOL.

Note also that Fortran is also in heavy use in various science fields. Rumors of its impending death have also been greatly exaggerated.

I'm not really sure why you can't hire a Ruby or Python programmer, if they are open minded about it, and train them. There's not that much actually different, compared to some other languages. In fact, I think the only major things you would need to teach them that they couldn't easily find from the docs would be about how context works, and references (references to a lesser degree, just make sure they get how they are used in complex data structures).

> I'm not really sure why you can't hire a Ruby or Python programmer

Yeah, sure, hire a Ruby/Python programmer ... and try to trick them about the language they're going to be using by being vague about it in the job posting and initial conversations until they got really interested?

Been there, done that. It was a stopgap measure that sort of worked and it wasn't that much fun.

No, just advertise appropriately. Some people don't apply because they don't want to use the language, others don't apply because they expect you to require skill in that specific language. Something along the lines of "Knowledge of Perl or equivalent dynamic language strongly suggested."

> I think the fastest way to convince yourself that Perl is obsolete is to try to hire for good Perl programmers.

Hire for good programmers, and they'll be good Perl programmers if they need to be.

A key component of the network side of our (https://scalableinformatics.com) SIOS layer is handled by Mojolicious running on Perl. Has been for a while (about 5 years).

I've taken some steps to do a re-implementation in node, but the perl version "just works" with very little fuss, under fairly heavy load, and is pretty easy to debug when I need to.

This is very much a microservice: tailored PXE environments as a service based upon a database, and the booting mac address as a key. A programmatic/database-based backend to a PXE server. It allows us to boot effectively anything that is bootable by modern hardware, from Linux, Windows, SmartOS/OpenSolaris, through FreeDOS, and other more esoteric systems. A number of our customers use this as a configuration tech underneath their own orchestration layers.

All Perl based, and using as modern techniques as possible.

We stopped using system Perl many years ago. Red Hat seems to like to ship not merely obsolete versions, but versions actually past their end-of-life, so that they are not really supported upstream anymore. They have a similar issue with Python and other languages. So, reluctantly, about a decade ago, we started building our own toolchain. First with Perl, then adding in Python, Julia, R, Octave, and other analytics codes. Our analytics tools use all of these as part of our SIOS rambooted appliances (http://scalableinformatics.com/fastpath), so we needed the updated toolchain. We are looking at Rust as well for future work, and have looked at incorporating Go, but we don't have any Go code developed/planned as of yet.

I have been using Perl to do similar things. I build everything internally in the same manner as one would a CPAN module, and this is checked into a private git repo.

From there it is simple to call custom tools to build and deploy services in Perl.

My favorite feature about Perl is that version 5.x is compatible with each release and our code has worked without issue for over a decade.

I recently integrated some Go code into part of my module and wrote a post about how to compile it as part of the build process here : http://www.tysonmaly.com/programming/go/using-go-perl-makefi...

It's often heard that CPAN is Perl's Killer App. No argument from me!

I believe what's even more impressive in the Perl ecosystem is CPAN testers.

When you upload a module to CPAN, lots of neat things happen. Of particular interest is all of the testing that gets done. For example:

http://www.cpantesters.org/distro/M/Message-Inform.html

As you can see, that's a pretty wide swath of OS and Perl versions.

This is an example of a failure report: http://www.cpantesters.org/cpan/report/e93a7034-cf3f-11e5-96...

This is an powerful resource. I've had random people approach me with suggested fixes to test failures on CPAN in the past. It's amazing!

I believe the Perl community is serious about testing and reliability in ways that few others are. For example, the default module installation command ($ cpan <module name> or $ cpanm <module name>) will NOT install if there are failures in any of the tests.

By modern standards, Perl5 has some silly characteristics, no doubt. But as others have said, the community is rock solid, and more lively than ever:

https://metacpan.org/recent

Perl is also one of the fastest things out there - if your problem is doing lots of string handling. I have this little test that does a lot of string concatenation and garbage creation, which I recently pulled out of the closet to compare Javascript on Node and Javascript on Nashorn (and also native Java, since it was already there). Even on Java 8, Perl still blows it away.

That said, I hate working with references in the Perl debugger, and, I would be lynched at most worksites, at least in Sacramento, if I suggested a Perl back end. ...Which is too bad - Java/ORM, "COBOL-fingers", bondage and discipline forever...

Can you expand on what you've run into with references in the Perl debugger?

print $ref

REFERENCE$HASH

Or something like that, it's been a while. I'm spoiled by the Javascript debuggers built into Chrome & Firefox, although I'm not sure about the Node.js debugger.

Is there a shell (either graphical, or "Borland Turbo Debugger for DOS" style) for perl -d ?

(even the "view locals" command in gdb for C presents data more quickly than the built in perl debugger)

Try 'x' instead of 'print':

ddiederi@lomin:~/t$ cat p1 #!/usr/bin/perl

my $a = 'hi'; $a = { hi => 'there' }; print "Done\n"; ddiederi@lomin:~/t$ perl -d p1

Loading DB routines from perl5db.pl version 1.49 Editor support available.

Enter h or 'h h' for help, or 'man perldebug' for more help.

main::(p1:3): my $a = 'hi';

  DB<1> n

main::(p1:4): $a = { hi => 'there' };

  DB<1> x $a

0 'hi'

  DB<2> n

main::(p1:5): print "Done\n";

  DB<2> x $a

0 HASH(0x29e6eb8)

   'hi' => 'there'

  DB<3> n

Done

Debugged program terminated. Use q to quit or R to restart,

use o inhibit_exit to avoid stopping after program

termination,

h q, h R or h o to get additional info.

  DB<3> exit

ddiederi@lomin:~/t$

What's microservice-specific about that post? It's mostly about pretty standard perl packaging and deploying, most of which would apply to monolithic equally well (even more so, if your dividing line is DarkPAN-backed modules and not HTTP-isolated (micro-)servers).

You are absolutely right!

The post's equally relevant to a monolithic setup as well. In such a setup, a code artifact (as mentioned in the post) will probably just be a script or a library (now I include a service too).

A toolset like this is a good-to-have for a monolith but a pre-requisite for micro-services (if you want to reuse code as much as possible i.e.), I feel.

The post has been written from a micro-services angle because that's what the setup is like here at Semantics3.

One the other hand, I've seen and read about companies where some forms of reused code are actually seen as code smells in this context. Especially when it comes to models, as one's service idea of what constitutes a 'Customer' might be different from others. Managing that (and redundant data stored in each microservice's private area) requires some heavy tooling...

So I'm always interested in the more specific approaches to communication and storing data when it comes to microservice architecture, which might be an idea for a sequel article ;)

It surprises me Perl isn't dead yet - in fact, German Bundeskriminalamt built a website for people having information about the German nazi hool riots during the France EM 2016 in Perl: https://www.bka-hinweisportal.de/

Why would this surprise you? Perl continues to improve and is battle tested. More Perl is written today than ever before (that said, much more software in general is written than ever before, and there is no doubt Perl's share has declined). Particularly if you're doing lots of text processing Perl remains a great choice.

> Particularly if you're doing lots of text processing Perl remains a great choice.

And, when you get right down to it, a huge chunk of web dev falls into that category.

A huge chunk of life is text processing nowadays.

Two great talks when it comes to the questionable greatness of Perl:

https://media.ccc.de/v/31c3_-_6243_-_en_-_saal_1_-_201412292...

https://media.ccc.de/v/32c3-7130-the_perl_jam_2

These are really not great talks at all. The speaker doesn't seem to actually know much about Perl, and it seems like the flaws he's found have little to do with Perl, and more to do with just writing insecure code.

See http://blogs.perl.org/users/joel_berger/2015/12/response-to-... for one response and some discussion in the comments.

Better rebuttals to those talks: https://gist.github.com/preaction/978ce941f05769b064f4

Perl is bundled with many linux distributions and is a hard dependency for many popular programs. It isn't going away.

On the other hand, bundled Perl versions have been a bane as much as a boon for the language. After a lull because everyone was holding their breath for Perl6, there's been a decent amount of development regarding the Perl interpreter, and if your target server still bundles 5.0008...

And upgrading that might risk incompatible system services, which is why sysadmins usually have way fewer problems adding something like Perl or Python. Isolated language environments (cf. perlbrew) are a nice solution to this, at least until you happen upon a particular paranoid admin.

Neither is m4!

Perl Just Works, it's fun to work with, and has either the same quality or better tooling that comparable languages.

Once you're used to Perl's testing tools, anything else feels chaotic and poorly thought-out.

This is one of the first articles that clearly and consistently lays out the actual concrete benefits of microservices instead of the cargo cult Medium drivel that usually plagues the ecosystem.

(Of course, these are still only arguments for SOA and there's no clear differentiation between SOA and microservices, because there is none.)

The term SOA was poisoned by the baroque standards that spun up around it several years ago. There's nothing wrong with the concept of a service oriented architecture...but, when someone said SOA a few years ago, it usually meant something that was very "enterprisey", for lack of a better word (and enterprisey isn't a very good word, either). Big XML schemas, extremely complex APIs, etc. were the hallmarks of the first round of SOA stuff. At least, that's how it looked to me. I didn't work with it at the time; maybe it wasn't as bad as it looked from outside.

But, small pieces inter-operating is very UNIX-y, and excellent for all sorts of tasks. I think the difference between microservices and SOA may just be the direction it's coming from. Microservice architectures are coming from the Open Source web and cloud culture; SOA came from enterprise and government and finance. But, the over-arching concepts, I think are the same. (I'm definitely not an expert on either, however. Just my gut feeling.)

It's not "punk" it's "new wave"

This is a great writeup. I especially like that you make the actual concrete case for micro-services instead of the hype driven garbage we usually see.

Do you have trouble hiring qualified perl devs?

Reply on news.ycombinator.com