Firms Struggle to Find Programmers Who Can Write Parallel Processing Applications | Hacker News Reader

Firms Struggle to Find Programmers Who Can Write Parallel Processing Applications

wallstreetandtech.com

18 pointsadammichaelc29 comments

I don't work in this industry (thankfully), but why work at a megafirm like Goldman Sachs or even the less elite ones like Morgan Stanley when you could work for a cool boutique rocket scientist company like D. E. Shaw or Jane Street? (Jane St uses OCaml very heavily.)

You'll have trouble finding good software people as long as you make them work deathmarch hours and treat them like glorified typists while acting like hypermacho traders are gods.

They will have problems as long as they view technology as a depressing cost instead of an asset.

It's pretty hard to get a job at D.E. Shaw...

Great point. Having been an intern at Wall Street firm in technology, sometimes I got the feeling that that the revenue generating branches are viewed as the champions of the firm, while the technologists are essentially (well-paid) stilts that hold up the business.

We'll see a schism form in the developer community. The elite (the people who were doing Java in 1996, i.e. those who actively pursue language evolution) will make the transition smoothly to Erlang, Scala, OCaml/F# and similar. In fact these people have already done so inside their own heads, they're just waiting for the job market to catch up with them. The masses (the "Java enterprise solutions architects", who just learn what they need to pass the test/get a job) will be stuck there churning out the same old same old.

Hm... seems to be an opportunity.

What is the best language to learn to take advantage of this market demand? Criteria: 1) manages complexity involving multiple processors 2) has utility libraries (or easy adapters can be written for existing C libraries 3) either has existing mindshare in the industry or at least is not too esoteric and can be sold there.

Any other criteria? What languages can we select?

If you don't mind being restricted to mostly-Microsoft platforms, C# is the race horse you'll want to place your best on. It's multi-threading support is great (the best I've seen so far w/ regards to the balance between control & simplicity). The way its delegates (callbacks), thread pool, and invokes work is quite impressive & powerful.

And, of course, C# can use any C++ library you throw at it w/ P/Invoke & a lot of marshalling.

I second that.. also check ms research parallel task library, very cool stuff.

Here's a primer video

http://channel9.msdn.com/Showpost.aspx?postid=384229

Actually, you'd be even better served looking at F#, since it has the functional programming characteristics that make functional programs more easily parallelizable than languages like Java.

Of course, you can also use things like LINQ and asynchronous workflows in C# to gain a lot of the benefits... but I'm really coming to like functional programming. I wish I'd known about it when I was building parallel applictions, it would have made the software I was developing much easier to write.

I also wish that I could be doing that again, rather than doing mainly maintenance work... even though I was recruited to develop parallel applications.

Many languages focus more and more on multi-core and distributed programming. Mainstream language like Java and C# will get more tools and libraries, but other languages will likely be better. One thing i like about startups is that you can freely pick technologies that are better or faster in solving a problem. You don't have to cope with the conservatism problem.

Personally i think that languages like Scala, F# and Erlang will be good to learn, depending on what platform you prefer. And Clojure also seems very cool. I know it's blasphemy around here, but Clojure is the only Lisp like language that i would like to learn.

Disclosure: I'm a Clojure user.

Clojure is cool, and definitely worth a look if you fancy using STM. However, it's still early days, and although single-threaded performance is reasonably close to Java's (recently, very fast numeric vector operations were added) I haven't heard much about performance of concurrent applications. I suspect it might be a while before the STM implementation comes close to the optimal case, but I think it's pointing in the right direction, promoting pure functional programming and lazy evaluation with the built-in immutable data structures. Plus you get easy access to all Java libraries.

Finally, the Clojure implementation itself is pretty hackable and the mailing list is active and helpful. (I've made some minor contributions to the code myself)

I've done parallel programming in C, C++, and Fortran, using OpenMP, Pthreads/Win32 threads, and even using lock-free techniques, but it's always been an incredible hassle. It'll be interesting to see if Clojure manages to gain any significant traction. Long-term though it will need to gain some kind of support for massive parallelism. Something better than MPI.

What languages can we select?

Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])

What languages can we select?

_What languages can we select?_

Writing parallel-processing applications isn't easy. I run a non-profit tech R&D organization; and we have internships where under-grad students can try their hand at coding in "a corporate environment" with joint development, toolchains, and the whole shebang.

We gauge the students by how much they accomplish in how much/little time and how elegant their solution is. And at towards the end of the internship, we invariably ask the best students to add some form of multi-threading or parallel-processing to the application they're working on (in C++, C#, or Java mostly) and even our best students have trouble with getting proper parallel-processing code working (and these guys had done brilliantly elsewhere).

I personally learned it the hard way, scouring the 'net for all the different methods until I built my own "templates" that work great for me.

Last semester I took a course on parallel computing and one of the things the Professor stressed throughout the course was that historically, parallel machines were built focusing first only on the hardware challenges and software/programmability was a mere afterthought. As John Hennessy related in an interview with the ACM: “Many of the early ideas were motivated by observations of what was easy to implement in the hardware rather than what was easy to use [vis-a-vis programming].”

We are going to see the situation change when there is a shift towards a more abstract/simplified approach to parallelism, where the minutia of parallelism is handled under the hood. I.e a return to the foundations of PRAM theory of the 70s. Otherwise, I don't see it getting much easier to find that kind of talent.

This is where companies like Cilk Arts are going to get things going, hopefully: http://youtube.com/watch?v=FjGKLYINGsg

Edit: To provide some more context, they work on tools and extensions for C++. Things like race detection tools and new language keywords that make it easier to parallelize algorithms. Based on this work at MIT: http://supertech.csail.mit.edu/cilk/

I'll see your ten minute video with annoying sound and raise you an hour long video about Erlang's background, how current parallel architectures work, and how Erlang nicely fits current systems requirements: http://www.infoq.com/presentations/erlang-software-for-a-con...

(plug: we also offer Erlang training http://blags.genges.com/itsgenetic/Bay_Area_Erlang_Training )

Fold!

Er, no pun intended. Thanks for the links!

Cilk looks like a useful abstraction on top of threading libraries. It will be quite useful for people who already know how to write proper parallel code. Concurrency-challenged programmers will need a language that better protects them from race conditions. I think they'll need a strongly-typed language combined with something like Cilk.

A new company has been spun out of MIT, with the goal of commercializing the Cilk technology.

multicore programming blog: http://www.cilk.com/multicore-blog/

we think the key to successful adoption of a multicore programming paradigm is the ability to maintain the serial semantics of the original app, rather than having to restructure code.

http://www.cilk.com/multicore-blog/

Wall Street will solve this exactly as they've solved every other talent problem - offer such ungodly amounts of money that every man and his dog wants to enter the field; skim the cream.

Are they unable to find programmers capable of doing that, or unable to find programmers who have "5 years experience writing parallel processing applications" on their CV?

I suspect the latter.

Parallel programming has been around for a long time. It just hasn't been mainstream, and, apparently it still isn't. The people who are really good at it are on the bleeding edge, and presumably don't fancy working for some bank.

Not saying that nobody could have 5 years experience or more with it. But I think if you put some faith into your engineers, they can also learn a new technology. Surely if you hire a decent developer, he or she will be able to master parallel programming within a reasonable timeframe.

There's going to be a lot of interest in Erlang in the next couple of years. It is going to take a couple of years for CIO's to realize what a nightmare shared state concurrency is to engineer. After a while, they will be forced to look at different languages to solve the problem.

Or, perhaps, they won't see the problem, and startups willing to use languages with a different concurrency model will come to dominate those markets.

soon we'll need a dedicated processor just to handle which data gets sent to the other processors.

It's here. Have you looked at the Cell processor architecture?

Reply on news.ycombinator.com