I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1]
The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that.
I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language.
I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end.
The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic).
The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2].
I looks like a good fit for your project.
[1] http://docs.julialang.org/en/release-0.1/packages/packagelis...
[2] https://groups.google.com/forum/?fromgroups=#!forum/julia-st...
mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with.
I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
(as in, I spent part of an afternoon end of last week skimming the entire code base).
This is troubling, because when you don't design your language to make it "manageable" to do static analysis / optimization on, it be becomes DIFFICULT to add that capability later. Julia is about as dynamic as JS or Scheme or Lua, plus it has all those runtime type tag cases in code... but whereas JS has relatively unbounded manpower (relatively!) to make a fast JIT, whereas lua is a small enough language that we have the amazing impressive LuaJIT, whereas Scheme is designed with very thoughtful semantics / specifications in mind, Julia lacks.
more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer. LLVM is a good backend, but it can't leverage the semantics of the input language, just the semantics of the LLVM bitcode you provide. You really really can't punt on optimization on the before LLVM side.
also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language.
LLVM is amazing, and i use ghc -fllvm exclusively, and while theres some really really awesome optimizations that LLVM does for haskell code (for example), a HUGE amount of high level code performance is really depending on the language (any language!) doing optimizations before passing things onto the LLVM.
The fact is that the performance of Julia is already quite good. Julia is semantically much closer to C than Haskell is, and it needs fewer optimisations to get reasonable speed.
The necessity of static analysis for performance is a myth that's been debunked for a while now. From the language semantics, I think that the runtime model of Julia must be about as complex as Lua's, including the LuaJIT FFI (swap metatables with type-based multiple dispatch).
As you said, though, whereas LuaJIT has its own tailored bytecode, Julia compiles to LLVM IR, and potentially useful information is lost in the translation process. LLVM-Lua is reportedly 2-3x slower than LuaJIT. I suppose that type specialized traces and immutable types help somewhat.
An area that clearly needs improvement is vectorised code. Currently, an new vector/matrix is allocated for each intermediate step. Bad. This is being worked on.
> also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
You're conflating type system with static type checking. Julia is dynamically typed, but its programming model (the UI of sorts) is built around the type system and multiple dispatch, it is one of its main features of the language, actually.
The first point is that Julia already has excellent performance on a par with most compiled languages, including, e.g. Haskell, whether they are using LLVM or not. Straight-forward Julia code is typically within a factor of two of C. That's shown in the microbenchmarks on Julia's web site [http://julialang.org/], but, of course, that's not entirely convincing because, you know, they're microbenchmarks and we wrote them. However, similar performance is consistently found in real-world applications by other people. You don't have to take my word for it – here's what Tim Holy [http://holylab.wustl.edu] has to say: https://groups.google.com/d/msg/julia-users/eQTYBxTnVEs/LDAv.... Iain Dunning and Miles Lubin also found it to be well within a factor of 2 of highly optimized C++ code when implementing realistic linear programming codes in pure Julia: https://github.com/JuliaLang/julia-tutorial/blob/master/Nume.... The benchmarks appear on page 7 of their presentation.
This statement about Julia's high-level optimizations is entirely wrong:
> i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization.
Julia does no tracing at all, so it's definitely not a tracing JIT. A relatively small (but growing) very crucial amount of high-level optimization is performed on the Julia AST before generating LLVM code. In particular a dynamic dataflow-based type inference pass is done on the Julia AST. Since Julia is homoiconic, this type inference pass can be implemented in Julia itself, which may be why you missed it: https://github.com/JuliaLang/julia/blob/master/base/inferenc.... Don't be fooled by the briefness of the code – Jeff's dynamic type inference algorithm is one of the most sophisticated to be found anywhere; see http://arxiv.org/pdf/1209.5145v1.pdf for a more extensive explanation. It's also highly effective: 60% of the time it determines the exact type of an expression and most of the expressions which cannot be concretely typed are not performance critical [see section 5.2 of the same paper]. You are correct that we leave machine code generation to LLVM – after all, that's what it's for – but without all that type information, there's no way we could coax LLVM into generating good machine code. Other very important optimization passes done on the Julia AST include aggressive method inlining and elimination of tuple allocation.
> more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer.
There is a very clear and thoughtful choice in static vs. dynamic semantics in Julia: all semantics are dynamic; there are no static semantics at all. If you think about your code executing fully dynamically, that is exactly how it will behave. Of course, to get good performance, the system figures out when your code is actually quite static, but you never have to think about the distinction. And again, Julia already has excellent performance, and we have accomplished that with an admittedly tiny and relatively poorly funded team. (All the money in the world won't buy you another Jeff Bezanson.)
> also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) )
The academic programming language community has gradually narrowed their notion of what a type is over the past decades to the point where a type system can only be something used for static type checking. Meanwhile, the real world has gone full throttle in the other direction: fully dynamic languages have become hugely popular. So yes, if you're a programming language theorist, you may want to insist that Julia has a "tag system" rather than a "type system" and other type theorists will nod their heads in agreement. However, the rest of the world calls the classes of representations for values in dynamic language like Python "types" and understands that a system for talking about those types – checked or not – qualifies as a type system. So, while you are correct that Julia doesn't do any static type checking, it is still understood to have what most people call a "type system".
[There's actually an important point of programming language philosophy here: one of the premises of Julia is that static type checking isn't actually the main benefit that's brought to the table by a type system. Rather, we leverage it for greater expressiveness and performance, leaving type checking on the table – for now. This emphasis doesn't mean that we can't add some type checking later – since we can infer exact types 60% of the time, we can check that those situations don't lead to errors. We can also provide feedback to the programmer about places where they could improve the "staticness" of their code and get better performance or better "checkability". This let's the programmer use a dynamic style for prototyping and gradually make their program more and more static as it needs to be faster and/or more reliable.]
> Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language.
I'll repeat myself a bit too. Julia has a very clear static semantics – there are none. The run-time does quite a bit of analysis and optimization after method specialization (aggressive run-time method specialization is incredibly important, however, so one shouldn't discount it). And, of course, Julia already has compelling performance compared with other languages, both static and dynamic, in benchmarks and real-world applications.
I think we've had 1-2 interactions where we've not quite gotten along, but I might have just been misinterpreting (or mixing up julia devs)
It would help if this explanation was a bit more prominent.
For example, one of the biggest challenges in dynamic type inference is getting the types of fields of mutable heap-allocated objects. We avoid this debacle by letting you declare field types, which also has benefits for code clarity and specifying memory layout. Everybody wins. But once you are writing types, you need fairly flexible types, to avoid being stuck with only Int and Any (everybody's favorites). We then follow the implications of this as far as we can.
As for "all those runtime type tag cases", the key is that they form a lattice, which feeds nicely into dataflow analysis. The lattice-theoretic properties of a language's universe of objects typically do not get enough attention, especially in dynamic languages, where it is actually most needed. Typically the lattices are either trivial (for example, scheme's fixed set of types), or highly uncooperative. You want to hit a sweet spot where you can compute greatest lower bounds that are actually somewhat interesting.
Is Cython or PyPy useful, or do you use Python more for prototyping, and rewrite some portions in C?
[1] https://github.com/numba/numba [2] https://code.google.com/p/numexpr/
WRT to the problem of deploying prototyping/research code, there are the following unsolved problems that I'm aware of:
* Going from matrix operations over the whole timeseries (taking care to avoid problems with your algo looking ahead) for speed in a research setting to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure.
* Concurrency. The options in python all suck to some degree - especially if you have to interact with C libraries or extensions. I don't think it will ever make sense to build your real time market date in python. Again here haskell has an advantage.
However the python ecosystem seems to be almost perfect for researchers:
* Excellent and flexible data slurping/munging/transforming.
* numpy, scipy, pandas, theano, scikits... 'nuff said
* ipython
* Cross platform
HTH
exactly my thought. Algo guys get stuck in matrix land because that's where their tools take them. Whereas this came out in R last week: http://cran.r-project.org/web/packages/stream/index.html
python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway.
My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.
Looks like a match made in heaven :-)
If you're interested in advice on how to bridge the two worlds let me know, there's a lot of of upcoming technology ( LLVM, Blaze, pipes, zeromq, cloud-haskell ) that could be very useful.