zygomega

thanks mangrish,

quant. How could you tell? I'm conecting with an aggregator cause the direct market feeds are monopolistic price gougers. It's an easy switch if we make it up that curve.

I'm not sure if I even understand what middleware is (I'm a quant), but I think the answer is the disruptor!

Wow, that's a pretty low ranking being shoved into the patent troll category of evilness. When someone goes and patents an obvious algorithm at some point in the future and my little project shows up as prior art would I be redeemed :)

So, yes, on my toy wannabe hft feed, I'm currently clocking about 10 million ticks per day on about 8 futures tickers I'm tracking. The raw feed is between 1 and 1.5 gig per day. I can imagine the petabyteness of the bigger guys.

It is - I can't imagine working without it. But I'm not sure I'm the poster child for how to use org-mode properly. Every piece of code in the repo actually resides in the org file which is a big monolithic journal of where I'm at.

o-blog is a great emacs package that lets you publish sites straight from org.

Classical landing page had me hooked anyways :-)

Isn't MtGox over as a bitcoin trading platform? I agree more generally that bitcoin could be a useful market to test out.

And your story fits exactly with a hands-off automated approach to trading. We wont ever be charting a price series in the production environment because computers don't get charts.

Yep, I've poured over the technology and concepts there and its awesome. So awesome I think you can increase complexity without too much of a latency penalty. Just behind the fast guys but way ahead of the straight algo guys is where I think the opportunities lie (I'm kind of answering your previous question about where the dials sit in my mind). I might be wrong but, and would like to test out all dial positionings at once before locking in. And that's where a flexible haskell version could shine.

I've been in the industry for too long (though not in low-latency) - I'm doing this to break out of the industry! And I think the finance game is long overdue for a good disruptive technologic event (not implying my humble project is it). R is a personal choice for data analytics (I love ggplot2). But the haskell thing is more than that. It's been done before (http://www.starling-software.com/misc/icfp-2009-cjs.pdf) and even goldman's has a thriving erlang/OCaml hacker ethic (http://www.zerohedge.com/article/aleynikov-code-dump-uncover...).

Yes, but the concurrency has been shifted to the two disruptors at either end of the business process logic. And I suspect the business logic is fairly simple and lends itself to the straight through solution. If you want to react to a news event (say), you have to somehow send a signal to the feed requesting recent history (last 20 minutes say), have something waiting to look at it, crunch the numbers when they arrive to see if somethings up, then interupt the existing trade decision/risk management schedule with a potentially better trade. Complexity increases quickly and concurrency helps do all of that at high speed.

Thanks Kasey,

I'm probably thinking of algos more generally and generically. For example, whatever the class of trades, I think that what you are doing is making a stochastic forecast of future price movements. So maybe there's a good separation possible between your algorithms that create a distribution and algos that determine how to trade once you have that distribution. But saying this probably reveals a bias I have to think statistically. And I totally agree the majority of the system build thats important is the boring stuff. An algorithm should be able to tell if the market feed has died or the market shuts down unexpectedly, right? And know what to do next. And good to hear that someone thinks the costs are much less than popular opinion suggests.

I like your thinking V, and that's not too far away from one of our ideas. Take some algos that are representative of typical market makers and trader behaviour and fit the parameters so that the simulated trading starts to look like the live data. Maybe it's a better idea to use real algo makers.

definitely. I intend to move all of that code into haskell once I get a handle on whether the algo design is on the right track.

yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.

Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)

I have no idea yet. I see you're geeking out in go and there's been a lot of advocacy coming through to look at go as an alternative.

No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.

Thanks Dan, the analogy is a stretch maybe, but most algorithms fail due to being narrow models of reality, and criticised by too few eyes. If I discover something, it's extremely unlikely that others will believe it anyway (unless I prove it by becoming filthy rich as a result in which case I'm just fine).

You're pretty right about good trading algorithms but that doesn't necessarily apply to good traders or good trading systems. A good system - hft or otherwise - needs to be flexible and evolve to stay in touch with an evolving ecology. And to do this, it needs to keep track of many different trading ideas and algorithms, each of which may have its day in the sun.

I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that.

I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts.

The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.

There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.

Sounds great, but way beyond either my paygrade or budget.

> Going from matrix operations over the whole timeseries ... to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure.

exactly my thought. Algo guys get stuck in matrix land because that's where their tools take them. Whereas this came out in R last week: http://cran.r-project.org/web/packages/stream/index.html

python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway.

My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.

I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen.

But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that.

I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.

We have some plans on the drawing board for market maker simulators that could do with some fast matching so I'll keep an ear open for that ping.

Yes, if the only edge you have is speed. But a good algorithm may well be the next battleground as the benefits of ultra-low latency reduce.

No, and I am making no claims. I have a market event feed coming in, a good idea of what the event processing looks like and a rough idea of how to send an order to a broker. I think the project needs to get to production fast though.

By putting it on FPGAs maybe. Stage one is to get a robust process for measuring and researching latency - I'm in Australia and US data is bouncing into my mac via a lousy connection and a tcp port in 250 millsecs on average. There's a long way to go. I suspect that if you can keep within a bulls roar of the low-latency crowd the big gains will be on the algorithm processing side. Algorithms still add up a few million numbers every time they recalculate a moving average - there's better ways to do it.

Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes.

The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.

Good points. To be more precise, I think that an open source approach to system research and design can be hyper-competitive versus closed-door secret-squirrel development run by a committee looking for short-term wins. This project benefits me firstly because I get feedback on my initial scratchings and maybe even collaboration on areas outside my comfort zone. And it could well enable me and others to avoid having to work for the sociopaths.

I am biased in thinking that monitoring the market and trading at low latency isn't fundamentally a bad thing, and shouldn't be taxed because others are too lazy to do the same thing. I certainly think that hft has been used by evil people to front-run unknowing third-parties, often in collaboration with middle men and women who turn a blind eye to the morality of their business models.

Investing in company stock is much more about investing with a global pool of liquid capital seeking risky asset exposures these days. When global capital changes it's mind (in whatever direction) those watching the market closely will be the first to know and the first to profit from the market signal. You can go broke investing in companies that look like they have a bright future.

I intend to publish and collaborate on private algos. Even if others have the code they still don't know where I choose to operate and deploy capital. And if they get rich on the code good luck to them - maybe they'll sponsor a dedicated server for 1.0