I might be a complete failure (after 8 years of work) | Hacker News Reader

I might be a complete failure (after 8 years of work)

pathdependent.com

126 pointschasingsparks114 comments

A lot of people are leaving comments critical of his goal of beating the market as playing a zero-sum game, just pushing wealth around, having no value for society, etc.

This strikes me as dubious and high-handed, to say the least.

First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function.

Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor.

I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.

All this is only undermined by the fact that the financial sector and the stock market in particular involves mostly fictitious money. There is no problem that it solves that could not also be solved by saying "here, I conjured you a million dollars out of thin air, act as if you had it". The market is, as it were, a MUSH.

There are certainly plenty of talented people working in finance. That talent is just completely wasted.

All money is fictional, in that it only has value to the degree that people agree and believe it has value. This is no less true for the US dollar than it is for gold, silver, oil, cattle, etc.

Not entirely true....not all forms of money are the same. Some can be conjured essentially from thin air (US dollar and all fiat currencies) while all others have supplies limited by nature (gold, silver, oil, cattle, etc.)

Now, this isn't to say that paper money shouldn't exist, but with its benefits come some serious risks and responsibilities.

Fiat currency production is limited by political capital; specie money, on the other hand, is susceptible to random booms (mineral deposit discoveries) and deflation (when economies grow faster than the growth of money supply). And it's not like mineral prices are predictable enough to be an entirely safe store of value.

I agree. Disparaging speculation as "zero-sum" and therefore inherently bad is non-sense. In almost all mature industries such as grocery stores, auto sales, lumber, appliances, etc., any individual seller's gain is some other seller's loss. So what? They are providing a valuable service and if profitable in the long-run are doing so at lowest cost, highest value to their customers.

Financial speculation has a long history of being disparaged by various anti-capitalist demagogues. Speculation is a zero-sum activity between speculators but, from an economy-wide perspective, speculators nevertheless perform a valuable function in the dissemination of new information and the formation of prices. This latter function is why it is so hard -- the identification of an arbitrage opportunity and acting in the markets to make profits means that price differentials fade away and thus the potential for profits. A speculator has to continually find new opportunities in order to make money over the long term.

> Speculation is a zero-sum activity between speculators

I agree.

> any individual seller's gain is some other seller's loss

I disagree. A farmer can sell to a grocer and both can gain.

I think you don't disagree. The grocer is not in a seller in the same ecosystem as the farmer. The grocer is one of the external people who benefits from the zero-sum game the farmers are playing.

Farming is pretty much the opposite of a zero sum game.

The growing part isn't, but the selling part can be seen that way in the right conditions. I agree, though, it isn't the example I would have chosen.

> any individual seller's gain is some other seller's loss.

How do you figure? Generally, the way one seller beats out another is by providing some combination of increased value or decreased cost.

In a mature market sellers are competing for a fixed number of customers. For example, a grocery store that wants to increase sales (above local population growth) must "steal" customers from its competitors. So my gain of a customer constitutes your loss of a customer, i.e. zero sum. Sellers achieve this (take customers from competitors) by offering higher value and/or decreased costs as I also mentioned.

After eight years, I have nothing concrete to show for my efforts.

So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose.

Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself.

When you do for others, you always have something concrete to show for your efforts. It may not be what you expected, but it will definitely be something worthwhile, I promise you.

This is a very valid way to put this, and it helped me. Thank you.

This is what happens when you play a zero-sum game. Even if you win, someone has to lose.

I don't see the causality. He has yet to succeed because it's an incredibly difficult problem domain. There are problem domains of similar difficulty that benefit people that he could have chosen and still be in this position.

Almost anything where you create something would be better.

Then again he/she is creating software and algorithms.

Yes, but to what end? When it comes to creating things that have value, I like objective standards and my favorite definition is Buckminster Fuller's: "Real wealth is whatever nurtures and accommodates human life." Anything else is pointless.

Incidentally, a recent study claims that 60% of UK workers feel that their work is of no use to society. Financial Services, Telemarketing, Spammers, High-Frequency Traders and many other 'professions' would fit this description. Here is the link: http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_wo...

Some problems just can't be solved with the current toolkit. Sometime you need a different toolkit.

For example, ESR derives a development methodology for open source software using Austrian economics.

He did? What does that mean?

> This is what happens when you play a zero-sum game. Even if you win, someone has to lose.

How is it always a zero-sum game to win? Fundamentally, trading stocks is not that at all. If I buy one of the first issued shares of "Tasty Ice Cream" for $1 because I want the company to spread their goodness as well as get a return on my money, no one has lost. If I lose my job the next week and need to sell my share which may be valued at $1.10 the person that buys has not lost, especially if he receives a dividend long enough to cover the $1.10.

I think this is the way capital markets were originally supposed to work; matching companies in need of growth capital with investors seeking to earn a risk-adjusted return on their long-term savings.

Unfortunately, the financial markets have morphed over time such that now, traders trade for the sake of trading. Instead of our work being our wealth, trading securities somehow became an end unto itself. What's worse is that there really are two markets today: one for the individual retail investor and a second for institutional investors.

Retail investors are taught to 'invest for the long-term', and 'not try to time the market' meanwhile hedge-funds and high-frequency traders play a different game with different rules. Their profits come primarily at the expense of retail investors and in many cases investment banks 'front-run' their own clients taking advantage of their customers in a way that would be illegal in any other industry.

John Stewart explains it better than I ever could in his face to face indictment of Jim Cramer of CNBC: http://ccinsider.comedycentral.com/2009/03/13/jon-stewart-an...

(Link for the Canadians: http://watch.thecomedynetwork.ca/the-daily-show-with-jon-ste...)

For the most part I totally agree. Rampant speculation combined with a fiat currency and too loose regulation is how we got so much money sloshing around, a deformed market, and this economic crisis (and a near collapse of the financial system). We've gotten too far away from fundamentals. We also see this in startups. For example, I totally agree with Mark Cuban who laments that modern founders often look at valuations, and indeed raise lots of capital, for possible IPOs without even discovering how to profit a single dollar.

>Retail investors are taught to 'invest for the long-term', and 'not try to time the market' meanwhile hedge-funds and high-frequency traders play a different game with different rules.

Exactly....that the financial services industry constantly advises the retail investor to "invest for the long term", when they do the exact opposite (while generally using the funds provided by these exact same retail investors), to me is a pretty strong indicator that the markets are something entirely different than how they are sold to people.

The markets were created to provide funding for companies in need of capital, and a branch of the financial services industry was legitimately required to facilitate this process between companies and individual investors, but now almost everywhere I look people are focused on finding new ways to skim money out of the system, under the guise of "providing liquidity", as if there was some problem with liquidity 10, 20, 30 years ago.

This is only true for long-term investors. Algorithmic traders have no vested interest in whether the underlying business succeeds or fails. They are only there trying to predict where the wind will blow next. They are quickly in and out of any potentially profitable company; quick to buy "puts" in a momentary time of weakness or "calls" in a time of temporary strength.

For algorithmic traders, it is indeed a zero-sum game. For the rest of us "investors", as I like to call it, to distinguish it from trading, there is potential for long-term good.

Not really. For algorithmic traders, the transaction costs will dwarf the dividends, so it's really a negative-sum game. Unless there's a very small volume of algo-traders, and a large volume of long-term traders who just happen to be re-adjusting their positions while the game is being played.

It is a zero-sum game as long as you define it as follows: a trader's score for each transaction is (price paid - closing price). If Alice sells GOOG at $100 to Bob, and GOOG closes at $99 that day, then we say that Alice won $1 and Bob lost $1. That's all.

Maybe 1 year later Bob will sell GOOG to Alice at $200, and the stock will close at $201. Again Alice won $1 and Bob lost $1.

Of course Bob made a lot of money in the process, so the zero-sum game definition might sound arbitrary. However, as far as trading is concerned, the definition as zero-sum game is useful. If Bob were a better trader (more skilled or luckier) perhaps he could have saved $2.

See The Winners and Losers of the Zero-Sum Game by Larry Harris.

http://www-rcf.usc.edu/~lharris/ABSTRACT/Zerosum.htm

So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose.

When you play football too, if you win someone has to lose. And yet there are positive externalities, to wit millions of people watching the World Cup.

In trading, the externalities might not be so obvious but they do exist.

Could you explain more please? Bonus points if you do it without using the phrase 'provide liquidity'.

A short answer would be: one of the externalities that traders like the OP provide is lowering transaction costs for other traders such as investors. Compare trading stocks with speculating in real estate: buying a $150k house and selling it later might involve (say) $10k worth of commissions to your real estate agent, while trading fees on a comparable stock transaction are usually much lower.

Please see my other comment below and (especially) the linked article.

http://news.ycombinator.com/item?id=1467171

Another positive externality of trading is lowering the cost of funding for enterprises. Companies go through a lot of trouble to make an IPO; they do so because getting listed increases their sources of funding.

A longer answer would probably require a blog post, so that we could discuss whether I'm "cheating" -- after all don't the points above boil down to providing liquidity? Am I really "cheating" or maybe your condition is unreasonable? Et cetera. If I write such a post I'll post it to HN, though I doubt I could do better than Larry Harris.

Eh, isn't providing liquidity reason enough?

I'm not interested in your bonus points, but in the correct answer. I believe this is it: Large, liquid markets reduce economic friction. They make an economy larger, more diverse, and more adaptable. The low friction allows small changes in supply and demand to quickly propagate long distances, producing responses where otherwise the information would have been lost to the friction. You don't have to look far in space or time to find places where this "zero-sum game" isn't or wasn't available. Just look for poverty and lack of social mobility.

There is an economic progression from the switch from barter to money, to lending of money, to spread of risk by enabling minority ownership of multiple ventures (early corporations), to easier transferal of ownership in smaller pieces (allowing more of society to participate), to large-scale, low-friction, highly-liquid markets used by everyone, directly or indirectly. Each step further reduces economic friction, which lowers the threshold for economic activity to take place, causing more of it to occur, increasing the prosperity of the whole society.

Even if you lose money personally in the "zero-sum game" with unfortunate investments, you're still probably losing only a portion of the money you made working at a company that wouldn't exist if not for large, liquid, low-friction markets.

> This is what happens when you play a zero-sum game. Even if you win, someone has to lose.

That's why I quit playing poker[1], even though I was a winning player after a lot of study. Zero sum games are no good. Creating things makes more winners and the capability for an unlimited amount of winning.

[1] Well, that and the culture with all the cigarette smoke, alcohol, late neon lights screwing up your sleep schedule, and general degeneracy. But that stuff didn't bother me before so much before I put it in greater perspective.

Trading stocks isn't a zero-sum game. The value you deliver is to (more) accurately price the shares. It also isn't zero sum in that the person on the other side of the trade values what you're giving in exchange more than you do, by definition.

Beating the market is a zero-sum game.

Calling trading a zero-sum game shows a fundamental lack of understanding of finance. We may have very negative views on the industry, but it exists because it serves many fundamental (and necessary) functions, including:

* Accuracy in pricing shares (and it is important to know the actual value of a company for many reasons)

* Enable market participation. In many illiquid OTC markets, it is all but impossible for anyone outside large institutional players to participate. More trading = lower transaction costs, more transparency, and ability for the little guy (me) to join the action

* Liquidity. And yes, this is very important. It is a common answer because it is a good one

* The same security can be worth different amounts to different people. By definition this precludes it from being zero sum (this statement is of course subject to what school of financial theory you subscribe to)

* Performance measurement. Stock options have gotten a lot of bad press, but compensating a CEO according to lets say, revenue, is much less unbiased and can lead to worse outcomes (e.g., grows business in unattractive segments or over-invests to boost top line)

* Others

The above are in no way MECE and overlap in many ways. Markets can definitely be better, but calling it a zero-sum game or completely value-less to society is a bit too far.

> * Accuracy in pricing shares (and it is important to know the actual value of a company for many reasons)

Define "accuracy"? Preferably in such a way that helps me reconcile it with events like the flash crash.

One of my biggest problems in understanding the financial markets is how you separate the signal (events reflecting actual change in company value) from the noise (speculation, flash crashes, etc.).

While I realize that there is real information (e.g. earnings reports) that goes into the pricing, it seems like there's also a hell of a lot of worthless noise (i.e. I see the flash crash as a massive noise spike).

It seems to me that separating the two is how people make money long-term.

It's more like, everybody who has a view on the value of a security can put some money where his mouth is. The price will move accordingly, until nobody who has a different opinion has any money left to put behind it.

If smaller differences between current price and opinions can lead to action--because of lower transaction costs--the market is better.

yeah open source part of your work. it should help a ton of people especially if it's a niche. plus it looks great on the resume

Question: has anyone managed to do this? (eg. do hedge funds do this, or do they combine algorithms with their own ideas and quasi-insider knowledge, plus massive amounts of timely data - inaccessible to an individual).

My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something more difficult that AI - ie, algorithmic inference without the data that your competitors have. Because of their better information, they will find a regularity (eg. 0.01% dip on Wednesdays) quicker than you - and trade it away.

It would be awesome if you could do it though - forget the philosopher's stone aspect of gold-for-nothing, this would be a very cool intellectual achievement.

Idea: focus on small markets/exchanges (unpopular metals, unpopular countries, even non-financial markets, like eBay) that don't attract the attention of big players. Easier to find arbitrage there; you can make some actual money, and then move onto a bigger market with your learnings, earnings, and confidence. Sort of a niche strategy.

Personal Note: My AI research depressed me, and I (wisely) decided to shelve it. It's been incredibly more satisfying to work with others, to achieve doable goals, that benefit others. Fortunately, AI comes up in everything; and my current approach (of not doing AI directly) may well be the most effective way to approach the goal. It might be the same with you - both markets and inference are universal.

EDIT So, Tantalus = low-hanging fruit + moving-goalposts http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus

Simply put, if anyone has managed to do it, you aren't likely to hear about it. That said, I know of two:

First was a math prof that would play the markets every couple years when he needed a little extra cash.

Second was someone who had isolated patterns of market intervention and figured out how to profit from them. But that's more like insider trading than beating the market algorithmically.

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse.

I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.

Long-Term Capital Management always springs to my mind. http://en.wikipedia.org/wiki/LTCM

Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of unexplainable randomness in those data. -- Wikipedia on Nassim Nicholas Taleb

People project patterns onto data when in fact, they don't exists.

"Fooled by Randomness" is a great book. But I sometimes feel about it the same way I feel about people who say lotteries are a waste of time: some people do win. And sometimes, despite everything, a transient pattern does repeat. How frustrating!

Edited to add this, which is sometimes also brought up: http://en.wikipedia.org/wiki/Illusion_of_control

Yeah, now Taleb is betting on the big random event, but what if the unexpected thing is that the unexpected thing never happens?

http://en.wikipedia.org/wiki/Apophenia

> People project patterns onto data when in fact, they don't exists.

So, the patterns we see in the Ulam spiral can be completely random?

We observe patterns. Human brains are really good in finding them. The problem is finding patterns in past data may or may not provide useful insights into future data.

With the resources being dedicated to this problem, I find it highly unlikely I could be able to offer any helpful advice that would prove more helpful than what already exists.

It would be hard enough to predict the market if the only factors influencing it were signals coming out of the companies we trade. Predicting what other algos may me doing in the space we are trying to examine and the influence their trades exert in its future is... Well... I won't try.

Yes, that is the main pitfall with machine learning in general. Usually you do two things to mitigate it (but they don't fix it entirely): regularization and a test set.

Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the complexity, and usually the value of C is pretty unimportante (it is empirically found that it only affects the solutions when you change it in orders of magnitude). Keeping a test set is always testing your predictions on held-out data that you didn't use to train the model and that is hopefully independent from your training data.

Still, if you come up with a huge number of models and test them on the same test data you're statistically bound to find a few that seem to work very well, until you try it in different data.

My feeling with most of these market-beating schemes is that just the price history of stocks shouldn't be that predictive of future prices (but aggregating real-world information should). After all, usually what creates spikes and valleys in the prices are some relevant news or decisions.

Isn't it better to adopt a Baynesian approach and directly model the complexity of the model rather than some arbitrary Ccomplexity?

To wit, P(model|data)=P(data|model)P(model) / P(data), which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using.

I guess the problem in all this is we want* the model to be very complex, but we can't tell that complexity from overfitting (and it may be that no model exists so the only thing you can do is overfit.)

Yes, but the formulation I showed you is equivalent to a bayesian prior. For example, if you want to learn a weight vector w that gives high likelihood to the data and has a gaussian prior with 0 mean and Cidentity covariance, the MAP answer is "minimize -log(likelihood) + C||w||", where ||w|| is the square norm of w. Equivalently, if the prior is a laplacian you just change the norm from the l2 to the l1 norm.

Being bayesian gives you an extra capability that is model averaging, and this does usually improve the behavior at a high computational cost.

I really like bayesian models, and right now I'm experimenting with one that should do unsupervised sentiment analysis without a priori knowledge of word polarity or things like that (yes, I'm a phd student in machine learning).

I do remember reading that the Bayesian approach leads to previous empirical formulas falling out. Is that the case here, or was that formula derived using Bayes? I'm a PhD student in something else, and I'm trying to do some machine learning. So, what should I read to make what you said make sense :)?

Yes, it is the case here that a bayesian approach can lead to a previous empirical formula falling out. What I was saying as well is that this regularization + test set approach is also valid (and sometimes slightly more or less general than the bayesian approach, since, for example, SVMs fall naturally out of thinking about regularization but they have no analogue in bayesian classifiers).

It also goes the other way, and some formulas are first proposed in a more bayesian-ish context and then extended to some simpler-looking empirical formulas (for example, the jumps from hidden Markov models to max-ent Markov models to conditional random fields to max-margin Markov networks to structured SVMs).

There are more approaches to machine learning, and in John Langford's blog there is a nice table showing the merits and flaws of many of them: http://hunch.net/?p=224 . But you must keep in mind that you can find many equivalencies between these approaches (boosting for example can be seen as a loss minimization with regularization, and max-ent can be seen as a special case of a bayesian model, etc).

There are, roughly, two schools in modeling (I'm not in the finance modeling field but I see no reason to believe it's different there): extrapolation and process modeling. Extrapolation becomes better with more data and more accurate descriptions of the curve. Process modeling becomes better with (duh) understanding the processes better. Of course in practice all models incorporate aspects of both. My point is that just adding data isn't the only way to improve a model, and in my opinion, it's even only relevant after you've achieved a sufficient level of accuracy in your process description. In my limited understanding of financial modeling, quants focus a lot on the mathematical aspects.

You've learned a lot, so you're obviously not a failure, but I think your post illustrates how draining a winner-take-all field like quantitative trading can be. You can never be sure if your wins or losses are real or just mirages caused by noise. Humans need the emotional encouragement that incremental success brings. Nothing get's me excited to work more than a good day of sales or a thoughtful comment from a customer. It takes a special type of personality to keep on persisting in the face of repeated failure. Certain start-ups also have winner-take-all properties. Anything that relies on network externalities or advertising revenue is more risky. The rewards can be big, but I prefer markets where the product's value isn't a function of the customer base and success is more incremental.

My mother was down on the floor of the NYSE the other month, and she got the opportunity to corner a trader who she noticed was relying heavily on algorithms.

She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful."

The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible.

If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful.

Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.

You are right that for a individual investor focused on the long term, a algo-order probably wouldn't make a difference in comparison to a market order with your E-Trade/Schwab account (most likely, your order-flow won't go directly to the market anyways; it is either crossed internally, or re-routed to a broker/dealer that's paying retail brokers for the order flow such as Timberhill).

However, I respectfully disagree that humans are better at making trading decisions than computers. The world of algo trading can be divided into two sides, a) high frequency traders, who through fast cancel-and-replace limit orders and colo with the market centers, try to act as virtual market-makers (or scalpers, depending on your perspective), b) buy-side institutional fund managers who want to complete their orders, without HFT predators and negative market pressure. Large block orders are spliced into small lots (i.e., VWAP) and sent to the market using intentional limit price over time to hide the movement of a huge buy or sell order on the market.

Human beings might be better than machines at picking single stocks for long term investing (although most people are probably still better off investing in an ETF). But that's not how sell-side traders make money in the first place. Guys like GS/MS/Timberhill make money by having the unfair advantage of faster execution speed, more capital and specialized trading algo's against the small retail investors.

Has there been a good writeup of what HFT "Is" at any point?

What is your impression of them as a pernicious/positive force? Are they essentially market making?

This is the kind of stuff I really should have an opinion about, but I haven't found anything to lean on in the public domain.

Has there been a good writeup of what HFT "Is" at any point?

http://vimeo.com/6056298

What is your impression of them as a pernicious/positive force? Are they essentially market making?

Open for debate. Depends on what you mean as a pernicious/positive force. Good for retail investors, institutional investors, stability of the market, or the sell-side? All of these are conflicting sides. It is generally SEC's mission to protect the small individual investors' fair access to the market, while trying to walk the fine line of not disrupting the big institutional investors/sell-side brokers' way of doing business (and their political lobbying groups).

Pro HFT argument: HFT are virtual market makers that through the use of technology and arbitraging through multiple ECNs, are decreasing the bid/ask spread of the traditional market makers and providing more liquidity to the market. They serve as stabilizing force during irrational exuberances.

Con HFT argument: HFT are bad predators who through technology, jump ahead of institutional investors' block orders and in term pass on higher priced liquidity to retail investors that no one needs. They don't serve as stabilizing force, as they stop trading as soon as they stop making money and in fact may fan the fire by employing high frequency short selling in a flash crash.

If you're doing what you love, you've already won.

I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I've been prototyping them, on and off, for about eight years too, depending how you count.

They've all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess.

This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you're lucky and it's a 10 year problem.

But even if your goal is impossible, it's very likely that if you work on it for 20 years, you will solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It's almost inevitable.

Regardless, I wholly support what you're doing. Keep going. And read widely... explore... make sure you're playing in the margins of your field, and exploring odd projects far outside the scope of your work. It's those places that you will make that accidental discovery... that in all these years of trying to "beat the market" you made a fundamental discovery in artificial intelligence, or network theory, or whatever.

And yeah, if you're doing what you love, you've already won.

It sounds like despite your ideas of what failure is, by age 25 you've learned and applied more than most people ever will.

Agreed. I'm 28, soon to be 29 and I have achieved a great deal less. That kind of experience, though sour, is pretty awesome, at the very least you should be able to get yourself into a highly paid quant position at a finance firm.

Many smart people have tried to "beat the markets" and failed. From what I've seen of it, this sort of activity is at best a form of numerology. Also, in trying to move money around slightly more efficiently than the next market analyst you're not really generating wealth as such, and your ideas and activities will leave no lasting value for society.

You may not be a failure, since the skills acquired in analysing trends can be utilised in other areas. The process of having promising looking preliminary results, only to find that they are illusory is familiar to many scientists.

Ultimately it comes down to what you really want to do with your life. Is your life only about moving money, or something else?

If I'm not mistaken, I believe the idea is to set up some automated trading bots that just simply make money for you on a continual basis. This constant revenue stream allows you to forget about making money and allowing you to use those resources to either fund a life of ease and luxury or other pursuits such as "making the world a better place".

Seems like it'd be a better bet to apply all your skills to creating value (wealth) instead of just trying to push it around a smidge more efficiently.

In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.

Hm, I accidentally upvoted this so I may as well leave a comment. The dichotomy between "creating" wealth and "pushing it around" is false, in my opinion. Lots of people make it and it's intuitive, I know, but figuring out the proper value of things, trying to make forecasts about the future, even profiting from differences that shouldn't be there, all this is part of the capitalist meta-system called financial markets.

One should not confuse the agency issues, borderline-legal information advantages, or even outright fraud in the markets with the basic activity that sustains them. Think about the alternative: Would you like to have to sit with a dimwit credit officer (= bureaucrat) from a big bank who can then effectively veto your plans? Or would you rather have a lively market that decides on such things? Do you want to do your IPO in a public equity market or would you prefer to try to break into an insider club of old rich people who may never come to understand your funny newfangled business model ("what's internet?!").

German sociologist Georg Simmel wrote an interesting book about money, one of the ideas I liked is that its fungibility dissolves the need for relationships, making us free to choose them, and thus effectively increasing personal freedom. The same, I'd venture, is true for capital markets. By spreading transactions around, they increase your options and freedom as entrepreneur. And for markets to work, you need people actually _in_ the market, like it or not.

Figuring out value (as opposed to the going price), identifiying and exploiting inefficiencies, trying to make more intelligent forecasts of likely developments, all these _are_ "real problems" in a market economy. Having intelligent, hard-working people working on them makes the system as a whole more intelligent.

I fully agree that finance is useful because it increases market efficiency. The problem is that like most fields, most of it is uncreative legwork that doesn't provide much leverage to get rich from. So a smart creative person should consider whether they're falling into the cognitive trap of "I want to make money, finance deals with piles of money, so I should apply my creativity to finance".

It's just a crapshoot unless you have some real information asymmetry[1] you can exploit to make money. This could be in the form of noticing some sort of systematic inefficiency, having a unique and effective strategy of gaining market information to identify mispricings, etc. Otherwise you'd be better off finding some unmet need to creatively address and avoiding the cognitive trap that leads from "I want money" to "I should go into finance".

1. http://en.wikipedia.org/wiki/Information_asymmetry

The median startup does not offer a better outcome.

If you're only going to get the median outcome, don't do a startup. If you have a real chance at making it, do a startup.

Yes, people mis-estimate their ability all the time, but that's part of life. If you want a safe good outcome, become a doctor or something.

The median guy who tries to hack the financial markets on his own has the exact same outcome as the median startup, and he's getting that outcome.

I suspect you could get a pretty good job as programmer for some hedge fund. Not only would it leverage your command of this domain, you might learn a few things that will explain why it is difficult to impossible for you to win in a vacuum.

There are social engineering aspects to this game that you have not been exposed to yet. Also, you may just not be sophisticated enough, or you have pursued a dead end approach. This is a very rich field of study.

If it's your passion, I say stick with it, but expand your circle of knowledge and influence.

And screw all this 'zero sum' crap, capitalism is the least zero-sum game humanity has ever come up with. And speculation, market-making, betting on winners & losers is one of the most important driving forces of capitalism.

"He[Dan Zanger] holds the unofficial record in trading stocks by turning $11,000 into more than $18 million in 18 months in 1999-2000. He grew that to an incredible $42 million in less than two years and has the tax receipts to prove it." http://www.investopedia.com/articles/trading/04/082504.asp

"Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest" http://en.wikipedia.org/wiki/Larry_R._Williams

Somebody has some sort of algorithm.

I find myself excited over the preliminary results of my increasingly sophisticated simulations

This. The vast majority of the money being made in the market is being made by people with what you may consider a laughable lack of "sophistication".

Start simple. Complex modeling of stock market behavior only makes sense if you think the stock market has a complex underlying structure.

By sophisticated, I meant refined and cultivated, not devoid of simplicity. It was a poor choice of words. At some point in my development, I started to eschew more complicated forms of AI and machine learning, in favor of simpler agent-based models with "laughably" simple assumptions. I think it's the proper direction, then again, the title I chose for the post was not arbitrary. ;)

The term "complete failure" is more revealing about your draconian idea of success. You should take a look at your own words : "they have never found their Holy Grail" [therefore, they joined the ranks of Academics as a testimony of failure].

If you're running after the Holy Grail, you have no choice but to be a hero. What kind of pressure is that ? In fact, most of the answers are in front of you, in the way you express yourself.

For instance, the myths of the Holy Grail and Tantalus are ones of insoluble challenge and dilemma. Therefore, when you are wondering if the Holy Grail exist, where it is, and if you'll ever reach it... You are probably asking the wrong questions.

Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do you want to "beat the markets" ? That's 2 different things, one is a mere consequence of the other. Where is your priority ? As it's not clear from your writing, I suppose it's not clear in your mind either.

The title was unnecessarily provocative, a reflection of my mood at the time of writing it. (It's Sunday; I'm hung-over; and I was looking at the results of a four day simulation run that were ambiguous.)

My priority is in "beating the markets" because it's a fascinating challenge. I also have interests that lie beyond satisfying my intellectual curiosity but require wealth. I find the idea of using my own wealth to solve problems (e.g. avenues of cancer research) far more appealing than fund-raising or requiring public funds.

Look on the bright side. Many people dedicated their lives to the search for a process to turn lead to gold. This seemingly fruitless search lead to all kinds of neat discoveries. You've learned lots of valuable things that you wouldn't have learned otherwise as well. Keep searching if you must but start benefiting from that knowledge.

The real secret about turning lead to gold is that it is possible, but by the time you have mastered the skill to do so, you no longer need to or want to.

Alchemy and transmutation of lead to gold is a pretty apt analogy for John Nelson's quest to turn his model of the market into gold.

a) The alchemical belief in transmutation was based on a thoroughly wrong understanding of the underlying processes. John's (and other's) lack of success indicates a faulty model.

b) Ironically, it transpired that, under true nuclear transmutation, it is far easier to turn gold into lead than the reverse reaction, which was the one the alchemists had ardently pursued. Nuclear experiments have successfully transmuted lead into gold, but the expense far exceeds any gain.

http://en.wikipedia.org/wiki/Nuclear_transmutation

If you look at the first sentence of the article, it is linked to the subheading of a blog post that explained my intellectual evolution -- the subtitle of which was "The Contemporary Philosopher’s Stone" which is how I refer to this problem. (http://en.wikipedia.org/wiki/Philosophers_stone) You're not the only one drawing the comparison ;)

Sounds like in that frame, most of the consoling messages here are along the lines of "our gold is not the common gold."

I hate to be the bearer of bad news but, based on the comments you've made in your post, you are most likely a smart fool (as you have so aptly put). In essence, you don't even know what you do not know. I've worked at top-tier Wall St institutions and have had first hand experience working with several of the largest quant hedge funds out there. Here's my advice: if you truly enjoy markets and are sure you want to take this path, you will be much better served by spending several years working at an established quant hedge fund. This will open your eyes to the theory, processes, data, and techniques required to earn alpha consistently and in a risk efficient manner. It's a long road and that road is full of extraordinarily hard working and intelligent people that one has to constantly compete against. If you can handle this though, you will learn a bunch and at some point in the future you can strike out on your own again. At that point you will at least have a chance of success. Good luck!

I tried to beat the market using all kinds of AI stuff off and on for a number of years. Learned a lot about AI techniques, and became a better programmer, which helped in my day job, but never got anywhere. It's a fool's game.

Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it.

Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.

Only very few people have commented that, you are only 25! (I guess given the nature of the HN community :)

I don't think anyone can consider themselves a failure at 25, having spent 8 years learning a lot of programming, algorithms, data mining / machine learning and complex systems modelling.

It would be easier to think that if you hated what you were doing. But it sure doesn't sound like you do. So you just need to decide what you really want to be doing.

If it's algo trading, I concur with another poster that says go work for a hedge fund (or bank, or prop trading shop). It is super competitive, but they have the technology infrastructure and most importantly the capital. Getting in is not easy, but simply show them all your work (it doesn't work anyway, but is indicative of your skills and way of thinking). You will learn a lot, you may hate the people and environment, or you may love it even more. And then yes after a few years of experience you will most certainly be in a better position to go off on your own again (or do something totally different, by then you will really know if you like it or not!). Many top hedge fund managers / traders only started their own thing at 30, 35, 40, even 50... I know a dentist who became a prop trader. Anything is possible.

A PhD would be a great option IF it's for the right reasons. But if you want to do a startup (sounds like you might quite like the idea and you posted here on HN, so...): - you already live on ramen, so no lifestyle change there; - bootstrapping something can't cost more than losing money with trading algorithms; - you already have many of the requirements: coding / technical skills, low-cost living circumstance, love to solve tough problems and a huge amount of tenacity in the face of failure and overwhelming odds; - bonus: your interest in social systems modelling etc ties in pretty nicely with what's big right now and in the near future.

So if that is what you really want to do, go for it either alone (or find a co-founder), or find a small/medium startup to work for. To make the transition a bit more natural perhaps focus on ones that are data-driven and have machine learning / modelling at the core of their business. Think recommendations, systems modelling (www.flightcaster.com) and weather (www.weatherbill.com). There are many many examples of YC and other startup companies of this nature (many focused on the social network space).

Good luck in whatever you do decide to do with the next 60+ years of your life. On your deathbed you can post about whether you think you are a complete failure or not.

John,

95% of people would agree with you. But they (and you) are wrong. What is your end game? Do you want to be worshiped on HN or do you want a six figure job some where?

If you want either of those, you could just as well write up all you know spit it out in blog posts and free eBooks. Someone is going to recognize your skills and take it from there. At this point the worst thing you can do is keep all your failures to your self, because then no one can benefit from them.

His approach to this - the goal of beating the market - has been totally wrong. It sounds as if he's been at this completely solo - he mentions in the comments that he worked alone as he didn't want to share his algorithms.

He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading.

Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year.

Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.

Well, he might just not be smart enough. Large financial institutions and many smaller hedge funds reap countless millions from the markets and much of this comes from proprietary trading (well, all of it in the case of hedge funds.) There's a massive brain-drain into financial institutions of many of our best and brightest, all in search of making a fortune by the time they're 30. So on the one hand you have top students from top institutions applying their considerable intellect and training into applying advanced mathematical methods against an extremely complex problem. On the other you have a guy who tinkers around himself hoping after hope that the amateur (for that is by definition what he is) attempts that he makes at beating the market are going to some day work out and make him his fortune.

Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage.

"I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.

Look to none other than Long Term Capital Management[1] back in the 90s to see how dangerous a game it is to either algorithmically predict or automate a thing as emotionally wrought and, at times, institutionally manipulated as the market.

[1] http://en.wikipedia.org/wiki/Long-Term_Capital_Management

What? Their time-tested bond arbitrage algorithms worked great. They beat the market month after month.

Only when they tried speculating on equities did they fail. And even then, their positions only took short-term losses that they would have recovered from had they not been leveraged 100-to-1. Their algorithms were correct, but they didn't have enough capital to see their positions through to the end.

The LTCM story is about greed and leverage, not algorithmic trading. Plenty of money can be made trying to beat the market: it pays the salaries of the hundreds of thousands of investment bank employees around the world. The problem is, you personally can't do the same things they do, because you don't have enough capital.

Raise a billion dollars and your days of commuting to work are over.

The most important thing in your life is you and the brain you are carrying around in your head. If your experiences have made you smarter, you DO have something concrete to show for the years you've put in.

Entire countries have had everything destroyed, but have come back to be very successful. And this happened because of what they carry around in their heads.

I find it inspiring that he's stuck with it, and wonder how's he's made a living in the meantime

I take short-term jobs and contract work for a few months, then quit everything for full-time research. Usually the jobs I take happen to require learning something tangential but possibly useful. It's a happy coincidence that I am perfectly happy with the absurd level of frugality required. (I live in a place that most people call the "House of Squalor. It's really not dirty, but it is semi-dilapidated.)

It sounds like a trolling headline - he himself acknowledges the value of what he has learnt.

Some suggested in his comments that he should join a firm so that knowledge and experience could be shared towards finding the algorithm[s]. His reply to Henrik, "I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms", is a factor in his progress, or lack of.

By working alone, he's more likely to earn 100% of nothing, when he could collaborate and earn a small % of a very large number.

That said, if he finds the algorith alone, then big congratulations and respect are in order for sticking to his convictions.

negative findings (and publishing those results) are just as useful as positive findings.

If that's true, I think it needs a lot of thought, then we at least know that negative findings are valued far less. I found that my TOE was wrong is not likely to win me a Nobel unless there are associated "positive" findings ...

I was obviously considering only negative findings with actual thought, research plans and analysis.

I've met people who claimed to make money with their algorithms, which really surprised me. Your post gave me an idea, though: What if instead of running your algorithm on the whole set of stocks, you create several random subsets. One of them has a good chance of being successful, at which point you can claim that your algorithm works and sell it for $$$. (I am not entirely serious, of course - but maybe some of the successful algorithms work that way?).

http://en.wikipedia.org/wiki/Monroe_Trout

Even people with very high level math knowledge fails to predict the market (remember the two noble prize scientists who failed?).Since you're out of school when you were 17, I doubt you'll able develop complex schemes (unless you are a very very rare genius).

Obviously he has the aptitude to learn things on his own. Sounds like he can learn better/faster on his own time than the large majority of students do at a school. So no need to hound him with pessimism for having an unconventional but pleasant lifestyle.

I was not out of school at 17. What caused you to infer that? (I'd like to correct it if it sounds that way.)

I am actually starting a masters/Ph.D. program in the fall in computational social sciences. Like I mentioned in the post, I am interested in social systems in general, not only markets. (However, I do find markets to be the most interesting.)

"Every few months, I find myself excited over the preliminary results of my increasingly sophisticated simulations, only to be disappointed a few short weeks later to find that I was simply wrong."

Isn't this why algorithmic traders "refit" their algorithms on a daily basis?

To beat the market you HAVE to predict what trades others will make in the future. It is easy to use math to make it appear to be a numbers game but that is just abstraction. The reality is that people buy and sell stocks for 1000s of different reasons.

I use much less math than your average quant. I prefer simulations with a lot of agents. Think Game of Life, not partial differential equations.

Any algorithmic method of beating the market can only be successful if it is a secret and if its trades are of insignificant volume. Otherwise, knowledge of this algorithm will be factored into the prices, defeating it.

this is really not entirely correct.

when things started melting down in 2008, it was largely attributed to the fact that hedge funds basically all had the same strategies in their book. When one started liquidating, many started losing value and liquidating in a disastrous feedback loop.

There are plenty of strategies that are very robust, and in hedge-fund land, its not uncommon that many players are playing very similar hands.

In fact, there even exist strategies that get better as more people run them.

Remember, no man is a failure who has friends. -- Clarence in It’s a Wonderful Life

An interesting post. I'd be interested hear what kind of background you've got.

I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship.

I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your background and what you know. If you think 'a random walk down Wall Street' is correct, it isn't. Have you seen the derivation of Black-Scholes? Do you know what most high frequency trading is based on?

I'd also be interested to see what type of stuff you like to read.

I didn't mean it in an insulting way. I'm actually starting a Masters/Ph.D. program in computational social sciences in the fall, hoping to fully immerse myself in an environment that also finds markets fascinating.

I have derived Black-Scholes; I study market micro-structure; I read as much as I can on anything related to social systems from volatility surfaces in options pricing to more generic machine learning methods such as PSO.

(I'd like to respond more, but I just noticed I am 20 minutes late for my fathers birthday party.)

Cool, definitely get back to me. Kinda random but I am very interested in complex networks as well as some finance stuff.

I don't think he was being insulting. He was just explaining that the incentives discourage anyone from publicly disproving the efficient market hypothesis.

This is where it gets fun though. An inefficiency being found amounts to you knowing something about pricing that others don't. You exploit it for profit, and the process of exploiting it removes or reduces the inefficiency. The only way the market is truly screwed is if consistent arbitrage opportunities become available, but the kinds of inefficiencies people find seem more transient than consistent.

On another note, betting markets are (or were) apparently far less efficient than financial markets, in that the true likelihood of a team/horse/camel winning in a competition may be different to the money which punters are willing to bet on it. I know of a guy who made his fortune after many gruelling years by finally coming up with prediction models which worked in these kind of gambling scenarios.

Could you elaborate on this? In which ways is "a random" walk incorrect? In what is high frequency trading based?

On the last question give me a little time to look up a paper. I'll post it. Essential it shows how the strategy of most long-short hedge funds work. I'd have to look again, but one of the major high frequency strategies uses a version of this.

I'd love to read it, but the easiest way to answer your second question is to point you to 'A non random walk down wall street.'

One huge assumption that Black-Scholes makes is that returns are normal distributed. This is not done for empirical reasons rather mathematical ones, namely it is so convenient.

Edit: Pdf http://www.newyorkfed.org/research/conference/2007/liquidity...

Some meta data and such if you don't feel like going with a pdf:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987

This type of thing has been done before:

http://en.wikipedia.org/wiki/FatKat_(investment_software)

http://www.idsia.ch/~juergen/finance.html

http://www.technologyreview.com/blog/guest/25308/

I'm sure there are other examples. I think part of the problem is this guy thinks he can get rich quickly with just the right set of programming instructions.

Nice quote from one of the links' links, from NY Times:

But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.

Reply on news.ycombinator.com