crntaylor

That's embarrassing - I thought that Shiller's "Real Price" had dividends already rolled into it!

Believe me, I am more than aware of the perils of overfitting!

The purpose of my comment isn't to say that using Shiller's P/E ratio is a good way to make money. In fact, I'm essentially saying the opposite - that while I think the research is good, it would be an inappropriate tool for most individual investors, because of the length of the required time horizon, and the length and magnitude of the drawdowns you'd have to endure.

As an aside, I believe that your criticisms of economics are misplaced. Certainly there is plenty of poor research and curve-fitting that gets published in journals of economics. But there are also extremely capable economists whose understanding of data-mining bias and overfitting is far in advance of yours or mine, and who are extremely careful when evaluating models using historical data.

Nothing fancy - the data is available on Robert Shiller's website[0] and I implemented the rule in Excel.

[0] http://www.econ.yale.edu/~shiller/data.htm

I believe that this is fundamentally good research. But beware if anyone thinks that you can use this to build a killer investment strategy.

If one tests a simple rule, for example "buy when the shiller p/e ratio dips below 15 and sell when it hits 20" then indeed, you would have seen some fantastic results - most obviously in the aftermath of the 2008 financial crisis, when after adjusting for inflation and dividends, you could have bought the S&P500 in Feb 2009 for 887.12 and sold it again in December for 1202.03, realizing a 35% return after inflation).

However, you also would have had the patience and determination to ride out some extremely tough times - you'd be long stocks for the 1987 crash, and completely miss out on the pre-2000 tech bubble, for example). You'd also have to ride out some drawdown that would have wiped out 50% of your capital before the eventual recovery.

In fact, you'd only have placed six trades in about 125 years --

1. Buy in April 1882 and sell in 1898, realizing 2.62% annualized, inflation-adjusted return over 16 years.

2. Buy in March 1907 and sell in 1928, realizing 1.01% annualized and inflation-adjusted over 21 years.

3. Buy in Sept 1931 and sell in 1936, realizing 10.1% annualized over five years (this was a great trade, as you bought in the immediate aftermath of the 1929-1930 crash, and just before the recovery of the early 1930s)

4. Buy in Jan 1937 and sell in 1961, realizing 6.79% annualized over 24 years (another great trade, but you would have had to hold your position over the second world war, which included some terrifying drawdowns).

5. Buy in 1973 and sell in 1992, realizing 2% annualized over 19 years. Again, you'd have to suffer losing over half your capital before you eventually came out on top.

6. Buy in Feb 2009 and sell in December the same year, realizing a 35% return in less than a year (the single greatest trade in the strategy's history, perhaps indicating how thoroughly depressed markets were after the 2008 crash).

This paper is really interesting - thanks. Some of my archaeologist friends have criticised Diamond's books for being too simplistic and using outdated techniques (even when they were written in the 80s and 90s) but I had never really figured out why they were wrong before.

My recollection is that Diamond's argument is slightly different. He argues that the east-west orientation of Eurasia meant that societies were more easily able to expand, because you experience relatively little climate variation moving along parallels than you do moving up and down meridians, which you are forced to do when you expand in a north-south oriented continent like America or Africa.

Similarly Eurasia had a wider variety of indigenous grains suitable for cultivation, and more large mammals suitable for domestication, and these could be taken with a population when the migrated (because of the aforementioned east-west orientation).

These advantages meant that populations were able to increase more rapidly, which necessitated an earlier development of agriculture, which lead to further productivity gains as improving food yields meant that people could spend more time developing more advanced technology.

The increased population densities had two further advantages - firstly, it meant that wars were more frequent, so Eurasians developed more advanced weaponry (the "guns" and "steel" of the title). Secondly, evolutionary pressures selected for resistance to diseases that could spread easily in crowded environments (the "germs").

So it's a little bit more subtle than just "our climate is inherently better for deep philosophical thoughts" or "our climate means that we are more productive workers".

One of his ideas is that the development of agriculture was "the worst mistake in the history of the human race"[0] contra the popular belief that agriculture improved food yields and freed people up to spend time developing science, technology and art -- and could therefore be considered one of the greatest achievements in the history of the human race.

I'm not saying that his argument is wrong, but it definitely runs against the prevailing wisdom. Similarly "Collapse" (the idea that the success or failure of societies is predictable) and "Guns, Germs and Steel" (the advantages of old-world people over new-world people in the 1500s were due to the influence of geography and climate, not inherent superiorities).

Maybe those things are obvious to you, but I don't think they're obvious to most people.

[0] http://discovermagazine.com/1987/may/02-the-worst-mistake-in...

I have heard the genre (which includes Malcolm Gladwell, Jonah Lehrer, the Freakonomics guys and Jared Diamond) described as "insight porn".

The idea is to deliver small chunks of insight that is just counter-intuitive enough to be novel, but not so counter-intuitive that we can't take it at face value. It's a market that's developed in response to the fact that nerds are addicted to feeling like they're just a little bit smarter than everyone else.

I think the same rush we get from learning something counter-intuitive and novel is the same reason that many of us have read pg's essays - particularly the early ones.

Edit: It occurs to me that this comment could be seen as an example of what it describes...

If you're using a logarithmic scale you shouldn't be using a bar chart, for the exact same reason as above - when you see bars, you naturally compare them on length. If the scale is logarithmic, then the form of the graph is in conflict with the data, and the graph becomes confusing. You should strive for every part of the graph to be in harmony and to repeat the same message.

The following paper on lucidity in scientific communication has been a big influence on my thoughts. Briefly, the author stresses that you should aim for lucid pattern repetition over gratuitious variation, i.e. the repetition of the same idea communicated in similar ways, over variety for the sake of appearing to be superficially interesting.

http://www.atm.damtp.cam.ac.uk/mcintyre/papers/LHCE/lucidity...

I hope that you are either (a) a professional investor or (b) buying passive index funds.

That isn't experimental evidence to the contrary at all, because the unpleasant jobs are done by people who don't have access to the government-mandated minimum income!

Your first point isn't too far off the mark. If goods are really 60% more expensive in Switzerland than in the US, then you'd expect people/corporations/institutions in Switzerland to use their CHF to buy dollars, buy goods in the US at a large discount and have them shipped to Switzerland, or for Swiss companies to hire US employees and pay them in USD, thus driving the USD/CHF exchange rate towards purchasing power parity.

Up to transaction costs (inc shipping) and trading restrictions, this really should happen, so it's interesting to explore why it doesn't. Typically the culprits are artificial limitations (e.g. international trade restrictions) or low-value goods where shipping costs are large relative to the cost of the item, or goods which spoil easily and so can't be easily shipped.

If you can't explain the disparity with transaction costs and trade restrictions, then there must be some other reason for CHF being so high relative to the USD. One possibility - CHF is viewed as a hedge for various risks (inflation, market crashes). It is typically anti-correlated with the S&P500, for example, and correlated with market volatility, so it's good to be holding it in a crisis. The same applies to JPY, and the opposite applies to AUD and CAD, which are seen as risk-on currencies.

The approach seems to be along the following lines:

1. Fit a model (in the case Zipf's law) to a semi-related field (the size of non-financial companies).

2. Find that the model doesn't fit reality in the field you're interested in (financial companies).

3. Posit that your model is correct and reality is wrong, and invent over $30 trillion worth of invisible activity to account for the difference.

Alternatively... your model is just wrong?

I have sent plenty of Facebook messages to dead friends and family. I know that no one reads them, but it makes me feel better.

I'm self-taught and a relatively late starter. I've also never studied computer science in a formal setting (i.e. at a university).

I started programming when I was studying for a PhD (in mathematics) around 2006-07. I was 23 at the time. That was mainly in Matlab and Fortran. My code from those 3-4 years is, frankly, pretty horrifying.

When I got my first job in industry (quant at an investment bank in 2010) I had to learn Java, and a language called Q[0], which is a very fast array-processing language with a functional feel and a built-in query language for the KDB database. I also picked up a little bit of R, mainly because I was building a lot of statistical models and it was far easier than doing it in Java.

That piqued my interest in functional programming, and I subsequently learnt Haskell using the book Learn You A Haskell[1]. I picked up Python around the same time when I was trying to write a web scraper, and finding it painful in any of the languages that I currently knew. For that I used Learn Python The Hard Way[2].

When I started a new job in 2011 I started writing Matlab again, causing me to finally learn it properly. Around that time I decided to try and learn a low-level language, so I picked C, mainly because there was an interesting edX course [3] that I wanted to try (the Harvard course CS50).

I'm at a weird in-between stage now. I can write comfortably in half a dozen languages, but there's a whole swathe of programming that is completely alien to me (web design, networking, and any kind of formal software architecture). I've written a lot of heavily numerical and mathematical code, but never written a CRUD app or a web page. I still don't know Javascript.

[0] http://en.wikipedia.org/wiki/Q_(programming_language_from_Kx...

[1] http://learnyouahaskell.com/

[2] http://learnpythonthehardway.org/

[3] https://www.edx.org/

I think what they mean is that the force we call gravity is an illusion that we experience because we perceive spacetime as flat. In a curved spacetime, you don't need to posit a force to explain motion due to a nearby mass - the mass curves spacetime, and objects move along geodesics in the curved spacetime. So while it would be incorrect to say "gravity is an illusion" I think it makes sense to say "the force of gravity is an illusion".

I don't write C++, but if it has linked lists then at least the first one should be trivial. The second one would probably involve function pointers.

Maybe his point was that you need to use a linked list rather than an array to get an efficient solution? Coming from a Scheme/ML/Haskell background I tend to view linked lists as much simpler than arrays!

Nice article. I don't really understand the final point, though. Here's a simple implementation of the collatz function that returns the entire list iterated through, using a simple data structure, and that is O(I) in the number of iterations I

    collatz_vec :: Int -> [Int]
    collatz_vec = reverse . go []
      where
	go acc 1 = acc
	go acc n = go (n:acc) (if even n
		then div n 2
		else 3 * n + 1)

You simply push new values to the front of the list rather than the end, and then reverse the list all in one go at the end. Sure, you could use a fancier data type like an IntMap, but why bother when you already have linear complexity with a simple one?

Edit: In fact, here's an even simpler solution that doesn't need to reverse the final result

    collatz_vec :: Int -> [Int]
    collatz_vec = takeWhile (/= 1) . iterate step
      where
        step 1 = 1
        step n = if even n then div n 2 else 3 * n + 1

It's not tail recursive (because it uses `iterate`) but it's lazy -- so if the elements of the list are consumed as they are created, the whole computation can be done in constant space as well as being fast.

Chasing down the references in that paragraph reveals nothing about the FDA, AMA, DEA or NIDA's opinion on the Anabolic Steroid Control Act (1990). The only citation leads to the THOMAS page for the act[0] which only contains the text of the act, and not any of the discussion surrounding it.

Remember that Wikipedia isn't a primary (or even secondary) source and shouldn't be used as one!

[0] http://thomas.loc.gov/cgi-bin/bdquery/z?d101:H.R.4658:

That article doesn't actually compare India to the US. In fact, it says that "It’s also likely the results aren’t India-specific." It would be interesting to see the results of a similar study carried out in the US or UK, for example.

In any case, there is a difference between the general level of competency of doctors in a given country, and the level of competency available to wealthy foreigners travelling to the country for medical tourism.

The problem with almost every language benchmarking blog post I see is that the author is generally an expert in at most one of the languages they are benchmarking, and so they write slow or non-idiomatic code in the other languages, leading to a useless comparison.

Here's one example that I picked up. The author writes that they had to refactor part of the Haskell code from (paraphrasing)

    if (checkColl tr rsDone || checkBound tr)
      then {- branch 1 -}
      else {- branch 2 -}

into

    if checkBound tr
      then noFit
      else if checkColl tr rsDone
        then noFit
        else {- branch 2 -}
      where
        noFit = {- branch 1 -}

because of "problems with lazy evaluation". But in fact the only problem is that the call to `checkBound` is fast whereas the call to `checkColl` is slow, and the || operator evaluates its left argument before deciding if it needs to evaluate its right argument (just like in C, and every other language I've ever used). So all that is required to get the speedup is to switch the order of the calls:

    if (checkBound tr || checkColl tr rsDone)
      then {- branch 1 -}
      else {- branch 2 -}

I also think it's a bit weird to flip direction of composition, but there are a couple of advantages --

1. It keeps the "imperative" feel, so that you can write

    person.address.zipCode

rather than

    zipCode.address.person

2. There's an argument that mathematical functions are "back to front" and that it would be more natural to write functions left to right, i.e. (x)f rather than f(x), so that the function composition f.g is "apply f then apply g" rather than "apply g then apply f" which is where we are at the moment.

An example of Simpson's Paradox[0] (N.B. not really a paradox).

[0] https://en.wikipedia.org/wiki/Simpson's_paradox

Yes, it's intentional. That's part of the 'joke'.

In fact, Elon Musk has said that the Hyperloop is not a vacuum tunnel[0].

[0] https://twitter.com/elonmusk/status/224406502188916739

This looks really neat. I produce a lot of plots in my day to day work (mainly in MATLAB) so I'm excited to see if there's an opportunity to use Plotly.

Can you point me to some examples of how you've brought Plotly into your workflow? Do you generally send data via the API and style it manually, or do you do all the styling in code? How easy is it to grab plots from Plotly to be included in e.g. a PDF?

I can't resist a quick plug for this beautiful O(n) implementation of the fibonacci sequence in Haskell. One line to define the list `fibs` -- an infinite list of fibonacci numbers -- and one line to define the function `fib` which simply takes the n'th element of the list.

    λ> let fibs  = 0 : 1 : zipWith (+) fibs (tail fibs)
    λ> let fib n = fibs!!n
    λ> fib 100
    354224848179261915075

I disagree with the advice to "focus on the positive feedback". You will already have an unrealistically optimistic view of your app, because it's your baby. Your purpose in showing it to HN (or any other audience) is to get a more accurate picture of both their positive aspects and their shortcomings. If you focus only on positive feedback, you get validation, but you don't find out how to improve.

Research has found that novices disproportionately seek positive feedback, whereas experts seek negative feedback[0]. Would you rather be more like a novice, or more like an expert?

Doyen of entrepreneurs, Elon Musk, also advices entrepreneurial wannabes to seek negative feedback[1]:

"Always seek negative feedback, even though it can be mentally painful. They won't always be right, but I find the single biggest error people make is to ignore constructive, negative feedback."

[0] http://blogs.cuit.columbia.edu/sf2559/files/2011/08/FF_JCR_F...

[1] http://news.investors.com/management-leaders-in-success/0821...

Nice, work! One minor gripe though - every tag I've tried (Java, C, C++, Python, PHP, Haskell, Scala) has had a dropoff in the number of questions asked over the past couple of months. This suggests that there may be some seasonality in the number of questions asked (or at least, a more general dropoff in questions asked over the past couple of months).

I think the results would be more meaningful if they either corrected for the seasonality, or presented the results as relative to the total number of questions asked.

Gwern wrote a lengthy blog post on improving recall[0] that is worth at least a cursory read. As usual, it is deeply researched and well referenced. The two main takeaways seem to be

1. Spaced repetition works, and

2. Intermittent testing works.

I don't think he says anything about summarization, but I'd be interested to read a study.

[0] http://www.gwern.net/Spaced%20repetition