buckie

This is not at all surprising. Statistics isn't the right tool for financial analysis... it's just the best tool that we have. Incredibly unlikely statistical events happen all the time in finance.

A great read on an alternative view on the topic of using statistics for finance is The (Mis)Behavior of Markets by Benoit Mandlebrot [1]. It's very well written, basic enough for most to comprehend and first book on finance I read in college (before I went on to major in finance + math).

The aforementioned book has some very interesting notions with Trading Time being my favorite. Basically one can near-perfectly "forge" financial data with fractal objects called "financial cartoons" [2]. The objects are composed to two distinct fractals - one for price vs trading time and another for trading time vs clock time [3]. The latter rescales the volatility seen former, either compressing or expanding it. Rescaling volatility isn't a new idea, but it was a parallel "discovery".

There has been some work on figuring out how to use Fractal Geometry to analyze financial time series data but it's still in its infancy. The problem is figuring out how to transform the data into the fractals domain + figuring out what the results from a fractal based analysis would mean for forecasting future events. I've been working on these problems for many years (in earnest in college and as a hobby thereafter) but made little true progress.

[1] http://www.amazon.com/The-Misbehavior-Markets-Financial-Turb...

[2] http://classes.yale.edu/fractals/randfrac/Market/Fake/Fake.h...

[3] http://classes.yale.edu/fractals/randfrac/Market/TradingTime...

This is all really great to hear, I've already begun the process of restarting the engine. It will likely take me 3-6 months just to get the momentum needed to reach out again officially. Oh the gov't...

> I'd love your input on how we can do better.

As I'm not in procurement and don't want to make assumptions about how your shop is run I don't have any comments of merit/meaning beyond my previous ones & saying an honest thanks for your reply and I hope that I can get my guys to reach out to you sometime next year.

Over a year I started the process of convincing the people I needed to at my job (Federal, Non Defense/Law Enforcement) to get an instance of Github Enterprise installed. Though it took a lot of time and effort, I managed to get both the approval and funds needed for a small scale - 10 user - install (on my 3rd try, gov't is really against change).

Our current "enterprise" VCS is PVCS (so old I had to look it up). The idea was to get github installed on a small scale, gain support and extend it to the entire agency over the course of a couple years. Everyone who deals with PVCS wants git, but the management don't care to put in the effort to get it initially, so gaining the support would be easy.

A couple months ago, the entire thing collapsed on my face and we went with gitlab instead. Why? Because github's sales team f*ed it up. Apparently, the $5k minimum order was too small for them to care (at least that's what my side says). Too bad, I really wanted to get them in the door.

I'm sure it's just a bad egg in the sales, but seriously there are some people trying to fix/modernize gov't tech from the inside and thing's like this don't help. I guess my only recommendation is that if your company usually sells to private and the gov't asks for a small-sized purchase please be flexible. The guy who caused this request to happen has likely been putting in a huge amount of effort and has larger plans for integrating your service over time. The person who is contacting you is not that guy, has no idea what your product (or any tech) is/why it matters and is usually doing the originator a favor.

For anyone curious, here's a comparison (that I wrote a couple days ago) of FizzBuzz in K vs. q vs. python... please note the insane syntax for K4 (which is a proprietary language of KX Systems and is thus undocumented).

K

    \c 150 100 
    c:{$[0=x-5*x div 5;$[0=x-3*x div 3;"CracklePop";"Pop"];$[0=x-3*x div 3;"Crackle";x]]}'1_!101 
    r:{x@&~x in "\""} 
    p:{-1 r .Q.s x}
    p c 
    \\

q (the "readable" wrapper around K4")

    \c 150 100
    c:{$[0=x mod 5;$[0=x mod 3;"CracklePop";"Pop"];$[0=x mod 3;"Crackle";x]]} each 1_til 101 -1 ssr[.Q.s c;"\"";""] 
    \\

Here’s a python (2.7) version of the program written in as unreadable as I could muster

    for i in map(lambda x: "CracklePop" if not x % 3 and not x % 5 else "Crackle" if not x % 3 else "Pop" if not x % 5 else x, xrange(1,101)): print i

Here it is in normal python (2.7)

    for x in xrange(1,101):
        if not x % 5: 
            if not x % 3: 
                print "CracklePop" 
            else: 
                print "Pop" 
        elif not x % 3: 
            print "Crackle" 
        else: 
            print x

Something to add to your 'bag-of-tricks' -- this was once handed down to me by a master by the name of .ike, as it was handed down to him before. Maybe you'd be interested:

"The 3-finger Claw Technique" The sweetest 3 functions, ever.

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "Cannot $*"; }

and always, exit 0

I use these daily, they're extremely cross-platform friendly in any bourne-derived shell I've used in the last 4 years.

  safexample.sh :
  --
  #!/bin/sh

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "cannot $*"; }

  # consider the following lines
  safe cd /some/dir
  safe tar xzvfp /my/big/tarball.tbz

  exit 0
  --

In the above example, using 'safe' suddenly gives the user the following special powers:

1. if `/some/dir` does not exist, the script will safely exit before that un-tarring does any damage.

2. the actual 'directory does not exist' will return to stderr for the script, (as opposed to just telling us the useless fact that the enclosing script failed). Now bourne shell starts behaving like a modern language! (ala Python/Ruby tracebacks, Perl errors, etc…)

Additionally, if you 'exit 0' at the end, you can run non-safe operations, and always be guaranteed that if the shell exits, it was 'complete'. An example:

  # consider the following lines
  safe cd /some/dir
  tar xzvfp /my/big/tarball.tbz

Now, safe was removed from the un-tar command, right? So, imaging tarballs created by people using a Mac (with HFS+ and some filesystem-specific sticky bit somewhere in the filesystem which was tar'd up). Now, you un-tar it on some BSD or other NIX box, and tar complains as it goes- and exits non-zero. One way to handle this, is to consider the tar 'reliable', and not use safe for it.

Now, if we continue to be conscious of this simple safe/notsafe distinction with these scripts, they can be called by other safe scripts- and behave just like any respectable UNIX program!

Now, consider this crontab entry:

  1       3       *       *       *       someuser flock -k /tmp/safexample.lock /path/to/safexample.sh

If it fails, (for bad perms, no tarball to unpack, etc…) cron will actually have a reasonable message to email/log on failure!

Thanks to the noble efforts of the Clan of the White Lotus, these 3 functions originate during the late thirteenth century. The original author, William Baxter, explained that these three functions took 15 years to boil down to what they are- I believe that after mercilessly abusing them for several years myself…

Below is another example with more bits.

  #!/bin/sh

  shout() { echo "$0: $*" >&2; }
  barf() { shout "$*"; exit 111; }
  safe() { "$@" || barf "cannot $*"; }

  for i in "before $@ after"
  do
   echo "arg <$i>"
  done

  for i in "before $* after"
  do
   echo "arg <$i>"
  done

  safe echo "this is ok"
  safe echo "this is ok, too" || safe echo "so is this"
  safe bad_echo "this is bad"

  exit 0

I've been working to get Haskell approved at my place of employment for 1.5 years but getting the US Gov't to change is rather hard; I've had to sit the "tech" people down and explain that javascript != java... with that baseline, explaining 'Why Haskell' is non-trivial. Come September, after 1.5 years of effort, I should have 'all the FOSS'. Until then I have to wait.

It's worth noting that iPython Notebook + pandas vs. cassava + conduits (even with iHaskell Notebook) serve very different ends. If I need to explore how to do something, I'd use Haskell. But I'm still in phase 2 (phase 1: collect underpants) and I've yet to find anything as powerful and flexible as the iPython Notebook + pandas + hdf5 stack that also just works. I can just move faster with that stack than anything else I've ever seen. That being said, I'm knowingly deferring bugs to the runtime -- 'tis the cost of python.

If you're unfamiliar with pandas, the "quick vignette" here[0] is decent enough. The reason pandas is awesome, IMO, isn't actually because pandas is awesome (which it is) but because it's embedded in a full language. Julia, R, etc... can do the same stuff (maybe faster), but I wouldn't also want to program, say, a production web-app in them (though I have high hopes for Julia).

Fuzzy parsing on dates: pandas by default uses dateutil[1] which is both awesome and slow.

10GB CSV... : yeah... it's "fast for python" but pandas is admittedly doing a lot in that time, namely putting it into a data structure that is very friendly to time series analysis.

[0] http://pandas.pydata.org/ [1] https://labix.org/python-dateutil

I tried out OpenRefine previously on the large CSV's I have to deal with at work largely on the recommendation of HN comments. On small stuff to medium size stuff, it's pretty nice. But for larger sets (+1GB) it starts to slow down and eventually will fail to commit changes when the set is big enough.

The only answer I've found to consistently work quickly, with the ability to explore the data, is the killer combo of iPython Notebook + pandas' read_csv[0] + a lot of RAM -- 10GB CSV on disk becomes ~20GB in memory (don't know why yet). When I say quick, I mean 10GB CSV un-cached disk to RAM in <5Min including fuzzy parsing on dates.

The nice part is, when you have things figured out, you can enable a chunked reading to get back in-core on machines of lesser specs. Further, you can dump the pandas DataFrame to HDF, thereafter having ludicrous-speed IO & 'where' queries.

Still though, OpenRefine is much more turn key and feature rich.

[0]http://pandas.pydata.org/pandas-docs/version/0.13.1/generate...