lpage

For those curious about the future of LIGO...

At present there are two LIGO facilities - one in Hanford, Washington and another in Livingston, Louisiana. This is necessary for both denoising (it's unlikely that a seismic event or random perturbation would effect both simultaneously) and triangulation via parallax.

Right now having just two facilities (that are relatively close together) limits localization to broad regions of the sky. Additional facilities are under way/in discussion for Europe, Japan, and India. This would significantly improve both the sensitivity of the array and its ability to localize events in a smaller region of the sky. Hopefully these projects get funded. LIGO stands to resolve some of the biggest open questions we have in cosmology.

> Such data is serially correlated (it's not good old iid) so already it looks like you have poisoned the test set with information from the training set.

Kind of. If it was that simple making money off of an autoregressive model would be trivial -> everyone would do it -> serial correlation would disappear.

I agree with your observation that figuring out what to feed the beast is one of the bigger challenges though. Case and point: train a mean reversion model on the last seven years of S&P data to buy dips and train a momentum model to buy higher highs. That equity curve would look very encouraging. Do it on a fifteen year basis, and not so much. Now the question becomes: how long of a lookback do you use when training your models? Chopping up data at random will mux out useful correlations. Subsetting into periods leads to poorly generalized models. Not fun.

> Knowing when you don't know the answer is something that has been lacking

Definitely, and I'm not sure why that space isn't getting more attention. For RL there's KWIK (knows what it knows) http://www.research.rutgers.edu/~lihong/pub/Li08Knows.pdf [pdf]. A large class of problems (the robo TA being one of them) lend themselves very well to AI with error bounds, for lack of a better term.

For anyone interested, the class, which covers interactive intelligence and knowledge based AI, is freely available at https://www.udacity.com/course/knowledge-based-ai-cognitive-... . It's a great starting point if you're looking to automate the sort of human intelligence tasks that don't lend themselves well to the traditional searching/planning/proving/minimax/retrieval route that underpins most AI.

I took it before they rolled out the chat bot TA, unfortunately.

Yes, but pruning (not considering everything) is as old as game tree search. Previous Go AIs used MCTS as well. What's new in AlphaGo is a more sophisticated approach to scoring game boards - policy networks that help the AI prune even more aggressively, and a value network that's used to "guess" the winner in lieu of searching to endgame. Note that guessing the winner is just a special case of an evaluation function. For any game, if you could consistently search to the end, your evaluation function is always a -1/1 corresponding to lose/win. AlphaGo is still using MCTS - just a more sophisticated form.

> he will later point out to as achieving very good results some 20 moves later

This. It's a fairly common feature of any AI that uses some form of tree search/minimax, and the effect is very pronounced in chess. Even the best human players can only think 6-8 plies into the feature versus ~18 for a computer. What we can (could?) do is apply smarter evaluation functions to the board states resulting from candidate plays and stop considering moves that look problematic earlier in the search (game tree pruning). AI tends to use very simple evaluation functions that can be computed quickly. They do so given that 1) it allows for deeper search, and a weak heuristic evaluated far in the future often beats a strong one evaluated a few plies prior and 2) for some games (like Go) it's really hard to codify the "intuitions" that human players speak of.

Because search based AI considers board states __very__ far in the future, the results are often completely counterintuitive in a game with an established theory of play. Those theories are born of humans, for humans.

The introduction of MCTS some years back was the first leap towards a human level Go AI (incidentally, MCTS is more human-like than exhaustive tree search in that it prunes aggressively by making early judgement calls as to what merits further consideration). AlphaGo's use of deep policy and evaluation networks to score the board is very cool, and the next step in that journey. What's interesting to me is that, unlike chess AI, AlphaGo might actually advance the human theory of Go. It's possible that these "strange moves" will lead to some very interesting insights if DeepMind traces them through the eval and policy networks and manages to back out a more general theory of play.

It's definitely possible but we're not updating real time at the moment...that will change soon. We can add endpoints to return the sentiment on a single story/collection of comments. Shoot me an e-mail (link in profile) and let me know what you need - hopefully we can accommodate.

Sorry about that. Design isn't my wheelhouse but I updated it to what google tells me is a colorblind friendly palette. I would definitely appreciate it if you could take a look and let me know how it is.

> Most of the comments seem neutral to negative to me?

Shameless plug, thanks to HackerMoods [1] I can quantify that statement: 0.85 neutral, 0.08 positive, 0.07 negative. The average Show HN is 0.17 positive and 0.04 negative, so your assessment is in line with the numbers.

[1]: https://news.ycombinator.com/item?id=11188633

We would love to add other datasets in the future and have plans to do so. That said, this is just a fun side project so we have to triage what data sources go in and when. NYT is high on that list.

Thanks for the suggestion, we very well might do so. The line does add something for the cases where a topic is becoming less neutral (stronger language is being used, either positive or negative) but it's unfortunate that the scales are quite different on the whole. We're going to work on making the system more intuitive and find better ways to highlight the most interesting trends.

We can definitely add some copy around that. The scores that we're using are normalized on [0, 1] and broken into three categories - positive, negative, and neutral. It's possible for something to be all positive or all negative, but most things are a mix. The relative differences between positive and negative sentiment are the most interesting. For example, if you search Obama, you'll note that in September '13 the average positive and negative sentiment trends cross. On average, sentiment expressed in comments on stories mentioning Obama were more negative than positive. They later return to being more positive than negative, but this is the sort of thing that's interesting to investigate. In this particular case there were a string of very negative stories involving the NSA [1], the assassination of U.S. citizens [2] and Obamacare [3].

In the case of React you'll note an overall trend of increasingly positive sentiment and less negative sentiment. Sometimes it's fun to figure out what the spikes were by searching on HN. The big positive spike on 20015-07-15 was [4], a Show HN with very positive reception.

[1]: https://news.ycombinator.com/item?id=6347790

[2]: https://news.ycombinator.com/item?id=6435963

[3]: https://news.ycombinator.com/item?id=6462339

[4]: https://news.ycombinator.com/item?id=9891989

edit: formatting

Thanks for pointing that out - should be fixed!

This was very much in line with expectations, ergo the muted reaction in the markets. It's worth noting that the Fed used gradual in lieu of measured to describe the increase. Measured implies a steady series of increases (announced every few meetings until the target rate is reached) versus a gradual approach in which there's a long term number in mind but no strict mandate on getting there - a dovish tone.

"Legal at 35=G Inc will allow me to publish this article only if this section is not present."

I love the FIX meta humor. Order Cancel Replace Request: tag 35=G.

I'm wondering about that as well. The closely related covariant entropy bound is predicated on both the holographic principle and our current understanding of quantum gravity, so I'm not sure if this finding calls that into question or not.

There was a good HN discussion on the theoretical limits of information density a few years ago: https://news.ycombinator.com/item?id=6466430

Maybe I'm missing something but this seems like a pretty perfect application for the bootstrap - a remarkably intuitive but powerful framework. Without loss of generality, imagine that you have two populations, A and B, and that you want to test some hypothesis about a statistic of A being different from a statistic of B (mean, in this case). Using the simplest form of the bootstrap you would do the following:

1. Pool and randomly label the data from A and B 2. Sample with replacement and form two partitions of the same cardinality as the original A and B groups 3. Compute the differences in mean 4. Rinse and repeat millions of times to form a distribution of mean differences 5. Check if the observed difference in means (from the true A/B labels) is statistically significant relative to the distribution found in (4)

This has some problems with fat tailed distributions but tends to work great otherwise. It's so simple that it avoids a host of pitfalls that can arise with other resampling schemes (what's being proposed is a type of resampling), and I love that it makes basically zero assumptions on the underlying data.

Many aspects of CV and signal processing in general produce almost magical results. It's a little less magical when you think back to stats examples of fitting a curve to some signal + (known) noise process. For linear models, the noise term can get really bad before simple regression won't persevere. Least squares in one form or another finds lots of use in image processing.

Images are just functions, and almost all of the mathematical techniques that you would use to analyze one domain hold for the other. Certain perturbations such as camera jitter are easier to deal with as "undoing" them is tantamount to assuming some underlying regularity/structure on the signal and filtering accordingly. Others, such as removing an occlusion, are harder. Humans do it well thanks to our power of inference, learned from the litany of visual examples that we take in over the course of a lifetime. It's not trivial getting an algorithm to visualize what should have been behind the person who photobombed your insta, but we do it somewhat naturally.

For occlusions and finding relationships between partially overlapping scenes, really interesting things are happening with deep learning. For noisy images, techniques continue to improve. Compressed sensing and signal recovery is an active area of research that's already paid huge dividends in many fields, especially medical imaging. I can't wait to see what becomes possible in the next five years. And, as has been noted - this article is dated. There are already more powerful techniques using deep learning for image super resolution and deblurring.

There's a highly tuned set of libraries available for computing HDR histograms with minimal overhead: https://github.com/HdrHistogram. This gets you a little closer to a continuous distribution, and does away with the branching needed for explicit buckets.

The race to zero (or more accurately c) on the networking front is seeing significant action, largely because it's the "lowest hanging fruit." Network latency is six to eight orders of magnitude higher than processing latency in most case. By Amdahl's Law, it's a very good system to optimize. This is especially true for places like Korea, and for the US where there's significant geographic distance between futures and equities.

Microwave is an "early adopter" technology that will soon be replaced with superior ones. The first microwave network was slated to cost 15M/yr at the time of its launch in 2012. However, competition dropped that number to around 2M by launch time, and within six months it was < 250K. Mind you, everyone was locked into a 1-2yr contract at the higher price points. For the U.S. Tradeworx (aka Thesys) is one of the fastest "publicly" available networks so they're commanding a premium at $250K, but microwave is now available for less than 80K a mere two years after launch. Same story for Spread - seven figures and a two year commitment at the time of launch, down to 15-60K.

On days with snow, rain, or wind kicking up pollen/dust, packets are dropped at comical rates. Dropping a packet is more expensive latency wise than sending it over Spread, and if the weather is bad enough, you're not getting anything through at all, which leads to dual Spread/microwave deployments. Spread knows this which is why the introduction of microwave didn't dramatically drop the price of Spread.

Bandwidth wise you used to get about 1.5Mbps over a shared link. That number is better now, but not substantially. Even with no loss there's barely enough bandwidth to carry L1 price updates on a single active market data symbol, so market data on the whole is a no go. From what I can tell most people have gone the route of transmitting many types of data over both fiber and microwave while leaving the responsibility of deduping to the receiver.

There's not much coverage on it but we'll see some interesting things on the fiber front soon. Photonic-crystal fiber is getting cheaper and we'll continue to see interesting advances on the hollow-core photonic-bandgap front in general [1]. These technologies can theoretically get very close to the speed of light (.997c) with extremely high bandwidth and low attenuation (repeaters in either fiber or microwave networks add significant latency) at which point the race to zero will stair step down again.

http://www.nature.com/nphoton/journal/v7/n4/full/nphoton.201...

This is a very simple and very pathological example that's easily ferreted out with a few more summary statistics (median, min, max) but it's a good illustration of the blind application of statistics. Short of visualization, non-parametric statistics really help with such things. Correlation is a fragile, linear measure, and things that are obviously correlated by inspection can easily appear mathematically uncorrelated -- points on a unit circle, for example. Likewise, the mean of any skewed distribution tells you very little, but that's the statistic that's always cited. Quantiles, medians, and non-parametric measures of correlation such as rank correlation are simple and often overlooked. They do a good job screening for pathological data sets like Anscombe's quartet and real world ones.

It's also worth mentioning "dumbbell" data sets. Two clusters of data, each of which have a independent, meaningful correlation in them, can easily leverage a linear regression into a meaningless line passing through the two clusters. That's a pretty common issue with high dimensional data (obviously you can see it in a 2D scatter plot), and it's not easily caught short of looking at regression diagnostic statistics.

Bertrand's Paradox addresses a separate issue but illustrates how subtle issues like this can be http://en.wikipedia.org/wiki/Bertrand_paradox_(probability) . The geometric interpretation that the OP offered in which he noted that the map had to be area preserving is a nice way of visualizing why the given implementation doesn't hold but that's just the start. As for the implementation this is also a great example of when something is hard to do in one coordinate system but trivial in another. Polar coordinates are the way to go.

"A great way to get into such a fund is to apply as a software developer, with aspirations of becoming a portfolio manager"

That's not a path well traveled, especially at banks. At good hedge funds roles are well delineated, and chances are that you'll be working hard enough at your day job that you'll have little chance to explore on your own. Policy may prevent you access to the interesting data sets (execution data and historical P&L is sensitive to reverse engineering, and market data is often subject to expensive licensing). For banks the issue is far more bureaucratic - different teams with different hiring budgets and lines of business yada yada. There are relatively few managers high enough to wall cross between back office (technology) and front office (trading) so there's a stochastic component to your success at moving internally based on what group you end up in and how well connected your MD is. It doesn't help that coders are usually the limiting reagent (and paid less than traders) so the firm's interests aren't necessarily aligned with your own.

If you're passionate about trading and coding, HFT is a great way to go. Money makers come from all backgrounds (grad/undergrad/super technical/creative coder) and you'll likely have better control over your coding/research ratio.

That was definitely the end result. For my inputs the expected survival time was about 5.5 games. For a pool with 100 players, taking the conservative assumption that every match is a coin flip, you would still expect a few people to outlast you. Even if the payout was high enough that your bet was +EV, your variance is huge, so this was definitely more for fun than for profit :)