Comment by yihlamur | Hacker News Reader

Comment by yihlamur | Hacker News Reader

Parent Full thread

Today we're excited to announce our provisional patent application and paper on arXiv: GPTree

GPTree outperforms the randomness by ~10x and the world's best experts by ~3x.

The cool part is that:

1. It's explainable, not a blackbox. A decision tree that humans can understand.

2. Human-machine harmony wins over machine-only and human-only decision making processes.

3. It's potentially applicable and high performant to any decision making use-case.

We fine-tuned the model for our own use-case at Vela Partners, picking outlier startups at their inception stage.

The reason why we love this research problem is that humans are so bad at picking startups at their inception stage.

For context, only 2% of the US-based investor-backed startups become an outlier return at the inception stage. Y Combinator and tier-1 VCs hovers around ~3% and ~6%, respectively. Ten-fold cross-validated GPTree is at 8% and most fined-tuned version is at ~18%.

Please take a moment to take a deep breath and let that sync in...

GPTree can find 1 outlier startup out of 5 of its investments at the inception stage. This may translate into a 10x+ return fund for whoever uses it if the future behaves as it forecasts.

Excited to hear the HN's feedback.

curious how the human intervention worked?

is there a risk of bias? when VCs make decisions, there is no hindsight. but when you ask humans to evaluate predictions, you necessarily are using past (training) data and the humans may recognize general successful patterns or even the examples themselves.

For example: the model outputs that it's considering an investment in a company that lets drivers pick up passengers on the way to their destination and earn some money as well. a person may think, "Duh, this is uber! invest!" Thus inflating "success" rate.

Great question.

We built a dataset of past success and fails with training, validation and test sets. Afterwards, the model got trained with some context from human experts at the beginning such as "Being a repeat entrepreneur is positive". Then, the model built out a explainable decision tree without human intervention. Lastly, a human looked into the models that machine came up with, and improve its logic further. For instance, the model might be asking a vague question such as "Is the entrepreneur based in an innovation hub?". The expert may prompt it to be "Hey, be specific and put specific cities or regions to improve this question".

Then the model would re-run and try to improve that question with a goal to increase the precision.

So this goes on and on with expert-in-the-loop.

Sometimes, expert may give wrong advice! :) And, in that case, the performance would decrease...

Lastly, your example is a great one. Human may introduce some bias into the process. For that reason, we also built a model with no human intervention with 10x cross validation. And, the model still outperforms all humans...

Given that expert-in-the-loop is a time intensive and expensive process, we did not do 10x fold validation for that. However, our initial observations indicate that the magic happens when humans and experts work together.

What about return on investment? You can pick more startups, but your avg return on investment might be lower.

We calculated what would happen if we were to run this with its precision and recall metrics.

The simulated fund with GPTree makes a 10x return vs typical VC funds return 3x, over 10 years.

Reply on news.ycombinator.com