Hacker News Reader

Something I'm starting to realize with all these AI APIs is how difficult it is to evaluate and compare the results.

How do you know their algorithms are better than the competition? The marketing buzzwords are all there. "Powerful algorithms", "never seen anything comparable" etc.

The worst part is the algorithms might work well with certain data sets but fail miserably with others.

View on HN

If you had an internal API as opposed to using an external one you'd have some sort of A/B or multi-armed testing in place to check if the model you've deployed is performing as expected. At this point you don't think about the theoretical best you could perform at for the said model without computing some sort of complexity measure. I assumed one would have to use a similar automated process for external APIs. I don't see why they have to be different per se.

Reply on news.ycombinator.com