RA_Fisher

Cool for the perspective, but also subsumed by the concept of returns to scale (that entails diminishing returns and includes constant and increasing returns to scale). ie- rediscovery of microeconomics

Sure, this chat describes what I have in mind, https://chatgpt.com/share/673e9290-a044-8005-995b-166efe653e...

For sure. There are ML folks with statistical learning backgrounds, but it tends to be relatively rare. Physics and CS are more common. They tend to view things like you mention, more procedural eg- learning to rank, minimizing distances, less statistical modeling. Humility around statistics is good, but statistical knowledge is still what's required to really level up these systems (I've built them as well).

Hi David. Nice to meet you. Yes, precision and recall are always in tension. However, both can be made simultaneously better with a more informed model. Using your example, this would be a model that encodes the concept of files in the context of a user demand surrounding AWS.

Best is to use one statistical model and encode the underlying aspects of the context that relate to goal outcomes.

Statisticians are famously disliked, especially by engineers (there are open-minded folks, of course! maybe they’d taken some econometrics or statistics, are exceptionally humble, etc). There are some interesting motives and incentives around that. Sometimes I think in part it’s because many people would prefer their existing beliefs be upheld as opposed to challenged, even if they’re not well-supported (and likely to lead to bad decisions and outcomes). Sticking with outdated technology is one example.

I don’t think it’s realistic to think that software engineers can pick up advanced statistical modeling on the job, unless they’re pairing with statisticians. There’s just too much background involved.

For sure, it’s very popular, just not the best anymore (and actually far from it).

I’m sure Search experts would disagree, because it’d be their technology they’d be admitting is inferior to another. BM25 is the workhorse, no doubt— but it’s also not the best anymore. Vectors are a step toward learning models, but only a small mid-range step vs. an explicit model.

Search is a useful approach for computing learning models, but there’s a difference between the computational means and the model. For example, MIPS is a very useful search algo for computing learning models (but first the learning model has to be formulated).

BM25 can be used as a starting point for a statistical learning model and more readily built on. A key advantage is that one gains a systematic way to reduce edge cases, instead of handling a couple, bc they’re so large as to be noticeable.

Google’s come a long way since PageRank + terms. Ancient doesn’t mean bad, but usually it means outdated and that’s the case here. Search algos are subsumed by learning models, our species can do better now.

BM25 is an ancient algo developed in the 1970s. It’s basically a crappy statistical model and statisticians can do far better today. Search is strictly dominated by learning (that yes, can use search as an input). Not many folks realize that yet, and / or are incentivized to keep the old tech going as long as possible, but market pressures will change that.

Well obviously Terence isn’t an economist. People don’t minimize risk, we maximize utility.

It’s cronyism.

Single functions and libraries can easily be imported in R, just like Python. It’s not necessary though, because the R community does a good job avoiding name conflicts (MASS aside).

I think the core issue is that the coordination benefits of having everyone use Python are overestimated, and the benefit of better statistical tools in R and SWE skilling up in statistics is underestimated.

Cursed in your view because it’s not Python?

Learning tech improves on search tech (and makes it obsolete), because search is about distance minimization not integration of information.

I’d vote for Sirious.

Indeed, it's exploitation.

Americans are relatively wealthy and hence choose to spend more on healthcare. That’s not surprising to me.

Of course they will, medical technology improves over time.

It may be that after analyzing the data, we still have substantial uncertainty, just depends on the process’s inherent variability and what the data provides in terms of information (a function of the skill of the person determining what should be collected).

Exactly. Continuing with the lease would be a classic case of sunk cost fallacy.

Sites are always getting new visitors, losing old ones and the ones they’ve observed return irregularly (or commonly, or somewhere in between). So it’s not realistic to assume a given sample of visitors is the population.

So sorry to read this. I'm thankful for the positive example she set for all of us. Gone but not forgotten.

I think you're in good company there. Kudos on seizing the opportunity. I built a tool that can help with this. It's free to try and I'd love to learn of any feedback you have: https://modelgymai.com/

Exactly! With even just 3 chained calls, 98% individual chance of failure ^ 3 calls ~ 94.1% joint probability of success or 5.9% failure global rate. That's a failure rate of 1 out of 17.

Problems like those and my experience as a reliability statistician are why I built ModelGym: https://modelgymai.com/

I want to help LLM app developers deliver on their visions and accelerate the realization of AI's potential.

It's free to try. If you have any feedback, I'd love to learn of it.

A multi-year contract could mitigate that risk.

I’m curious, how do y’all keep track of performance and reliability?

I ask, because I think it’s going to be a big challenge, so I built a service to record feedback / acceptance data: https://modelgymai.com/

If you think it can help, I’d love if you’d try it out and let me know if it helps.

Who has higher inflation?