The Only Way to Know If GPT-4 Got Dumber | Hacker News Reader

The Only Way to Know If GPT-4 Got Dumber

latent.space

16 pointsswyx16 comments

LLMOps / PromptOps, seems to be a new monitoring tool segment which is heating up. And pretty much essential as AI models are constantly being updated "in production" these days.

Seeing that you interviewed LangChain, which just launched LangSmith back to back?

How do you think this space will turn out - cause I find it very hard to tell how the two of them differ.

Additionally, what happens when incumbent monitoring tools start entering the scene (DataDog, Elastic, etc) ?

thanks! i address that a bit in the essay - basically langchain will try to build a verticalized solution for langchain, humanloop will be more general but for llms, datadog will be offering the same thing but for existing datadog customers. i think all three groups can win for a long time but when they start competing it will come down to customer personal preference and ability to sell and ship compelling features

makes sense, considering there lacks an open std to port back and forth.

people stick to datadog / elastic / etc passionately, to avoid "redoing their charts"

also low hanging fruit to evals prompt and check (does it JSON or not), is significantly more useful then folks give credit for Q_Q

i would love to help define an open standard with the AIEF...

Hi HN! proud to present my latest interview with Raza Habib of Humanloop (YCS20). he went viral on the YC youtube a few months ago (https://www.youtube.com/watch?v=hQC5O3WTmuo) and in general has been the first and best known LLMOps/PromptOps/Foundation model ops platform out there (having bet on NLP before GPT3 and then ChatGPT blew up).

I caught up with him in london and covered the origin story of how they pivoted from an automated labeling product into Humanloop today, and what they are seeing in both the AI Engineering and UK/Europe tech scene vs SF. They also launched their new pricing (free tier) and evals product (which a lot of other startups are trying to build) so we got into that a bit and I chose to highlight the "GPT4 nerfing" debate as that has been a topic of interest here before (https://news.ycombinator.com/item?id=36134249).

happy to take qtns!

> They also launched

Couldn't not mention that, could ya

why is that bad? it literally is what they just did lol

> please use the original title, unless it is misleading or linkbait; don't editorialize.

Original title is:

"Building the Foundation Model Ops Platform — with Raza Habib of Humanloop"

The anchored section linked by the OP has the same title.

Oh, it's an ad.

this is very cynical and hurts everything i am trying to do with this podcast. humanloop did not pay for this, i invited them on because they are one of the top companies in their space, and thought it might be of interest to HN because of their YC connection + perspective from working extremely closely with OpenAI (many might remember https://news.ycombinator.com/item?id=36177895 which we chose not to discuss on the pod but I asked him literally everything else). it hurts when people dismiss my work so casually as it was done in complete good faith and genuine interest in how founders are building in the new AI world.

It has an editorialised title, it doesn’t explain what this is really about. It’s basically click bait.

It’s either an ad for human loop or an ad for your podcast.

It’s not meant to be hurtful, it’s just the truth. If the title was instead more descriptive of what the content was. You wouldn’t get the cynicism.

as mentioned by @htk above - it links to the exact section of the essay, with that exact title, i wanted to offer for HN interest because it directly addresses active conversation topics that have been here before, which i know from myself being a longtime community member.

I see, so anchor tagging a clickbait piece of text within an article makes it not clickbait?

Sorry, while I don’t like hurt feelings and I feel bad if that is the case. And I believe you are being genuine. I think it’s still something a bit off here.

The argument wasn’t over it being clickbait. It was claimed that this is an advertisement (it being paid is implied).

The NY Times uses blatant clickbait, but I still respect them as real journalism and not just rewritten PR ad pieces producers.

Sorry I don’t draw a distinction here. Clickbaiting is an advertising technique. This was advertising the author/affiliate. The main point isn’t wasn’t about the content (for me).

Agree I guess there is a borderline where the title is both compelling to click, but still informative of what the overall article is. That wasn’t the case here. NYT does this better, agree.

Reply on news.ycombinator.com