Hacker News Reader

Comment by drabbiticus | Hacker News Reader

I really wish people would define terms when using math. What is y? What is LLM(x)? Presumably it evaluates to some real number so that it can be fed to the logistic sigmoid function. If it is the logistic function, then why does beta going to infinity matter? It seems to just collapse the output of the sigmoid function to 1 and make the value of LLM(x) meaningless instead of their claim that it recovers the LLM classifier. What is the function I()?

Maybe these are well understood terms in some field? Maybe I'm just lost?

View on HN

x and y are common shorthand for inputs and outputs respectively. Saying "y=LLM(x)" denotes that the input tweets (x) are fed to a black-box function (LLM) that returns a classification (y) for that input. Given that we're talking about binary classifiation it's expected for y to be a value between 0 and 1, which leads to defining LLM(x) as "a function that receives a tweet x as input, uses an LLM, and returns a value in the [0, 1] range representing its classification as ironic or not".

The point about beta (which I think the equation doesn't actually reflect) is just to indicate that this approach risks nothing because, worst case scenario, the weights you assign to the model can simply return the original LLM prediction. Keep in mind that 0*inf=0 and that LLM(x) only returns 0 or 1, so it's not accurate to say that the sigmoid would always return 1.

The function I() is the indicator function [1] which, in this case, returns 1 if the argument is true and 0 otherwise. It's only there to convert booleans to integers because summing booleans is not defined.

[1] https://en.wikipedia.org/wiki/Indicator_function

Thanks, a lot of this helps.

Having said that, if

    - y is a binary "ironic"(1), "not ironic"(0) label and
    - x is the content of a tweet and
    - LLM(x) is meant to represent some probability range [0,1] representing p(y=1|x)

then it seems that p(y=1|x) === LLM(x) already.

Why bother with p(y=1|x) = σ( α + β*LLM(x) ) as in the article? You are right that I missed the case where LLM(x) = 0, but that just means that p(y=1|x) has exactly 2 values when defined as above and β -> inf.

    - when LLM(x) = 0 then p(y=1|x) = 1/(1+e^-α)
    - when LLM(x) = (0,1] then p(y=1|x) = 1/(1+e^-inf) = 1

This is substantially lossy and converts a continuous output LLM(x) of range [0,1] to a step function not even defined as a set {0,1} but instead the set {1/(1+e^-α),1} for unclear gain. It also loses all sensitivity to LLM(x) near 0, and only detects/changes value when LLM(x)=0 exactly. It might make more sense if LLM(x) is not limited to [0,1] like you have claimed because then at least the logistic regression is clamping the output to [0,1]. If you were wrong about the range of LLM(x), this would come back to asking authors to actually define their terms.

My point is that when math is used to justify something or communicate something, it should be explained or very apparently right. If when someone goes to try to understand the math it doesn't match the claims being made in the article ("recovering the LLM") then it throws the rest of the article into doubt.

Reply on news.ycombinator.com