mbowcut2

oh, I think the horror element is there.

Absolutely horrifying. Well done.

This was my first thought too. AFAIK each layer encodes different information, and it's not clear that the last layer would be able to communicate well with the first layer without substantial retraining.

Like in a CNN for instance, if you fed later representations back in to the first kernels they wouldn't be able to find anything meaningful because it's not the image anymore, it's some latent representation of the image that the early kernels aren't trained on.

Brand new account + strange overreaction = maybe bot?

This is my main point. Seems like gripes about the quality of published research should be directed toward the publisher.

I understand the complaints, but I don't think the Freakonomics podcast should necessarily be expected to meet the high standard of peer review. Is it really that bad for Levitt to take published research at face value, trusting that gatekeepers upriver have done their due diligence?

Making things easier to google is IMO the least-impressive use of LLMs. I'm still waiting for a Roomba I can have a conversation with.

Numberphile did a video on this a while back. https://youtu.be/mceaM2_zQd8?si=0xcOAoF-Bn1Z8nrO

I guess we need to find a way to incentivize good practice rather than interesting results? Turns out that science is so hard that people cheat.

The biggest difference is when you feed a sequence into a decoder only model, it will only attend to previous tokens when computing hidden states for the current token. So the hidden states for the nth token is only based on tokens <n. This is where you hear the talk about "causal masking", as the attention matrix is masked to achieve this restriction. Encoder architectures on the other hand allow for each position in the sequence to attend to every other position in the sequence.

Encoder architectures have been used for semantic analysis, and feature extraction of sequences, and encoder only for generation (i.e. next token prediction).

THIS. People don't realize the importance of Mamba competing on par with transformers.

Touché. I put the first paragraph in GPTZero, and it says 100% AI generated. Unfortunately, AI detectors are notoriously bad, giving all kinds of false positives, which goes to my initial point that it is largely a gut feeling. But if you look at other articles posted on the site by the same author they also come up as 100% AI generated on GPTZero, and they also fail the gut test.

I’m similar. Never been very over weight but I’ve put on 20ish lbs in the last few years. My wife and I started tracking calories and exercising a month ago and it has been so beneficial. I lost about ten lbs, but even better than losing the weight has been building a better intuition of what foods are calorie rich, higher in protein, etc.

oh I just realized that 'isNew' in the response refers to a global set, not the user set. So, I guess it's doing exactly what I said lol.

This is fun. It would be interesting to build a single graph of concepts that all users contribute to. Then you wouldn't have to run LLM inference on every request, just the novel ones, plus you could publish the complete graph which would be something like an embedding space.

This makes sense. Add in that he was featured on the JRE (which is where I first learned about him) which no doubt gave him a nice audience boost.

I've been long time listener of his show, so I don't necessarily disagree with your points. It just seems like he punches above his weight when it comes to guests.

This is perfect. We need more hobbyist apps. Minimalist design, simple and solid functionality, and no ads in sight. Shake off the shackles of the advertising dystopia!

Next they'll announce GPT-4 is the new CEO.

But really, how accurate are your results if they depend on strong assumptions about your data?

For what it’s worth, my undergraduate was in Economics with an emphasis in econometrics and this article touched on probably 80% of the curriculum.

The only problem is by the time I graduated I was somewhat disillusioned with most causal inference methods. It takes a perfect storm natural experiment to get any good results. Plus every 5 years a paper comes out that refutes all previous papers that use whatever method was in vogue at the time.

This article makes me want to get back into this type of thinking though. It’s refreshing after years of reading hand-wavy deep learning papers where SOTA is king and most theoretical thinking seems to occur post hoc, the day of the submission deadline.

Looks like a duck, quacks like a duck.

This is the move right here. And it doesn't have to be a GPT-4 I-know-absolutely-everything model. I think they could train a <1B param model that is capable of being cute and interactive.

Thanks

So, the argument is: "I worked on old tech that didn't work without human's in the loop therefore no technology will ever work without human's in the loop"? I don't follow.

Master Foo and the Recruiter walk into a Bar.

I recommend this book: https://math.byu.edu/~doud/transition/

We used it in our intro proofs class in undergrad and it will get you from basic logic to introductory analysis.

So, for people unfamiliar with deep language models like GPT, it's essentially a program that takes in a prompt and predicts the next set of words based on a training corpus -- which in GPT-3's case is a large portion of the internet. In these examples GPT is not executing any python code, it has just been trained on enough Python code/output to successfully predict what kinds of outputs these functions would produce.

I just can't get over the fact even fusion reactors are basically just steam engines. We haven't figured out a better way to produce electricity than boiling water?