longdog

I feel the webpage strongly hints that sparse autoencoders were invented by OpenAI for this project.

Very weird that they don't cite this in their webpage and instead bury the source in their paper.

What are those other areas which display age-based creative decline? Other creative fields I can think of off the top of my head - scientific research, animation, fiction writing, architecture - are overwhelmingly dominated by older people.

Even in pop music, I'd argue that artists are doing very little of the actual heavy lifting compared to the producers and the writers. Pop singers have a much shorter shelf life than producer/writers due to the importance of image in appealing to younger fans. See https://en.wikipedia.org/wiki/Max_Martin for an example.

I've been playing with Llama 3 8b instruct but I've found it to be surprisingly low quality compared to some of the better Mistral 7b finetunes (zephyr, dolphin, openorca). Rather surprising because there's no way Mistral or any of the organizations doing the finetuning did even a fraction of the training volume that Meta did.

Population in Japan has barely fallen (yet). So far it's only a ~2% decline from peak population, but there will be a 20% decline in the next 20 years.

There is a long lag between below-replacement fertility and actual population decline. Because of how compounding growth works and the length of human lifespans, sub-replacement fertility won't result in population decline (for a previously fast-growing country) until 40+ years after the fact. Japan is only just now seeing the effects of lowish fertility from the 70s and 80s.

Note that one of the other consequences of population math is that if a country has been previously declining in population for a while, it'll continue to decline for decades even if the current fertility rate is at or slightly above replacement rate. This means that population decline is essentially an inevitability for most East Asian and European countries for the next several generations.

None of us knows what will happen when populations are falling by 5%+ per decade which is now the inevitable future of many countries the next few decades..it's totally unprecedented in human history (excluding cases like war/disaster).

The author is way overselling how controversial Haidt's claim is.

- The effect of social media at a societal level isn't something that can tested experimentally. And like all non-experimental research, results depend heavily on modeling methodology (for example, the meta-analysis linked by the author is literally just a regression of facebook DAU on life satisfaction polls [1]). As a result there will NEVER a universal consensus, same as with most "macro" level studies in social sciences. If you wait for researchers to reach an agreement, you will wait forever.

- The most credible research I've seen on this is this quasi-experimental paper [2]. Since the timing of Facebook's rollout was staggered across schools, it can be used as a natural experiment. Schools should've seen declines in student mental health that correspond with the date of Facebook's rollout on their campus, which is indeed what happened.

- The effect of using social media at an individual level (as opposed to a societal level) IS known and it is very clearly negative. See [3] for an example.

- Most importantly, the fact is that mental health and suicide rates have been rising significantly since exactly when social media gained popularity (mid-to-late 2000s). The effect is global so you can't blame country-specfic policies. And the rise is most significant with demographics most exposed to social media (young people, especially girls). There's not a single other explanation that makes sense. Frankly, I think people are afraid to admit that social media is the problem because so many people are tech addicts and don't want to admit that their own addiction is part of the problem.

[1] https://royalsocietypublishing.org/doi/10.1098/rsos.221451

[2] https://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.20211218

[3] https://www.aeaweb.org/articles?id=10.1257/aer.20190658

Interesting, but I'm very skeptical. There are over a dozen transformers-based foundation time series model released in the past year and without fail, every one of them claims to be at or near SOTA. For example:

- Time-LLM (https://arxiv.org/abs/2310.01728)

- Lag-Llama (https://arxiv.org/abs/2310.08278)

- UniTime (https://arxiv.org/abs/2310.09751)

- TEMPO (https://arxiv.org/abs/2310.04948)

- TimeGPT (https://arxiv.org/abs/2310.03589)

- TimesFM (https://arxiv.org/html/2310.10688v2)

- GPT4TS (https://arxiv.org/pdf/2308.08469.pdf)

Yet not a SINGLE transformer-based model I've managed to successfully run has beaten gradient boosted tree models on my use case (economic forecasting). To be honest I believe these foundational models are all vastly overfit. There's basically only 2 benchmarking sets that are ever used in time series (the Monash set and the M-competition set), so it'd be easy to overtune a model just to perform well on these.

I would love to see someone make a broader set of varied benchmarks and have an independent third party do these evaluations like with LLM leaderboards. Otherwise I assume all published benchmarks are 100% meaningless and gamed.

You don't need to be a cutting edge research scientist to train a SOTA LLM. You just need money for scaling. OpenAI's "secret" was just their willingness to spend tens/hundreds of millions without guaranteed returns, and RLHF/instruct fine tuning, both of which are out of the bag now.