sweezyjeezy

This feels a bit tenuous. What world do you envisage where it's a completely level playing field? Do we ban talking, music, sound in movies etc etc??

The task is not to compress general data by a factor of 200, the task is to compress a very domain-specific kind of data by a factor of 200. Presumably the hope is this data has lower entropy than e.g. the Hutter prize data.

If I tell you to write an image compression algorithm, you aren't going to be able to do much with a bitmap of uniform randomly generated pixels. However if I tell you that in the domain I'm working in there are only two colors white and black, immediately I can reduce storing each pixel from 24 bits to 1bit, saving a factor of 24. If I tell you further that >99% of pixels are going to be black, more compression tricks become possible, etc.

I don't have expertise in this particular problem, but a priori dismissing it by comparing to Hutter is not valid.

> Space in academic journals is too precious to waste on inessential content

Not the biggest issue in maths - the arxiv version usually won't match the journal version 1:1

Example: the Crab Nebula was the result of a supernova that was seen by Chinese astronomers about 1000 years ago: https://en.wikipedia.org/wiki/Crab_Nebula

One approach I've made work a couple of times is to pull all of the data going in and out of a system for a time period, and rerun it after the refactor. This gave me a lot more confidence than unit tests.

100% - our daily stand-ups are a waste of time usually, but we have fantastic team camaraderie and I think us shooting the breeze for 5 minutes every day is a huge part of this.

> While cosine similarity is invariant under such rotations R, one of the key insights in this paper is that the first (but not the second) objective is also invariant to rescalings of the columns of A and B

Ha interesting I wrote a blog post where I pointed this out a few years ago [1], and how we got around it for item-item similarity at an old job (essentially an implicit re-projection to original space as noted in section 3).

https://swarbrickjones.wordpress.com/2016/11/24/note-on-an-i...

Interestingly it used to be quite standard with 'small' language models to use a search algorithm to render a full block of text, the most basic being beam search. Then you can get better with more processing power to do a wider path search. This is not what OP is talking about, it just means generating a larger number of candidate continuations. However it's not necessary or optimal for newer LLMs, because it tends to siphon the LLM into quite generic places, and it can get very repetitive.

re: caching python functions, is it referring to functools.lru_cache maybe?

Yeah but try getting e.g. Dall-E 3 to do photorealism, I think they've RLHF'd the crap out of it in the name of safety.

I don't think this will refute the article, but I would have found it more convincing if the benchmark had included a single setitem assignment as well, to be sure that the difference wasn't python doing a lazy dict-or-set type assignment on {}

FWIW I think df.column.value_counts() is better to use here in pandas.

haha that's an interesting point! Recompress as a jpeg maybe.

I'm not sure that screen-shotting the image will work FWIW - any rescaling interpolation in rendering the image on the page or loading it for a model will likely reduce or nullify the effect.

Also these perturbation based adversarial attacks are often model specific. You take the model's gradient at each pixel and iteratively perturbate the image to make it more and more confident that it's e.g. a cat.

There's a couple of false dichotomies here

- to say that because we're "more optimised" we must be the most optimised. Our brains are optimised well for certain things, sure, but computers are far more efficient at e.g. crunching numbers than we are

- to say that there's no success in a machine that can't currently beat us at math - this year has already proven that false

I think the word "sentience" is a red herring. The more important point is that the researcher at Google thought that the AI had wants and needs 'like a human', e.g. that if it asked the AI if it wanted legal representation to protect its own rights, this was the same as asking a human the same question.

This needs much stronger evidence than the researcher presented, when slight variations or framing of the same questions could lead to very different outcomes from the LLM.

Here is something that I think would be a big breakthrough:

I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt.

Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.

"Is this AGI"? doesn't seem like a useful question for precisely this reason - it's ill-defined and hard to prove or falsify. The pertinent questions are more along the lines of "what effect will this have on society", "what are the risks of this technology" etc.

But they haven't legalised their behavior... if they were to continue doing it, the fine and punishment goes up - it doesn't become a cost-of-business you can just eat going forward.

Also CZ has just stepped down and will be getting legally charged now.

I feel you're going to have to do a bit more work to explain why this lobbying -> reward, vs. crime -> financial penalty, analogy makes sense.

Yes they are different - example: https://en.m.wikipedia.org/wiki/Parallel_postulate

I'm not sure I see why the axioms were not also discovered though? Choice between irreducible assumptions does not seem to make them any more 'invented'.

Wework started spiralling in 2019, it was never going to stay afloat.

The article starts by saying mathematicians want things to work for large numbers, but that doesn't really get to the crux of the issue with primes. Infinite sequences are ubiquitous in number theory, and in general it will be infeasible to have a test for inclusion in these sequences for large enough numbers. But often they have structure that we can use to characterise them very precisely - think about square numbers, it's easy to say what the trillionth square number is, what it's remainder when you divide by 13, etc.

What makes primes hard, and also interesting, is that they seem to be extremely unstructured, we believe they behave like a kind of random number generator, even though they are clearly not random. In fact many of the theorems and conjectures mentioned in the article actually hinge on this. Random numbers are unpredictable on a small scale, but on a large scale they have very nice distributional properties, whereas more structured ones of similar growth rate will often have undesirable restrictions on them.

5 years for me, but yeah live coding is always done on a screen these days. I mean

- whiteboard / paper is just so slow

- extremely poor editing functionality on whiteboard / paper

- some people's handwriting is not legible

Does vision solve it? How does one ask a question through images alone?

There's a key difference between writing for the screen and writing for software - making software engineers more efficient makes software cheaper will raise its demand. This is similar to how there are more accountants now than pre- spreadsheets. But writers in TV and film are not the bottleneck, making them more efficient will not lead to more demand for entertainment. It's clearer to me that the latter camp are in serious trouble, whereas we may just need to adapt.

That's probably more because of RLHF though, they've optimised for certain kind of responses rather than simple model loss on internet text.

I think there's a middle ground here. The original quote does not mean DRY=bad, abstraction=bad. The point is there is a non-zero cost to these things. A bad abstraction can, as you say, accumulate to something terrible through inertia or inexperience. A bad abstraction, even if caught early, was probably not worthwhile - I mean, it took time just to make the original one right? This does not mean that we should be scared of abstraction in general, but in my opinion abstractions that are purely for the sake of reducing duplication should be viewed with an extra level of apprehension.

I've found these archive links have basically stopped working for me on all desktop browsers recently. The link opens fine on my mobile but not desktop chrome or FF.