pk-protect-ai

Please, don't give them ideas!!!!

I'm refusing to consider the runaway AGI a possibility within the next 20 years. However, your comment shifts the discussion away from the actual issue. Altman's hype lines were "regulate us," which is what this hypocrite rode the wave of hype with. Now he is forbidding reporting on safety risks to the regulator... which is illegal in the first place, and yes, the entire business of OpenAI is so overhyped due to Altman's manipulations.

I'm so glad that I no longer need to use OpenAI models; there are far better alternatives available, thanks to Meta and Anthropic. Private

I will never allow my linux to update my fonts ever again ... Arbitrary code execution in its finest form.

I can't understand ... Why can't these guys cut the losses and leave the ship, instead of leaving their families without income for years?

That is not 100% what I read in this paper. There are several takes:

1. LoRA makes transformers linear versus pre-training that keeps non-linearity (in 3.1 and 3.2). What is kinda to be expected.[One more insight is that the combination of seemingly linear blocks can lead to non-linear output]. Thus you can replace part of the layers in fine-tuned models by nn.Linear for inference ...

2. There is a way to make LoRA keep non-linearity by changing loss function and improve performance of the model (in 4, Cosine Similarity regularization term).

3. Small models are surprisingly unaffected. But IMO that may be because of small number of layers and weights overall and the adapter layer being much larger in comparison to the model size.

That is what I have repeated so many times in the last 2 years over and over. I consider Yi Tay's response [1] a mere technicality that is actually irrelevant. What is relevant is how predictable "interpolatable" the data are, how predictable we are.

1. https://twitter.com/YiTayML/status/1783273130087289021

This is quite an old book. I wish I had an access to or even knew about this book during my school days 40 years ago. When I discovered this book, I was already a middle-aged engineer and was just looking for books for my kids. The first two pages blew my mind. If only I had this book back then... However, I remember now the several sleepless days and nights in sequence when I tried to make sense of finite fields at university. I could not sleep; I could not rest... And then I understood that you can create your own algebra whenever you want; you just need to follow the rules. This was so mind-blowing, and at the same time, no single teacher even tried to point out this wondrous fact that actually changed my mind in such a significant way. It could literally have saved a couple of years of my life if I had read this book back in high school. I'm not a mathematician; math exists for me only when it is applicable to what I have at hand. But you realize that you missed a HUGE AMOUNT OF TOOLS, too late in your life.

I would say they need to "unfuck" more than 100 years of pseudo science ...

I believe it is totally the opposite. Every single one of the voters has read the bill and agreed unanimously on passing it through, because it makes a lot of sense. Modern psychology is a little better than astrology back in the day. Psychology is yet to become a proper science, once it accepts proper scientific methods and discards the voodoo of past days.

Until then, hereby I vote for them to wear wizard hats in public!

I'm impressed with the model's output quality. In some cases, the answers from the 70B model are far better than those of GPT-4.

Absolutely amazing!

There is something wrong with these arithmetic: "(32000 * 20) * 2 bytes = 2.3e78 TiB of storage" ... The factorial is missing somewhere in there ...

I had repeating dreams about int 10h and EGA back in 1991 ... It was magic ...

Hey, nobody knew what to do with GF(2^x) up until mid last century either... Oh wait, CS was not really a thing almost up until mid last century...

> By scaling the numbers first by taking the log, multiplication becomes addition and addition becomes x + log1p(exp(y - x)).

Isn't this the same approach as in GF(2^x), which has been in use for decades? The only limitation that comes to mind is the field size.

Who are they? Why do you mix up the guys who prepared the data with the other guys who used this data and making money from a vague memory of that data?

I know. But here where I am, using torrent means participate in distribution of the content and that is where I'll get huge bill for illegally sharing this file.

they have stole nothing, they make no profit from it as well.

I wish it had included the books3, but it doesn't anymore. I wish it was possible to download that 36GB books3.tar in the wild these days. Herewith, I promise to use this dataset according to the "fair use" only...

Do not forget that you are charged when you publish your work in scientific journals. Would you not expect your published work to be preserved?

This might be true as well along with what I said.

Actually, we can't be certain that humans themselves are not reactive. It is just that their reactions are either built in (self-preservation, reproduction), or based on much broader input (sensory, biochemistry, etc.). The current level of reactivity of the LLMs is very limited by their architectures, though, and as long as these architectures stay in place, you can't expect them to be "self-driven".

This isn't true. Benchmarks do not define the usefulness of the models. Mixtral is much more useful for me right now than GPT-4. Look at LLaVA, or the new one, and the very impressive LWM (the text-only version is LLaMA2-based with a 512K context, which does not require TPU to run the inference). The fine-tuned LLaMA 34B is much faster than GPT-4 Turbo and less annoying, providing pretty impressive quality of results.

The agents and action controllers are the reactive components in the toolchain, aren't they?

> But luckily we got LWM, and that means we have only been 10xed.

If I remember the paper correctly, it was something about a 4M context in there. So not 10x, but 2.5x.

> What is really insane is that, we have had LLAMA2 for over a year now, and nobody else figured out how to get this result from it, despite it being around so long.

This isn't true. For now, the task of extending context to 10M tokens is brute-forced by money (increased HW requirements for training and inference and increased training time are also a financial domain). And for now, there simply is no leapfrogging solution for open source or commercial models, which will decrease the costs by orders of magnitude.

Yeap. that's how it should be.

No single neural network currently has the capability to destroy the world, and it will not occur until the architecture for NNs is altered to become self-driving rather than reactive. Even then, it will not be sentient. The deception occurs when the attention shifts from the people operating new powerful technology (tools) to this technology itself.

They are not leading their competition in terms of technology; they are ahead in a public theater and as the first to market. The claims of "regulate us," "it's almost AGI and it's dangerous," and "let's put a moratorium on larger models" are all just hype, deceiving the public. They utilize the work of scientists and other companies worldwide without acknowledging their contributions. Certainly, they have done an impressive job, and the scientists and engineers working there deserve respect and recognition. However, they are not significantly ahead of current research. They are not even weeks ahead of research. The current landscape is a game of budgets, winning public opinion, and securing investments.

It is indicated by the magnet link which they have posted on twitter (see comments above).

Oh, dude, you'll easily end up with $200 per hour if you are working on generating synthetic data...