1,271 karma · joined October 10, 2016
Q. When you are training with a context length of 128k, is the attention in the global layers dense or sparse ?
If dense, would the attention memory requirement here would be O(n^2) where n is 128k for each global layer ?
The implementation of OpenBSD predates many safer systems languages but I think OpenBSD should now start moving to a checked variant of C or a safer language like Rust/OCaml/Odin/Zig/Something else.
The conversion can start with some OpenBSD user space programs. I notice a steady stream of C related security fixes in the OpenBSD changelog. Many of these could have been probably avoided if the implementation language was more “safe” by default.
I doubt that this is going to happen but I think it is fair to point out that using C does give you some additional security headaches by default.
Unsloth allows you to give it a transformer model and additional training data to do LoRA/QLoRA. LoRA/QLoRA will keep the weights of the model as constant but as output some low rank adjustments to the weights which serves as the weight "delta".
Typically one would do SFT with the training data. But Unsloth allows you to do RL (Reinforcement learning) specifically GRPO on the model + training data you give it also! The output of the GRPO here is again in the form the LoRA/QLoRA weights.
You have found a way to reduce the memory requirements for GRPO.
Question: How does one decide whether the training data will be SFT (Supervised fine tuning) or GRPO ? When will you get better results with SFT and when with GRPO ?
So, if we simply wait for <EOS> to be sampled in a block, that could be the place to stop. My answer above should be the same (modulo this correction) -- simply keep obtaining the blocks in a semi-autoregressive fashion.
In a normal transformer the incremental message is a single token while in LLADA the incremental message would be an additional fixed size block of tokens. If you do inference in the semi-autoregressive strategy outlined in the paper you could start by adding block after block. When a block is fully unmasked it essentially becomes part of the prompt. You would stop the "turn" as soon as there is some sort "end of reply" token decoded in your block. This token could appear anywhere in the block.
Questions: (1) Is there such thing as a "end of reply" token ? (I'm not taking about End of Sentence <EOS> which is obviously different)
(2) What about the junk tokens that appear after this "end of reply" tokens. I could image the unmasking transformer generating some stuff for those tokens.
TL;DR: This is a very interesting paper about attention calculation in transformers. It shows how attention can be calculated over a large token window without saturating memory and/or GPU arithmetic abilities.
Usually attention is a sliding window of tokens. The window can turn out to be too big due to the quadratic nature of attention which increases the amount of computation required. There are many papers on how to get some of the benefits of transformers by doing "sparse attention" -- i.e. avoiding some of the quadratic blowup.
The solution in the paper is first divide every `x` tokens into groups or "blocks".
(1) Capture long range conections by compressing blocks of tokens to a single token
(2) Select important tokens by only choosing the tokens in the "important" blocks
(3) Select recent tokens by using a sliding window (like normal transformers)
Compression of a block of tokens to a single token in (1) is done by an MLP that is trained during normal training time.
Now attention scores can be done for an incoming token with the preceding block of tokens. Select only the top-k blocks which have high attention scores for (1).
Finally combine the results of attention of incoming tokens with (1), (2) and (3) to give you a final output token. You get long range coarse attention, attention to selective blocks and the usual sliding window attention. Awesome !
This is sort of engineering type paper also with lots of low level details.
Question for the authors: Why not do the experiments with MHLA also (multi head latent attention) that is there in DeepSeek V3 and R1 ?
During suspend, for graphics cards, GPU VRAM needs to be transferred to system RAM.
However, during high memory usage scenarios the VRAM + RAM usage could exceed system memory -- this would ordinarily involve system swap coming into play and handling the temporarily out of memory issue. However system swap was already deactivated when it came time to suspending the AMD card causing all sorts of problems.
The fix was asking the GPU to evict its VRAM to system RAM via the hook ("suspend prepare") before swap was deactivated in linux kernel.
SSoftMax_i = exp(s log(n) z_i) / sum (n is length of embedding).
Normal softmax (with temperature)
SoftMax_i = exp(z_i / T) / sum (T is the temperature).
Here Temperature is a hyperparameter. Having a temperature as hyperparameter does not seem too different to me than having `s` as a hyperparameter. I personally don't understand the benefits of SSoftMax. During hyperparameter search you would find the optimal `s` as you might find the optimal temperature `T`.
This approach should make the LLM deterministic regardless of the temperature chosen.
P.S. Choosing lower and lower temperatures will make the LLM more deterministic but it will never be totally deterministic because there will always be some probability in other tokens. Also it is not possible to use temperature as exactly 0 due to exp(1/T) blowup. Like I mentioned above, you could avoid fiddling with temperature and just decide to always choose token with highest probability for full determinism.
There are probably other more subtle things that might make the LLM non-deterministic from run to run though. It could be due to some non-deterministism in the GPU/CPU hardware. Floating point is very sensitive to ordering.
TL;DR for as much determinism as possible just choose token with highest probability (i.e. dont sample the distribution).
> A soft fork might clarify things like the old gcc fork did.
Nice idea but in practice there is absolutely no chance that this would be a successful strategy for those people in favor of Rust.
Being out of mainline tree would perpetually put Rust in the "future" (like Nuclear Fusion -- always just "20 years away"). Meanwhile Linux would continue to evolve in ways that could be very disruptive for Rust integration. For example, Rust interop needs to think very carefully about locks, memory order etc. Any change in the mainline kernel could really mess the assumptions made in the Rust fork. A year or two of evolution of the slightly independent evolution of the two branches may make it almost impossible to merge the two.
Only companies as large as Google could maintain the Android fork. The RFL team is small and I don't think large vendors would like to invest their time and energy on forks. Who would pick up and package the fork ? Red Hat ? Debian ? Arch ? They are already complaining about compilation and maintainability of Rust when it is in the mainline kernel. No chance that they start using the down stream rust kernel in their main kernels.
The only way Rust for Linux succeeds is if it has the blessing and push from Greg/Linus.
Joking aside, Rust for Linux seems to be entering a phase where rubber is hitting the road and some maintainers who thought that Rust would just go away are having to express their opposition more honestly. There are definitely reasons for and against Rust in the Linux kernel and I feel for them.
Actually this pattern happens in all spheres of life. Some change is introduced. First, people ignore/deny it thinking it won't affect them or they feel that the system is too strong and the change won't really happen. They are (probabilistically) right because successful change requires really superhuman drive which the Rust for Linux maintainers by luck/unluck (depending on who you are) happen to have.
Now real change is coming to the door of some maintainers. Some of them hate the melange of Rust and C the Linux kernel is likely to become in the coming years. So they are hoping that their subsystem can escape. But it can't if Rust is really going to be a viable rather than toy option.
I think Rust has been baking in the kernel for a long time -- The BDFL/DFL needs to informally poll the maintainers and take a definitive decision to cast Linux's lot with Rust (or not!?). The Experimental tag in Rust gives some people the hope that this Rust experiment could be ended. This experimental tag should be replaced with in-progress/not-mature tag to further send the message down the line.
Another way to understand AI in my view is to look at (often smaller) resource rich countries around the world (Oil, Minerals etc). Often the government is more worried about the resource rather than the people that live in the country. The government often does not bother educate them, take good care of them, give them a voice in the future of the country etc. because those citizens are not the ones that pay the bills, or are the main source of GDP output or source of political power etc.
Similarly in an AI heavy economy, unless systems are not designed right governments might start ignoring their citizens. If democracy is not robust or money has a big role in elections the majority voice of humans is likely to matter less and less going forward.
Norway is a good example of resource rich country that still looks out for its citizens. So it should be possible to be resource rich/AI rich and have a happy citizenry. I suppose balancing all moving parts would be difficult.
The way to deal with the risks of AI would be to make AI available to all -- this is my strong belief. There are more risk to AI being walled off to select nations / classes of citizens on grounds of various real / imagined risks. This could create a very privileged class of countries and people that have AI while the others citizens don't / cant. With such huge advantages, AI would wreak greater havoc on the "have-nots". (Over)regulation of AI can have worse consequences under some conditions.
Yes, it is maybe monolithic (but so is the Linux kernel). Its philosophy may differ from unix's "get one thing done well" too but integration of various functionalities comes with its benefits.
Some people say it is bloated. The substitutes to systemd are lightweight but less featureful. Maybe some of them will get bloated as they achieve feature parity.
People have a right to build substitutes and replacements -- I believe in the "Let a hundred flowers bloom" philosopy. However I can't understand why systemd is the point of so much disagreement.
But why do I think that DeepSeek did this themselves? I think the amazing decision of DeepSeek to release its full model in the open shows how confident and proud of the work they are. This model by all accounts is a good one. If large scale distilling happened (hundreds of thousands of training outputs taken from closed models where this is expressed disallowed by the said closed model) I don't think DeepSeek would have been so open with its models and technical papers. They would have walled it off and just provided a chat interface. This genuinely seems like a labour of love. You can deeply inspect the model. I'm sure OpenAI experts and others are all over the model since the weights are freely available.
Another point: the field of deep learning has a vast literature by now and a lot of openly available supervised training datasets. There is enough training data available during the pre-training and post training stages if you can gather a large team. Now DeepSeek has a large team -- they would have been able to get enough training data legitimately.
Lastly DeepSeek in their technical papers have shown how they bootstrapped this model. Training data for R1 was also synthetically generated by previous models of DeepSeek (R1-Zero/ V3 / both ?). Synthetic data generation with light human curation I can imagine would have been entirely possible to do legitimately at scale.
Finally I feel that we should resist the urge to think if something came out of China then it must have been ripped off from US, involved large scale industrial espionage etc. By now China legitimately produces a lot of cutting edge products and original science. Without specific proof shown by experts I would hesitate to adopt this viewpoint.
Also consider this: China is the largest producer of electricity in the world and the cost of every material input is quite competitive. Labor costs are quite low also (Few Deep learning engineers there earning million dollar plus salaries).
DeepSeek papers also describe how their adopted various inovations to keep GPU costs low (Multi head latent attention to reduce GPU resources, use of FP8 instead of FP16/32 etc.). All this would have kept the costs low.
On a closing note, this model represents a gift to humanity. Not unlike Google and OpenAI's research papers. When I stop thinking about this model merely as "Chinese made" but (also) "Human made" my heart is filled with admiration of this achievement. This model can be owned by humanity just like the Linux kernel is.
We are forgetting that China has a whole hardware ecosystem. Now we learn that building SOTA models does not need SOTA hardware in massive quanties from nvidia. So the crash in the market implicitly could mean that the (hardware) monopoly of American companies is not going to be more than a few years. The hardware moat is not as deep as the West thought.
Once China brings scale like it did to batteries, EVs, solar, infrastructure, drones (etc) they will be able to run and train their models on their own hardware. Probably some time away but less time than what Wall Street thought.
This is actually more about nvidia than about OpenAI. OpenAI owns the end interface and it will be generally safe (maybe at a smaller valuation). In the long term nvidia is more replaceable than you think it is. Inference is going to dominate the market -- its going to be cerebras, groq, amd, intel, nvidia, google TPUs, chinese TPUs etc.
On the training side, there will be less demand for nvidia GPUs as meta, google, microsoft etc. extract efficiencies with the GPUs they already have given the embarrasing success of DeepSeek. Now, China might have been another insatiable market for nvidia but the export controls have ensured that it wont be.
I think it goes like this: The major market regulator could say directly/indirectly: Hey you can merge in the US ... but good luck operating in our geography in a frictionless manner if we are against your merger. As a regulator we can make life hell for you if you don't obey our anticompetitive laws. Since you derive a high percentage of your revenue/profits you must listen to us !
It all depends on the percentage of sales in the foreign geography. With EU/China it can be quite high -- especially for tech companies.
So yes, foreign powers can and often do block companies from merging.
In AI models, I think the pictures the AI generates is biased to generate is also a form of "airbrush" except the model makes the reflectivity of the images high -- simply to hide the fact that there _arent_ any imperfections that would make the photo more realistic.
In other words, gloss is just a form of airbrushing -- AI does it to hide the fact that there are no more details available.
I would guess that AI models could make the airbrush more like the airbrush human photo editors do by changing some hyper-parameters of their models.
Many of memory/load stores will be removed later in the pipeline and changed to registers. That is something for the optimization pipeline. Thats when phi nodes etc. will be insert and come into play.
I like languages that disallow null by default (e.g. Rust, OCaml etc) because it seems to be a huge source of errors.
This is not true in my personal experience.
As has been famously said (paraphrased): Functional programming makes tough problems easy and easy problems tough.
In other words the value of functional programming depends on your domain.