Kubuxu

Majority of good spots for hydro were already built up, and if they weren’t, good luck with NIMBY.

Beam Search sampling is sometimes getting used

The question is, "If I look at this repo, who owns the copyright?" Sure, you could move the original LICENSE into a directory. Still, if the files are intermingled, you should have a prominent notice that says, " Hey, these files have mixed copyright ownership."

In case of deps, the dependency comes with its own LICENSE file.

In this case the code is essentially forked, integrated and intermingled, so that is why it should be in the LICENSE file.

If it was file or two, it would be fine to add a comment pointing to the license file in the repo, if it was a directory, or to copy it verbatim to that file. It all the copied code was in a directory then having it in directory would be fine.

In this case it looks like they took the original code and heavily modified it, so the simplest way to solve it is one LICENSE with both notices.

The question is who does the copyright belong to in this repository. It is both original author and Microsoft (because they took authors code and modified it). So the License file should mention both.

TME is European as well, and I think Arrow has warehouses in Europe.

Could you list few? I’ve stopped following media in general because everything I could find was profit maximisation through outrage and sensationalism. So I would really appreciate something that isn’t that.

I think what would be useful is a histogram of population densities, or for example a percentile based metric: population density covering 99/95% of population.

Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

I don't think entire GPU is specialised nor a singular token will use the same expert. I think about it as a gather-scatter operation at each layer.

Let's say you have an inference batch of 128 chats, at layer `i` you take the hidden states, compute their routing, scatter them along with the KV for those layers among GPUs (each one handling different experts), the attention and FF happens on these GPUs (as model params are there) and they get gathered again.

You might be able to avoid the gather by performing the routing on each of the GPUs, but I'm generally guessing here.

Not sure about DeepSeek R1, but you are right in regards to previous MoE architectures.

It doesn’t reduce memory usage, as each subsequent token might require different expert buy it reduces per token compute/bandwidth usage. If you place experts in different GPUs, and run batched inference you would see these benefits.

I remember reading a paper that showed that giving models even a a few filler tokens before requiring a single phrase/word/number answer significantly increasee accuracy. This is probably similar.

Chain of Thought was known since 2022 (https://arxiv.org/abs/2201.11903), we just were stuck in a world where we were dumping more data and compute at the training instead of looking at other improvements.

You can create contract, but contracts require consideration, and I don’t see how you do consideration in a case like this without it being a bribery.

A lot of people don’t opt-out themselves, but the society opts them out. Be it a bully or stigma in school, not fitting in high-school or uni.

Yes, depth of focus will be larger, as signified by the larger f-number.

FFWD later is independent of context size, each processed token passes thought the same weights.

It is quite ironic to get a cookie banner on tat page.

Probably forks/duplicates of repos in the dataset.

It would double the size of the KV cache, which can be significant (multi-GB) at larger context sizes.

Hmm, 0.8 works well, but let's try setting lower layers to lower initial value. Let's say 0.2. Ok, I need a formula that will go between 0.2 and 0.8, slowly approaching 0.8. Starts fiddling with numbers for 20min, I guess this can work.

They thought it was dead end, that is why they released it :P

Don’t worry they wrote 50-500 more of them.

AFAIK Intel and AMD have cross-licensing deals for x86 and AMD64 which become invalid in case of acquisition, on the side that gets acquired. This means that you cannot acquire rights to manufacture x64 by acquiring either of them.

You could have an O(N^2) algorithm outperform an O(N) on the scale of 10,000 (or whatever scale you want to imagine). The big-O notation compares only asymptotic behaviour, and sometimes the lower power factors are overwhelming.

The gap between fine-tuning API and weights-available is much more significant than you give it credit for.

You can take the weights and train LoRAs (which is close to fine-tuning), but you can also build custom adapters on top (classification heads). You can mix models from different fine-tunes or perform model surgery (adding additional layers, attention heads, MoE).

You can perform model decomposition and amplify some of its characteristics. You can also train multi-modal adapters for the model. Prompt tuning requires weights as well.

I would even say that having the model is more potent in the hands of individual users than having the dataset.

That is partially on PlatformIO. AFAIK, they started pushing hard the scheme "be a partner (meaning pay us) or we won't even accept external contributions to support your platform".

ESP had a pretty public dispute with them (payment wasn't explicitly mentioned but can be inferred).

AB+ is the universal recipient of red blood cells, O- is the universal donor. It is reverse for plasma.

The blood rejection is reaction between blood plasma (with cells produced by your immune system) and red blood cells.

Zero-knowledge cryptography frequently deals with polynomials of degree 2^20 - 2^27.

FFT is used extensively in curve based zero-knowledge cryptography, not for the purpose of splitting a single large number into smaller ones, but to interpolate and evaluate very large polynomials (for example of the degree 2^27).

All of this happens in a field of an elliptic curve so the complexity reduction is greatly appreciated.