cztomsik

I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training.

The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention itself is not that unique/important.

https://substack.recursal.ai/p/qwerky-72b-and-32b-training-l...

BTW: It could be also interesting to try use already trained attention and see how long the FFN itself takes in the gpt2 speedtraining (it would be against the rules but still very interesting IMHO - definitely something I'd like to read paper about) https://github.com/KellerJordan/modded-nanogpt

Also, I read yesterday that at some point, the embeddings across all of the models are (very) comparable/similar, and a simple converter can be trained, and if both of these statements are true maybe we could just train everything much faster just by sharing fixed embeddings and attentions.

More tokens -> more compute involved. Attention-based models work by attending every token with each other, so more tokens means not only having more time to "think" but also being able to think "better". That is also at least part of the reason why o1/o3/R1 can sometimes solve what other LLMs could not.

Anyway, I don't think any of the current LLMs are really good for coding. What it's good at is copy-pasting (with some minor changes) from the massive code corpus it has been pre-trained. For example, give it some Zig code and it's straight unable to solve even basic tasks. Same if you give it really unique task, or if you simply ask for potential improvements of your existing code. Very, very bad results, no signs of out-of-box thinking whatsoever.

BTW: I think what people are missing is that LLMs are really great at language modeling. I had great results, and boosts in productivity, just by being able to prepare the task specification, and do quick changes in that really easily. Once I have a good understanding of the problem, I can usually implement everything quickly, and do it in much much better way than any LLM can currently do.

Thank you, so many great features. My personal favorite is .declLiteral() syntax, which makes webserver route-configuration very easy and readable!

If you are ever in temptation to do a builder for internal DSL, .declLiteral() might be a better way. I think it might be very cool for GUI structures too.

I did a quick PoC for SQL query building, but unfortunately, any more advanced queries are better to copy & serialize their arguments for safety reasons, so not that useful there, but still, very hyped about this particular thing.

I suspect those so-called, "zero-cost abstractions".

Rust is generally safer than Zig, but unsafe Rust is/was (mid)designed in such way that it's almost guaranteed to blow in your face eventually.

I don't know if mutable noalias has been already enabled for good, but even if there are no changes between compiler versions, those mutable noalias rules are super-tricky to get right.

In Zig, a pointer is just regular thing, you take care, but most pointers are arena-allocated anyway, and that's a bit like runtime-branded lifetime (which is great and Rust cannot do that, at least not in idiomatic Rust).

Nope, it's quite obvious why distillation works. If you just predict next token, then the only information you can use to compute the loss is THE expected token. Whereas if you distill, you can also use (typically few) logits from the teacher.

"My name is <?>" without distillation has only one valid answer (from the dataset) and everything else is wrong.

Whereas with distillation, you get lots of other names too (from the teacher), and you can add some weight to them too. That way, model learns faster, because it gets more information in each update.

(So instead of "My name is Foo", the model learns "My name is <some name, but in this case Foo>")

That's what I meant to say. There are 2 parts, and both need to work correctly, otherwise you're wasting power.

+1, no idea.

But maybe to add a little bit of context, "damage tracking" means for example, that if there is any ongoing animation (like a spinner), then only a small part of the screen will be re-rendered (with proper vertex-time scissors, so only relevant pixels will be computed). I am not sure if it makes sense in the context of a terminal emulator, but it's certainly a big issue for any non-toy GUI toolkits.

GPUs are incredibly fast parallel computers, so you will likely not observe any perf difference (unless you need order-dependent transparency which you don't), but it might improve your battery life significantly.

esbuild is a bit unfair example because Go is really great fit for this kind of job, it does a lot of I/O, it allocates a lot, and then it just dies. evanw even tried to use Rust at first but it was slower than Go version.

IMHO, if JS had struct types, the difference could be much smaller. Dead proposal here: https://github.com/rbuckton/proposal-struct

IIRC there's `zig reduce` builtin in Zig :) https://github.com/ziglang/zig/blob/master/lib/compiler/redu...

You pay for a car. This is free and OSS.

explicit about branching and allocations, not so for types. we've recently got .decl() syntax, which is even more implicit than .{}

See https://news.ycombinator.com/item?id=41712239

I think around the BLOOM models (2022) it was found out that if you train english-only, the model performs worse than if you have even little mixture of other languages.

Also, there were other papers (one epoch is all you need) where it was shown that diverse data is better than multiple epochs, and finally, there was paper (textbooks is all you need) for famous Phi model, with conclusion that high-quality data > lots of data.

This by itself is not a proof for your specific question but you can extrapolate.

Exactly, they should be sued for this. I think the only people who are defending Apple are the ones who did not read the ToS carefully.

If you mean that LM-head is just inverted embedding matrix then this was already done in GPT-2.

Unfortunately, the only thing I found out about this is that bigger models benefit from separate layer. But this was only mentioned somewhere in discord, so no paper to read and my personal hunch is that it should work for bigger models too. After all, GPT-3 was just scaled GPT-2.

From my personal experiments, models learn better if you give them harder task. And tied weights could be one of such things. Multi-token prediction could be another and bitnet could be also considered such... (and dropout too)

can you comment on what those 500 lines actually were? react+redux?

How is that specific to React? And who would use webapp technology for a website?

FYI there's also std.mem.zeroInit(T), or std.mem.zeroes(T) for C-style zero-init

I don't like where this is going. There's nothing wrong with htmx except that aggressive marketing. SPA frameworks are overused but we got there because React was pushed everywhere exactly like htmx is being pushed right now.

SPAs are useful, not for everything and certainly not for most sites, but bashing SPAs without further thought will only lead back to dark ages of smart backends, postbacks, and huge session states with impossible to scale infrastructure.

Just my 2 cents.

the cool thing with sqlite is that you can compile your whole app into a single binary, so you don't need docker for example (or it's trivial to dockerize it afterwards if you really insist)

can you elaborate? I was living under impression that what sqlite has, is exactly what covering index is...

column reordering is simple to fix with this migration script https://david.rothlis.net/declarative-schema-migration-for-s...

if you are using Zig (and like to live on the bleeding edge), you can also just use my library which includes similar script and also a simple query builder https://github.com/cztomsik/fridge?tab=readme-ov-file#migrat...

Why? From what I read those architectures have many similarities (and same weaknesses)

> Importantly, these recurrent and convolutional forms, which I like to call “RNN mode” and “CNN mode,” are mathematically equivalent. This allows S4 to shape-shift depending on what you need it to do, with no difference in its outputs.

Is this really true? Because it seems to ignore hardware and data type precisions entirely. I mean computing same math thing in a different way with floating points often leads to different results.

There is also thorvg by Samsung (or the authors work in there and they are also using it, not 100% sure but it's production ready)

https://github.com/thorvg/thorvg#dependencies

AFAIK they have experimental GPU backend but I'm not sure how far they are with it.

DPO is not RLHF.

Thank you for your kind words :) I have so many ideas how to make it even better but the time has not been my friend lately :-D

Hm, thanks for pointing that out. I need nginx for SSL anyway, so I was not even thinking about going raw :)

I'm wondering where exactly is the problem in the std.http.Server because it might be easy to fix. Maybe it's just that nobody cared yet... But 2x slower than Ruby sounds awful.

> Possibly you can re-use your ThreadContext

Yes, this is the idea, and extending the injector for request-scoped and thread-scoped dependencies.

It's using comptime inline for, see https://github.com/cztomsik/tokamak/blob/main/src/injector.z...