cztomsik

TLDR; do not trust Activity Monitor.

> I gave up Sublime because of memory usage

Isn't this just because of GPU-mapped memory/textures? Because you can't get much better with any other app, for example text-rendering is not a solved problem yet, there are few nice tricks but they always come with some tradeoff so if you want to support non-english scripts, you usually just go with glyph atlas which needs to include 1:1 glyph-size (retina is big) + up to 3 extra variations because of kerning for each used glyph. System apps do not report this if they are using cocoa but a similar overhead is still there.

Or because of the emoji font, which alone takes 200M!!! And if your app opens it (just to support emojis, not necessarily to draw them), it gets reported as its own memory even when it's usually mmapped and shared between processes (and might not even be loaded in the memory).

What kind of memory usage are we talking about? And how did you measure?

I can't promise anything but there's apparently some problem with intel macs and I only have older machine here (with catalina) so I might give it a shot and if it works I can publish the build.

EDIT: it's too old, sorry.

This is macos application, you download it, double click it and it should work.

Sorry, I thought I have answered already. No, everything runs as one native-like (mix of native and webview) application.

Thank you, and I wish you luck with your project too, of course :)

It depends, what version do you need?

Thanks for the feedback, this is what I was hoping for.

> Scrolling up should stop auto-scroll in the chat

This is at the top-of the list for today, check the website/discord/twitter for updates. I want to do both - pin the button at the bottom, and also make sure that if you scroll, it will stay there. Currently, it's one component which contains the button and also scrolls and it was a bad idea.

> Blurring the window while installing

Agreed 100%, I definitely want to improve downloader a lot more but this was good enough for first release.

> Maybe an uninstall button for models is appropriate.

It should be there (in the upper area), I hope the 24h cache in cloudfront is not too much but the link should be pointing to 09-18 version which has Delete button.

Exactly! I don't want to subscribe to anything, I want to pay once and have the license forever (for that specific version + bugfixes).

But that makes OSS more complicated - people usually don't pay for something they can clone & build themselves.

BTW: Linux and Windows will definitely happen, I just didn't have time yet. Sorry, it's a lot of work and time so I had to pick macos first, and even that alone is a lot of testing on different configurations.

Thank you for your kind words, I couldn't say it better.

Ok, let's try this, if this was (fully) open-source, would you pay $5/month for an access to a premium discord/formus with videos, examples, ability to propose new features or vote on the backlog?

Not really, my closest competition is LM Studio, which is closed-source too. There's also Machato, which is also closed-source (and not even free).

> IMHO, at some point, it's just about trust

Yes, but you are also literally going to multiply several billions numbers together and pray for a random outcome.

I don't know, maybe some middle ground would be feasible - like have the server-side part open-source, because that's the only part where anything harmful might be (and it's also fairly thin BTW).

But no matter what, the project needs to be profitable for me, sorry to say that, but let's be a bit realistic, you wouldn't work for free either.

Thanks :) love your podcast BTW.

See https://news.ycombinator.com/item?id=37562287 - I can't tell much more yet, but I feel like LLMs are way more useful than for a chat-bot or a vscode extension.

No, you are right, it's a new term I made up. The idea is that I see this as a RAD tool (for non-programmers). So you would have this app open all the time and it would serve different tasks you have prepared before. But that part is not finished yet and it will be limited in the free version.

Fair point, but if this was open-source, would you really go and read all of the source code in several programming languages and also audit all of the dependencies?

IMHO, at some point, it's just about trust.

Now, to be 100% honest, I'd love to release this as OSS, but my current business model is based on freemium. And, sell it as white-label for companies.

EDIT: if you have better idea, I'd love to hear about it.

It has playground with prompt saving and (simple) templates, chat saving, builtin model downloader. It's also very small, most people don't care about that, but I do.

It works with current version of GGUF models for llama.cpp - you can find them on huggingface, or you can convert them manually.

Only a few download links are baked-in at the moment but whatever *.gguf file you put in your Downloads folder should appear in the dropdown.

Agreed 100%, and some of the "simple" approaches like repetition penalty, can harm output, because for example markdown tables repeat a lot of characters, and if you just blindly apply repeat penalty, it will just stop doing what you've asked for.

My guess is that they have some kind of state-aware sampler, and they know if they are in the table, etc. Because then you can sample in a much better way. Just like with grammars, but grammar itself is probably not enough.

Sampling and tokenization are IMHO the biggest open problems. We have something which kind of works but it's nowhere close to be perfect.

It is very common for small models, and sometimes even 7B model gets stuck.

There was an article/paper showing that GPTs (whole family) quickly get confident in looping if there is anything loop-like in the window. So basically, as soon as it loops once, it will never get out of it.

Note that sometimes loops are desired, like with docstring in the code, always starting before the function definition, and other structural things.

FYI llama.cpp can do that for a "while" https://github.com/ggerganov/llama.cpp/pull/1773

Somebody is also working on a whisper.cpp version, which is maybe even more interesting because if you have grammar you can speak not only JSON but also a code (or anything)

Yes, it's true that you can do arenas in Rust, but they are harder to use in Rust, due to borrow-checking. For example you can't store arena in a struct without lifetime.

So in a language which actively makes your life miserable in the name of safety, you will likely just use Vec<T> because it's easy. Hence, back to the previous point - Vec<T> is slow -> your code is slow and it's not even obvious why because everybody does that so it looks like you're doing it right.

The point was typical, typical Rust code is using Vec<>, typical Zig code is using arenas. Typical Rust code is using smart pointers, typical Zig/C code prefers plain, copyable structs.

It is 100% possible to use such style in Rust/C++ (and then the performance will be same or maybe even in favor of rust, it might be the case) but people usually do not do that.

I don't need any data to say that Zig is typically faster than Rust because I know that Vec<T> will do a shitload of useless work when it leaves the scope, even if the T does not implement Drop. You can do vec.set_len() but that's unsafe, so... Typical Rust is slower than typical Zig.

Zig does not do any work because it does not have smart pointers and it does not try to be safe. It tries to be explicit and predictable, the rest is my job.

BTW: This is very real, I've been profiling this and I couldn't believe my eyes. And even if they fixed it already there would be many similar footguns. You can't write performant software if you can't tell what work will be done.

I don't need to prove you anything. Go and give it a try yourself.

1. It's typically at least as fast as C, unlike C++/Rust

2. You can do type introspection (and switching) during compile-time, and it's not just some stupid TokenStream transformer, you really have type information available, you can do if/else on the presence of methods, etc.

3. There are no generics, but your functions can accept anytype, which is still type-safe. See https://github.com/ziglang/zig/blob/9c05810be60756e07bd7fee0... and note the return type is "computed" from the type of the input.

4. Types are first-class values (during comptime), so any function can take or return a new type, this is how you get generic types, without (syntax/checking) support for generics.

5. You can easily call anything which does not allocate in these comptime blocks.

6. There's @compileError which you can use for custom type assertions -> therefore, you have programmable type-system.

7. It's super-easy to call C from Zig.

8. This is subjective: You don't feel bad about using pointers. Linked data structures are fine.

The question is: Who needs to abtract over 10 lines of code?

There were other frameworks before React: AngularJS, Knockout.JS, Backbone.JS, ExtJS, and I'm probably missing few others... Most of them had a concept of props/dirty-checking/retained-mode and they were definitely faster than naive React.

What VDOM made easier (in terms of performance) was memoization, you could get faster if you carefully evaluated your dependencies, but to be fair, it introduced many edge-cases and many people avoid memoization to this day.

VDOM also made react-native possible, and it made cross-browser programming bearable (but other frameworks at the time did that too)

Now, one REALLY COOL thing about VDOM is fragments, because being able to render multiple elements from one component, makes certain layouts very easy using plain CSS - either with simple > * or even with CSS GRID.

I am not sure, if Svelte 4 can do that, but it's very simple thing to do with VDOM. And it certainly was not common feature for a very long time.

I am not an expert, but from what I've seen, PyTorch is mostly a thin wrapper over the C++ libtorch. The same is true for DOM as well, but nobody uses DOM directly, whereas everybody uses PyTorch.

The big shift is Jupyter, but that's mainly for exploratory programming. If you already know what you're doing, there's no reason why C++ should be worse than Python for training. It's likely that most ML engineers do not have experience with C++.

I don't have that experience either but from what I've seen, C++ is very powerful, so once you subtract the jupyter, there's not really too much left.

BTW: You cannot use DOM/CSSOM from C++, the only API is JS, so your argument is theoretical.

You're not entirely wrong but pretty much everything you use from python is written in C++ anyway, so what's your point?

In my case, Zig was super-simple to pickup, I've been using rust for few years so I was already in the systems programming (as a hobby), and I also had to read some C/C++ during that time, that probably helped me too.

Other than that, I was and I am, mostly a frontend developer with ECMAScript and TypeScript experience. I think Zig is very close to both, because Zig has anytype, so you can do duck-typing like you do in JS, and it has programmable type system, just like TS. Not to mention that reflection is basically your daily bread in JS/TS.

TLDR: If you have some systems programming experience and you've done a bit of TS, I'd definitely recommend you to give Zig a try. One weekend should be enough. I did that literally single-handed, as I was recovering after wrist surgery.