164 karma · joined August 25, 2019
So I'm extrapolating this same idea to LLM inference.
That's why I haven't fully understood yet how working like this is simpler.
So batching requests is always something I think should increase performance by a lot, but most server implementations make this pretty difficult, but the thing I struggle the most to understand is how to keep the latency down if you have multiple clients request all batched together? The total amount of latency for all clients is always the latency for the slowest.
It's been fun dealing with memory and C's weird design in this age of agentic coding.
Also, if it's execution is purely deterministic, you probably don't need non linearity in the layers, right?
Obviously that could only work in a high trust environment, that why open source suffers so much with AI submissions.
I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and many times it ends up not even testing the thing that actually matters.
I try to make integration test whenever possible now, even if they are costly I find that the flexibility of being able to change my implementation and not break a thousand tests for no reason much better to work with.
Edit: I must qualify that this is for software developers only, we did hired juniors for things like data engineers, security, IT and such.
Eventually you might start adding more things to it because of needs you haven't anticipated, do it.
If you find yourself building the tool that does "the whole thing" but worse, then now you know that you could actually use the tool that does "the whole thing".
Did you waste time not using the tool right from the start? That's almost a filosofical question, now you know what you need, you had the chance to avoid it if it turned out you didn't, and maybe 9 times out of 10 you will be right.
But I do think that spawning a goroutine just to do a non-blocking task and get its return is kinda wasteful.
Unfortunately I don't have the results (or the test code) anymore, but it shouldn't be hard to do again (casually at least).
Yes, I guess. I wouldn't be surprised to see "Jai inspired" languages coming out before Jai itself, since some of the ideas look pretty good.
> How does one even get the compiler and build programs ?
Don't know how seriously you're asking this, but I believe you can apply to use it and if Jon likes your credentials and what you want to try it on he might let you have the compiler for some version. Don't know why he won't just open source it and say it's a early version passive of change, but game development is not usually very open source friendly compared to web dev.
So there's a lot that is different with Jai, it's more like a highly anticipated game than a language.
All IO is synchronous? Writes don't actually write when they say they do? Running 201 queries is as "fine" as running a single one with join?
It seems as they got every bad ideia in the book and somehow made it into something that works (that is, if it actually does).
Now he's not here anymore, and posts in his linkedin about simplicity and how people overcomplicate things in software so much. This shows me that you should be careful when taking advice from other engineers, they learn and move on from what was previously a "best practice", while you might get stuck thinking it's worthy it because "that one very good engineer said it was how it should be done".