nikki93

https://github.com/terryyin/lizard has been useful to track when functions get too convoluted or long, or when there's too much duplication -- in code generated by agents. Still have to see how well it works long term but it's caught things here and there, I have it in the build steps in my scripts so the agent sees its output.

A relevant paper: https://arxiv.org/abs/2306.11644 -- the Phi models (and many others too) are based on this idea.

Pasting a comment I posted elsewhere:

Resources I’ve liked:

Sebastian Raschka book on building them from scratch

Deep Learning a Visual Approach

These videos / playlists:

https://youtube.com/playlist?list=PLoROMvodv4rOY23Y0BoGoBGgQ... https://youtube.com/playlist?list=PLoROMvodv4rOwvldxftJTmoR3... https://youtube.com/playlist?list=PL7m7hLIqA0hoIUPhC26ASCVs_... https://www.youtube.com/live/uIsej_SIIQU?si=RHBetDNa7JXKjziD

here’s a basic impl that i trained on tinystories to decent effect: https://gist.github.com/nikki93/f7eae83095f30374d7a3006fd5af... (i used claude code a lot to help with the above bc a new field for me) (i did this with C and mlx before but ultimately gave into the python lol)

but overall it boils down to:

- tokenize the text

- embed tokens (map each to a vector) with a simple NN

- apply positional info so each token also encodes where it is

- do the attention. this bit is key and also very interesting to me. there are three neural networks: Q, K, V – that are applied to each token. you then generate a new sequence of embeddings where each position has the Vs of all tokens added up weighted by the Q of that position dot’d with the K of the other position. the new embeddings are /added/ to the previous layer (adding like this is called ‘residual’)

- also do another NN pass without attention, again adding the output (residual) there’s actually multiple ‘heads’ each with a different Q, K, V – their outputs are added together before that second NN pass

there’s some normalization at each stage to keep the numbers reasonable and from blowing up

you repeat the attention + forward blocks many times, then the last embedding in the final layer output is what you can sample based on

i was surprised by how quickly this just starts to generate coherent grammar etc. having the training loop also do a generation step to show example output at each stage of training was helpful to see how the output qualitatively improves over time, and it’s kind of cool to “watch” it learn.

this doesn’t cover MoE, sparse vs dense attention and also the whole thing about RL on top of these (whether for human feedback or for doing “search with backtracking and sparse reward”) – i haven’t coded those up yet just kinda read about them…

now the thing is – this is a setup for it to learn some processes spread among the weights that do what it does – but what those processes are seems still very unknown. “mechanistic interpretability” is the space that’s meant to work on that, been looking into it lately.

I think a 'linear types' ish model where the compiler flags an error if you didn't write the explicit unlock call, and only compiles if some acceptable unlock call (or set of calls) is added, would be a good design. It can / would also prevent use-after-consume. I do want the static checks, but I don't think that means that implicit function calls need to be generated on a } or a = (assigning to a variable causes a drop) etc. This is what Rust already does with `.clone()` -- needing the explicit call -- and I think it makes sense for a lot of cases of implicit drop. I have seen discussions about implementing this for Rust and I have it implemented in a C borrow checker experiment I'm trying. ATS is also an example of this, and going further is something like Frama-C or seL4 and so on.

The main point being: the implicitness is not necessary for "compiler makes sure you don't forget". So the original comment about how usage of the explicitly named and paired APIs can clarify intent both for the writer and reader can still stand while not implying that forgetting is involved. I see this dichotomy often being drawn and I think it's important to consider the language design space more granularly. I think a reminder is a better fix for forgetting, rather than assuming what you wanted and doing it for you.

(the explicit calls also let you / work better with "go to defintion" on the editor to see their code, learn what to look up in a manual, step in with a debugger, see reasonable function names in stack traces and profiles, pass / require more arguments to the drop call, let the drop call have return / error values you can do something about (consider `fclose`), let the drop call be async, ...)

Frama-C uses CIL.

GCC and clang (and maybe others) have 'statement expressions': https://godbolt.org/z/sqYnbh4Ej

This would be nice also to have an API that's usable from C/C++ code running in Wasm. I see libraries often do this where they expose a C/C++ library like Postgres to Wasm and the main / documented API is JS and you have to dig a bit to find the C/C++ API if it's possible to access it that way.

I use this static reflection hack in C++ -- https://godbolt.org/z/enh8za4ja

You do have to tag struct fields with a macro, but you can attach contexpr-visitable attributes. There's also a static limit to how many reflectable fields you can have, all reflectable fields need to be at the front of the struct, and the struct needs to be an aggregate.

Nice! Ebiten is a super nice API for Go. Lots there to be inspired by in API design. Another API I like a lot is Love for Lua (which also actually can be used from C++).

Re: the comments on here about the GC etc. -- I've posted about this a couple times before but I've been using a custom Go (subset / some changes) -> C++ compiler for hobby gamedev, which helps with perf, gives access to C/C++ APIs (I've been using Raylib and physics engines etc.) and also especially has good perf in WebAssembly. Another nice thing is you can add in some reflection / metaprogramming stuff for eg. serializing structs or inspector UI for game entity properties. I was briefly experimenting with generating GLSL from Go code too so you can write shaders in Go and pass data to them with shared structs etc.

The compiler: https://github.com/nikki93/gx (it uses the Go parser and semantic analysis phases from the standard library so all it has to do is output C++) (it's a subset / different from regular Go -- no coroutines, no GC; just supporting what I've needed in practice -- typechecks as regular Go so editor autocomplete works etc.)

A game made with it a while ago for raylib game jam along with the raylib bindings and other engine stuff: https://github.com/nikki93/raylib-5k

Very old video of a longer term game project I'm using it on which also shows state-preserving reload on rebuilds and also the editor inspector UI: https://youtu.be/8He97Sl9iy0?si=IJaO0wegyu-nzDRm (you can see it reload preserving state across null pointer crashes too...)

Hello world just needs to call console.log, so doesn't need libc. Here's an example that builds without libc / emscripten to produce a very small wasm hello world: https://github.com/nikki93/cxx-wasm-freestanding There's actually some other stuff in there right now but console.cc is the main thing -- it calls consoleLog which js exposed to wasm from the js code at https://github.com/nikki93/cxx-wasm-freestanding/blob/master... .

You do need some JS code that asks the browser to run the wasm blob. You can't eg. just have a script tag that refers to a wasm blob yet.

libc does help with things like having an allocator or string operations etc., or for using C libraries that use libc. And that's where emscripten becomes helpful.

Browser functionality like the console or making html elements is exposed through JS interfaces, and the wasm needs to call out to those. But they may be directly exposed to wasm later (or they may already be at this point in new / experimental browser features).

Does Grasshopper in Rhino count?

I do also really like Go for various reasons, and have been working on a Go -> C++ transpiler and associated ECS libs to make a personal game project with. I used it to make a game for Raylib game jam earlier this year too: https://github.com/nikki93/raylib-5k You can see what the development workflow looks like in this video (the ECS stuff also has a built-in editor like a much more minimal version of Unity's): https://www.youtube.com/watch?v=8He97Sl9iy0

I'm trying to decide how much time I should devote to making this easier to set up / use by other people in the medium term, since it's just a side project for me. Might make a codespaces template so it's quick to get started.

I've been using my own little Go (subset / my own extensions) -> C++ compiler -- https://github.com/nikki93/gx -- and found it to be a fun way to add some guardrails and nicer syntax over C++ usage. You get Go's package system and the syntax analyzers / syntax highlighters etc. just work.

Re: perf for hobby gamedev, I basically agree for native builds, but lately I've felt like Wasm support seems key for hobby gamedev (so you can have more people play your game / without downloading it / it works directly on mobile too without dealing with app or play store). And Go perf in Wasm unfortunately is not so good (I was hitting big GC pauses when trying to make a game with Ebiten and large images).

I ended up writing a Go -> C++ compiler. Go's standard library parser/typechecker made it very doable. The games I've done with it don't use the GC at all but also don't manually manage memory -- they use an ECS api which helps. https://github.com/nikki93/gx -- the README links to development workflow video and complete example game code. I get the perf, interop/libs and portability of C/C++ but with Go's developer experience (well the build system involves C/C++ of course but I have something set up there that I now just use).

For the client I use a simple go -> c++ compiler I wrote and compile to wasm from that actually, on my side projects. It had zero overhead interfacing to / calls to C/C++ (including generics<->templates) since it's just generating that. Example web game made with that: https://github.com/nikki93/raylib-5k

I think I've seen wasmtime before. If I needed to interface to any C/C++ things on the server I would probably just write in C/C++ (or Gx) yeah.

Personally I'm not super familiar with its benefits if any on the server and would actually not use it on the server myself and just build binaries directly, probably using Go. But I've seen some references to Wasm on the serverside for something similar to containerization or loading plugins. It does seem less obvious to me than the client side.

What makes you say Wasm is a server side runtime / imply that it's meant to be one?

GC implementations over Wasm definitely have perf issues. eg. Go compiled to Wasm has these with its GC. The main way when compiling to Wasm now is to not use a GC, which is reasonable depending on the application in question. It's quite reasonable for games or graphics applications.

But yeah Wasm has definitely been used successfully with performance results in production, eg. in Figma: https://www.figma.com/blog/webassembly-cut-figmas-load-time-...

The extent of Wasm's availability on browsers is quite big right now. Both iOS and Android and all major desktop browsers. That sort of reach is what I meant by the term "the browser" used generically. Different from being an extension to one or a few browsers or something like that.

One of the things that killed Flash was not being supported on iOS while Wasm runs fine (and actually pretty well!) on iOS which is pretty good for reach.

I think you missed the part about it running in the browser? But yes, some previous technologies are similar to some new technologies, that's not really an insight at this point. Especially not about bytecode interpreters which seems like a standard practice.

The main benefits are being able to use the same codebase for native and web builds and also not needing to use a GC and being able to do memory management yourself. The latter stuff really makes a positive difference for Wasm in eg. graphics / games. Can be less pronounced if the app mainly does DOM manipulation potentially due to the API boundary.

For DOM you can get pretty far binding http://google.github.io/incremental-dom/ to C (it's all just procedure calls).

But other than that there are OpenGL bindings (eg. in emscripten) and things for audio like SoLoud etc. You can usually put off writing JS for a while.

You can compile C, C++ and a bunch of other languages to Wasm, while you can't do that with Java as a target. A lot of existing C code compiles unchanged. And Wasm also actually runs in all browsers nowadays without the user noticing a difference (eg. iOS Safari too).

Sweet! I've been using it for the same. Example game project (did it for a game jam): https://github.com/nikki93/raylib-5k -- in this case the Go gets transpiled to C++ and runs as WebAssembly too. Readme includes a link to play the game in the browser. game.gx.go and behaviors.gx.go kind of show the ECS style.

It's worked with 60fps performance on a naive N^2 collision algo over about 4200 entities -- but also I tend to use a broadphase for collisions in actual games (there's an "externs" system to call to other C/C++ and I use Chipmunk's broadphase).

I've been using an immediate mode api that generates html elements in web, so that you do get the things that html offers. I basically have a thin wrapper around http://google.github.io/incremental-dom/ (I'm using it from wasm) -- calling it per frame has actually been reasonable perf-wise.

I've liked Raylib lately for writing graphicsy applications. Here's a small example that does 3d, runs in the browser, code shown below: https://www.raylib.com/examples/core/loader.html?name=core_3...

Here's a little template that uses Raylib: https://github.com/nikki93/raylib-template

You can render OpenGL with C++ and compile it to Wasm but alsk have it build and run natively (Emscripten or your own bindings or whatever). So this can be a way to do both, while keeping the reach of web. Depends on what kind of application you're building and whether this makes sense for that, for sure.

I also tried running vanilla Go in Wasm (one of the targets I want to support) directly and was hitting GC pauses and frame drops every second. Garbage generated due to temporaries has also been an issue in an engine I worked on that had Lua scripting. This isn't just a theoretical thing, it's clear it's a problem. When you say "programs that are tight but" and "allocation discipline" it's like why add the cognitive overhead when you can not have the problem at all. I want to / want people that use this engine to be able to just write naive / straightforward code and get a high perf ceiling off the bat. GC is not actually helping in the game case since game logic is ultimately explicit about entity lifetimes.

So there isn't a reason to use it and then work around it. If anything I think it's because the ergonomic language work after C++ (C#, scripting languages) tended to include GC so using them meant having it. This is an exploration in having a language that doesn't do that and preserves ergonomics (and it's working / promising).

All that said, the GC thing isn't the main or only reason that motivated this approach, it's just one of the points among everything else. Portability (the resulting C++ compiles and runs in Wasm, native desktop, mobile is supported) and control (being able to decide language and resulting execution semantics, having direct integration with tools) are the main things, at a high-level.

I don't think goroutines / channels is a clear win or even that relevant for data-oriented gameplay code. I go into this more in-depth in this comment: https://www.reddit.com/r/golang/comments/r2795t/i_wrote_a_si... (it addresses a bunch of points)

This is what the game code looks like: https://gist.github.com/nikki93/0425d9ead9eb7810075434d006f3... -- I don't think CSP helps much to improve on that while keeping serializability of state.