1,589 karma · joined October 12, 2016
Researcher at https://www.sift.net/
Email is concat ["me@", username, ".com"]
Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z)
in Decoder step 7 instead be Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z_self_attention)
? Also, is layer_norm missing in Decoder step 8...In code I'd consider high-quality, there'd be a comment like this [1]:
// SAFETY: inner is only mutated by foo, bar, and baz, all of which
// ensure the pointer is valid.
[0]: https://doc.rust-lang.org/std/primitive.pointer.html#method....[1]: https://std-dev-guide.rust-lang.org/policy/safety-comments.h...
Doesn't OpenJDK install a SIGSEGV handler, and generate the exception from that on a null dereference?
(AFAIK, a lot of runtimes for GC'd languages that support thread-based parallelism do so anyway, because they can use mprotect to do a write barrier in hardware.)
98 won't have moves, so I'd imagine you'd end up boxing most of your structures to avoid needing to copy construct then all over the place.
You still have to manually delete things but now there's extra indirection.
Templates might be nice, but having STLy containers is where the kernel would need them most, but then above boxing issues start to exist.
From what I know of each, they're fairly close together, but a few small things like the clever encoding of logical immediates (e.g. and x1, x1, 0x3fff70000000 is a single instruction; you don't need to waste a register to load the immediate) in AArch64 make it seem friendlier to me.
So like, at the end of all the decoders, the model gives you an output vector; you multiply this by your embeddings to get your token probabilities, then you sample from them to choose a token.
Instead of sampling, you could just look at the probabilities for the tokens "positive" and "negative" and return whichever of those two is highest.