424 karma · joined December 6, 2010
self.w2(F.silu(self.w1(x)) * self.w3(x))
I.e. the nonlinearity is a gate.https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
Log also means a "look" is not baked into the image so, since you're starting from scratch, it's 1) easier to tweak the images so you can cut between two cameras from different manufacturers without distracting differences and 2) you can give the image more of your personality.
As a general note, I've found that in the world of "cinematography", tech terms aren't used very rigorously and there's a lot of cargo cult which comes from the benefit of one tech being conflated as a benefit of something else. It's often hard to sift through the noise when learning.
This is true of tools like excel, but for things like server infrastructure (at least in big tech companies) they are very hesitant to put something into production without at least source code access (much preferring being able to compile it themselves).