Crudely? Because you can't grep a sequence of latent states for variants of "If I kill all the puny humans, I can <achieve my current goal>."
3,071 karma · joined November 27, 2008
In 2000, I was the 20th-or-so full-time engineer at VMware, where I worked for 9 years. Then was at Facebook from 2009 to 2016, where I worked on the search backend (now replaced), HHVM (which still runs the Big Blue Application, a shrinking portion of the Meta Empire), and started FAIR in 2015 (which finally seems to have turned around the "open" sign with Yann's departure).
In 2016 I started at Slack as Chief Architect, where I mostly did not write a ton of code. I worked on a job queue scheduler which I would not be surprised to find has been replaced. And after that I was mostly encouraging/advising people doing Real Work.
All of which is to say, it is quite possible that the last code I've worked on professionally that is out there running on customer machines ... is that libpthread mutex bug fix from when I was barely old enough to drink.
We needed to port all the user-level fork(2) calls to vfork(2) when working on uCLinux, a port of Linux to MMU-less microcontrollers[1]. It used to be that paging MMUs were kinda expensive (those TLBs! so much associativity!!!), and the CPU on your printer/ethernet card/etc. might not have that much grit. Nowadays not so much.
Still. A hard-and-fast use for vfork(2), as requested perhaps.
From the article it doesn’t seem as though photorealism per se was a goal in training; was that just emergent from human preferences, or did it take some specific dataset construction mojo?
When Diego first showed me this animation, I wasn't completely sure what I was looking at, because I assumed the left and right sides were like composited together or something. But it's a unified screen recording; the right, generated side is keeping pace with the riffing the artist does in the little paint program on the left.
There is no substitute for low latency in creative tools; if you have to sit there holding your breath every time you try something, you aren't just linearly slowed down. There are points that are just too hard to reach in slow, deliberate, 30+ second steps that a classical diffusion generation requires.
When I first heard about consistency, my assumption was that it was just an accelerator. I expected we'd get faster, cheaper versions of the same kinds of interactions with visual models we're used to seeing. The fine hackers at Krea did not take long to prove me wrong!
The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct...) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovatio...), the idea that each tenant is fully isolated was increasingly false.
Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-v...
HHVM has an embedded web server (the folly project's Proxygen), and can directly terminate HTTP/HTTPS itself. Facebook uses it in this way. If you want to bring your own webserver, though, FastCGI is the most practical way to do so with HHVM.
[1] https://www.zdnet.com/article/slack-brings-shared-channels-t... [2] https://sec.report/Document/0001628280-19-004786/
The toolchain for HHVM is all installed as a single big deliverable, which gives you the language engine and supporting runtime libraries itself, an in-address-space web server (Facebook's Proxygen), a debugger in the form of hhvm -a, and the Hacklang toolchain accessed via hh_client and appropriate editor/IDE integrations.
I share your intuition that there is actually a glittering core of "stuff-that-makes-you-successful" hiding in the incidental complexity of PHP, and we wrote this blog post trying to put some substance behind that intuition: https://slack.engineering/taking-php-seriously-cf7a60065329
I've seen Hack used in a multi-tens-of-MLOC codebase to gradually insert types. It made huge differences in the kinds of changes that were possible; you can, e.g., rename a class or method, or change the order of its arguments, with confidence comparable to that in a C++ codebase. Most developers hack-ified everything they could get their hands on, and did all new work in Hack, without any external encouragement.
I was hoping that their might actually be a linear type system hiding out in there, but alas, such is not the case.
The theoretical hand-wringing about SGD for neural nets is that their loss surface isn't convex. It turns out this doesn't matter. The loss surface is a high-dimensional egg-carton, and you need to get winning-the-lottery-while-struck-by-lightning unlucky to find a significantly shallow local minimum. There are lots of saddle points, and you need to do something to drive out of those, but stochasticity seems sufficient in practice.
Rotation-invariance is probably not really a thing you want. The visual world is not, in fact, rotation-invariant, and the "up" direction on Earth-bound, naturally-occurring images has different statistics than the "down" direction, and you'd like to exploit these. Animal visual systems are not rotation-invariant either; an entertainingly powerful demo of this is "the Thatcher Effect" (https://en.wikipedia.org/wiki/Thatcher_effect).
Reflection across a vertical axis, on the other hand, often is exploitable, at least in image recognition contexts (as opposed to, say, handwriting recognition). If you look at the features image recognition convnets are learning they are often symmetric around some axis or other, or sometimes come in "pairs" of left-hand/right-hand twins. As far as I know nobody has tried to exploit this architecturally in any way other than just data augmentation, but it's a big world out there and people have been trying this stuff for a long time.
The win here was about individual page load time. And page load time is just as important, if not more so, for something new trying to vigorously grow as it is for the big sites.
(Disclaimer: HHVM alum.)
From my point of view, most of what has changed is resources and the immediacy of our survival-level concerns. Four years ago Google had declared nuclear war on us, we had far fewer users, we were not profitable, there were constant fires to put out with basic production operations stuff we've gotten better at, and we were enormously more under-staffed. I was working on HHVM already, but it was in a million little pieces spread across Drew's, Jason's, and my desks. The tools we're open sourcing over the last two years mostly did not yet exist, and if they existed, it was in some primordial form. We also have gotten much, much better imho at being good stewards of our open source projects; HHVM's predecessor system, the HipHop compiler, was also open source, but we people were spread way too thin to be able to respond to bug reports, pull requests, get FB's latest code into public hands, build binary packages for popular distros, etc. on a timely basis. Huge props are due to all of the technical people on our open source teams.
And yes, you could try to share tracelets whose bodies are identical, but, unless their successor tracelets are also identical, you'd need to "dynamicise" the dispatch so you go down Path 1 when you're really tracelet 1 and Path 2 when you're really tracelet 2. We normally chain tracelets together by either falling through (if it's unconditional and the successor tracelet could be placed right next to it) or with jmp or branch instructions.
As usual, the abstract really isn't enough to draw big conclusions from. The concept of a latent type applies to arbitrary expressions, not just variables. All the values flowing through a PHP program implicitly share the same union type; they can be floats, strings, arrays, etc. The latent types are the narrower types that can actually flow through the program in practice.
To be clear, it is more general than just things like:
$a = 0; // $a is an int!
g(foo() . bar());
The scientific revolution in computer vision right now is due to deep convnets, trained in a supervised way using backprop and SGD. All of the systems we're talking about that have started blowing away records are members of this family, and were trained this way. If Alex Krizhevsky had not entered ImageNet 2012, we would not be having this conversation (in part because I probably would never have gotten curious enough about it to leave my home territories of systems and programming languages). Second order methods, unsupervised pre-training, RBMs, graphical models, etc. etc. etc. were exciting for those inside the field, definitely provided encouragement to those optimistic about deep models, and still might prove important, but they have had little impact and visibility to skeptical people outside, in the way that entering a computer vision competition and murdering all the computer vision systems did.
WRT the Glorot and Bengio paper, it’s true that there was a lot of excitement surrounding unsupervised pre-training of DNNs, but this mostly preceded the current wave of successes. The big differences between the architectures that are working on image processing today and that paper are:
1. Moar data. The datasets this paper was looking at were on the order 10^5 or so images. 10^6 is a different ballgame.
2. Convolution. Sharing weights really is special. This means there are far fewer parameters to learn in the early parts of the network, and so pre-training seems less necessary.
3. RelU activations. The survey of activation functions uses only smoothly differentiable ones whose gradients get tiny as you are far from zero. RelU has fewer problems with the gradient getting tiny or huge at idiosyncratic points, and also has the virtue of sparsifying the gradients as you backprop (since anything that landed in the negative tail has zero gradient).
So yeah, we really do do entirely unpretrained learning of low-level features, straight from RGB values between 0 and 256, and it works! Isn't that cool??