Also discussed a couple days ago: https://news.ycombinator.com/item?id=29795910
1,200 karma · joined July 23, 2021
To be clear that is not being disputed. IMO the central thesis of the paper is:
> the progenitor of Omicron jumped from humans to mice, rapidly accumulated mutations conducive to infecting that host, then jumped back into humans
That hypothesis is primarily supported by statistical evidence related to the frequency of certain amino acid mutations in the viral genome. For example:
> Considering that these two amino acid mutations are uncommon in human patients infected by non-Omicron SARS-CoV-2 variants (0.005% and 0.002%, respectively) we proposed the hypothesis that the progenitor of Omicron evolved in mice.
> searching against the GISAID database using Omicron’s backbone sequence. The top hits were again from the B.1.1 lineage, which differed from Omicron by 31 mutations, indicating that human SARS-CoV-2 variants reported to date could not provide a backbone for Omicron
While the paper certainly presents intriguing preliminary evidence, critically there was no analysis performed which accounted for variation or underestimates in viral mutation rates, number of infected hosts, or gaps in genomic DBs, etc. As such, there are certain hidden assumptions that were not explicitly addressed or conclusively proven.
Moreover, the "pleiotropic effect" is a significant confounding factor:
> predictions showed that the adaptation of Omicron to mice also promoted its adaptation to other species, such as humans, camels, and goats, via stronger RBD-ACE2 interaction (Fig. 6). Such a “pleiotropic effect” of mutations was likely caused by structural similarity of ACE2 across species
Finally, given that selective pressure due to mass distribution of S protein targeted vaccines was also not considered, I think it's fair to say that the evidence pointing to mice is fairly tenuous at this point. But the paper is still valid & interesting (I did not intend to suggest otherwise).
One hypothesis involves the fact that current mRNA vaccines induce an immune response highly targeted toward the spike protein RBD.
In contrast, natural infection induces a broader immune response that includes antibodies targeting other proteins (Nucleocapsid protein being one of the most notable; it is commonly used to determine if a prior infection occurred).
Omicron in particular is a variant that has accumulated a significant number of mutations in the S protein region.
Is this coincidence or are we seeing selective pressure due to mass vaccination campaigns?
> We found that mutations accumulated in ORF S at a rate of ∼0.45 mutations per month on average. In sharp contrast, 27 mutations accumulated in ORF S in Branch O during the 18 months spanning May 2020–November 2021, equivalent to ∼1.5 mutations per month, or ∼3.3 times faster than the average rate of other variants
> We found that the Omicron spike protein sequence was subjected to stronger positive selection than that of any reported SARS-CoV-2 variants known to evolve persistently in human hosts, suggesting a possibility of host-jumping
Although host jumping is undoubtedly occurring (eg infection has been serologically observed in white-tailed deer and other animals), I’m skeptical that it is the primary factor as this paper suggests.
Another possibility is that viral mutation rates are underestimated, and/or infections are systematically underestimated. There is evidence of both in the literature.
Combined with enhanced selective pressure on the spike protein RBD due to mass vaccination campaigns, these are scientifically supported alternative explanations for the observations in OP.
>* The Retrieve Metadata feature uses a Zotero web service to find item metadata. The Zotero client sends the first few pages of text from the PDF to the web service, which uses a variety of extraction algorithms and known metadata from Crossref, paired with DOI and ISBN lookups, to build a parent item for the PDF. The Zotero lookup service doesn’t require a Zotero account and doesn’t log any data about the content or results of searches.*
Does anyone know of open source alternatives which are offline friendly? I’ve tried tesseract and was impressed but the layout analysis and OCR made enough mistakes that tasks like bibliography parsing became considerably more difficult.
It’s alarming because “they don’t believe in science” has become a divisive weapon that hand waves away all scientific nuance.
The vast majority of young healthy people are completely asymptomatic, they don’t need a vaccine, nor would it be a wise investment of limited doses.
Also, previous infection confers protection that is at least as effective as vaccination. This has been consistently observed since the beginning of the pandemic.
Vaccines can save the lives of many vulnerable people though; it’s an amazing tool and its use is certainly justified by science.
But using economic and financial engineering to force everyone to get vaccinated? There is no scientific evidence supporting that decision, and the second and third order consequences are completely unstudied. (evolution of viral resistance is one example; many experts are aware of the phenomenon but commenters here have been misled to believe that an airborne virus circulating the planet can be eradicated ‘if only everyone got vaccinated’)
Genuine talent tends to recognize and avoid these types of orgs, and the vicious downward cycle you describe seems unstoppable, at least in my experience.
To be fair we know that mobile hardware resources can be extremely limited, and I’d wager a server side model will always be bigger and therefore better.
But recently there’s been a lot of amazing progress in techniques to shrink large models down while preserving most of the accuracy (eg quantization aware training, etc).
With some additional constraints on scope (maybe only supporting the handful of languages the user needs?), I believe a sharp team of a few experts could deliver this fairly quickly, with reasonable results that would of course improve over time.
> We don’t know for sure why OpenAI hasn’t shown its work in a more reproducible way. But this step is definitely done to stimulate the further openness and progress of such models.
Super interesting and great commentary, thanks for sharing!
But IMO you’ve raised the important long term consideration - do graph based schemas and query languages obviate the need to model foreign keys explicitly? If this JOIN FOREIGN proposal is an incremental step forward, what’s the next big leap?
Does anyone else feel like a caveman when modeling a many to many relationship in a normalized schema, and then querying via SQL?
I’m surprised graph DBs aren’t more popular for this reason alone. Maybe it’s a far fetched dream, but perhaps a graph frontend can be slapped onto the Postgres backend.
For example, naively searching a string of length N for P different substrings will yield an algorithm that has roughly O(N * P) worst case time complexity.
But implementing that algorithm as a DFA yields an efficient construction that has O(N) worst case time complexity.
This results in considerable speedups for complex pattern recognition tasks on large inputs, and yields other useful properties.
As a real world example, a CSV file can be parsed in parallel on a GPU by using a DFA based algorithm [1].
One downside to DFA representations is that they can be difficult to work with as linear text in your editor. Tikz produces great visualizations but I feel the syntax is a bit cumbersome. I’m still searching for that perfect text based format for DFAs - ideally one that can be visualized but also executed.
[1] ParPaRaw: Massively Parallel Parsing of Delimiter-Separated Raw Data https://arxiv.org/abs/1905.13415
> However, the risk of myocarditis following vaccination is consistently higher in younger males, particularly following a second dose of RNA mRNA-1273 vaccine.
This is an important reminder for policy makers and the public that vaccination policy cannot be reduced to a simple dichotomy.
Vaccines can be beneficial for the overall population AND present excess risk for certain sub-populations. Stratifying by age and sex is essential for discussions on this topic to be accurate and productive.
Moreover, it takes a long time and a large sample size for these anomalies to be detected with significant statistical power. Let's do our best to avoid frustrated bickering, we need all the patience and empathy we can muster.
To keep things simple, I'd say the true "primitives" of ML models can be reduced to mathematical formulas. For example, a plain old feed forward network is implemented as matrix multiplication. Sprinkle in a bit of calculus to analytically derive the formula for back-propagating errors (aka training), and you have the basic building blocks of modern deep learning. Convolutions, Transformers, etc are just a bit fancier spins on the same mathematical foundations.
Hyper-parameters are essentially tunable variables in a formula. I'd say your instinct is spot on - they are absolutely necessary to capture for reproducible results.
If you have the code and the data the answer should be yes. You should be able to take that PyTorch code and translate it to MyFavoriteMLToolkit to obtain numerically identical results.
In practice, we face the same universal difficulties as other computer science based research: fighting inconsistencies in software, hardware, all the way down to the physics of the universe with cosmic ray induced bit flips, etc.
According to the latest ImageNet standings [2], ViT appears to have slipped to second place in Top-1 Accuracy. CoAtNet-7 is the new leader, but only by a slight margin and at the cost of what appears to be a significantly larger model.
[1] Scaling Vision Transformers https://paperswithcode.com/paper/scaling-vision-transformers
[2] https://paperswithcode.com/sota/image-classification-on-imag...
https://news.ycombinator.com/item?id=19054501 (Feb 1, 2019) 411 points, 23 comments
https://news.ycombinator.com/item?id=23391934 (June 2, 2020) 304 points, 21 comments
[1] Attention Is All You Need (2017) https://paperswithcode.com/paper/attention-is-all-you-need
Introduced the Transformer architecture and applied it to NLP tasks.
[2] The Annotated Transformer (2018) https://nlp.seas.harvard.edu/2018/04/03/attention.html
An “annotated” version of [1] in the form of a line-by-line Pytorch implementation. Super helpful for learning how to implement Transformers in practice!
[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018) https://paperswithcode.com/paper/bert-pre-training-of-deep-b...
One of the most highly cited papers in machine learning! Proposed an unsupervised pre-training objective called masked language modeling; learned bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.
Bonus: https://nlp.stanford.edu/seminar/details/jdevlin.pdf
See the above slideshow from the primary author, noting the remarkably prescient conclusion: "With [unsupervised] pre-training, bigger == better, without clear limits (so far)"
[4] Conformer: Convolution-augmented Transformer for Speech Recognition (2020) https://paperswithcode.com/paper/conformer-convolution-augme...
Proposed an architecture combining aspects of CNNs and Transformers; performed data augmentation in frequency domain (spectral augmentation).
[5] Scaling Laws for Neural Language Models (2020) https://paperswithcode.com/paper/scaling-laws-for-neural-lan...
Arguably one of the most important papers published in the last 5 years! Studies empirical scaling laws for (Transformer) language models; performance scales as a power-law with model size, dataset size, and amount of compute used for training; trends span more than seven orders of magnitude.
[6] Language Models are Few-Shot Learners (May 2020, NeurIPS 2020 Best Paper) https://paperswithcode.com/paper/language-models-are-few-sho...
Introduced GPT-3, a Tranformer model with 175 billion parameters, 10x more than any previous non-sparse language model. Trained on Azure's AI supercomputer, training costs rumored to be over 12 million USD. Presented evidence that the average person cannot distinguish between real or GPT-3 generated news articles that are ~500 words long.
[7] CvT: Introducing Convolutions to Vision Transformers (May 2020) https://paperswithcode.com/paper/cvt-introducing-convolution...
Introduced the Convolutional vision Transformer (CvT) which has alternating layers of convolution and attention; used supervised pre-training on ImageNet-22k.
[8] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (Oct 2020) https://paperswithcode.com/paper/pushing-the-limits-of-semi-...
Scaled up the Conformer architecture to 1B parameters; used both unsupervised pre-training and iterative self-training. Observed through ablative analysis that unsupervised pre-training is the key to enabling growth in model size to transfer to model performance.
[9] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Jan 2021) https://paperswithcode.com/paper/switch-transformers-scaling...
Introduced the Switch Transformer architecture, a sparse Mixture of Experts model advancing the scale of language models by pre-training up to 1 trillion parameter models. The sparsely-activated model has an outrageous number of parameters, but a constant computational cost. 1T parameter model was distilled (shrunk) by 99% while retaining 30% of the performance benefit of the larger model. Findings were consistent with [5].
[10] ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing (August 2021) https://paperswithcode.com/paper/prottrans-towards-cracking-...
Applied Transformer based NLP models to classify & predict properties of protein structure for a given amino acid sequence, using supercomputers at Oak Ridge National Laboratory. Proved that unsupervised pre-training captured useful features; used learned representation as input to small CNN/FNN models, yielding results challenging state of the art methods, notably without using multiple sequence alignment (MSA) and evolutionary information (EI) as input. Highlighted a remarkable trend across an immense diversity of protein LMs and corpus: performance on downstream supervised tasks increased with the number of samples presented during unsupervised pre-training.
[11] CoAtNet: Marrying Convolution and Attention for All Data Sizes (December 2021) https://paperswithcode.com/paper/coatnet-marrying-convolutio...
Current state of the art Top-1 Accuracy on ImageNet.
It’s worth noting that GP may have been referring to the physical distribution of active ingredients within the mushroom body. The “cap” of a mushroom almost always contains significantly more active ingredients.
Thus, 1 gram of stems may yield a significantly different dose than 1 gram of “caps”.
> We encourage you to take an attitude of curiosity and acceptance toward everything that happens during your session. Whatever comes up has some kind of meaning or wisdom that you can learn from, even if this meaning is not immediately obvious.
> You may have bizarre sensations and experiences, and you may experience frightening images or thoughts. These may alternate rapidly. This is normal and does not mean anything is wrong.
> We encourage you to “go with” or surrender to difficult experiences rather than fight them. Approach rather than flee; accept rather than reason away, lean into whatever comes up, including any impulse to run away.
> There are methods for grounding and calming yourself, if you would like to use them.
Do you prefer smoking over edibles because of the instant rush versus a slow ramp up?
So many people claim that Cannabis helps them sleep but the science just doesn’t support that idea.
Smoking a gram or two would be enough to get a newbie high as hell for the entire day. For chronic users it’s more like a cup of coffee that wakes you up for the day. And in that respect, many chronic stoners are forever chasing that feeling of the first time they got high. You can feel that way again, just take a break!
> It was not our priority early on. Now that we have had a product up and running successfully for many years, it’s time to pay more attention to it.
This is a hard earned lesson but one that is firmly rooted in the reality of the industry IMO.
I’ve had the good fortune of working on a few greenfield products and bringing them to market. Time and time again I see technical and executive leadership steer teams in this direction. The fact is that writing good tests is hard and expensive. And most new software products survive less than a handful of years.
Accepting this reality will help ease the pain of working on that spaghetti codebase at your day job. At least it did for me.
Good on stackoverflow for paying back some tech debt in their successful product. I certainly appreciate the tradeoffs that come with building a business, but they can be difficult to swallow for engineers who take pride in producing high quality and well tested code.
Oof that Fastly XSS seems particularly nasty, and no mention of the bounty.
Am I understanding correctly that any website hosted on Fastly was vulnerable to XSS because of this?
Personally I find the leading hypothesis a bit absurd or at least unbelievable. The idea that colliding ice crystals shed electrons and form ionized air in masses great enough to trigger the most energetic phenomenon on the plant seems … like a Newtonian hammer looking for a nail.
It’s well known that the sun emits a powerful electric field which causes the “northern lights” to appear at the poles of the planet. And it’s well known that solar activity is correlated with large scale weather patterns and atmospheric thermodynamics.
How is the Sun not a viable explanation for the origin of lightning? Cosmic rays initiating an electron cascade is interesting, but I would expect the sun to play a bigger role than that.
Contemporary machine learning instead prefers to create complex model architectures and loss functions which imbue certain inductive biases that guide a model toward learning useful transformations.
In this way, modern machine learning is just “meta” feature engineering. It’s one level abstracted, but still feature engineering under the hood.
Of course feature engineering will always be around, especially in the abstract sense of “transforming data”, which the OP alludes to with examples like transforming CCD sensor readings to RGB pixel space.
Viewing organizations and software engineering through this lens has been a game changer for me personally. As an individual contributor I now appreciate horrendous technical debt and glacially slow development as systematic organizational issues, rather than blaming burnt out or lazy engineers.
Now I’m much more aware and critical of technical and executive leadership. “A fish rots from the head” is one of those quotes that resonates with me more and more everyday.