nil-sec

Sorry maybe I should have added more explanation. One way to think about attention, which is the main distinguishing element in a transformer, is as an adaptable matrix. A feedforward layer is a matrix with static entries that do not change at inference time (only during training). The attention mechanism offers a way to have adaptable weight matrices at inference time (this is implemented by using three different matrices, K,Q & V called keys query and value in case you want to dig deeper).

I guess I was more thinking about self attention, so yes. The more general case is covered by your notation!

Feedforward: y=Wx

Attention: y=W(x)x

W is Matrix, x & y Are vectors. In the second case, W is a function of the input.

I had the same experience a couple of years back when people started discussing AI. I try to keep this in mind but somehow I keep forgetting. It’s genuinely difficult to filter good from bad takes without expert level understanding of a topic unfortunately.

It’s quite nice to have something irreversible though. It gives you time. Also nobody really thinks QM is the end of it, assuming semi classical physics under the hood is just odd to me. There is something below QM that we don’t understand yet (AdS/CFT looks like a good start to me) and personally I think the whole interpretation of QM debate will look stupid in retrospect. Yeah collapse is odd, but it just shows us this isn’t it. Reality is much weirder than we thought and giving up on realism is just the beginning.

It’s funny because, for me, this was one of the major confusions when I moved from Europe to the US. In Europe, there are a lot of prominent subcultures, particularly in college. This was totally absent in the US in my experience. Even at places where you would expect it, e.g. underground, hard Techno warehouse raves in Baltimore. Instead the same mainstream people & opinions were there too.

This isn’t true, the quality of images generated by DALL-E are really good, but they are an incremental improvement and based on a long chain of prior work. See e.g. https://github.com/CompVis/latent-diffusion

Insane take, the west vs Russia in open confrontation ends in a nuclear winter.

I have no background in control theory but this sounds very similar to the identifiability problem in nonlinear ICA. Are those equivalent?

This is a good point which I think stems from wrongly equating human level intelligence to AGI in the popular literature. It’s not at all clear what a general intelligence should be and it’s much less clear that humans have general intelligence. In my view we have a set of very good innate priors (e.g. space/time continuity, intuitive physics) that have been optimized by evolution for thousands of years. These priors in turn allow us to learn fast from unlabeled data, but would anyone call such a system general intelligence? I’m not sure.

While I agree with the general point of this paper I don’t think it’s quite right to compare the current situation with the last AI spring. It’s not AGI but it’s very good narrow AI that has real commercial value right now. The systems back then did not to the same extent, and for this reason I don’t see funding dry up for current ML approaches.

For a given causal model it is (1) in my understanding.

For one it lets you avoid controlling for the wrong variables and causing e.g. spurious correlations by doing so. In fact this is one of the best examples of why a causal model is necessary, because without one you can easily end up with a correlation that doesn’t exist as is illustrated quite nicely in his book.

It is incomprehensible to you, because you just simply do not understand what your parent is talking about. You are the ignorant one here and indeed quite rude. Doesn't matter that genetics is not natural language. The point is we can train large transformers auto regressively and the representation it learns turns out to be useful for a) all kinds of supervised downstream tasks with minimal fine-tuning and b) interpreting the data by analysing the attention weights. There is a huge amount of literature on this topic and what your parent says is quite sensible.

Agreed, I trained a 3D version of b0-b2 on a classification task I worked on and besides being very slow to train they did not outperform a simple baseline VGG architecture. Interestingly training time was much improved by setting the cudnn benchmark flags in pytorch. Haven’t seen this reduction in training time for any other architecture but for b2 it went from about 12 sec per iteration to about 0.8. I guess there is more margin optimizing NAS models.

Location: Zürich

Remote: Yes

Willing to relocate: Yes, within Europe.

Technologies: Pytorch, Tensorflow, Python, C++, JS, HTML/CSS, Gurobi, MongoDB

CV: https://nilsec.github.io

Email: nils [dot] eckstein [at] googlemail [dot] com

Machine Learning Researcher in the last stage of my PhD, where I designed terabyte scale computer vision algorithms for Connectomics (Neuroscience). Looking for Machine Learning Research/Engineer positions and interesting problems.

They could have used all negative samples for testing (and even training if they would have done it better), yes. But once your test set is large enough, whatever that means, its not that relevant anymore. They are anyway "under sampling" by not recording data from all humans that are negative right now.

And no, it's not a strong claim to make. Of course the network learns the distribution of your training set. That's why you want it balanced. But during successive applications of inference the weights do not change, it has no state. So it cannot, for example, store that it just predicted 90% negative and now it would be time again for some positive prediction.

1. Isn't an issue. They make inference on a sample by sample basis. The network has no memory so it won't expect a 50/50 distribution on the test set just because its trained like that. Having a balanced distribution is the exact right thing to do because you do not want the network to be biased to one or the other class for any given sample. If it were unbalanced the network could achieve almost 0 training error by just predicting negative all the time. This is not what you want.

Would love to play around with this data but there are no seeds for this torrent. Anyone here who can provide this dataset?

Let me be very concrete: There is currently a lot of research being done in so called “contrastive learning”. This is an unsupervised technique in which you train a network to build good representations of its input data. Not by explicitly telling the network, “this is a dog” but by telling it “this image is different from that image and similar to this other one”.

How this works, why this works, coming up with the technique itself, are all data agnostic. All you did so far is write down a function f(x) -> y and a loss L(x,y), with x the input and y the output, specifying your model.

Of course you use a specific dataset to train your model in the end and see if it works. But the model and the technique itself are not grounded in any specific dataset and thus nothing in this model perpetuates bias.

Now usually the next step for you as a researcher is to evaluate the performance of your model on the test set. Lets take image net. Now there are 3 situations.

A) Your Train set is biased & your test set is biased in the same way.

B) Your train set is biased & your test set is not.

C) Your train set is unbiased & your test set is biased.

With biased I mean any kind of data issue such as discussed in this article, e.g. no women in the class “doctor”.

In situations B) & C) your model wont work well so you actually have an incentive to fix your data. This will happen to you in production if you train your tracker only on white people say.

Situation A) is likely what’s happening with imagenet and other benchmark datasets. In this case your model learns an incomplete representation of the class “doctor” and learns the spurious correlation that all doctors are men. This will work on the test set because it’s equally biased.

You go, get good test results and publish a paper about it, unaware of the inherent dataset biases. (You could have done all this also on MNIST or a thousand other datasets that do not have any issues with societal biases because they are from a totally different domain, but that’s another point).

In this entire process of coming up with the model, training and evaluating it, there is no point at which the researcher has any interest, or benefits from, working with biased datasets. Furthermore, besides potentially overestimating the accuracy of your model, there is nothing in here that would hurt society or further perpetuate biases. That is because models are generally not designed to work on a specific dataset.

Again, this is a different story when you use your model in production. In this case you are in situation B) or C) and here now lies the crux. If you can make money from this bias or maybe it perpetuates your own biases well you might keep it like that. This should be fixed. Here now is a real argument for why there should be diverse populations working on AI systems that are used in industry.

Of course having diverse populations in science is also our goal. But not to fix our datasets but to do better research.

I happen to work in AI research and what you are saying isn’t true. There is theoretical machine learning and applications of it. They are distinct.

The former is largely task agnostic and deals with fundamental issues such as, how to train networks in an unsupervised way, how do you do hard statistical inference for intractable models in a approximate but well enough manner, how is gradient descent behaving exactly, why does it work, are there better ways for optimizing nns, what about pruning? The list goes on.

On the other hand you have applications of AI to other fields that require their own research. In biology for example you may want to segment cells and their compartments or design better point spread functions for you microscope or classify cell types. These are applied problems.

Again, as stated in my original post, striving for more diversity is a good thing and should and is done. Why make it about AI ethics and bias though when large portions of this field have no contact point with it?

In industry, yes. I had the impression this article was aimed at academia and research in AI.

The fraction of people in AI working on problems that need to consider diversity/fairness/etc. is rather small. Yes, those people, these specific applications, should be designed with care, should be overseen properly and in the best of all worlds should be unbiased. However, the recent discussion around this topic is framed like AI research in general is somehow unethical and should be informed more by minority populations. While this is something to strive for, for the sake of having a more just society, the impact of diversity on an AI system that say classifies mitochondria in yeast is not clear to me. I’d argue the majority of problems in AI right now are of this form and not of the form addressed by the author. If you want to push quotes, increase minority representation for occupations, do so. If you want to address specific biases in very specific applications then do that. Don’t use the latter to make an argument for the former though. These are two different issues and while I recognize the author sees a link here I’m not so certain about that. To be clear I think that both are worthy goals. But it’s somewhat disingenuous to make an inherent political argument, people may disagree with, and justify it by overgeneralizing a specific, niche technological ethics question to an entire field.

This is a very one-sided view of what this article is talking about. That there is systemic racism in the US is not up for debate. It's a fact. Even if your back of the envelope calculation there were correct, you have to ask yourself why there are more black people committing crimes. The reason isn't skin colour, but socio-economic factors which themselves are a result of a long history of oppression.

It’s an interesting description of the space of computation and I quite enjoyed this article even though, as many, I have issues with the way wolfram participates in science. Certainly what he is describing can be a way to view the universe but that is somewhat trivial. It is not surprising that you get something like the universe when you start considering universal computation. The fact that we can write down e.g. equations of motion means there is a Turing machine that computes it. The real test is whether this perspective is useful in the sense that we can predict properties of the universe with it. That test is still out and until it’s addressed there is no way to evaluate or criticize this model. It’s too flexible to be useful at this point.

Turing completeness isn’t necessarily an interesting thing to have in common. Many (very simple) models of computation are Turing complete but have vastly different properties. Take for example a cellular automata, a Turing machine, Wang tiles, (cyclic) Tag systems, Fractrans, Register machines, string rewriting systems. All of these are Turing complete. Yet they are miles apart in how they carry out computation. In order to understand and do what the brain is doing we have to figure out the brains model of computation. It will also be Turing complete but it will look very different than a Turing machine.

Why exactly shouldn’t scientists be politicians? It’s not like most of the people who are politicians have a formal education in political science. In an ideal world the ministers should come from exactly the background they are tasked with. This seems to fit in Sweden. I don’t know why you believe an economist or lawyer would make better decisions here.

This is the point. There are many people at these protests. This is a heated, complex situation and at any large protest the people there are not homogeneous. The majority wants to peacefully protest, some loot, some want violence, the police is aggressive. Given that it is understood that every protest is a complicated social phenomenon, the narrative pushed by major news outlets in the US, and echoed in parts here, that somehow this is led and organised by Antifa, or the far right, is delusional. There are protests all over the US. In suburbs. How do you imagine this is working? The Antifa/Right has some chapters in every suburb of the US, just waiting to be unleashed upon the masses and this is the reason why protests turn violent? It's a gross oversimplification of the actual dynamics, it's a child's story that is easy to tell and easy to digest.

It is utterly ridiculous to try to blame either, far right or far left wing organizations for what is happening in the US right now. None of these groups have the power to cause that much damage. I’d be surprised if the number of antifa/black bloc people in these protest is more than a tiny fraction of people, like in any protest. Yet the discussion is shifted to these groups as an explanation for violence and this entire thread is full of people arguing about whether the Antifa is an organization. The antifa doesn’t even matter. What is happening to this discourse?

I read this when I was very young and had something like a pre puberty existential crisis. Like you, this book and the thoughts expressed therein resonated with me and it is a view of the world that I have held dearly ever since. I always liked it particularly, because it makes life a comedy instead of a drama. Nihilism is much darker and feels very depressive to me. Existentialism is meaninglessness too, but with a smile.