kirillkh

Replace "New York" with "Tel Aviv", and nothing else needs to change.

> There is a dissertation in psychoanalytics just waiting to be written.

Just don't wear a Hawaiian shirt while giving interview about it on TV.

I used to be like that when I was younger and in earlier stages of my career. As time passed, the pride of achieving the status worn off, as did the novelty of problems I was solving. Many problems turned out to be reoccurring, so I moved on to harder and less mundane problems. I keep doing that to this day - moving away from mundane work to something new and/or more difficult.

At the same time, as I grew older, my social life improved, and I learned to understand humans, so that part became easier.

From my experience as a programmer, while some level of communication is certainly important, most of my time and effort is spent concentrating alone at the task at hand. That's the central part of the job, communication only supports it. Communication is necessary to divide work across the team and exchange experience; it also provides some psychological relief and motivation. While all of these are important, I consider the concentrated mental effort a far more important and difficult part.

> There's no evidence supporting the supposition that there is an innate ability gap.

First, this is a straw man argument: I never argued for "innate ability gap". I argued for "innate affinity", which I understand as (quoting myself) "they do not like working in it".

Second, I never claimed there was evidence to support the correctness of "innate affinity" argument. I only claimed that it is a possibility, and OP should not have ignored it.

Third, there is no consistent evidence supporting "social explanations", and that's why people resist attempts to "change the field to be more welcoming" at the expense of hard-working, deserving white males.

> any argument for innate characteristics would have to explain why the rates started going down in the 1980s despite the field becoming increasingly popular

http://slatestarcodex.com/2017/08/07/contra-grant-on-exagger...

It's when you talk about young men in kinder gardens that you realize how ridiculous this whole discussion truly is. There will never be 50:50 young men to young women in kinder gardens. Not even 20:80. That much is obvious.

Most young men do not like to be with children. Most young women do not like working in car repair. While none of these claims are sufficiently substantiated in research, if the first can be true, then surely the second one can be, as well?

Addressing your main points.

> why we have any reason to believe that programming is a “masculine” profession

By exclusion: we have checked everything else we could think of and found no other logical explanation for the disparity of sexes in STEM. That doesn't mean women's preference is the true underlying reason, but then, we don't have a better explanation, or even any other explanation consistent with facts. Still, AFAIK, Damore never claimed it was THE reason, he just raised it as a possible and the likeliest explanation - given no other explanation seems to work.

> But in India, the vast majority of teachers are men.

I don't think India is a valid example here, because there is still a lot of inequality in that society. Let's talk about countries on the higher end of the equality spectrum, like Finland or Sweden.

> Also, the truck analogy has been debunked

[Source missing]

Not sure how from

> those fields are cognitively more demanding than commercial software development or, for that matter, undergraduate computer science

... you arrive at

> No cognitive ability or innate affinity explains the degree of disparity in computer science as practiced in industry.

Even if software development is "cognitively less demanding" in every sense (though I'm not convinced there is just one universal kind of cognitive ability), it may still be that women do not possess the "innate affinity" for it - namely, they do not like working in it, preferring other fields instead. To my understanding, there is nothing to contradict this explanation, and it makes perfect sense.

Not sure what all of this has to do with the linked article, which discusses purely the scientific merit of claims of bias in STEM.

Score: 5, Funny

Soderbergh's Schizopolis has a lot of scenes with fake or, better yet, subtext language. One of my favorites: https://www.youtube.com/watch?v=9eOLEdTG7Gc

How about the moral imperative "we should treat men and women the same", as opposed to "women are heavily oppressed, we must discriminate in their favor"? Would that imperative be justified by stating a difference?

Perhaps it's better to set it to 2 years, then. A clock that's counting time towards its own death. There is some grim poetry in that.

Do you just use development boards with these? And which ones?

Can any of these be programmed with Rust?

Ah, you're right, word2vec assigns one vector to each word, as opposed to one vector to each meaning. Then the problem remains: we can't differentiate between homonyms.

But it seems it's been solved, too: https://github.com/sbos/AdaGram.jl

> Finally, conceptual search can also help with precision - by matching across all concepts within a document, you can help disambiguate its meaning, when you have words that have multiple meanings

Thanks! I've written up something along these lines here: https://news.ycombinator.com/item?id=15592196

I'd love to hear your opinion if this is going to work.

Actually, I have an idea, albeit not without some doubts.

Let x1 be the number of vectors matching A, x2 the number of vectors matching B, etc, till xn. Let c1..cn be a particular selection of vectors. Now my main assumption here is that in order to determine which of these vectors are most often encountered together in the same context [1], our goal is to find j that maximizes sum_{i from 1 to n, i!=j}[d_i], where d_i=(c_j dot c_i) if the dot product is nonnegative, otherwise d_i=0. I'm not sure it's true primarily because I don't know if by summing up these dot products we add apples to apples or apples to oranges.

Then in order to find the best selection of vectors c1..cn we can iterate on every vector v_k matching A and dot v_k with every vector matching B, then pick the maximum m2 (or 0 if it's negative); dot v_k with every vector matching C, then pick the maximum m3; etc. Thus, for k'th iteration we obtain the selection of vectors that maximizes M_1k=sum_i[m_i]. After we're done with all c1 iterations, we pick the best such selection M1=max_k[M_1k]. This is all done in O(x1(x2+x3+...xn)) time.

Next, we repeat the above process for all x2 vectors matching B and obtain M2, etc, etc. Ultimately, we pick the selection of vectors that produced the highest M_t across all choices of t. Overall, we get O((sum_i[xi])^2), which seems fast enough. What do you think?

[1] One obvious problem is this limits the number of contexts we match against to just one.

Also, this makes me wonder what other things you can do with vectors. If you compute dot product between a verb or a noun with vector "singular"-"plural", will it give a positive value for plurals and a negative for singulars (or vice versa)?

Thanks! That paper is extremely helpful. Still, there is one thing missing to complete the picture for me right now. At the input, I have a list of words that I want to index or query. When indexing, they usually form a sentence, when querying, they might just be keywords. But in both cases the words will usually be selected by the user/author in such a way that a human that reads all the words from the input together is able to disambiguate the meaning of every word. This is precisely what I'm missing.

Let's say the user entered three words: A B C. You look up each of them among the vectors and discover that there are three matching vectors for A, four for B and five for C (and for the sake of generality let's assume that there are more words than just 3 in the input, so it's impractical to test every subset of these words for co-occurrence). How do you jointly select the correct vector for each of the words?

Thanks, I know about HebMorph. Its authors don't want it to be used for commercial purposes (at least for free), so that limits its usability beyond simple experiments. As to your second link, it confirms my suspicions that lemmatizing is important for Hebrew, but the code they reference in the footnotes is equally hostile to commercial usage. I was really hoping word2vec or other new tools would enable building lemmatizer from scratch without much hassle.

Thanks for your advice, anyway.

Please forgive me attempting to milk as much as possible from this discussion - I just don't have many opportunities to get useful advice on this subject, and I've been mulling over it for a long time.

> But maybe lemmatization would be better than stemming

You're right, I'm using "stemming" and "lemmatization" interchangeably where I shouldn't. What I mean is lemmatization.

> It is also possible that it is an unnecessary step for clustered word vectors for your use case

I don't focus on a specific use case, I'm just trying to find a way to enable full-text search for Hebrew. Searching based on concept similarity is a very cool addition, though, and I do have some use cases in mind for it specifically. But I'm just thinking what a typical cluster would look like, and I imagine 99.9% of it will be different forms of the same handful of base forms. Furthermore, telling Lucene to match based on all these forms will inevitably create a large number of false positives due to the aforementioned abundance of homonyms. So I can see a clear problem here even now. That's why I keep reiterating my original question of whether this system can first be used for lemmatizing and then everything else.

4) But recall that the language in question is such that stemming is hard. If you expand every form of every word in Hebrew, you obtain something like 600,000 words. And many of them have completely different meanings due to syntactic coincidences and short roots. So, ideally, the first step would a) determine what exactly each word in this document means in the given context, b) replace it with an unambiguous identifier.

For example, in Hebrew the word BRHA can mean several things: "pool", "blessing", "in soft" and "her knee" (no kidding).

Some clarifying questions:

1) Do words in a generic corpus (such as Wikipedia) actually form well-separated clusters?

2) Is it correct that you find word clusters in the corpus as a preprocessing step (as opposed to at indexing or query time)?

3) Do I understand correctly that you use all words in clusters as synonyms and pass them to Solr at query/indexing time? Is it query time, index time or both?

4) Given a language where words have many syntactic forms (e.g. buy-bought-buying), how does it work with clusters? Do both syntactic forms and synonyms end up in the same cluster? Wouldn't it be beneficial to treat many of these different forms as the same word (i.e. perform stemming) and only list truly different, but closely related concepts as synonyms?

Thank you! This sounds exactly what I was imagining. Very exciting!

If I understand correctly, this forgoes Lucene entirely. I would really like something that can be integrated into Lucene/Solr due to the availability of all the infrastructure build around it.

> works faster than grep

I didn't quite get the connection to grep.

How can this be used for full-text search, e.g. with Lucene? The first step in indexing a document for full-text search is reducing each word to its base form, and similarly for a search string. While it's not a difficult problem in English, in some languages (e.g. Herew) it's notoriously hard to figure out the base form of a word and further disambiguate its meaning, as the only way to do so is based on context. So how can you easily build a stemmer/lemmatizer on top of these instruments to perform such task?

Then we have no hope of ever engineering Mars to have enough air for us to breath?

Asking as someone who barely has any clue in this field: is there a way to use this for full-text search, e.g. Lucene? I know from experience that for some languages (e.g. Herew) there are no good stemmers available out of the box, so can you easily build a stemmer/lemmatizer (or even something more powerful? [1]) on top of word2vec or fastText?

[1] E.g., for each word in a document or a search string, it would generate not just its base form, but also a list of top 3 base forms that are different, but similar in meaning to this word's base form (where the meaning is inferred based on context).

Other people gave you enough arguments against censorship above. I'll just add that "Inciting violence, threats to ones well being or insults towards a group" are all quite vague and thus contradict the goal of having a clear ruleset.