Differences between the word2vec paper and its implementation
github.com
github.com
In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode easier to describe).
Then, over time, things start to get further away from that. You start to see papers published with algorithms that either don't work as described, or are so inefficient as to be unusable. Like literally cannot work. Where you can get source code, later looking at the source code shows that's not what they did at all.
One infamous example of this is SSAPRE - to this day, people have a lot of trouble understanding the paper (and it has significant errors that make the algorithm incorrect as written). The concept sure, but the exact algorithm - less so. Reading the source code to it in Open64 - it is just wildly different than the paper (and often requires a lot of thought for people to convince themselves it is correct).
It's not just better engineering/datastructures vs research algorithms.
The one shining counterexample is the Rice folks who wrote their massively scalar compiler in nuweb (one of many literate programming environments), so the descriptions/papers and code were in the same place - these are very very readable and useful papers in my experience.
Nowadays it's coming back to the earlier daysdue to github/et al. People seem to try to make the code more like the paper algorithm since they now release the code.
Word2vec appears to be a counterexample (maybe because they released the code they didn't feel a need to get the paper as right)
I expect if you email ken kennedy over at rice, he may be able to find it as well.
https://www.amazon.com/Engineering-Compiler-Keith-Cooper/dp/...
And the example of the paper with literate code:
https://www.cs.ucsb.edu/~ckrintz/papers/shared.ps.gz
Still it would be nice if somebody decides to get and also make available the papers or code for which the links are now dead, e.g. those linked from here:
The new table structure is supposedly more memory efficient than anything else out there. That's the whole point of it.
So it's nice that they leave out "btw only works with size power of two". I got 2/3 of the way through an optimized implementation before I realized it wasn't any more efficient than the previous contender, in fact much less so, if you couldn't fit your set into exactly a power of two size.
Not outright lying but creative ommission for sure
It's very niche, I doubt many will run into it. Writing your own hash table is generally considered a bad idea :). I was just very bored and annoyed with memory limits. It will only affect implementors, which is less than 5 people in 4 years. Good example of implementor beware though, I'm sure it's a common issue
I don't think it's creative omission if it's a complexity improvement but it's definitely worse if the performance gains were just empirical but relied on that one condition.
PS: Sometimes I think some researchers think they're helping themselves keep their research proprietary so they will able to monetize their special knowledge or implementation, especially if no one else can make it work ("knowledge" (job) security/silo). Why do the hard work of figuring out how to make a novel AI/ML algorithm if it can be readily commercially monetized without recompense? (Modern Western civ doesn't have a good patronage system to uniformly support arts, trades and sciences.)
I might be naive in this regard, but isn't that the main point of doing research?
Are you sure you have thought this through?
This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.
It would be a big change, but this is actually possible, and it would make things easier for everyone down the line.
Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered valid and by the community but sometimes cannot be replicated.
They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.
Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it for them, and make the results public. Then everyone could check the code and results, reviewers included.
To me, getting the right people to review the paper is more important than reproducible code.
There's also a big difference between accepting a paper and judging if it is good. At a top conference like NIPS, lots of good papers will get rejected. That's the nature of being a top conference in a hot field. It's similar to a high school valedictorian not getting into Harvard: it's not a judgement that they are a bad student. But when you're an elite thing, and lots of people want in, you end up looking for reasons to reject rather than accept. You will end up rejecting good papers/students in such a case. NIPS, or NeurIPS now, is a particularly interesting example, as a colleague recently told me that they just got 12,000 submissions this year.
There's also an issue in computer science papers where people may not like your paper based on personal taste. They may not be convinced you're solving a real problem, and if your paper involves a new artifact, the design of it. I refer to these as "your baby is ugly" reviews. They don't think you're wrong, they just don't like it.
I think people's time is a bigger thing than the wrong people. One problem, I think, is that we tend to compete for these once-a-year conferences, causing an enormous time crunch for reviewers. VLDB has a great model where they do rolling acceptance throughout the year, accepting a few papers every month, even though the conference happens once a year (http://vldb.org/2019/?submission-guidelines). I believe this is the best hybrid between the typical CS conference system, and the more common journal system in the rest of science and engineering.
A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here on HN, or do something else to influence the acceptance decision). Obviously there should be methods to prevent abuse: moderators, reputations of reviewers, restricting reviewers to those with specific publications (e.g. based on keywords), etc. Btw, code reproducibility could be used to give some weight in such public review process.
That's not how the VLDB process works. What they did was establish a PVLDB journal which has a monthly deadline, and it accepts 5-12 papers a month. The papers are public on the website a few months after acceptance. (See: https://vldb.org/pvldb/vol12.html) The VLDB conference is then all of the papers that appeared in PVLDB in the past year.
I would be in favor of exploring your model as well, but I also see the hybrid model developed by VLDB as superior to the standard conference submission and review process.
Anyhow, I think Qworg has a point in that the editors do have responsibility for what they accept. It's true that referees do not have the resources to reproduce results: I for instance have limited time on the shared high-performance computing resources we have, so I'm not going to use it on someone else's paper. However, as another commenter says, this could be paid for. A journal could potentially "sell" this as a competitive advantage: papers in this journal were actually reproduced, by paid staff who are neither editors nor referees, people paid in currency that can be exchanged for goods (not academic currency).
My very limited experience says it is true. While our team worked for a commercial company in a certain highly competetive industry, I've both read many papers describing algos that on a closer look couldn't work as described and myself was forbidden to publish anything allowing our competitors or potential buyers to reproduce our algorithms. Our team ended up submitting a paper that was turned down by a reviewer and called an advertisement for the lack of details. None of us wanted to 'fix' that by adding fake details as in the papers we had read, so we ended up not publishing anything while working on the projects in that industry.
Seems like this could be useful for writing specification documents or anything safety critical..
As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it:
https://youtu.be/aRqn8t1hLxs?t=2650
It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word.
To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec:
https://arxiv.org/pdf/1310.4546.pdf
On page 3, just beneath equation (2).
Why so surprised?
I call them insincerely opensourced projects.
They didn't even bother to merge my pull requests.
It's just good news when you can find source code at all instead of just being told vague things about something the author did.
Absolutely correct, and not in a good way
Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)
This is extremely common, and I don't see why it should be allowed. I could easily write a paper claiming some kind of interesting result, fake a few "ground truth vs my algorithm" pairs, and talk a bit of mathy-sounding piffle about what it does - and it would be indistinguishable from many papers I have read.
In my opinion - provide runnable source, or what you have isn't a paper, it's a boast.
That should be left to TensorFlow model zoo or GluonCV model zoo. I simply look at these research "open-source" as reproducible research.
This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.
This sad situation is both bad for open source and science. Bad for science since inaccurate results that probably aren't even reproducible aren't the knowledge advancements we need. Funding agencies like NSF need to change their policies to address the root cause here. Universities will follow where the incentives lead them.
One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.
Geoff Hinton interview:
https://www.wired.com/story/googles-ai-guru-computers-think-...
I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!".
But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks.
If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.
Fully agree, and it's a necessary disease in young fields like ML (akin to grid search in fact).
But at some point, there will need to be some sort of theoretical foundation brought to bear or advancement will grind down to a halt
And the academic reward mechanism needs to start reflecting that fact.
You can get around it if you show BigName is wrong but if they've been vague enough that's hard.
> I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.
Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.
The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.
Sorry but I don't trust your research just based on yours words and cherry picked images.
Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.
A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand.
The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been.
It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another.
On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does.
The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks.
The code should come first and your advisor should pressure you to publish that first (assuming it actually works).
If someone wants to read your poorly worded explanation of how it works, fine, publish a paper.
Show me the code first.
Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.
I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you.
So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?
This is the actually important definition of "reproducible".
"has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description.
The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.
Isn’t the whole point of a paper, to obviate needing to talk to the authors? Like, so that science is a ratchet that doesn’t slip backward the moment the authors die?
Word2vec is a great example of a piece of research that conveys a great idea with plausible results, where in fact, the numbers are less useful than the story behind them. Of course, google is a big corp that can just put out useful interesting stuff without worrying about how to play the science game.
Now, that said, reproducibility is terrible in many fields. CS has an opportunity to act as a trailblazer here, but it should be noted that this would be holding themselves to a higher standard than their peers in other fields. As a result, there's going to be a learning process for everyone as they figure out how to make this all work. :)
And some pretty good science got done before computer scientists were gifted to the world.
I'm genuinely skeptical that modern software engineering practices are a good way of thinking about reproduction in science. Even in computer science. There's a lot that scientists can learn from software engineering (and in fact I've helped run workshops in the past on exactly this topic), but science is not engineering.
I'm happy to talk about this if you want. One of the most important aspects of this work was that people like Dijkstra started using notions that approached what real computers could read while remaining human-readable. This is some measure of classical "reproducibility". And work like McCarthy's was revolutionary in part because it was a definition of reproducibility as a result!
I can give examples of shockingly good papers that are struggling to see the light of day in their industry because they're written in ways that make them hard not only to understand, but to reproduce.
So don't presume to lecture me about this. Part of the reason the word2vec paper stands out is precisely because this is such a deviation from the norm to have a paper misrepresent its most fundamental component: the algorithm.
But similarly, if you say, "Here are our statistical models, they're in a notation for a private, custom MCMC system you can't use" that's probably failing the standard even if the work is good.
I am one of those people who actually did the extra weeks/months to properly test/review/document/release my code and data sets (you can apt-get install my "research artifacts").
In retrospect, it was a poor use of my time and a poor use of my sponsoring institution's time. "apt-get install" is NOT what we mean by "reproducible" in science.
High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.
You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.
But that's not even the important point. The really important point is that, in most cases, high-quality code is not even sufficient for a result to be reproducible! See: the article we're discussing.
IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".
> So what is the optimal outcome here?
An optimal allocation of scientists' time and effort.
As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases. That time is much better spent on writing and other dissemination explaining the ideas that make the code work (in some cases well-documented source code is the best description but in other cases prose is much more effective and illuminating). Or on coming up with new ideas that are even better than the old ones.
Which I guess is just another way of saying that scientists should spend their time on science, not engineering.
P.S. When shitting on "scientists" for not being good enough software engineers, please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science. I.e., treat scientists who maintain their code as you would treat FOSS contributors who are making 5x-10x+ less than you while working longer hours. Because maintaining high-quality code is something they are almost certainly doing in their free time.
Which failed here. However, you're going to have a difficult time convincing many people that the attributes you described would be bad properties, just that they might not justify their expense.
> You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.
By the same token though, the LHC repeats experiments and solicits feedback on how to improve their methods, which they go to great lengths to publish and simulate, because they're aware of this problem.
> IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".
Ah, yes. Yes. "If this code is TOO reproducible then people might reproduce it, and handwave handwave the quality of papers would decline.
That's certainly NOT the case in pure CS papers, which have only improved since the days when folks felt that "Lenses, Bananas and Barbed Wire" was how folks should go about writing papers.
Now, physics might be different. But there is surely a middle ground between, "I've shipped you a LHC just plug in in lol" and "This paper doesn't even remotely describe how we achived the results."
If you believe that wasting the time of scientists is bad, then surely you're for clear papers with accurate descriptions of the methods so that those who go and reproduce your work are not sent on wild goose chases?
> As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases.
No, we got that part. But surely someone does and maybe you can design your work to leverage that rather than reproducing and discarding scaffolding. My big concern here is that a lot of scientists (like you claim to be) are underqualified and unpracticed at software, and thus are surely seeing at least some aspect of their work distorted by software and hardware issues.
> Which I guess is just another way of saying that scientists should spend their time on science, not engineering.
Scientists are not going to be able to escape engineering. No one else is going to build what they need besides them.
> please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science.
Yeah, I'm aware. I suspect their lot would be better if your attitude wasn't that their work is disposable and unimportant.
Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and understand it in a few minutes. The most puzzling part with framework code is figuring out the shape of tensors and, sometimes, what each dimension means.
Regarding reproducibility - it's hard to achieve on account of parallelisation. You see, neural nets use floats, and floats are not real numbers. Half the float numbers lie between [-1, 1] and the rest outside. If you do 1e-10+1e10 you get 1e10. So if you add 1000 times 1e-10 to 1e+10 you still get 1e+10, but if you add them up together first, the result is different. So float summation is not commutative. Depending on race conditions the order in which summation occurs could change and the result change as well, even if you use the same random seeds. And if we give up parallelism then we can't run the experiments any more.
That's if float rounding issues are even a big problem in the first place. If your results are within .1% over a few runs it's reproducible enough for most purposes.
How wrong should the original work be before it ceases to be a case of 'clean your own pipettes'?
Could you PLEASE consider reading the article we're all discussing before you roll into the comments section of an article about it with your strong-but-loosely-held opinions? You're arguing against a point that almost no one is putting forward (that software should be one click).
Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuable enabling other people's work to be a principle researcher themselves. I've heard from a number of people that they avoid hiding their programming skills to avoid that trap.
This has many consequences beyond the obvious direct ones like papers with unusable implementations.
Academic work is often valuable even when it isn't practically useful but in many cases where academics are /trying/ to do something practical they fail because their community lacks the engineering experience-- I've seen a fair number of papers presenting optimizations as useful engineering when in reality their approach only makes sense because of inefficient non-programmer tools make a weird set of primitives fast (e.g. using a matrix multiply in matlab where in C you'd simply write a loop). As a practitioner this is frustrating because it sometimes requires re-implementing the approach to realize that it was only 'fast' compared to a pants-on-head-foolish approach.
It also can create essentially fake results.
Many times the lack of software engineering is compensated for by mocking components out in ways that would give faithful results if the researcher understood everything, but the whole point of expirementation is that the researcher doesn't! This mocking is seldom disclosed in papers. For example, I've encountered multiple papers that claimed to implement some enhancement to Bitcoin and tested it in a test bed but in reality they just added sleep()s with appropriate times, guessed out of specious reasoning like "our function X should be 10x slower than a signature", and hardcoded constants (computed in mathmatica or sage) for their messages. ... not realizing that sleep() isn't the best analog for cpu busywork in a multithreaded program!
Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers.
I don't have any proposed solutions but I think it's important to acknowledge that it is a common and serious limitation to the usefulness and accuracy of contemporary research.
In the past academia and government agencies developed the internet and its communication protocols; this was (imho) a much better internet than the corporation-dominated internet we have today.
I'd like to see a resurge of this type of internet. Let us use well-researched open protocols again; and let us use applications which are really built for people, not advertisers. Let corporations build the hardware, but let us keep our data far away from them.
And besides the computer science / software engineering branches, I'd like to see other fields (like industrial design, and for instance even sociology) to join the development of a better digital future. The kind of future where everybody profits, not just shareholders of big companies.
This happened so many times when I was working on NLP/CV algorithms - I've read and implemented many algorithms from papers just to find out that they only produce the amazing improvements on cherrypicked dataset. On pretty much all other practical data the algorithms performed worse and in many cases even crashed!
To compound the problem, the fact that you're competing against many of your peers for the few jobs there are, and jumping positions every 1-3 years after grad school, means that best practices are not disseminated by working with a stable set of coworkers. This is similar to your 4th paragraph point.
As a mentor of grad students on projects that these days are increasingly using machine learning, I am trying to figure out how to add this stuff to my teaching load, essentially, because the pain of my students using nested for loops when they could just use a single vectorized expression is real... but I myself have no formal CS training......
That problem is compounded with the issues you describe, which I'd categorise as probably the best case. I did not find even a single case of a properly opensourced repository that just compiled somewhere that wasn't the researcher's laptop.
[1] google: "neural networks vector initialization" [2] http://deeplearning.ai/ai-notes/initialization/
[1] https://papers.nips.cc/paper/5021-distributed-representation...
All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.
That's a nice minimal way to demo/test the idea, but limited and fragile in other ways. The initial gensim implementation did something similar, then I changed it to use a separate doc-vectors space, to better support a lot of options (including the PV-DM mode with a concatenative input layer – which has never been confirmed to perform as well as the original paper implied).
Embeddings are shallow, and what comes after them is usually less than 5 layers of LSTM, not a deeeeeep neural net (maybe, deep only horizontally, on the axis of words).
So, I had a very bad impression of the code. But then, I checked the thesis, and I found it awesome. The amount of tests and implementations the guy made, and how he showed in practice how better results could be achieved in a good number of different setups... I found it really impressive. But such great work paired with such bad code! I was just a CS student, so I found it shocking. Nowadays I realize he was simply focused on a different thing, and the results he obtained were indeed outstanding and talk for themselves.
It's easy to look back and criticise the code, but when you look at the work he did in perspective... it's completely unfair to ask more from him (admittedly, they had time to address some of the issues later, but they probably had better things to do too).
I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algorithms were patent pending and I assume the authors were hoping to make some money from the patents. Compared to that, it's a lot better nowadays where most of the popular papers come with published code.
Fortunately the trend seems to be towards better levels of openness, but it varies by subject. Stories like the BASF team being unable to reproduce vast numbers of established published techniques are way too common.
> The authors are ... constrained by space due to paper limitations
This is very real. Different journals have different criteria: word count; formatting; the number of tables you're permitted to include; etc. It's archaic and daft but that's the truth of it. And that's before we even get started on the really bonkers stuff like author order, impact factor, reviewer workload vs lack of pay, publish or perish, and so on.
It's a shame because I understand the idea of the paper, and have an excellent place to apply it, but I lack the DSP background, so I can't really rebuild the code from scratch -- so the work is not able to be used.
The basic context is verification code delivery -- I'm playing pre-recorded samples of numbers to users, and can't control or sample the noise (either transmission or environmental), but would like to enhance intelligibility to reduce user effort, improve experience, and reduce costs.
[1] https://www.research.ed.ac.uk/portal/files/17887878/Cooke_et...
[2] https://web.archive.org/web/20131012005150/http://listening-...
I try to make my students release polished and easy to use code, but it can sometimes fall through the cracks due to deadlines, etc. Many projects are the output of a single PhD student.
Also, I think it is something of a fact of life that you can't put all of the details of your algorithm in the paper, not just paper length limitations. I have actually tried to do this in two papers by putting all of the details in the supplements and the work to explain all the details and justify my choice of parameters and decisions for edge cases is almost as hard as writing the main paper. It becomes hard to justify the time spent pretty fast. Also, putting this in the main paper makes your beautiful explanations be tinged with edge cases and digressions haha.
There's only a second vector for a word in the (common, default) negative-sampling case, where each predictable word has a distinct "output node" of the neural network, and the second vector is the in-weights to that one node. Still, most implementations don't emphasize this vector – the classic "word-vector" is a word's representation when it's a neural-network input. And in the hierarchical-softmax training mode, there's no clear second vector.
I suspect the original word2vec authors left out a clearer description of the initialization as they were following some oft-assumed practices implied by their other descriptions.
Another minor difference between the literal descriptions, and original C implementation, was a slightly different looping order in skip-gram training: holding a target-word, and then looping over all context-words, rather than holding a context-word, then looping over all neighboring target-words. One of the authors once mentioned that the shipped approach was slightly more efficient – maybe it was due to CPU cache issues? In any case all the same context->target pairs get trained either way, just in a slightly different order.
word2vec is a multi-class classification problem with a softmax output layer and cross-entropy loss. The novel part of word2vec, in my opinion, is two:
1. dataset (proximal input word & output word) generation from documents eg: skiagram, CBOW, etc 2. engineering speedup for softmax: Approximate Softmax eg Negative Sampling using NCE, hierarchal softmax, etc
If you just build word2vec w/o step 2, it's a easier to understand. Then when you get that working, add in the negative sampling speedup trick which isn't core the theoretical algorithm.
The implementation can differ, they got time to refactor/optimize it after the publication. But they can't probably revise the paper itself. As long as the code is there and can produce said/better result, then it is probably your responsibility to keep the differences in check.
It is actually quite common for deep learning papers overall, the github repo gets updated after the paper is out, and you will find the divergence lying there.
The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed.
W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_w.
Am I wrong?