GitHub releases an ImageNet for code and a CodeSearchNet challenge

From the paper, it's not really clear how the resulting binary or intermediate representation fits into the picture? Wouldn't a robust model have to exploit features of the underlying computer architecture as well?

Besides the possible applications mentioned such as recommendation systems and code generation. I am wondering about automatic code annotation and documentation. With the eventual goal of creating a "code tutor" that would assist students in real-time as they type. I think what's interesting about this is that the most talented CS grads tend to have a high correlation with deep understanding not only of the history of digital logic design, but also math, physics etc.

Fantastic work ;)

Something like a robots.txt for github projects. Not that anybody would really care, only to make my intent clear that I don't support this sort of mass data gathering nonsense.

I wonder if we’ll eventually see a system where instead of writing code you describe in natural language what you want the program to do and then ML is applied in order to generate the code for that.

I mean, a lot of people have been interested in the past in making human programming languages, and had varying degrees of success.

Personally I love writing code but, it could be, couldn’t it?

Write some unit tests, a human description of what it does and based on the source code and description of existing software the system would basically “debug the program into existence” for you.

That’d be kind of freaky, kind of cool and a little bit scary.

1. We could log additional information about the model, such as inference time, number of parameters, memory usage, etc. and have the primary metric be overall efficiency (best NDCG with fewest parameters/fastest runtime/etc).

2. We're experimenting with different kinds of benchmarks, and I am most excited about explicitly collaborative ones. In these there is no contest/prize (hence no incentive to cheat/withhold information); only the shared goal of improving the model and our collective understanding of the problem. I hope we can incentivize information sharing by tracking and acknowledging individual contributions to the eventual best model in the benchmark. We could approximate individual contribution by seeing which scripts, code segments, workflows, architectural changes, writeups, or discussion comments other participants rate as the most helpful or choose to include in their experiments most often as the benchmark evolves. Of course this could only be an estimate--as Shawn says above, any idea could have "actually happened in a hallway conversation". Still, this is much easier to achieve in a logging/visualization platform like W&B than in the current paradigm of "read research papers, clone relevant repos, spend weeks trying to synthesize/reproduce their results, run your own experiments, write them up in a research paper, hope it gets accepted to a conference before other people publish the same idea, try to integrate your changes/publish your own repo, repeat"--and for hundreds of practitioners, ranging from brand new students to PhDs, working on related problems. This cycle is especially challenging for folks who are new to, working outside of, or trying to collaborate across the relatively few established/well-funded academic/industrial teams.

Collaborative benchmarks can be especially impactful for social good projects, where the primary incentive is to figure out and broadly implement the best solution ASAP (e.g. climate change!), not to make money or argue over the credit attribution. So, my long-term goal is for as much sharing of information and collaboration from as many folks as possible--the more inclusive and transparent the field of deep learning research becomes, the safer and better its outcomes. Very open to ideas on how to help make this happen.

~Stacey, deep learning engineer at W&B

- All the system logging information including CPU/GPU utilization, with the runtime and type of GPU card used - Extensive logging of model training progression - All of the model artifacts and metadata - A link to the code on GitHub with the code that ran that data. - Anything emitted to stdout (for logging) - etc.

This allows for extreme reproducibility and insight that is very helpful. With these tools, the community can see if an "ensemble everything: method is used and how long the model takes to train and what resources are consumed, etc.

Good question!

We've considered benchmarks that proceed in phases: a closed competitive phase for 3 months, then award a prize to the top result, and another prize for best user writeup. Follow that by a collaborative phase where it's more about sharing, teamwork etc. Rinse and repeat.

The question of attribution is really interesting. Who made the largest contribution to the development of a model? It could have actually happened in a hallway conversation, or something equally as untrackable. We'd love to hear other peoples' thoughts on this.

Stacey on our team has put a lot of thought into these topics and may have more to say here!

Resources and training time are also kept even between submissions.

We did consider adding StackOverflow questions. Some of our queries in the CodeSearchNet challenge do actually come from StackOverflow (via StaQC [1]). It's certainly interesting to see how all other SO data can be useful for this task. Thanks for the suggestion!

The reason we didn't try this at this point:

Many people in research have tried working with SO data. In my experience I have observed an interesting problem with the data: it's deduplicated! This is great for users but bad for machine learning, since the data looks "sparse" (roughly, each concept appears once). Sparsity is an obstacle, since it's hard for most existing machine learning methods to generalize from sparse data. In contrast, in natural language there are (e.g.) multiple articles describing the same event more or less.

[1] https://ml4code.github.io/publications/yao2018staqc/

I am playing with `todoinator` - it is a way of finding all the "todos" I leave scattered around my code, but also gives me ways to rank my code - its not quite where you are but I think the principle of having almost everythin derived from code is the guiding light here.

[#] https://github.com/mikadosoftware/todoinator/blob/master/tod...

    code:github.com/kstenerud/bit-tricks/absolute_value.h/absolute_value

There are many interesting ideas that you could build on top of this kind of data, and we only scratched the surface so far.

For example, the simple "search" technique we are using as a baseline is based on the idea of joint embeddings: We learn functions f_query and f_js/f_python/... to map from the inputs into some vector space such that, for example, for a (python method, docstring) pair, f_query(docstring) is near to f_python(method). To search given a query, we just do f_query(query) and look for nearest neighbours in all the code we indexed before.

Now, we could also just do f_python(def bubblesort(...): ...) and look for the nearest neighbour that is in C#, and should get out a C# implementation of bubblesort. Similarly, we could apply all kinds of filters on the results (code from highly-starred repos, code that uses framework X, ...) to do more interesting thing.

Someday I’d like to learn Rust, for example.

If I could leverage the languages that I already know, I could more quickly build something more useful.

Behind all the hype, predictive text is something machine learning models are beginning to do very well. G-mail has rolled out a lot of similar features from advancements in deep learning models.

That's certainly true for simple use cases. Our goal here is to eventually also capture the long-tail of queries about a codebase. Often, within the domain of a project there is a set of natural language terms/jargon that describe complex tasks specific to the domain. Imagine for example a developer joining a mid-sized project and trying to find how to achieve some simple but project/domain-specific task.

I had to use a tool for work (name withhold to protect the guilty) that has awful documentation. The tool allows you to write snippets of your own code, but provides no IDE and no documentation (AFAIK) of anything but the most trivial aspects of the API.

I started using Python, but with no debugger and no interactive shell there was no way I was going to guess the names of the functions I needed. Lucky for me, someone uploaded the Javadoc of an older version of the API, and that was the missing piece of my puzzle: having the function names, the return types, and Java's stack traces, I now had all I needed.

Back to the topic: like you, I sometimes wonder if there's a downside to not having to scroll through hundreds of manual pages anymore. But until someone shows some kind of evidence of something being lost, I won't worry too much.

That said, I definitely wish more companies would make their documentation available offline, if only as a static version of the online version. For those of us who regularly program in trains and planes, offline docs are a lifesaver.

We also provide a way to evaluate how well your machine learning model works. That's why the blog post says "We’re announcing the CodeSearchNet Challenge evaluation environment."

The released data and code (and hence, the announcement) are meant for data scientists and ML researchers who want to work on this problem, and the rest of the world does not need to care. There are no products or applications of this work at this time. If the terms in the blog post don't mean anything to you, then you are not in the target audience for this announcement, and that's OK.

0: https://hacks.mozilla.org/2018/10/webassemblys-post-mvp-futu...