What a simple idea that will go a long way. There are great many algorithms/models out there that are seemingly great and do wonders on toy data, but in the real world fail not only because the data isn't comprehensive (it isn't sampled well), but also because people misuse existing models and tweak them enough to work for that data and those tweaks make it on acclaimed journals/conferences. These kinds of efforts will help tremendously to filter out all the noise because now, you'll have to show results for real-world data to argue that your algorithms can generalize and/or that your tweaks actually matter.

Besides Amazon, Google has taught us that more data is better and sometimes less complex algorithms that scale but give you only a marginal hit on accuracy are preferred; ie, cost functions matter, and that's something you don't get practice with in the academic world.

When you think about data sets to boost performance, think outside the directly dependent ones. There is so much data out there now and it's not being used. This I think will change soon.