The differences between tinkering and research (2016)
togelius.blogspot.com
togelius.blogspot.com
I have come to my personal conclusion that there is a difference between 'science' and 'the academic world' in much the same way as there is a difference between 'religion' and 'the church'. One can be very religious without going to the church and doing all the rites that are required by the church. In much the same way, a 'tinkerer' (or 'heathen') can be scientific without being academic. I believe that wiki summarizes it quite well for me: "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe". There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
I want to solve real world problems and the way I want to do that is in a structural and repeatable approach. Science helps me doing this. Of course I'm not living in a vacuum, so I will look up as much as I can about the subject at hand as is reasonable within the available amount of time. Interestingly enough, I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera. Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
So the quote (which you don't attribute to anyone other than "wiki" - a citation here would be useful) really proves my point. If you're doing not doing your scholarship, you are not doing science, or research, you are tinkering.
Again, there's nothing wrong with tinkering.
In the sense of presenting the research in a manner that other people can digest, academia seems to be more of a ancient guild than an organisation for expanding knowledge.
You and parent are saying the same thing. Just because they are written for experts does not mean they are not written in an opaque manner.
And as someone who was once in academia, they really are written for 2 reasons:
1. To get past the peer review process.
2. To be written in the minimal time possible.
Enlightening peers comes a distant 3rd.
Example: It took days for a grad student/professor to derive a formula that is included in the paper.
Professor insists the derivation not be included in the paper. Insists the student not even mention in a few sentences the steps to get to it. Claims "any expert should be able to do this. No need to add it to the paper."
It took that expert days to do it. Unless it turns out to be a seminal paper, I guarantee that in most cases, no reader of the paper will even try. An error in the derivation? No one would catch it. Clearly, the professor is not even writing for his peers.
I do agree with the parent - the practices approach that of a guild more than any objective measure of explaining things.
You say there's nothing wrong with tinkering but it's a fairly derogatory word. From dictionary.com [0], it's specified as:
> an unskillful or clumsy worker; bungler.
Certainly here in HN and in computing in general, the term hacker might be more fitting.
Come to think of it, many outside the hacker community would perceive "hacker" more negatively than "tinkerer", because many people still think of hackers as criminal. At least that's my perception. I guess words have different values in different contexts.
https://en.wikipedia.org/wiki/Science
> Science (from Latin scientia, meaning "knowledge")[2][3]:58 is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[a]
Incidentally, the "wiki" as a system separates "scholarship" from "authoring". "scholarship" is done collaboratively post factum, given substance "authored" by various overlapping sources.
Crediting someone is not about vanity, at all. It's so in the future someone else can see where to go for ideas you've used, what the general thought and progress was that lead to an idea. See "scholarship" in the linked article. Crediting people facilitates scholarship and knowing the context of an idea. In theory you should be able to trace all human thought that lead to an idea, and what else those people were thinking at the time. Amazingly in modern science that's essentially possible. Most papers have an acknowledgements section where you even acknowledge conversations or private letters that lead to an idea. It's not vanity, even in the world of tinkering its worth crediting people with the input for your ideas and the source of your material as it makes it easier for someone else to riff off of your tinkering.
It's not about avoiding reinventing the wheel either. It's about knowing all the criteria for wheelness, knowing what has been done before and what is widely accepted as a wheel rather than a roller or a track. Is there a body of related work that might apply in the abstract, if rollers are like wheels perhaps roller maths would be useful, perhaps a development in roller technology has not been applied in principal to wheels and could be a breakthrough.
Both of these together are how you actually push things forwards. Giving credit especially is highly misunderstood. If I want to know if one paper Im using is a weird off shoot of thought of the author or not, what else were they working on, where did their inspiration come from, is there more I can get from their direction, or is there somewhere I can go they never dreamed of etc. That transparency is actually amazingly powerful for the creative process.
It's a case of expectation mismatch from your end. A scientific article is aimed to encapsulate all information which can help push the frontiers of human knowledge. Science is not meant to sell a product.
>There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing.
The article isn't a discussion about what science means. Its a discussion about the principles of research that all of our current scientific progress is based upon. One of those principles is the organization and structuring of knowledge. Citations are one of the means of linking knowledge that allows easy navigation between topics and ideas for people both inside and outside the field. Because of the collective effort of people citing other work, it saves time and money of everyone involved since they can easily hop between topics and ideas without having to read a million unorganized pages of scientific data and figure it out every single time.
> I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
Um, you don't really get to tell someone they're doing it wrong, when what they're doing has worked out brilliantly for the human race.
In regard to saying that something is seriously wrong; There is no doubt that the academic world has produced a lot of valuable knowledge during her existence. It's the current academic world that is in crisis. With the H-index as its ultimate false god. The H-index exists only since 2005. Why should I not be allowed to criticize that? If the system is still good, it will survive the criticism.
Okay, now I understand the point. But why is it necessarily the responsibility of a researcher to find industry applications? The industry has to figure out which portions of these ideas can be realistically turned into applications or products based on market criteria. Of course industry can and often does contribute papers to further our knowledge. And when they do invest in doing research they're probably going to be motivated to find applications.
> It's the current academic world that is in crisis. With the H-index as its ultimate false god. The H-index exists only since 2005. Why should I not be allowed to criticize that? If the system is still good, it will survive the criticism.
I was using 'don't get to' as a rhetorical device. In my view it was like walking up to Usain Bolt and telling him how to fix the problems with his form cause hes doing it all wrong. Sure, you can do that, but most people won't take you seriously. Or in other words, simply being able to identify surface level problems, doesn't mean much without a ton of analysis, explanations, etc, etc. But yeah, obviously you can opine on any subject you like, and people too have the freedom to respond.
That is probably related to your field of work. I once had to write a pathfinding algorithm for multiple robots in a tight space. With such problems, you will end up with research papers. I can imagine lots of other technologies (such as everything related to AI) where papers will help you the most.
Blog posts etc are a solution to common problems. Sometimes you indeed find some gems on architectural level, or for game development there's lots of valuable content out there.
But once you go to complex algorithms, that's the space when research becomes handy.
I wish that google scholar showed bidirectional links though. I suspect they might have concluded that the list of citations included in a particular paper is part of the content of the paper (not just metadata) so that showing it might infringe copyright.
I thought you were going to go the other way on that one... going to church doesn't make you religious. I might even say not all churches have religion. And the analogy still works, 'the academic world' is hardly a subset of 'science'
As for the rest, the goal of academic research, lately, seems to be to produce as many as possible units for the benefit of the publishing industry. In the case of very successful articles, you can fit all the readers in a (small) bus. Most of the articles are badly written, because they follow old habits from the age when printing was expensive. Citations are done in a very selective way. Not so long ago there was no need to give the source of an "unpublished" article, "preprint" was enough. Tinkerers work is almost never cited, even if it is read and used. Blogs? are you kidding me? that's not "scholarship". Heck, even arXiv still seems exotic in some fields, despite the fact that it is a good, standard communication mean for the researchers in other fields. Finally, peer review. How does it work? Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could.This passes for validation. Magic.
In conclusion I believe the future of academic research comes from "tinkering" with a long time attention span. The dissemination is technically trivial. Peer review will be supplemented by more rigorous validation (although there is no absolute solution here). The same kind of validation will be applied to tinkerers results.
Many aspects of "scholarship" become inevitable when "tinkering over a long timeframe". You build up a knowledge base about the problem and/or methodology you're tinkering with, you eventually form a community around your tinkering, etc.
Lots and lots of tinkerers gratify themselves without contributing any modicum of knowledge to the world. But the most successful tinkerers are almost always effectively scholars in the sense of this post. In the sense that they're familiar with a body of knowledge/the people producing that knowledge. And they contribute genuinely new ways of doing things, or new ideas of things to do, or excellent executions on existing ideas that achieve the aspirations of previously articulated ideas. E.g., Linus is certainly a successful OS scholar in the sense of this post.
> In the case of very successful articles, you can fit all the readers in a (small) bus
This is off by at least an order of magnitude for any even remotely reasonable definition of "successful".
> Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could
First, the author explicitly kinda-sorta agrees with you here (...Third, it's not really the publication...), so it's pretty clear that publication venue != scholarship.
As an aside, peer review != validation!!! This is something that every young Ph.D. should explicitly learn early on.
This means that code and data is typically hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use (or often not publicly inspectable at all). Often the folks doing the grunt work are graduate students, who leave after a few years, leaving a new batch of graduate students to deal with the code without much guidance, so it tends to accrete features but seldom get any refactoring love. Nobody is paying for bug fixes or software maintenance or documentation.
Someone coming from industry (or even just a tinkerer) who wants to use the same idea in a production environment often realizes halfway through reimplementation that it isn’t actually better than existing alternatives, and that the analysis/benchmarks in the paper were limited or misleading for a general context. Sorting the wheat from the chaff can be pretty demanding, ideas can sometimes be obfuscated by abstruse mathematical formalism, and academic researchers don’t typically go out of their way to make their output usable for non-academics.
Far Better would be to compare Maker vs Researcher.
How "Production Ready" is the result of both activities is the interesting question. How well covered with tests? How maintainable? How re-usable in broader contexts? How readable and understandable?
I used to live on the margins of the Physics community and blamed myself for being a really really Bad Researcher for having no patience with the Physics papers I read.
Then one day I had an epiphany.
If I were to look at the physics journals the same way as I look documentation for software or hardware.... I would be really really Angry.
The average research paper is extraordinarily badly written and unreadable. They have adopted as best practices, styles and habits, that you'd never forgive in a datasheet.
If you picked up a datasheet written like a physics paper....
You'd throw it against the wall and spec a different part from a semi-decent manufacturer.
(Hint: Datasheets are notorious for being terse and indigestible and badly written.)
If you are not from a first world university with a first world library at hand with paid for subscriptions to the premium journals and a fat budget to cover pagefees... Scholarship be damned. It's a rort. It's a racket supporting a closed shop guild.
It's not Good Manners except in the sense of Manners defining a "Gentlemens Club".
You've got traditional iterative methods through to bayesian methods and machine learning methods. Each of these approaches seemed to be relatively isolated from each other and largely cited from their own subfield, despite all solving variants of the same problem. When they did cite outside the field, they tended to cite less relevant or far from state-of-the-art papers.
What was amazing was the number of papers where you would have a plot showing how amazing the new deconvolution method was in the presence of noise compared to other methods, often using a vaguely defined metric. This plot would take up about half a page, usually following a few example images that were set very very small, often ~2cm and usually in a grid. That was always deeply suspicious.
The other thing I remember were models with tons of unexplained parameters, which were carefully tuned to give good results for that image. But really it was guesswork. Sometimes the authors would carefully avoid well-cited work in another field that they could easily have found.
As a tinkerer with a scarcity of time you learn where not to waste your time and move on.
Which is to say that no all incremental, scholarly research is necessarily what's needed. In fact, I think what the OP describes is effectively what Thomas Kuhn calls "normal science" [1], which can be great until it becomes terrible - ie, "normal science" periodically reaches crises that require a different approach (inspiration, insight, iconoclasm, etc, whatever).
But that shouldn't detract from the point that anyone should know what traditional research, that if someone is going to engage in something other than normal science, they would benefit from being really aware of why and how they are doing this.
You say that like it's not still state-of-the-art in machine learning and AI today. (And piloting self driving vehicles around public roads...)
Journals sometimes require the authors to pay "page charges" of sometimes hundreds of dollars per page. I know of some (relatively high impact) journals in image processing related fields that require this. If the author is not working under a grant, then there's not much incentive to add to the page length.
Of course, this is insane that this is even a thing in the digital era, and it feels very mobbish. Idk. Maybe there's valid reason that I'm not aware.
This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.
"$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later."
I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.
In this instance you need to pay the price of working in an obscure domain, research literature, and experiment what works and how.
As a concrete example, boolean operations on polygon meshes still have even not one "de facto" implementation. You can find papers from early 70:s, but each method has one catch or another. Compare this to, for example to sort alogrithms, where just quoting the method name gives sufficient information - "quicksort or mergesort"?
The only source are few obscure books,a ton of research papers, and some open source libraries with very little documentation on the algorithmic specifics.
In this instance I'm very happy the research papers exist - they give lots of information and ideas, on what has been tried and so on. If not for the papers, the occasional tinkerer would have all of that ... decades of academic research ... to fumble through, all alone.
They are trying to communicate with other researchers who actually care that the proofs are correct. Maybe you don't, if you're simply trying to reimplement some algorithm, but criticizing research papers for using mathematical formalism is a bit ridiculous.
Having both mathematical notation and prose can help uncover gaps in understanding and works much better as a system of organising human knowledge.
One thing I'd note is that before that Internet era, the scholarship needed to discover who had done X previously took a really significant amount of time. The Internet has changed that a lot, at the same that it has tempted people into tinkering and overall wingnuttery. I remember in 80s, having professor who functioned as little more than a search-engine, tell him an idea and, after a couple of puffs of his pipe, he'd give you a list of people who'd worked on it. He was marvelous despite apparently never having done much more than providing this kind of information.
Another thing to note that scholarship implies a "community of researchers". If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. The article kind of says this but also consider that the people who have worked on this stuff exist, they have publications, conferences, meetups, etc and you can go to these or look at these and find-out whether or not your idea was considered and rejected, think about what's different between your idea and the idea the "field" rejected and so-forth.
At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly:
- each of the earlier people were approaching that problem from a specific angle, and new people might be approaching it from an untried angle
- the earlier people might have had limited time to really investigate the matter, so there may be unturned stones
- advances since the earlier work may give newer researchers a leg up.
Note that all of these apply even if (as is likely) the earlier researchers were very smart.
> At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly
Well, if you use such an approach of considering previous work, you will have the tools to show you why your work shouldn't be dismissed (whether you are believed or not is a different matter). If you aren't looking at earlier work, not only do people have good ammunition for dismissing your position but it really is hard to be sure you haven't just repeated something.
That doesn't mean you'll be listened to. You might, but that's no guarantee.
There's also a situation you're overlooking: people being discouraged for the cited reason before they even start. "What's the point, if people as smart as you -- and possibly smarter, more senior, and with more of a reputation -- already tried to think of everything?". This can be something that the researcher can tell themselves.
I was explaining why there can still be room for progress, even if lots of smart people have already explored the problem.
> If you aren't looking at earlier work
At no point was I suggesting people shouldn't look at earlier work.
Conveniently, computer programming and invention offer other ways for what the OP called "tinkers" to demonstrate the effectiveness of their ideas. Science is imperfect, especially isn't necessarily about the latest, most exciting ideas but about making ideas reliable and the problem with being willing to quickly accept even plausible-sounding ideas is that they can also include a lot of dreck. But hey, there are many alternatives to scientific acceptance.
It's exactly the opposite. You can now spend years just reading all the recent papers relevant to a quite specific topic of interest, and 20-80% of your time keeping up with new results.
If you narrow your research so much it's manageable you're immediately rejecting lots of directions right from the start.
I appreciate this dismissed as a straw man, but there are definitely online communities (and individuals I know) who reflexively discount comments from people outside academia.
I understand the instinct, it's a reasonable immune response when everyone thinks they're an expert because they read a blog post or a pop sci best seller.
Sometimes, though, the antibodies kill perfectly healthy tissue too, like valid contributions from bright generalists.
Now it's harder to make breakthroughs without specialized equipment, and the area of interest has narrowed to a tiny sliver.
OP's distinction between tinkering and research seems wishful thinking at best.
There are many grievances people have with research (and it's easy to think of examples), but I find most tend to fall under:
* (unaddressed) RISK: poor result analysis, cherry-picked examples; glossing over edge cases and failure modes; focus on novelty and "academic writing" at the cost of repeatability and conceptual clarity. The Mummy Effect of research that looks great but crumbles to dust when you actually try to touch it: https://rare-technologies.com/mummy-effect-bridging-gap-betw...
* (missing) OWNERSHIP: cadence dictated by grant cycles rather than one's intrinsic motivation to solve a problem; slow or missing feedback cycles; no dog-fooding => poor code and documentation; few incentives to make it easier for others to pick up or validate the results
In fact, both of these seem better addressed by tinkerers than by institutionalized research. No wonder people react more positively to "tinkerers" and "hackers"! Academia has a lot of positive inertia (and taxes) going for it, but all goodwill has its limits.
The game-like qualities of surviving peer-review are just that: a set of pretty arbitrary hurdles to cross. If you work in a narrow enough field, your set of blind review peers is tiny and their critique, even if masked, can seem very personal at times. I don't like that much.
I don't overall judge my attempt a success. I think this is an endevour best started early. Late stage career, trying to demonstrate you understand the rules of science, is hard.
Obviously the intrinsic quality of your work matters. But you’ll more often see work of marginal value that properly follows the form than work of great value that does not follow the form. In fact, in my field (academic HCI), I have never seen the latter, yet see plenty of the former at every conference I attend.
This is more important than you'd initially think. Incentives matter, and professors are rewarded for publishing papers that get cited by other papers. So if you cite your reviewer's paper, it's in your reviewer's best interest to get your paper through review. Hence, quote a ton of works from that venue and hope to get lucky like that.
I'm actually really curious to know if anyone has first-hand knowledge of a tinkerer (by this blog's definition) getting their work published.
As the GP said, understanding the rules is one thing, but demonstrating their understanding is another. And as you said, matching the style seems to be more important than the value of work that is demonstrated.
I see the value in matching terms, jargon, style etc just so the reviewers can standardize their thought processes. But as someone who's been a tinkerer for ages now it's hard to change styles for no immediate benefit.
Maybe some examples will inspire me to try :)
My second attempt was 'this is a specific thing it can do' combined with a much more rigorous academic 'this is the analytical technique' and 'this is a polemic about lack of statistical rigour in results, here is our data, you repeat it' -Which interestingly got panned as 'too much tutorial, too much argument, more results' -which of course this time, we addressed instead of walking away. Result? we didn't get in the first journal/conference, we made the second. I'm reasonably content, but this feels like 'learn the rules of the game' a lot more than 'say something of merit you find personally interesting'
Oh, and 'this technique is interesting' doesn't seem to cut it as a paper subject.
> They wanted a lot more demonstrable outcomes. I walked away. I found peer review very upsetting. It felt like nobody actually cared about what I was trying to say.
> Oh, and 'this technique is interesting' doesn't seem to cut it as a paper subject.
Well, yeah? To invoke an HN cliche, "ideas are cheap". Why should anyone else care about your idea if you can't be bothered to show it actually does something interesting on some specific problem(s) or even motivate why it might be expected to do something interesting in light of what's already out there? Without any expectation of experimental validation, conferences would basically be giant circle-jerks filled with completely inconsequential "interesting ideas".
And it sounds like you took the feedback from your first round of peer review and revised your work in light of those critiques and got your resubmission accepted. That seems like a pretty good experience to me, knowing many academics with multiple experiences of resubmitting work 3+ times (with new results and revisions each iteration) before acceptance. I'm not saying that any peer review process is perfect by any means, but it's a very important filter and in this case it honestly sounds like the criticism you got when your paper rejected was pretty fair...
One is a crude and improvised or temporary solution to a problem, designed to be more functional and timely than precise. Something done with little forethought, organization, planning or precision.
Whereas the other is meticulously documented to note the choreographed nuances.
It was the latter bit that really challenged 3D printer makers. So many 3D printers would work one time and not the next because there were so many parameters that they didn't specifically account for or respond to.
In academia, you are allowed to cherry-pick an artificial problem and work on it for 2 years. The result needs to be novel, and you need to research previous and similar solutions. The solution needs to be perfect, even if not on time.
In industry, you should solve a given problem end-to-end. Things need to work, and there is little difference if it is based on an academic paper, usage of an existing library, your own code or an impromptu hack. The solution needs to be on time, even if just good enough and based on shady and poorly understood assumptions.
So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science.
(from http://p.migdal.pl/2016/03/15/data-science-intro-for-math-ph...)
Science in a nutshell is to take a hypothesis, test it rigorously and validate/invalidate the hypothesis based on the results. The results of this experiment could be extremely practical. Data science that follows the scientific experimental design, does science. The one that doesn't is equivalent to reproduction of past experimental results OR putting a blindfold on your eyes and throwing darts around until it hits the bull's eye.
Building engines or cars (involving a lot of tests!) is considered engineering. And as any serious engineering it's not purely random trails&errors. A/B tests are not a sufficient criterion to make something science. And data science (in business) is focused on solving practical problems (which may or may not be research) rather than open-ended research (which may or may not be practical).
Source: I worked in academia. I now work in data science. You?
- giving talks at universities (then I admit that "data science" is more engineering, compared to academic science)
- people who had only experience with software development and then any research-like stuff (or simple mathematics) suddenly becomes "science!"
In reality, neither is really deficient, they're just aiming at different audiences which need different things.
I didn't mean to imply that tinkering was inferior to research - the whole premise was just to tease out how they're different, with different audiences, as you say. Interestingly, the discussion here has been dominated by people who think that I look down on them. People who've discussed it in other fora have not read the post that way.
The Show More links under his youtube MarI/O video link directly to the Stanley Miikkulainen NEAT paper.
and he links the wikipedia page on Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution
which cites two [1][2] of your papers among others :
yet you say : " it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. "
a bit harsh perhaps ? At least mentioning the foundational NEAT paper is not a complete lack of scholarship.
I feel you have overlooked that Seth does try and provide good citations to his audience and it would kind if you mentioned him by name in your article rather than " some guy "
He did inspire other twitch streamers to experiment with neuroevolution and neural nets and benchmark many SNES games.[3] although downstream this follow up does goes uncited by yourself.
[1] "Neuroevolution in Games: State of the Art and Open Challenges" https://arxiv.org/pdf/1410.7326v3.pdf
[2] "Countering poisonous inputs with memetic neuroevolution" (PDF)https://www.academia.edu/download/30945872/poison.pdf
[3] Mario Bros mari/o https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat
https://www.youtube.com/watch?v=bRxUQNFxAWc Mari/o kart winterbunny
https://clips.twitch.tv/HilariousPolishedZucchiniHassanChop - mario kart RNN mariflow
I'm old enough to remember "shared source" and the classification of "hobbyists" in Microsoft's licenses. In that case the intent was to try to create a division between "professional" programmers, who would pay for useful access to source code and "hobbyists", who supposedly had no need for useful access to source code (because they aren't "professional").
I'm quite certain that's not your intent (and you say so several times in your article), but the implication still lingers. My area of interest is in language design and practitioners have made significant contributions to the field. Researchers are enabled because the endless experimentation of practitioners have narrowed the search space, even though it may have been done in an inefficient manner (by reinventing the wheel many more times than necessary). In turn practitioners have benefited greatly from the exhaustive knowledge and documentation of the researchers.
Both scholarship and practice are useful, but it is understandable that one group may emphasise one over the other and get different kinds of useful results.
I have met humble academics, I have met many more who are arrogant know-it-alls. It behooves all to understand that each one of us has a limited set of knowledge. Especially when one becomes more "expert" in some field.
I think one reason that various groups of people has a distaste for science and academics is the fairly common "I'm better than you because I've earned a degree or two or three and know more than you do" attitude displayed.
One aspect of research and publication which has been highlighted many times is that replication of experiments that have been published don't get any funding or recognition.
In that regard, what seems to happen in research is that instead of further investigating the slight anomalies that are found, it is often just let go because it is not within the scope of what was initially being investigated. It seems to be a rendition of the attitude of "Move along, there's nothing to see here".
Your post did a nice job explaining some of the reasons that academics do the type of work they do, and how that differs from hobby side projects. I think it’s more interesting to discuss the trade-offs involved instead of just passively agreeing with the OP. [Also, many non-academic projects are more organized than someone’s weekend hobby hack (large scale volunteer efforts, funded by donations and sponsorships, directly corporate-run, etc.), so it is worth discussing those as well.]
In my opinion, the major problem with your post is that your view crystallises the ivory tower mentality instead of really discussing the difference between "researchers" and "tinkerers". In reality, statuses and titles aside, there is only a fine line between a seriously good "tinkerer" and a seriously good "researcher", and the assertions you have made are simply shallow and narrow.
Your attitude towards opposing, but non-hostile opinions is also rather off-putting—you are simply making light of people who wish to add to the discussion and dismissing them instead of addressing them. It doesn't seem to be that of a reasonable person who is willing to consider different views and explore different possibilities, and definitely not one of a good researcher.
That one math formular which is nowhere explained? Yeah good...
I often enough, get the idea and details like results but thats it.
Those papers are not tutorials and they are not written / done to be reproduced and easy understandable. They are, in my opinion, written for other science people who spend there work time doing something similiar or need to solve the problem what the paper solves and rebuild those results in time consuming work.
Probably still better than having no paper but still way more work than just using it.
i do read "the morning paper" (https://blog.acolyer.org/) and he is really good in reevaluating papers. I'm very curious how much time he spends reading and analysing them.
... and hard to resist the temptation to go jump out the window when you find out that the thing that you've been trying to figure out for a year has already been figured out...
I think its important to note that that he does reference NEAT (and the paper it comes from) and the emulator he used. This is building on what others have already done. While the 2009 paper from the post author would definitely be worth note, I feel "atrocious" is a bit harsh.
## Scholarship
To begin with, there are many "tinkerers" out there who practise good scholarship and at acknowledge prior art, as well as provide references. Anyone who is courteous and has the awareness that the world doesn't revolve around her would honestly document these things even if she is a "tinkerer".
And just as there are "tinkerers" who practise good scholarship, there are "researchers" who practise bad scholarship as has been pointed out by @jacobolus.
## Testing
I'm not sure what makes the author thinks that rigorous testing is only a "researcher" quality. The article just insulted all the "tinkerers" out there by saying:
> "Here's another big thing. A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens."
A "tinkerer" doesn't just make something and puts it out there. A serious "tinkerer" would not want to put something that doesn't work out there—it's embarrassing and it's just, wrong. To make something (reasonably complex) that works usually involves a lot of testing, even if the goal isn't to publish a scientific paper it would still involve a lot of testing.
## Goals
This bit really lost me. It sounds like a bunch of self-contradictory, "researcher"-glorifying statements one after another.
> Usually, goals in research are not just goals, but ambitious goals. The reason we don't know what the results of a research project will be is that the project is ambitious; no-one (as far we know) has attempted what we do before so our best guesses at what will happen are just that: guesses.
A "tinkerer" can have ambitious goals. A tinkerer may do something that nobody has attempted before and, naturally, in that case she would just have to make guesses.
If, hypothetically, I set a goal to investigate the possibility of using CSS to make high-quality 3D games that can run at 60 FPS in web browsers, does that make me a "tinkerer" or a "researcher"? What if, in the process of doing so, I practise good scholarship, document everything carefully in a well-organised format, but just simply have no interest of publishing it in a scientific journal—does that make me a "tinkerer" or a "researcher"?
On this:
> However, if you read a scientific paper those are usually not the stated reasons for embarking on the research work presented in the paper. Usually, the work is said to be motivated by some scientific problem (e.g. optimizing real-value vectors in high-dimensional spaces, identifying faces in a crowd, generating fun game levels for Super Mario Bros). And that is often the truth, or at least part of the truth, from a certain angle.
The stated reasons in scientific paper is usually after written after the research has been carried out and they are designed to tell a reader why the works is important. In a space where everyone is struggling to keep up with publishing so that they can keep their job, who in the right mind would begin a paper with "our research group embarked on [insert amazing thing] because it seemed fun to do so and has never been done before"? Actually, I would love to read papers like that, because most of the time you know it's the usual standard crap when academics start a paper by stating a problem—it probably means that they haven't solved the problem and are not even close.
Also, I would like to draw your attention to the last, contradictory sentence. It seems that the author isn't quite sure either.
## Persistence
If a someone spends 10 years modifying and perfecting something that has never been attempted before, but does not care for its scientific value, does that make her a "tinkerer" or a "researcher"?
Bottom line, anyone who appreciates, and wants to do, good work will naturally do all of those things above—"tinkerer" or not. To throw away sentences like:
> Probably the most importance difference between tinkering and research is scholarship.
> A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens.
> While tinkering can be (and often is) done for the hell of it, research is meant to have some kind of goal.
> Tinkerers are content to release something and then forget about it. Researchers carry out sustained efforts over a long time, where individual experiments and papers are part of the puzzle.
... is just simply conceited.
Near the end of that blog post you find:
>>Reading the book, I felt that most of my research is not science, barely engineering and absolutely not mathematics. But I still think I do valuable and interesting research, so I set out to explain what I am doing.
Of course, both the blog author and the book author are right in a way. But probably the useful thing to do here would be to come up with new names for these different kind of activities, rather than destroy the language by trying to just place one more thing into "research in general" category.
> The video certainly reached more people on the internet than my work did; it makes no mention of any previous work.
Video
>"I didn't come up with this on my own, it's based on an algorithm called NEAT based on ap a paper by..."
>Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
Then publish your damn results in a damn open journal, dammit.
In machine learning and computer vision, the default is to put your work on ArXiv immediately upon completion. There is a lot of openly available research depending on the field. I find most fields in computer science are good for this.
In the case of the Mario example, I very much doubt that the author was not able to find other work because of closed journals.
Research can seem non-transparent to non-researchers because when problems are new, they are often poorly understood. Academic papers discuss novel problems and contextualize them based on other cutting edge work. Reading and understanding these papers requires a lot of context and takes time. Research is a skill that takes years to learn.
After some time has passed and we collectively gain a better understanding of a problem, academic papers may seem abstruse and overly complicated, but when these works were first published, this was the best way to understand them. For somebody looking for a recipe solution to a problem, an academic paper is likely not an the ideal place to look, which is why we write books, blog posts, etc... as we come to better understand a problem and its solutions.
Discoverability is hard. I mean, forget academic papers, take a look at just the web, and how many years it took Google to create a search engine that made discoverability easy, and that exploited (originally) the graph structure of the web. Papers do cite other papers, but how do you check a paper's relevance? Moreover, papers often describe something very specific, a specific approach, and searching for that approach is a lot harder than looking for a page that is "Haskell tutorial".
An open journal isn't the answer here (it is the answer for other things though imo) and I'm not sure why you're getting so angry.
You also can't make use of data if it doesn't have appropriate metadata indicating how it can be used. Metadata is more than just "good"- it's essential for actually being able to run an analysis. Take fMRI for example- someone could just throw a brain image up on the internet. In fact, one particularly uninformed individual (who possibly broke Virginia state law) did so here: https://github.com/dcunited001/mri-scans
However, the raw data is not useful without knowing what scan sequence was used, slice-timing order, etc. Some of this is encoded in the data itself (the example above uploaded DICOM, which has somewhat rich metadata, although by no means is it comprehensive enough), but other data that is necessary is not (Phillips in particular is somewhat infamous for doing non-standard things with their scanners that aren't recorded in the DICOM headers, and in general it's a Herculean task getting MRI vendors to agree on standards). You could try performing an analysis with contextless data, but you're likely just going to make really spotty conclusions.
> I'm not in academia
Why should that matter? The circumstances of data collection are also important for business intelligence data or financial modeling data. I'm certain that the lack of this also contributes to poorly formed conclusions that cause real hurt to businesses out there.
In any semi-decent peer reviewed venue, you would cite a wide variety of papers that solve a similar or related problem or introduce a concept related to the method in the paper.
The related work section of a paper is one of the most important parts since it puts the research into context. By citing other work, the authors explain what has already been done, and what contribution their work makes.
A related work section should also illustrate the downsides, limitations, and differences of other cited research. Limitations of other works are often poorly understood since very few people have had the time to evaluate them beyond the initial experiments done before first publication.
Research is not simply about presenting new techniques but also understanding the trade-offs that arise when choosing different solutions to a problem.
This of course comes up during the peer review process as well. The referee informs you of a paper that is tangentially relevant, but you couldn't find that during your literature search because it is paywalled. How relevant was it actually to the work you performed or to people reading your paper if it is not readily possible for you (and possibly your readers) to access the paper?
Citations in academia are often times more the currency of the trade than actual scholarship. There is a reason review articles exist, not every paper needs to be a review article in its own right.
Reproducibility is important but a completely different topic and I don't know why you're bringing it up here.
Note that I don't say that SethBling is wrong. Seen as tinkering, this is perfectly fine, and he did a great job with the video. No hard feelings. I just use it as an example of how it is not research, because of the lack of scholarship.
Mayhaps, you need to do a bit more "youtubing" to get your stuff out there and more accessible. A couple of my current favourites on youtube is 3Blue1Brown and Mathologer. They are both entertaining as well as being informative and have set me on investigatory paths for other research in various fields.
And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh.
Maybe we can refresh some standards for academic writing while we're at it.
The number of pretentious title-holders (even at the lowest minimum levels) I've met seems unending. I could get into that one... but I won't.
Instead you've just reminded me of yet more subjects to stack on my reading list.
tl;dr Seth Bling directly references NEAT and his links[Show More [1]] link to 2 Togelius papers[13][14] among many others !
To wit, in OP, Togelius mentions Seth Bling's MarI/O[1] as lacking scholarship because he didn't mention Togelius:
" seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. "
Rather disdaignfully (sadly/ ironically/ poigniantly) Togelius refers to Seth not by name but only as " some guy ".
Yet Seth Bling[2] in his short video, mentions and explains NEAT and directly links Ken Stanley's and Risto Miikkulainen's foundational NEAT paper[10] at U.texas which Togelius builds on [hint: click SHOW MORE under Seth's video]
And much more, i.e.: ---- the innovative toolchain Seth uses: Bizhawk to script SNES emulators in Lua [12]!
---- Seth links the wikipedia page on Neuroevolution[11] which directly references 2 of Togelius's own papers[13][14]
Seth's [Show More] Links on the MarI/O video[1] are very very comprehensive and (IMHO) indeed a great example of scholarship !
Togelius is a top flight scholar, and publishes in the field of AI game players and level generators[15] so I hope he will welcome some random tinkerer pointing out his error.
[full disclosure: I am a Togelius fan]
To give Togelius his due he does at least link to Seth's MarI/O video.
Perhaps Togelius failed to click / read the Show More links ? The charitable benefit of doubt principle is accidental oversight.
Seth Bling is primarily a Twitch Streamer[2] and Speed Runner[3] who often popularises complex programming[4], math and academic ideas[5] to a primarily young audience using Mario and Minecraft[6] and takes requests via his subreddit[7]
Seth's very popular Youtube video MarI/O which uses NEAT (Neuro Evolution of Augmenting Topologies] to learn and play Super Mario World, has led to a huge number of streamers on twitch playing other MarI/O levels[16], using RNNs, LSTMs, and other SMW levels and SNES games, like RNN multiplayer x4 Mario Karts[17].
The OP article is great, academic rigor and bibliographic scholarship might sometimes differentiate the full time university scholar from the amateur researcher, enthusiast or tinkerer.
Seth and his many followers may not be 'scholars' as in academic research but it is an amateur community of mostly youngsters doing their own research by having fun and sharing ideas, mostly undocumented outside their own transient twitch streams.
Compare this to the well respected amateur astronomy community and perhaps a responsible academic approach is to try to reach out and tap in rather than adding to the distance.
By his own OP standards of Togelius is lacking in that hasn't discovered/ mentioned the work done by the Twitch/ Bizhawk community - it is not entirely undocumented [16].
Togelius your AI Mario competition[8] is legendary in academic AI circles but not necessarily widely known by youngsters playing Mario ( and to be fair AFAIK you don't publish in their forums like twitch) although the code Infinite Mario[9] you base your work on is by Marcus Persson[10].
Though Persson is better known as Notch [10] ( author of Minecraft ) who is more famous than perhaps even Alan Turing , Yann le Cun, Jurgen Schmidhuber, or Geoff Hinton to the under 16 crowd.
Perhaps Togelius could mashup some of his work and do a crossover episode with Seth or relaunch the Infinite Mario competition[8] / benchmarks to this new younger crowd evolving Neural Topologies to play Mario games - or even help document some of their fun.
As public servants funded by tax dollars it is perhaps the responsibility of the academic to reach out rather than blaming fun loving amateurs popularising and doing research / 'tinkering' for not putting in the due diligence of scrupulously mentioning every source.
[IMHO] I agree amateurs don't always do full and proper scholarship and links are food for hungry minds - but they aren't publicly funded researchers but amateurs working for love of the subject in their spare time, for free, or for streaming donations and youtube hits, using transitory, live media and whose audience must be constantly engaged and may have shorter attention spans than the average academic.
[full disclosure I am an amatuer neural net nerd]
[1]Seth Bling's MarI/O https://www.youtube.com/watch?v=qv6UVOQ0F44
[2]https://www.twitch.tv/sethbling
[3]https://www.youtube.com/watch?v=-spFoon7klA sub 1 min SMW via credits warp
[4]Bling's arranging shells in SMW to human inject Flappy Bird code into SNES https://www.twitch.tv/videos/57032858
[4]Bling's Atari 2600 emulator built in Minecraft https://www.youtube.com/watch?v=5nViIUfDMJg
[5]Seth Bling's RNN Mario Cart race https://www.youtube.com/watch?v=Ipi40cb_RsI
[6]Bling's Minecraft Redstone tutorials https://www.youtube.com/watch?v=DzSpuMDtyUU&list=PL2Qvl4gaBg...
[7]https://sethbling.reddit.com
[8]Togelius 2009 Infinite Mario AI competition http://julian.togelius.com/mariocompetition2009/
[9]https://web.archive.org/web/20080423023424/http://www.mojang...
[10]Stanley Miikkulainen NEAT paper http://nn.cs.utexas.edu/?stanley:ec02
[11] https://en.wikipedia.org/wiki/Neuroevolution
[12] Scripting Lua in Bizhawk Emulator http://tasvideos.org/Bizhawk/LuaFunctions.html
[13] Risi, Sebastian; Togelius, Julian (2017). "Neuroevolution in Games: State of the Art and Open Challenges" (PDF). IEEE Transactions on Computational Intelligence and AI in Games.https://arxiv.org/pdf/1410.7326v3.pdf
[14] Togelius, Julian; Schaul, Tom; Schmidhuber, Jurgen; Gomez, Faustino (2008), "Countering poisonous inputs with memetic neuroevolution" (PDF), Parallel Problem Solving from Nature https://www.academia.edu/download/30945872/poison.pdf
[15] Togelius' Game AI book http://gameaibook.org/
[16] https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat
[16] https://thenextweb.com/artificial-intelligence/2018/01/03/th...
Anyone interested in adding a soupçon of scholarship to Seth's project?
Words do mean different things to different people, in different contexts..
There is a recent body of literature that explores the modern "maker" [3] movement. However, "maker" as a term may not have been a good fit for the OP's argument [1], which contrasted (academic) researchers with so-called "tinkerers".
An alternative term for "tinkerer" might be "bricoleur", a loanword from French. (Roughly, it still means one who tinkers: https://en.wikipedia.org/wiki/Bricolage but has other meanings depending on the academic lens.)
Given that we are discussing AIs that play, in the context of education, we can also go back to Seymour Papert's work on https://en.wikipedia.org/wiki/Constructionism_(learning_theo... .
Originally known for work on _Perceptrons_ with Marvin Minsky, AI researcher Papert later adapted theories from education towards the vision of "learning-by-making" and the (young) bricoleur [4]. This can approach can be seen in the evolution from 1960s graphical [turtle] https://en.wikipedia.org/wiki/Logo_%28programming_language%2... to Lego https://en.wikipedia.org/wiki/Mindstorms_(book) to modern day efforts to encourage coding-for-kids [5,6,7].
One of Papert's later collaborators, Sherry Turkel, discusses bricolage as it applies to programming -- https://en.wikipedia.org/wiki/Bricolage#Internet .
When it comes to early education, Turkel argues for epistemological pluralism [8] and cites anthropologist Levi-Strauss in comparing analytic science with a "science of the concrete".
We can appreciate both Seth Bling's concreteness [9] and Togelius's original papers for academics. Almost a decade ago, Togelius introduced Super Mario Brothers as a benchmark for reinforcement learning [10] and, with Karakovskiy, for AI more generally [11].
deepnet what's your interest in neural nets?
[1] http://togelius.blogspot.com/2016/04/the-differences-between...
[2] https://news.ycombinator.com/item?id=16744694
[3] For example, https://scholar.google.com/scholar?&q=maker+movement
[4] http://www.papert.org/articles/SituatingConstructionism.html
[6] https://www.apple.com/swift/playgrounds/
[7] https://github.com/google/blockly-games/wiki
[8] http://www.papert.org/articles/EpistemologicalPluralism.html
[9] https://www.youtube.com/watch?v=qv6UVOQ0F44