Handwriting Generation with Recurrent Neural Networks
cs.toronto.edu
cs.toronto.edu
Edit:
holy crap I had a revelation (that a lot of people probably already had before). Consciousness is lagging after actual decision making process (sometimes up to 150ms according to some studies). What if consciousness is an additional layer of back propagation mechanism for our brain? Analyse the decision post facto taking bigger picture into account, make corrective action if bad outcome. Its just a small process governing a bunch of primitive NNs.
Oh! I see what you're saying. What a crazy neat way to think about consciousness. Are we all error correctors?
Incidentally, these circuits are chaotic. They are extremely sensitive to initial conditions. These conditions are in part the output of other circuits, which are chaotic themselves. The circuits can be trained to perform better and better, by constantly comparing actual performance to emulated performance. However they will rapidly degrade. This is why even top athletes benefit from warming up before competition: training their circuits to reach maximum performance, because they will have degraded overnight. That degradation may be marginal, but at elite sports level those margins make all the difference. The best athletes are able to train their physical circuitry as well as their emotional and cognitive circuitry.
The central nervous system contains emulators (already suspected by some researchers). Sensory feedback after completion of a goal-driven task is compared against output from an emulator that has been asked to complete the same task, in parallel The emulator runs faster, partly because the circuitry will simply be shorter.
This is suspected in musculoskeletal control, and I believe it may happen outside the musculoskeletal domain too.
Consciousness may only lag behind the emulator, not the "real world" sensory feedback.
This may offer a counter, or adjunct, to the proposal that consciousness is simply the brain experiencing itself.
The lag may be back propagation. It certainly seems to be some sort of computational overhead.
-of course I like her because Im into big butts, and 3 is totally my lucky number!
I'm not an expert but... Fly brains and human brains already have have neural networks for common tasks, like sensing motion, baked into our genetics. Evolution has already trained these networks to an extent. Training a computer-simulated neural network seems to yield similar results to what nature has done.
My personal uninformed opinion is that we'll start to see how much human experience and expression is driven by our brain wiring. That we'll discover how similar we all are to each other based our wiring. And that we'll find that approximating advanced cognition will be the result of putting that wiring into a computer model -- and having a computer powerful enough to run it.
Typical machine learning problems deal with isolated training sets and isolated problems. This approach seems strange to me; in the case of neural networks, this is somewhat analogous to a newborn child who is deprived of all senses except the limited training data to make up their world, and good/bad feedback from the loss gradient. How can one expect this hypothetical newborn to learn any meaningful representation of the world with which our machine learning problems are derived?
I think the first step towards realizing anything like "Hollywood General AI" will be a system that spends an early portion of its existence ingesting a universe of contextual data, before it is presented with a problem to solve (at which point it can make use of seemingly unrelated information to do something like handwriting). Andrew Ng's work on self-taught learning (built on transfer learning) is particularly relevant here, but I think those ideas could be taken a lot further.
We want an AI trained with ML to keep focusing on flying plane. We may not want an AGI pilot who can also get bored just like humans and can get distracted playing games.
In other words, we will start thinking about people as if they are mere things. Complex things, but things nevertheless.
My mind will change when my brain does.
I wouldn't say that this is cognitive but it definitely learns to perform a skill in a similar fashion to intelligent beings which is remarkable. Extrapolating that behavior to self-awareness isn't likely however (of course this could lead to a philosophical debate)
Human language is not based on set theory. There is a huge difference in connotations here.
You seem to be using "merely" to mean "nothing more specific than," which is a very different meaning that would not be appropriate when talking about things which are subsets of other things.
"Are we sure what is being simulated here is cognitive? It seems to me merely mechanical."
This is analogous to asking "are you sure this is done by humans rather than animals?" The question makes sense and is valid. To answer it with "humans are merely animals" would not address the subject at hand and, again, would have specific connotations.
I see that this kind of semantic acrobatics is extremely common in discussions of AI on HN.
1. Have trained a networks to generate handwriting. Handwriting can be objectively judged, unlike those images. The interactive demo does look pretty darn cool.
2. Made it so the network can be "primed" by small samples of new handwriting. This is the most impressive part, because it is something that actually qualifies as semi-generic AI. You train it on one thing and then it does another thing.
Actually, the second part is bit too impressive, compared to everything I've seen from NNs before. I've read the paper, but I don't understand how this priming works. How big is the primer? Is it just the sample shown on the website? Are there any constraints? Can I simply supply a sample of my writing with a textual "translation" for this network to pick up some aspects of my writing style?
Google wasn't training their NNs to 'generate weird-looking images'. The NNs were for image-classification (which is pretty impressive), the weird images were just a visualisation/artefact of the training process.
The difference between the two demos is that the impressiveness of those images is mostly subjective, while handwriting replication has pretty well-established criteria to judge it by. Also, it sounds like priming is fast and does not require tons of samples.
For an easy example look at human babies - they're not conscious (maybe self-aware is a better word?), but we can see this development happen over time. Maybe sleeping has some role in recalculating the weights for the neural net? Eventually something like a feeling of self emerges as the evolved mechanism for working with social feedback. Maybe this is the general way intelligence works.
It does seem to make sense when looking at how it develops in animals.
I'm not exactly a deep-learning fan, but could you post the links?
I'd go on to venture that our minds probably use discriminative models for "intuitive" judgements, while reasoning that can explain the variance in percepts uses generative models, thus obtaining better performance where necessary despite using more energy. Or possibly, the generative models can be used to train the intuitive discriminative ones, slowly allowing the less intensive part of the mind's processing to adjust its class boundaries to suit what's really known.
The killer app in this space will be when someone figures out how to extract a vocal model from existing recordings of singers. Vocaloid already synthesizes singing quite well, but a human singer has to go into a studio and sing a long list of standard phrases to build the singer model. The next step will be to feed existing singing into a system that extracts a model usable for synthesis.
The RIAA is so going to hate this.
I understand it's hard but it also sounds like a fun project for people with the relevant know-how.
Reminds me of this article from 5 years ago: http://www.dansdata.com/gz103.htm
A cover band has to license the underlying composition, but not the recording they're covering. (This means ASCAP gets royalties, but the RIAA does not.) In the US, there's a compulsory license for compositions, and you can record and distribute any song by paying a relatively modest fee set by law.
This is just automating the cover band industry.
In ten years or less, this will be a common feature in DJ consoles, and we'll hear songs from musician A as if performed by musician B.
The RIAA is really going to hate that.
http://i.imgur.com/cFrlyy8.png
The input was copied from the instructions - "Type a message into the text box, and the network will try to write it out longhand". But you can see it skipped the "e" in "Type" and added an "h" after the "w" in "network", and pretty clearly spelled "to" as "du".
It also tried to cross the first vertical line of the "w" in "network" in lieu of adding an actual "t" beforehand (which is arguably an idiosyncrasy a human's handwriting might have, if a rather odd one); and stuck a big phantom stroke/letter between "T" and "y".
Unless I misunderstood how this was developed...that could be the case ha!
And so begins the devaluing of that proof. Just like when marketers started reproducing the "signature" on every sales letter with blue-colored toner, mimicking the authenticity of a hand signature.
I don't write handwritten letters, and I don't romanticize the past. But our dwindling ability to assess the authenticity of incoming communication is slightly unsettling.
Relative paths allowed from user input is usually a HUGE warning sign. Are you sure I can't make it open arbitrary files? What happens to your cgi script if it reads a file in the wrong format?
Text entered: this is a test of handwriting generation
Style sample #1 selected.
All other settings at default.
Edit: I've tried a couple other styles and haven't duplicated this craziness.
Since the net can't really be 100% sure of anything, it leaves itself a little uncertainty in case the human does something crazy. So sometimes the result will seem to be a little random than real writing because of it's uncertainty.
The bias parameter tries to fix this by biasing it's output towards more probable sequences.
I don't know what data it's using - it's quite possible it's corrupted by people drawing randomly or something.
lol
!@#$%^&*()_+
As I turned up the bias, it seems to trend to the letter g:
Some symbols seem to just fail, I'm guessing that's a bug in the serializing.
I could see it being used in games to generate hand written notes from data files.