supercarrot

I've put fracking numbers to point out the comparison.

Animal agriculture is being done everywhere.

I'm doubting people are eating meat from Brazil, Australia and Denmark in US, Asia or Africa so much that these are the main dots of world animal agriculture.

fracking - 70 to 140 billion gallons anually

animal agriculture - 34 to 76 trillion gallons anually

at least we know what wastes a lot of water, the solution is simple, but is not as tasty.

as for heating, yes, it's quite a problem.

withouthotair.com is a great resource for the approximate numbers.

This is implied through research on reductions in machine learning.

That simple models can solve complex tasks.

For example, you can do multiclass classification, cost sensitive (importance weighted) multiclass and binary, quantile regression, structured prediction (as is done with HMMs, CRFs, MEMMs, structured SVMs etc.), just using a binary classifier.

So, if your implementation of that binary classifier is efficient and performant, you'll be (given that your reduction is consistent) efficient and performant on any of the above tasks.

What the authors of the paper above did, is that they rediscovered some old tricks, removed the theory of reductions, and that's that - without referencing vowpal wabbit that does way more useful tricks. I'm not sure why, because VW team consistently references Leon Bottou (out of all others) that is member of FAIR, and has been using implementation tricks for decades.

Their log(k) implementation is probably less performant than the one-against-some consistent reduction in VW due to the latter having better theoretical bounds on performance.

as much as it's not part of the standard, __restrict__ allows limiting the effects of pointer aliasing in C/C++. [1]

[1] : https://en.wikipedia.org/wiki/Restrict

> IMO RNNs do need some kind of structured loss (more than per step likelihood) to be competitive with HMM approaches using Viterbi decoding

This is exactly what they do with the dependency parser I've cited, so your opinion is definitely valid. Although their approach is not general, given the fact that they approximate hamming loss with log-loss and again make it work only on sequences.

http://arxiv.org/abs/1502.02206

paper above also has a very good analysis on how to remove the search component of the inference and allow linear time complexity with competitive results.

consistent (as it is used in the machine learning theory of reductions) reduction from structured learning to multiclass classification seems to be possible. I just haven't seen anyone couple the learning procedure with neural networks. (Daume did mention they trained RNNs with the reductionist approach but seems that the code didn't make it to vowpal wabbit).

the approach above works with any loss you want (from F-score to any weird thing you might think of), the loss doesn't have to decompose over the structure (one can just announce the loss after the labelling is done and learn from that loss), it can work on any kind of structure, from images to sequences to documents for translation. it can also use a O(log n) consistent reduction of multiclass classification if speed is of the issue and if number of classes is large. It can easily work as an online method too, not requiring the full structured input.

for example, simple sequence tagging works (depending on the number of possible labels) around 500k tokens per second :D word count is only 2-4 times faster than that :D

there still aren't any papers using the above consistent reduction in the framework of NNs but I guess they'll soon be coming.

Still, beam search can be simulated and improved by running decision processes in parallel.

For example, instead of learning the sequence labeling as a sequence of n decisions (where n is the length of the sequence) you can learn sequence labeling as a sequence of 3n+1 decisions where you make 3 decisions for each sequence element and after 3n decision pick one out of three decision streams that minimizes loss using an extra decision. (when inference is done then the classifier will, hopefully, pick the stream that minimizes test loss).

This simulates a beam search and can be done during learning and inference and is probably more effective than picking confidence scores of particular decisions and keeping a beam of most confident partial sequences.

Bean search is a heuristic thing that improves performance and is done mostly to allow you to correct mistakes you made at the beginning of the process.

http://arxiv.org/abs/1603.06042

they illustrate the problem well (label bias).

the question remains the same, for example, in the paper above they approximate the partition function of CRFs with a beam but get superior results to other structured prediction methods.

Interesting observation of computational complexity.

HMMs require Viterbi algorithm to find the desirable sequence. The complexity of the algorithm depends on the number of previous decisions on which we condition the current one. Time complexity is O(n^(k+1) * S) where n is the length of the sequence, k is the number of previous decisions we condition on, and S is the number of possible decisions.

Now, why does HMM require the Viterbi search procedure, and LSTM/RNN doesn't?

Inference in LSTM is linear in the length of the sequence -- ignoring the classifier decision time.

Why does this work without Viterbi?

Given the fact that even Conditional Random Fields need dynamic programming, and that Maximum Entropy Markov Model suffers the inability to learn quickly without dynamic programming and make inference not suffer label bias.

What is so special about LSTM that it doesn't need DP.

RNN obviously learns a much better representation of features. Still, why then won't HMMs work with that same representation without the search part?

Really baffles me.

Although, there exists a process that doesn't use NN for sequence labeling and it is linear in the length of sequence and works well. Instead of using a simple classifier as is the case in MEMMs, one can use a cost-sensitive classifier and learn for each decision associated future loss and greedily avoid the loss. This allows the usage of a simple multiclass classifier, removes the search part and can work as well as NN given the right features.

Another interesting thing is that perceptron can approximate any non-linear function with just a single hidden layer.

http://cstheory.stackexchange.com/questions/17545/universal-...

2048 or somewhere around that is the time when all of the ocean ecosystems will be completely destroyed.

we're living in some crazy times.

2011. I'll edit it.

"China aims to have 100 gigawatts (GW) of wind power capacity by 2020, and the nation’s leaders plan to expand installed solar capacity to 20 GW during the same period. These are truly astonishing goals, and, if China even comes close to accomplishing them, it will become the world’s renewable energy leader. But there is a problem. Total Chinese electricity generation capacity is 900 GW currently; with seven percent [GDP] growth, that means the nation’s electricity demand in 2020 will be something like 1800 GW. Wind and solar together would supply less than seven percent of that. The only thing likely to boost that percentage much would be a dramatic reduction in growth of energy demand to, say, two percent annually."

I like this from The End of Growth (2011) by Richard Heinberg.

As much as the numbers are incredible we are really undermining the amount of energy produced by fossil sources.

Similar problem with nuclear.

Are we really on the exponential road to sustainability? I hope so.