The Man Who Invented Information Theory (2017)
bostonreview.net
bostonreview.net
He also did early work in cryptanalysis, AI (minimax chess algorithm and a learning robotic mouse). His student was Ivan Sutherland. More or less, whether its networking, signal processing, compression, crypto, machine learning, circuit design, basically anything to do with the digital age, you'll find Shannon did important foundational work there.
Pretty much anyone doing signal processing, which is part of the standard physics curriculum where I'm from, will be introduced to the Nyquist–Shannon theorem while Church and Turing did fundational work in logic and computability but with far less practical application.
The reason Claude Shannon is a legend has a lot to do with the fact that his ideas are not just correct and draw from his multidisciplinary knowledge, but also expertly communicated. It is an incredibly readable paper and only absolutely requires a little bit of mathematics when he is describing how to convert a state machine for Morse code to a matrix which would allow someone to calculate how many bits per time unit can actually be transmitted in Morse code -- and this is not absolutely essential, it is calculated in a different way and also it is communicated that you can easily take his word for it that it is whatever numerical value it was. A lot of the arguments about fitting a stream of symbols into an encoding for noisy channels have these lovely diagrams that help to elucidate exactly what he's talking about.
If you want to make that same sort of impact, it is not just important to be a great mind, but to spend a bunch of time practicing how you communicate that information.
Sadly, many extremely smart and profound scientists are quite incapable of (or not interested in) conveying clearly their thoughts to general audiences.
The path integral formalism or the diagrams, albeit being mathematically obscure at first where quite clear from the intuitive viewpoint once he explained them.
The most insidious cause of this is that getting a permanent position in a department still often depends on publishing in the top journals within that department's subject, which means explaining it only to the point that it is coherent to that subject's experts. Any further elucidation is sometimes seen as simplification or over-analysis, and to the detriment of getting an article accepted in subject-leading venues.
Or, to put it another way, putting more bits down the channel than are needed for comprehension by the editor is seen as unnecessary redundancy.
On the other hand, with Information Theory, Shannon was considered to have been ahead by decades.
The closest related work that is considered to come from an independent line of inquiry (taking place in the USSR), but similarly fundamental is Kolmogorov complexity (algorithmic entropy) by Andrey Kolmogorov [1] in 1963.
While Shannon made the huge leap on his own, it should be noted that Harry Nyquist [2] (his colleague at Bell Labs) laid essential foundations through his Nyquist Stability Criterion and studies on bandwidth. This came after harmonic analysis (Fourier transform, etc.) appeared in the 1800s.
[0] https://en.wikipedia.org/wiki/Herbert_Matar%C3%A9
(emphasis in the original)
I find that second sentence humorous every time I read it... I wonder if he had seen the current state of internet discussion if he would have changed "frequently" to "occasionally".
Which is accurate. A lot of Internet communication is a tracker phoning home, TLS handshakes and other communication with content but no "meaning" per se. In Shannon's day, such industrial communications--over telegraph--were common enough to warrant mentioning but not prevalent enough to make up the bulk of human communication traffic.
The bits are the same, the meaning is different.
[1] https://www.amazon.com/Introduction-Information-Theory-Symbo...
If anyone is interested in learning more about Bell Labs and the folks who worked there, “The Idea Factory” by Jon Gertner is a fantastic book written on the subject. It’s not comprehensive but it’s a very inspiring read.
There has always been something very vexing about Bell Labs’ legacy though. They had everything they needed to start the personal computing revolution: engineers, scientists, equipment, a nationwide telephone network for god’s sake. What happened?
Even though that brought the lab's demise, its then-younger staff and their mentees continued their work at other organizations, including many at Google.
A formal study of some of the consequences:
https://economics.yale.edu/sites/default/files/how_antitrust...
Unfortunately, a lot of this research was buried in microfiche in university basements as Bell Labs' legacy was traded from company to company without a viable distribution mechanism. As a result, although this work is now all available on-line (sadly not in open-access form), there are still more forgotten gems in there than many researchers realize.
Another book with a lengthy section on Claude Shannon is James Gleick's The Information: A History, a Theory, a Flood [1.]. A shorter but still nice explanation in on Brain Pickings [2.].
1. https://www.goodreads.com/book/show/8701960-the-information 2. https://www.brainpickings.org/2016/09/06/james-gleick-the-in...
... about the guy who gets credit for figuring out BOTH how to make computers do math, AND how information can be encoded, transmitted and manipulated.
Also: as much as people talk about information theory in various contexts, I doubt that many take the trouble to understand it better. Like thermodynamics, people want to take away an overly broad, folksy interpretation and apply it everywhere without stopping to think if it really applies in the way they claim.
>I don't see anyone claiming it applies in places it doesn't.
It applies in a lot of places, but probably in a more narrow way than people think. You can say "information theory applies here" and probably be right, but to be specific about what that means requires some work. Shannon says that himself in that piece.
Thermodynamics is an easier one to see, because people always make claims about something "because of the 2nd law of thermodynamics", something about everything always becoming more disordered. But that's too simplistic a view, and fails to consider what is a closed system and so on.
https://books.google.com/books/about/The_Idea_Factory.html?i...
If I remember right after 20 years.
I think this "optimum discernment" at base 2 you remember is nonsense? could you provide a pointer?
I think he was uncomfortable with the fact that the continuous signal story was not as well fleshed out. He came to revisit that after about a decade in "coding theorems for a discrete source with a fidelity criterion". This did for lossy compression what his 1948 paper did for lossless.
It rarely happens that a paper that creates a field also resolves its most important questions. That was Shannon's standard, I guess. That he revisited the question perhaps speaks to his inner discomfort that the story was not complete.
That was nearly 20 years ago.
Everyday I appreciate that statement, and Claude Shannon, a little bit more.
https://www.worldcat.org/title/grammatical-man-information-e...
[1]: https://www.amazon.com/Fortunes-Formula-Scientific-Betting-C... [2]: https://en.wikipedia.org/wiki/Edward_O._Thorp
When I saw that masters thesis. I literally hyperventilated.
https://medium.com/@_NicT_/subtractive-adversarial-networks-...
https://www.boardgamegeek.com/geeklist/143233/claude-shannon...
> Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state.
https://arxiv.org/abs/1806.08053
Roughly speaking: The amount of computation or energy needed to perfectly reproduce a random source, such as a coin flip, is high, while the significance or meaning, for the average receiver, is low. Natural language text requires less computation to reproduce [1], but, for the average receiver, the significance is higher.
Also, what about crystalline forms, which are very orderly and require minimal computation to reproduce, but are equally insignificant for the average receiver?
More or less correct. The key difference is that you could not compress a random coin flip sequence (and that a compressed text is meaningless until decompressed to original).
> all minimal programs are by definition Kolmogorov random
Compression provides an upper bound to K. Kolmogorov Randomness itself is not computable. AKA: You can't ever know if you have a minimal program.
> Crystalline forms
It is possible to both have low significance and low information content. Crystalline forms were very significant to Turing though: https://en.wikipedia.org/wiki/The_Chemical_Basis_of_Morphoge...
With this in mind, you can see text as the output of an algorithm (brain) taking many such decisions. The information entropy contained in this text reveals the complexity (amount of distinct binary decisions) which needed to take place in the machine (brain) in order for the text to occur.
At my uni they only had communications theory, which covered stuff like software defined radios. Information theory was a significant part of it though.