Ask HN: Older textbooks/papers you consider classics still worth studying today? | Hacker News Reader

Ask HN: Older textbooks/papers you consider classics still worth studying today?

237 pointswebmaven85 comments

Claude Shannon's original 1948 paper "A Mathematical Theory of Communication" launched the entire field of information theory. It's 50 pages, highly readable, and pedagogical. The source of its magic is that Shannon introduces and concretely grounds an essentially new ontological concept of vast applicability. And it has 100,000 citations.

http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

This could have easily been 3-4 landmark papers, but instead its packed into one cogent idea.

That common interview question about query autocomplete/sentence completion? Shannon solved it and demonstrates it in this paper, almost a decade before FORTRAN existed. New grads still struggle with that problem. PhD's still struggle with that problem.

Pretty much every machine learning classifier is using a loss function described in that paper.

I always thought it'd be a really cool start-up idea to have a service that prints, binds, etc.. and mails you within a few days a printed paper like this.

I probably have 10 papers floating around, loose pages. Annoying. I print them when I want to read them but of course rarely can immediately.

After skimming the first page I now feel slightly foolish for not having realised before that a bit is short for binary digit.

Modern Higher Algebra by A. Adrian Albert (1937, Dover/Cambridge). It covers both abstract algebra and linear algebra.

Most modern textbooks tend to approach linear algebra from geometric perspectives. Albert's text is one of the few that introduce the subject in a purely algebraic approach. With a solid algebraic foundation, the author was able to produce some elegant proofs or results that you don't often see in modern texts.

E.g. Albert's proof of Cayley-Hamilton theorem is essentially a one-liner. Some modern textbooks (such as Jim Hefferon's Linear Algebra) try to reproduce the same proof, but without setting up the proper algebraic framework, their proofs become much longer and much harder to understand. Readers of these modern textbooks may not realize that the theorem is simply a direct consequence of Factor Theorem for polynomials over non-commutative rings.

With only about 300 pages, the book's coverage is amazingly wide. When I first read the table of content, I was surprised to see that it not only covers undergraduate topics such as group, ring, field and Galois theory, but also advanced topics such as p-adic numbers. I haven't read the part on abstract algebra in details. However, if you want to re-learn linear algebra, this book may be an excellent choice.

Sorry about that. I shall try to do better.

This is the piece I love about HN. A comment refers to an accomplished person and he/she happens to be right there. And either supplying a correction or taking the feedback positively.

For those who are reading this, let me stress that it is not that Prof. Hefferon's proof of Cayley-Hamilton theorem is bad (it is actually better than some really horrible proofs that appear in some well-received textbooks), but that Albert's treatment is superb --- it is far better than the treatments of the theorem in most modern textbooks, including Prof. Hefferon's. Also, I was certainly not commenting on the overall quality of Prof. Hefferon's book, and I thank him for offering his textbook for free.

Oh, forgive me, no offense taken. I should have put a smiley. I read your post with interest and shall check out the book.

(As you no doubt know, different books have different audiences. Before I wrote my Linear book, when I looked at the available textbooks I thought that there were low-level computational books that suited people with weak backgrounds, and high-level beautiful books that show the power of big, exciting, ideas. I had a room with students who were not ready for high. I wrote the book hoping that it could form part of an undergraduate program that deliberately worked at bringing students along to where they would be ready for such things. Naturally, with that mindset I read your post as meaning that the audience for the book you described is just different. Anyway, thanks again for the pointer.)

"The Art of Electronics" Paul Horowitz, Winfield Hill. Electronics for people who want to do stuff.

I loved this book as a teenager and recently got reacquainted with this after years by the almighty AvE on youtube when he took apart this an old Helicopter Radio/Telephone here: https://www.youtube.com/watch?v=6eoBj5W7Vdc

Great book. Do get the most recent edition, because the recommend parts in the older ones are quite outdated.

I adore the "good circuit / bad circuit" ideas interspersed throughout that text.

When I was an undergrad (enrolled in 2011) I decided to screw my professors book recommendations and lookup what the internet seemed to think was the best book in each subject and read that instead. Most of them were fairly old. Some examples that I remember off the top of my head:

Intro to CS: SICP (1979)

Algorithms/data structures: CLRS (1989)

Theory of computation: Sipser (1996)

Compilers: Dragon book (1986)

Calculus: Spivak (1967)

Linear Algebra: Dover's by Shilov (1971)

The given year is for the first publication, some of them are still being updated and I probably read a newer edition.

Code: The Hidden Language of Computer Hardware and Software by Charles Pretzold

Reading the book is the most beautiful and simple way that a person can really understand what a computer and come to the realization that it is not black magic.

https://www.amazon.com/Code-Language-Computer-Developer-Prac...

This is a famous one from 1999 - "Why Johnny Can't Encrypt" [1]

There's a couple of follow-ups too, such as "Why Johnny Still Can't Encrypt" [2], and "Why Johnny Still, Still Can't Encrypt" [3].

[1] https://people.eecs.berkeley.edu/~tygar/papers/Why_Johnny_Ca...

[2] https://cups.cs.cmu.edu/soups/2006/posters/sheng-poster_abst...

[3] https://arxiv.org/abs/1510.08555

"There's Plenty of Room at the Bottom"—Richard P. Feynman (1959) http://www.phy.pku.edu.cn/~qhcao/resources/class/QM/Feynman'...

"Zen in the Art of Archery" — Eugen Herrigel (1953)http://www.ideologic.org/files/Eugen_Herrigel_-_Zen_in_the_A...

"What is it like to be a bat?" — Thomas Nagel (1974) https://organizations.utep.edu/portals/1475/nagel_bat.pdf

"The Tragedy of the Commons" — Garrett Hardin (1968) https://www.hendrix.edu/uploadedFiles/Admission/GarrettHardi...

World of Mathematics - An amazing compendium of accessible content straight from the masters - Poincare, Jonathan Swift, Neumann, Bishop Berkeley, Cayley etc https://www.amazon.com/World-Mathematics-James-Newman-Hardco... I believe it is available on archive.org

What Is Mathematics? by Richard Courant and Herbert Robbins published in 1941. One of the most beginner friendly yet rigorous books out there for a survey of many areas in mathematics.

You can find hundreds of gems here from erstwhile Soviet Union - https://mirtitles.org/

K&R C is still an excellent resource for learning. C may not be the flashiest new thing, but it's still a very useful skill to have.

For sure get the second edition. The first edition is way, way out of date.

But the typesetting took such a dive downhill!

Yes, it’s a spectacular introduction not just to C but to programming in general.

PAIP - Norvig 1992. It appears to be available online https://github.com/norvig/paip-lisp

Yes, this is the classic. PDF could be found in bittorrent

I worked with an ex-ecology professor on some 'data science' projects at work, who suggested this older book I enjoyed called "The Ecological Detective: Confronting Models with Data." Good read imho for ideas about generating hypotheses, exploring data, and comparing models to explain the data, and not just in ecology (though that is the context obviously).

https://press.princeton.edu/titles/5987.html

Structure and Interpretation of Computer Programs (SICP)

https://mitpress.mit.edu/sites/default/files/sicp/index.html

Computer Graphics: Principles and Practice in C (2nd Edition) is an incredibly deep look at the cutting edge of computer graphics technology as it stood in the late 1980s.

It's full of beautiful renderings and diagrams, covers the core algorithms of 2D and 3D graphics, introduces the mathematics required, and many other related subjects such as user interface design.

Apparently there is a 3rd edition from 2013 which looks at modern GPU-based rendering, though I don't own a copy.

https://en.wikipedia.org/wiki/Computer_Graphics:_Principles_...

I’ve studied both. They are totally different books. The 2nd edition is the “Art of Computer Programming” for computer graphics. The 3rd edition is a fantastic overview of GPU based graphics libraries (the fundamentals not the APIs). Which is to say, they have almost nothing in common.

The Structure of Scientific Revolutions (1962) by Thomas Kuhn is a fantastic book that explores the history of science while also debunking the commonly held belief that discoveries (of gravity, oxygen gas etc) are instantaneous observations, instead of a gradual weaving together of several seemingly contradicting observations.

https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...

If you haven’t read all of the Turing award papers yet, I highly recommend forming a study group if you can and working from the beginning.

Where can I find a list of those papers? I just spent a minute trying to find something using Google but didn't manage to find anything

This book has the first 20. I have read through the first 5 or so and highly recommend it. https://www.amazon.com/ACM-Turing-Award-Lectures-1966-1985/d...

Thanks! I might end up reading that

https://amturing.acm.org/alphabetical.cfm links to a page for each awardee, with a link to their lecture.

Thanks a lot! I totally overlooked the lecture links

The art of getting money (1) by PT Barnum is definitely one of those books. Even though written in 1880 most of the advice transcends time and technology and still evergreen as ever.

(1) https://books.google.co.in/books/about/Art_of_Money_Getting....

"Science and Sanity" — Alfred Korzybski (1933) http://pialogue.info/books/Science-and-Sanity-Korzybski.pdf

"The Presentation of Self in Everyday Life" — Erving Goffman (1956) https://monoskop.org/images/1/19/Goffman_Erving_The_Presenta...

It would help to read your commentary on these.

Well, I don't know if mine will help. I love Goffman's book, it's super-readable, enlightening. It's about the different roles and..settings people operate in, and the rules and customs of those places, e.g. backstage, shopfront, military ranks, hospitality etc. Full of great stories quoted from an impressive number of sources, very diverse.

Korzybski's book used to be huge, recommended by all kinds of famous people. I spent a few hours reading in it one day, to see for myself. (Plus had heard a fair bit about it before.) Korzybski basically seems a huge crank, who thought himself and his baby General Semantics[0] as important as Aristotle. The quote one always hears from it is "the map is not the territory", and well, that's about the only thing worth quoting from it. Plus he tried to get rid of "is" from the language, i.e. "A is B".[1] Seemingly because such sentences are deceptive - if you say "The car is red", well, it's many things besides red, so the sentence is a lie is many ways. It's a very strange objection. As if it's bad because it doesn't say everything, just one thing. Aristotle he's not.

Also there's an interesting contraption featured in the book, made of metal with holes, strings, plugs, used to make maps of levels of concepts. I don't know if it's practically useful.

Apart from that, what makes it a big slab of a book, are a host of chapters on different academic subjects serving as introductions to those subjects, e.g. one on maths, calculus I think, supposedly illustrating general semantics applied there. These seem mostly intended to give the impression Korzybski is a genius polymath. People who didn't know anything about that subject might learn something from that, and feel the book taught them something. But it's nothing to do with Korzybski's theories.

[0] https://en.wikipedia.org/wiki/General_semantics

[1] This was extended by a student of Korzybski's to E-Prime, a language without any form of the verb 'to be'. https://en.wikipedia.org/wiki/E-Prime

http://www.feynmanlectures.caltech.edu/

Not sure you’re looking for philosophy, but I keep a translation of the Tao Te Ching nearby at all times. It’s helped me stay centered and humble.

Just to go meta: whatever book you learned something from originally/in college you should keep. It might not always be the best, but keeping the context of your original understanding can really help and speed up recollection when needed. (This probably applies most to textbooks used for whole classes as opposed to minor topic references.)

Good idea! I am a complete sucker for tracking down CS textbooks from my courses 20 plus years ago. And I like the hard copy versions. My outlook is, an engineer is known by the books he keeps.

A Relational Model of Data for Large Shared Data Banks (1970)

E. F. CODD

https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf

Nice choice.

I would add: An Introduction to Database Systems - Date

Also a few unmentioned so far: Discrete Mathematical Structures - Kolman

Introduction To Systems Analysis And Design - Hawryszkiewycz

Modern Operating Systems - Tanenbaum

How Computers Work By Roger Young, Here is an online version: http://www.fastchip.net/howcomputerswork/p1.html

anytime worth reading, simply tells you how everything in a computer work like memory and processor.

The Dragon book - Compilers. I got more out of it reading it after using it in a university course.

Mechanics and Thermodynamics of propulsion by Peterson and Hill explains most of the rocketry systems still in use today and the first edition was published in 1965. I think it is interesting that fundamental boost rocketry has changed very little since that time.

Very famous paper, one of my favorites - https://en.m.wikipedia.org/wiki/No_Silver_Bullet

Previous posts have nicely covered almost all of the textbooks and papers I would have mentioned myself, including K&R's _C Programming_, Brooks' _Mythical Man-Month_, Feynman's Lectures, etc. The only glaring omission was any mention of the classic (1965) FFT paper by Cooley and Tukey: "An algorithm for the machine calculation of complex Fourier series." _Mathematics of Computation_, 19(90), 297–297. doi:10.1090/s0025-5718-1965-0178586-1 (https://www.eit.lth.se/fileadmin/eit/courses/eit085f/Cooley_...) I might also have added the _CRC Handbook of Chemistry and Physics_ that seems to have been on every working scientists's and engineer's bookshelf (including mine) during the mid-to-late 20th century. Still in print, nearing the 100th edition. Also, some of my old favorite textbooks that I used at University were not mentioned, including Thomas' _Calculus and Analytical Geometry_. I still have my red cloth covered Addison-Wesley 3rd edition, no longer in print, but a much later edition might still be in print.

Euclid's Elements was a pretty foundational text. Also Philosophiæ Naturalis Principia Mathematica

Chandrasekhar's "Newton's Principia for the common reader" is a reading of Newton's book in modern mathematical notation and with commentary on the methods Newton was using.

"Reflections on Trusting Trust" (1984) by Ken Thompson

"The Design of the UNIX operating system" by Bach holds up well. Feynman's lectures on physics (3 volume set). Electrodynamics by Jackson is also up there.

"Mathematics Useful for Understanding Plato", 100AD, by Theon of Smyrna. Amazing.

Something I had read long time ago and also recently recommended by Ed Witten is John Wheeler's essay - "INFORMATION, PHYSICS, QUANTUM: THE SEARCH FOR LINKS" also famously known as the "It from Bit"essay (which now probably is "It from Q-bit" as our current understanding).

http://cqi.inf.usi.ch/qic/wheeler.pdf

"An incomplete list of classic papers every Software Architect should read", https://blog.valbonne-consulting.com/2014/06/09/an-incomplet...

[1] is a repo of papers people...love. Maybe you can grep for dates to find the older ones. The nice thing is that they are sorted by category.

1: https://github.com/papers-we-love/papers-we-love

"On the criteria to be used in decomposing systems into modules" (1972) - because the core principles of modularity haven't changed [https://www.win.tue.nl/~wstomv/edu/2ip30/references/criteria...]

"The Mythical Man Month" (1975) - because human nature hasn't changed [https://www.amazon.com/Mythical-Man-Month-Software-Engineeri...]

"The History of Fortran I, II, and III" (1979) - because this historical piece by the author of the first high level language brings home the core principles of language design [https://archive.org/details/history-of-fortran]

"The Unix Programming Environment" (1984) - because the core basics of the command line haven't changed [https://www.amazon.com/Unix-Programming-Environment-Prentice...]

"Reflections on Trusting Trust" (1984) - because the basic concepts of software security haven't changed [https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...]

"The Rise of Worse is Better" (1991) - because many of the tradeoffs to be made when designing systems haven't changed [https://www.jwz.org/doc/worse-is-better.html]

"The Art of Doing Science and Engineering: Learning to learn" (1996) - because the core principles that drive innovation haven't changed [https://www.youtube.com/playlist?list=PL2FF649D0C4407B30] [https://www.amazon.com/Art-Doing-Science-Engineering-Learnin...]

"xv6" (an x86 version of Lion's Commentary, 1996) - because core OS concepts haven't changed [https://pdos.csail.mit.edu/6.828/2011/xv6/xv6-rev6.pdf] [https://pdos.csail.mit.edu/6.828/2014/xv6/book-rev8.pdf]

Smashing The Stack For Fun And Profit by Aleph One - http://phrack.org/issues/49/14.html

Structured design by W.P. Stevens, G.J. Myers, and L.L. Constantine

On politics and management:

'The prince' - Machiavelli (early 16th century)

Best book I know to understand how the world really works. Here is a link to an abridged version: http://sqapo.com/machiavelli.htm

Politics yes but management? unless you mean managing a country, not a company.

Edmund L. Gettier: Is Justified True Belief Knowledge? (1963)

http://www.ditext.com/gettier/gettier.html

https://doi.org/10.1093/analys/23.6.121

Alan Turing's "Computing Machinery and Intelligence" (1950). Describes the "Imitation Game", aka Turing test.

https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...

L.J. Savage, Foundations of Statistical Inference, 1962

and (if this counts as old) Berger and Wolpert, The Likelihood Principle, 1984

Turing _On computable numbers, with an application to the Entscheidungsproblem_ (1936)

Sipser's _Intro to the theory of computation_ (1996; 3e in print)

Aho, Sethi, & Ullman's _Compilers: principles, techniques, and tools_ - 'the dragon book' - (1986; 2e in print)

Various authors' _Handbook of theoretical CS_ (2 volumes, 1990-1)

Hamming, Richard R. Art of doing science and engineering: Learning to learn. CRC Press, 2014.

...and for papers, I like the curated list in Fermat's Library: https://fermatslibrary.com/journal_club

Evar D. Nering, Linear Algebra and Matrix Linear Algebra and Matrix Theory*

Kenneth Hoffmann And Ray Kunze. Linear Algebra, 2nd Edition, Prentice-Hall, Englewood Cliffs, New Jersey, 1971.

https://www.zuj.edu.jo/download/linear-algebra-2nd-edition-k...

Halmos, Finite Dimensional Vector Spaces

George E. Forsythe and Cleve B. Moler, Computer Solution of Linear Algebraic Systems

Paul R. Halmos, Naive Set Theory, Van Nostrand, Princeton, NJ, 1960.

More has been done since this book, but this book is a gorgeous introduction to axiomatic set theory. So, even people who want to dig into the latest work would do well to have this as the first book. And for people wanting to read any of the more advanced material here, knowledge of this book will be from good to have to important.

https://piazza-resources.s3.amazonaws.com/is25bi6c6o1oh/isv1...

Walter Rudin, Principles of Mathematical Analysis

The third edition is a lot better than the first two.

H. L. Royden, Real Analysis: Second Edition

Beautifully written, elegant, but maybe don't work way too hard on the exercises about upper/lower semi-continuity, and there is a better summary than Littlewood's three principles.

Bernard R. Gelbaum and John M. H. Olmsted, Counterexamples in Analysis

John C. Oxtoby, Measure and Category: A Survey of the Analogies between Topological and Measure Spaces

Walter Rudin, Real and Complex Analysis

Walter Rudin, Functional Analysis

Leo Breiman, Probability

Kai Lai Chung, A Course in Probability Theory, Second Edition

Jacques Neveu, Mathematical Foundations of the Calculus of Probability

Erhan Cinlar, Introduction to Stochastic Processes

J. L. Doob, Stochastic Processes

I. I. Gihman and A. V. Skorohod, The Theory of Stochastic Processes I, II

Donald E. Knuth, The TeX book

Donald E. Knuth, The Art of Computer Programming, Second Edition

Leo Breiman, "Statistical Modeling: The Two Cultures," Statistical Science, Vol. 16, No. 3, 199–231, 2001.

Paul R. Halmos, "The Theory of Unbiased Estimation", Annals of Mathematical Statistics, Volume 17, Number 1, pages 34-43, 1946.

Paul R. Halmos and L. J. Savage, "Application of the Radon-Nikodym Theorem to the Theory of Sufficient Statistics", The Annals of Mathematical Statistics, Volume 20, Number 2 (1949), 225-241.

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177730...

Here are URLs of PDFs of two of the references above:

Leo Breiman, "Statistical Modeling: The Two Cultures," Statistical Science, Vol. 16, No. 3, 199–231, 2001.

http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?vi...

Paul R. Halmos, "The Theory of Unbiased Estimation", Annals of Mathematical Statistics, Volume 17, Number 1, pages 34-43, 1946.

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177731...

Finite Dimensional Vector Spaces is a jewel. But I wish the typesetting was updated to something more modern. Same applies to Rudin.

I find well typeset TeX a joy to read. Whereas FDVS is a bit cramped and looks antique.

FDVS was written in 1942 when Halmos had just gotten his Ph.D. from J. Doob, author of Stochastic Processes in my list, at U. Illinois, and was an assistant to John von Neumann at the Institute of Advanced Study in Princeton.

IIRC Hilbert space was a von Neumann idea: It is first, just a definition -- complete inner product (dot product in much of physics and engineering) space. But the good stuff is (1) importance of the examples and (2) the theorems that show the consequences, e.g., in Fourier theory.

Well, the vector spaces of most interest in linear algebra are actually (don't tell anyone) finite dimensional Hilbert spaces. So, one role of FDVS is to provide a text on linear algebra that is also an introduction to Hilbert space, that is, that tries to use ideas that work in any Hilbert space to get the basic results in linear algebra.

The treatment of self-adjoint transformations and spectral theory are likely the most influenced by this role.

This role is accomplished so well that sometimes physics students starting on quantum mechanics are advised to get at least the start they need on Hilbert space from FDVS.

Sure, a better start is the one chapter on Hilbert space in Rudin's Real and Complex Analysis. The chapter there on the Fourier transform is also good, short, all theorems nicely proved, the main, early results made clear.

Also a good start on the basic results of self-adjoint matrices are the inverse and implicit function theorems given as nice exercises in the third edition of Rudin's Principles .... And spectral theory is in Rudin's Functional Analysis. Also get a bonus of a nice treatment of distributions, that is, replace the Dirac delta function usage in quantum mechanics.

For how to get the eigen value and orthogonal eigen vector results for self-adjoint matrices from the inverse and implicit (these two go together like ice cream and cake) function theorems is in Fleming, Functions of Several Variables. Then you will be off and running on factor analysis, principle components, the polar decomposition, the singular value decomposition, and more.

Tao te Ching. If you like that there's also the Art of War and Book of Five Rings.

I once read original works of Sir William Rowan Hamilton and was amazed at the clarity of thought he demonstrated, a truly open mind to understanding nature.

Books by Max Born (Atomic Physics, and Theory of Relativity) are absolute awesomness.

I didn't "get" calculus until I went and read the Principia.

In other fields, the Standardized Barbers Manual (for barbers) and The Modern Tailor Outfitter And Clothier (tailoring) are still extremely relevant

"The Unix Programming Environment" would be my choice. Love it.

General Chemistry by Linus Pauling.

Zen and the Art of Motorcycle Maintenance (Philosophy).

Several reviewers at Amazon said that General Chemistry is unsuitable for beginners or self-learners. Is that true?

It’s funny , I get the feeling that 50% of the contributors on this thread are would-be philosophers :)

Gettier cases and the life and times of a bat are far less interesting to “civilians” ;)

Differential geometry of curves and surfaces by Do Carmo

"Computable numbers with an application to the Entscheidungsproblem" - Alan Turing

Gravitation, by Charles W. Misner, Kip S. Thorne, and John Archibald

The Works of Archimedes by Sir Thomas Heath

A couple of papers on my reading list:

From information theory:

  A mathematical theory of communication (information theory, Claude E. Shannon) [1]
  Three approaches to the quantitative definition of information (A. N. Kolmogorov) [2]

And from inductive inference and computational learning theory:

  A formal theory of inductive inference (Ray Solomonoff, 1964) [3]
  Language identification in the limit (Mark E. Gold, 1967) [4]
  Inductive Inference of formal languages from positive data (Dana Angluin, 1980) [5]
  A theory of the learnable (PAC learning, Leslie Valiant, 1984) [6]
  Occam's Razor (Blumer et al, 1987) [7]

Bonus: a deep learning paper

  Long Short-Term Memory (Hochreiter and Schmidhuber, 1997)

The first two papers - well, one launched information theory and the other is Kolmogorov's paper where he introduced the idea of Kolmogorov complexity.

The second batch of papers start with Solomonoff's inductive inference papers, kinda important if you want to learn things from other things. Mark Gold's paper proves that it is impossible to learn a non-finite automaton from examples. Dana Angluin's follow up extends this with learnability results about various classes of CFG. Any time someone claims that their deep neural net has learned a CFG, point them to these two papers.

Valiant's paper is the theroy of machine learning as we know it today. It basically relaxes the assumptions made in inductive inference and introduces the notion of error. If you can't learn some concept perfectly, what degree of error is likely from some set of training data? Blumer's paper discusses a further bound on that amount of error that follows an Occamist bias (simplest truths are better) and is a basis for understanding overfitting (error increases as the hypothesis space does).

These two sets of papers probably look disconnected - but, learning is compression. Compression, with generalisation, I guess. Anyway, no, they're not unrelated.

The final paper is the one that introduced LSTMs and the, er, "constant error carousel" (the solution to vanishing gradients, which this paper is worth reading for).

These are papers that one must read if they're interested in machine learning. Carefully so. They're not even "old" papers- more like, essential ones.

I'm totally omitting a whole bunch of others, obviously.

___________

Online pdfs (not all free):

[1] http://www.math.harvard.edu/~ctm/home/text/others/shannon/en...

[2] http://alexander.shen.free.fr/library/Kolmogorov65_Three-App...

[3.1] https://www.sciencedirect.com/science/article/pii/S001999586... (Part 1)

[3.2] https://www.sciencedirect.com/science/article/pii/S001999586...

[4] https://www.sciencedirect.com/science/article/pii/S001999586...

[6] http://www-personal.umich.edu/~yinw/papers/Angluin80.pdf

[7] https://www.sciencedirect.com/science/article/pii/0020019087...

Calculus made easy

K&R