Information Theory, Inference, and Learning Algorithms (2003) | Hacker News Reader

Information Theory, Inference, and Learning Algorithms (2003)

inference.org.uk

237 pointsmalshe30 comments

Great book by a great teacher. I read this during my masters and was my route into machine learning.

David did many great things but sadly passed away too young in 2016, surely would have continued to have a great positive impact in the world had he lived longer. His other projects included the sustainable energy book https://www.withouthotair.com/ and information theory inspired keyboard software for people with severe disabilities https://www.bltt.org/software/dasher/

I had the honour of being a PhD student in Cambridge during his last year of life, and saw him give talks a few times in our lab. It's truly a remarkable experience to see someone who is a passionate explainer in action. Like how I imagine it must have been to see Feynman in the flesh.

This is a great short video to get a feel for his style of explaining: https://youtu.be/UR8wRSp2IXs

(Writing this post feels wrong because he's had such a positive impact on my life and those of many others, and this post definitely can't do it justice)

His books are fantastic and his writing is captivating. It's such a shame he passed -- cancer is a bitch, but the blog post about the radiators in the hospice that he died in are one of the most heart rending pieces of writing I've come across from a mathematician.

Is this the post you mean? http://itila.blogspot.com/2016/04/appendix-two-open-letter-t...

Yes!

Thanks for sharing your experience. The video link is fantastic!

As a follow on from Sustaianble Energy he also drove the original version of the my2050 game, which communicates the tradeoffs we need to make in decarbonizing to the general public very well. More recent update here https://my2050.beis.gov.uk/?levers=433233211331111

I've taken this as inspiration for my own transport models, though I have a model with some sliders to drag around I don't have a flashy game yet!

A great communicator and a kind man, indeed.

It's been a privilege to see him lecturing at the Cavendish in 2002.

Here's his last video interview: https://www.youtube.com/watch?v=sCyidsxIDtQ

There's also a nice set of recorded lectures available by Professor MacKay covering the core material from the book, available on videolectures.net.

http://videolectures.net/course_information_theory_pattern_r...

Fantastic. Thank you for sharing this!

"Your browser does not support playback of available video formats. Please install Adobe Flash player or upgrade to a more modern browser."

...Okay?

I found the videos on youtube as well: https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n...

From lecture 1 [1]:

> Ok, so, we have an answer of 1000 plus or minus 30 thanks to "Mr. Binomial", the inventor of the distribution that bears his name.

Lol, this is great.

[1]: https://www.youtube.com/watch?v=BCiZc0n6COY&t=971s

Huh. Weird. I'm using Firefox 109.0 on Linux and the videos on videolectures.net play fine for me.

Yes these are so good! Highly recommend.

I attended an amazing talk by David circa 2006 at Princeton where he demoed his probabilistic text entry system called Dasher [1,3]. I think it was made into an iOS app later.

I also fondly recall his videos on Gaussian processes - best explanation on GPs I ever came across. [2]

[1] https://www.inference.org.uk/dasher/

[2] https://www.youtube.com/watch?v=NegVuuHwa8Q

[3] https://dasher.acecentre.net/about/

EDIT: Fixed the youtube link in [2], added [3].

Would love to see the GP video, but [2] is points to an unrelated video.

Fixed

This book shows up every year or so on HN and I will never stop upvoting it. My favorite textbook of all time, along with my favorite quote:

“Marginalization is a macho activity, appropriate only for those fluent in definite integration”

"Exact marginalization over continuous parameters is a macho activity enjoyed by those who are fluent in definite integration."

http://www.inference.org.uk/itprnn/book.pdf#page=331

I absolutely adore this book and its approach to information theory.

I will also note the author made available the (non compilable) latex source [1] of the text for easy searching!

[1]http://www.inference.org.uk/mackay/itprnn/booktext.txt

Information Theory, Inference, and Learning Algorithms (2003) [pdf] - https://news.ycombinator.com/item?id=11289488 - March 2016 (9 comments)

Information Theory, Inference, and Learning Algorithms (free ebook edition) - https://news.ycombinator.com/item?id=852562 - Sept 2009 (21 comments)

I remember him (David Mackay) leaving comments here on HN, but I don’t remember the username. I wonder if you can find them somehow.

If anyone can find this I'd be very grateful, would be cool to read his comments

Tried Googling "mackay site:news.ycombinator.com" and found a lot of times where this book has been mentioned; I didn't see anything that clearly said "I am David Mackay" but that might be a good start.

(You'll also find the occasional reference to Charles Mackay's book Extraordinary Popular Delusions and the Madness of Crowds.)

I did some searches but couldn't find anything. If someone else does and the thread is closed for replying, let me know at hn@ycombinator.com and I'll reopen it.

one of my favorites (even though the pdf is free i keep a copy of the book). very clear introductory chapters in probability leading into a really nice blend of applied math and signal processing, finishing out with neural networks.

his father published was something of a figure in the fields of perception and cognition as well.

Cybernetics is alive the same way punk is alive: people say it's dead, but then there are punk bands and punk records and just like a truckload of punks. This book, like those written by his dad, is cybernetic theory in everything but name. I absolutely adore it.

From the preface:

"My work has depended on the generosity of free software authors. I wrote the book in LATEX 2ε. Three cheers for Donald Knuth and Leslie Lamport! Our computers run the GNU/Linux operating system. I use emacs, perl, and gnuplot every day. Thank you Richard Stallman, thank you Linus Torvalds, thank you everyone"

His books have taught me soooo much; in addition to information theory, I have learnt many tricks of LaTeX typesetting (the info theory book's LaTeX source is open) and how to estimate (from his other book on sustainable energy)

David was a fantastic ultimate frisbee player. He taught me how to properly throw forehand and overhand.

The book compares quite well to Harry Potter, unless you need the Latin translation: https://www.inference.org.uk/mackay/itila/Potter.html

Was the David MacKay on Stargate: Atlantis named after the author?

Reply on news.ycombinator.com