How You Know
paulgraham.com
paulgraham.com
I think that perhaps very squishy subjects like politics are particularly vulnerable to this sort of disconnect, where a complex viewpoint is formed based on the hot topic of the day, and this viewpoint persists for years or decades even if the basis for its formation is completely forgotten.
(Spoiler alert don't read till the 26th)
Do you believe in Santa? I believe in Santa. Oh how nice you also believe in Santa. Aren't we all happy members of our club? It would be so antisocial for someone to say Santa doesn't exist, because then how would we tell who is in our club? This is the best club ever. Santa? Who's that? We just talk about him to pledge allegiance to each other, we don't actually care about fat dudes in red suits.
Of course, it's one of the earliest results in psychology that social conformity has an influence on people's beliefs, even on readily accessible questions like whether line A is longer or shorter than line B. To me, if somebody professes a belief, acts on that belief, and works to convince others of that belief, then they effectively hold that belief.
The longer it has been since you checked, the further off the number will be.
And then, when you finally accept that you don't grasp it and start calculating with a formal growth rate, the growth rate suddenly changes.
[1] Nobody does. Some people know they don't, some are completely naive, the others lie to themselves. The first group have a chance of dealing correctly with it.
The need to reread and reconsider books, ideas, etc. is not to check if the facts have changed, but to make sure you didn't miss a point because of lack of knowledge or bias.
For example (on a very basic level), I find that if I reread a book on a programming language after using it for some time I notice things which I missed on a previous reading or wasn't able to appreciate due to lack of familiarity with the language. The same can apply to history - or any other study - where one only appreciates certain details after understanding the larger picture and surrounding events.
For this reason there lies an advantage in both rereading the same book (immediately or after some time) by itself and reading after studying more on the topic from other sources.
For me, this quote was the most powerful. I strongly and immediately agreed, yet I hadn't consciously considered the idea before. I now ask myself the question, "Which ones were the important books?" Some seem obvious, but I may have a deeply rooted worldview established a long time ago that needs to be reevaluated. I may have read books that added support to that worldview, and have since forgotten from where that support came.
I'm not very well read yet, but I have a question for those of you who are. What are some methods you utilize to remember which books are the important ones?
Edit: The comment at https://news.ycombinator.com/item?id=8753656 contains a great idea. I agree that taking notes and writing a journal can be great solutions, but I don't often read the notes and entries I've written. A personal wiki that is searchable and contains references seems very interesting.
I'm not certain if this question was meant for me directly. If it was rhetorical, I apologize for misunderstanding and answering.
I prefer not to define "important" in this context to avoid excluding any definitions subjective to those who may respond to my question. That way, I may learn both what makes a book important to someone and tips on remembering which were important.
Edit: Now that I know the question was intended for me, I'll provide a little more depth to my answer.
The first thought I had when considering what makes a book important is how strongly it resonated within me, and the intensity of my emotions when reflecting upon what I've learned or how my perspective changed shortly after reading it. I know those stronger emotions may derive from a bias I had at one point in my life, and may no longer have.
Therefore, I can't help but think my definition is wrong because it's relative to the period in my life which I read the book. So some books that were important before may not be now. That's why I was curious to learn others' definitions of "important" books, and how to identify them for rereading.
When I look back, I find that most of the things I've learned that I enjoyed the most, am the most proud of, or have been most helpful, have in fact been accidental, and not a result of planning. I found so-and-so article then downloaded so-and-so package, experimented with it and realized it could benefit a certain project I was working on. Sorta like going down a Wikipedia hyperlink rabbit hole -- very in the moment, just chasing your will and not questioning yourself, doing simply what feels right.
Look back at your bookshelf and pick what feels right. If you start reading it and it feels wrong, then put it back. Being told what to read is annoying, so why would you do that to yourself? Do what you're doing when you playing music improvisationally, but with your life instead.
Original Paper: http://pss.sagepub.com/content/16/3/190
How facts backfire - Researchers discover a surprising threat to democracy: our brains
"In a series of studies in 2005 and 2006, researchers at the University of Michigan found that when misinformed people, particularly political partisans, were exposed to corrected facts in news stories, they rarely changed their minds. In fact, they often became even more strongly set in their beliefs. Facts, they found, were not curing misinformation. Like an underpowered antibiotic, facts could actually make misinformation even stronger."
http://www.boston.com/bostonglobe/ideas/articles/2010/07/11/...
The extreme cases can be found in cults where followers cling tighter to their beliefs once exposed.
A great example are the followers of Harold Camping, the Christian radio broadcaster who predicted the end of the world a few years ago, and kept re-predicting when it never came.
People's belief systems about the relative threats of terrorism and pedophiles is thus distorted by the media because the amount of time they spend watching terrorist attacks or episodes of "To Catch A Predator" is massively disproportionate to the actual level of risk posed by those kinds of things, and so especially is the emotional impact of that media.
I have, on a few occasions, found myself relating some 'fact' or 'story' and realized immediately that it was likely an urban legend. Both times, the detector was right, and a quick search cleared it up.
I wonder if there is a good way to intentionally 'paw through' your repository on purposes, and apply your life experience to them on purpose.
"The extent of and continuing increase in inequality in the United States greatly concerns me. … It is no secret that the past few decades of widening inequality can be summed up as significant income and wealth gains for those at the very top and stagnant living standards for the majority. I think it is appropriate to ask whether this trend is compatible with values rooted in our nation’s history, among them the high value Americans have traditionally placed on equality of opportunity." Janet Yellen
The best people adapt to new information, no matter their age.
But only if you truly realise that you are not infallible, can you learn new things, and continue to grow.
Also, quotes are a good way of appearing wise.
http://www.newyorker.com/science/maria-konnikova/i-dont-want...
There are always areas where we do forget the provenance. In my case I stood up amoung friends in my thirties and announced that Ice cream was made from seaweed. After a few blank stares I remembered my father explaining to a much younger self on the beach that I did not want an ice cream - because they were made from that horrible floppy green muck. And he did not have to make a half mile trek back up the beach. Yet that "fact" and it's provenance stayed at the back of my brain for twenty years.
But even so the provenance was quickly recovered. I am not convinced we easily lose our source code. We can usually point to the module or package even if the line numbers are hazy.
But trauma does rewire us, and quickly, and is not to be discounted - I just think it is not the normal course of events.
http://subterraneanpress.com/magazine/fall_2013/the_truth_of...
My explanation is linked because I don't want to spoil the episode. (Also spoils 15 Million Merits.) http://paste.lisp.org/display/144706
I think that the efficiency of technology shown in TV shows and movies shouldn't be assumed to imply something about the world therein, unless it's explicitly called out in the work itself. The writers are much more likely to have gone with something because it sounded cool than because there was a sound technological reason for it. The bit in the matrix where someone refers to someone else as "copper-top" is an example, I think.
This is what the episode all the more disturbing to me. This would be the far-west's answer to a demand for less retribution-oriented prison designs, like a virus becoming less deadly so that it can infect more people. (How many on Hacker News would tout the virtues of free services as "you only have to watch ads, you're not forced to buy anything"?) You have "free choice", but it is only within a heavily curated model of illusory flavors; yet the vast majority of prisoners in this system are content, even euphoric, within it.
As Eddie Izzard put it:
> I know a lot of people who'd love to be under house arrest! They bring you your food... "Just stay here? Oh, all right. (laconically) Have you got any videos?" You know, you just sit there all day...
It's possible that the bikes are used to generate electricity, but that would be very inefficient and, also, not a very creative interpretation. Maybe they're part of a fitness experiment - the participants have an exercise regiment that they keep and are being monitored for the effects the excercise has on them. In return, they are provided with lodgings and food. After all, people have been paid for weirder things in modern times. I can think of a few more scenarios where the bikes are a part of some data gathering/social experiment thing.
Overall, I think, Black Mirror has a trait that distinguishes good fiction from bad - it manages in each episode to create a world that is abstract and unfamiliar enough to play by its own rules, but at the same time, allows parrallels to be drawn between the fictional world and our reality.
Ted Chiang is always worth reading. I think he captures the nuances of the impact of technology better than any other writer I've read.
Making an erroneous assumption or a mistake can mean that a particular sentiment is expressed "incorrectly", but that doesn't mean it's existence is useless. It may prove to be very useful eventually.
The idea that mental models can be incorrect by being compared to other mental models is a strange concept to me. It requires assumptions that can not be proven in their entirety.
Even if accidentally.
Not only disconnect: What I've read of psychological studies indicate the mind amplifies those facts that support it's current model of the world and actively ignores those that do not.
Which goes a long way to explain why it's nearly pointless to argue with fanatics of one kind or another.
Many times we tune out feedback that isn't congruent with our worldview and get locked in. That's why it's good to follow Paul's lead on rereading and revisiting. Sometimes it's hard to remember the anecdote or story that steers us, but finding it again allows us to question it. There was a specific retelling of a Vonnegut (of all authors) story that got me interesting in technology. Rereading the story in it's original form 20+ years later allowed me to understand the first better.
The method of loci, for building/encoding hierarchical "memory palaces", works well for remembering key ideas or facts and building upon them to compose your models. Also, using software like "The Brain" [1] and other mind mapping tools are useful aids for organizing information so you can easily go back to remember.
That sounds... time consuming.
> "Philosophizing in midstream"
Since you're picking axioms/first principles, isn't it all philosophizing in midstream to one degree or another?
It seems as if becoming proficient at something indeed involves moving more knowledge into intuition, same as how you had to purposely look at the rear view mirror and watch cars closely as you learned to drive but now it becomes something of intuition. You had to pay special care to syntax and to grammar initially, but now all of that is habit and you can concentrate on the 'design' of a program, or the 'characters' and 'themes' in a piece of literature.
What if these 'logical arguments' we make for things are simply retrofitted justifications on top of our feelings? Pro-choice 'feels right,' but of course that won't fly in court and so I'll make up some argument for it, becoming disingenuous thus not just to others but even to myself, distancing myself from who I am and causing some amount of strife within. I hope to think I am a perfect rational thinker, but am I really? I am driven by drives, by passions. Maybe I should become aware of this and proceed, acknowledge the person inside me rather than attempt to destroy it with reason. Maybe I should just stop and listen and stop trying to rationalize, both to others and to myself, stop thinking, open my eyes and see life in HD.
"Only those questions that are in principle undecidable, we can decide.
Why?
Simply because the decidable questions are already decided by the choice of the framework in which they are asked, and by the choice of rules of how to connect what we call "the question" with what we may take for an "answer." In some cases it may go fast, in others it may take a long, long time, but ultimately we will arrive, after a sequence of compelling logical steps, at an irrefutable answer: a definite Yes, or a definite No.
But we are under no compulsion, not even under that of logic, when we decide upon in principle undecidable questions. There is no external necessity that forces us to answer such questions one way or another. We are free! The complement to necessity is not chance, it is choice! We can choose who we wish to become when we have decided on in principle undecidable questions.
This is the good news, American journalists would say. Now comes the bad news.
With this freedom of choice we are now responsible for whatever we choose! For some this freedom of choice is a gift from heaven. For others such responsibility is an unbearable burden: How can one escape it? How can one avoid it? How can one pass it on to somebody else?"
This is such a common occurrence in board gaming that it's frequently abbreviated as just "AP".
http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_t...
The main problem with this approach (I'll call it the Cartesian approach because it was most famously used by Descartes) is that human beings are less than 100% reliable at logical reasoning. If you make an error anywhere in your chain of reasoning, your conclusions are going to be off and there's not going to be any way to check them. It's just like writing 10,000 lines of code without ever actually compiling it, let alone testing it. You also develop a sort of foolish confidence about the correctness of your own beliefs, which makes it even easier to be wrong. If you make enough wrong turns, you become Ayn Rand.
That's why empiricism is so good. It's not that empiricists don't make mistakes too, but when they do, they find that they are surprised by concrete facts that they observe, and know when to go back and reevaluate.
Another helpful trick is to understand that there are degrees between 0% and 100% confidence. I can entertain an proposition as being possible or likely rather than simply true or false based on the recognition that I have incomplete information. If you tried to take this approach you couldn't derive anything logically because you would just have a multitude of possibilities in front of you. Formal logic only works with statements that are 100% true. Otherwise you're stuck with Bayesian reasoning, which is even more mentally taxing to derive information from.
(Or, as an alternative response)
Please derive for me, from "first principles", why it is imperative to build one's mental model from first principles.
A benefit to a digital mental model is that it could be updated based on new data independent of the owner which has all kinds of (possibly disturbing) implications.
"Your program says I should now believe that it's morally acceptable to kill infants. I don't agree with its reasoning."
Once you have read something and your mental model of the world is adjusted to include the new information, you have a difficult time understanding why others don't see what you see. This is compounded by the fact - as highlighted by pg in the essay - that you also forget how and when your mental model changes.
This is one reason why not every expert is a good teacher - as they fail to see the world from the point of view of students.
But it is also relevant and useful to remember this in the world of startups. Established large companies routinely get disrupted by novice startups - often because the experts at the large company fail to see problems the way novices do. It is impossible to become an expert at something while continuing to view the world from the eyes of a beginner.
Reading this made me think of poker. Calibrating to the skill level of lesser players is often very difficult for intermediate and lower-advanced players. Being able to synthesize the less sophisticated thought technologies beginners are using is surprisingly difficult. Failure to adjust often leads better players to play incorrectly against newbies. Anyone who has experienced the frustration of beating medium/high stakes cash games only to lose in home games with your friends for 1/1000th the stakes will know what I mean.
http://www.imdb.com/title/tt2089050/
http://www.netflix.com/WiPlayer?movieid=70264856&trkid=33258...
Fantastic episode.
I think this audience would especially like the series:
'Black Mirror is a British television anthology series created by Charlie Brooker that shows the dark side of life and technology. Brooker noted, "each episode has a different cast, a different setting, even a different reality. But they're all about the way we live now – and the way we might be living in 10 minutes time if we're clumsy."'
Hence the important art of keeping a journal. You can keep a transaction log of changes. The act of replaying the journal allows you to identify patterns in your thought processes and identify cognitive dissonances. The very act of reading should induce a reactive compulsion to write.
As Burroughs taught in his later creative writing courses -- in order to become a better writer one must first learn to read (I'm paraphrasing here).
Part of becoming a better thinker is learning how to think. In order to do that one must catch one's self in the act.
But I think a journal is the wrong model (i.e. time ordered entries, either electronic or paper). I have used a paper notebook in the past, but I would rarely go back and look at things, and it's not searchable, and paper is not editable.
For the last 10+ years, I've used a Wiki. Hyperlinks are huge. They really do model the associations your brain already makes. I have wiki pages that are 10 years old and that still grow new associations. I think it takes a big load off your brain to have all that stuff written down, and searchable with ease. (I had to write my own Wiki to get it fast enough though.)
I started it 10 years ago, and side benefit was that writing a Wiki is a good project to learn about web programming. The first version was of course riddled with XSS and escaping problems :-/
I think writing a Wiki is still a good exercise now. I'm not a front end person per se, but every programmer should know something about the web. I'm always a little taken aback when I meet some back end guy who doesn't know how HTTP or the browser works.
And IMO there is too much bloated JS on the web now. I think people forgot how to make a simple web app with a form and plain buttons. There are too many fast-moving frameworks, so just doing it "raw" (or to WSGI) is a good learning exercise.
However when I'm reading specifically I keep notes in paper journals. First I find the immediacy of pen to paper to be intuitive. Second, and more importantly, the serial nature of the journal forces me to explain my thoughts. It is this specific constraint that allows me to see the process I went through to arrive at my current self.
It's not indexable or searchable, that's true. However I don't need the process of reflection to be fast and instantaneous. I have a lifetime to work it out. I don't mind reflection and introspection to be slow and tedious.
I wonder about future technology when PG says "Eventually we may be able not just to play back experiences but also to index and even edit them." I also wonder about the iterations between now and then.
What do you think could be an intermediate step (company idea) between your wiki and something like a searchable memory catalog that would do what PG describes?
1945, As We May Think, http://www.theatlantic.com/magazine/archive/1945/07/as-we-ma...
I think if you just use this system for awhile, tons of ideas will come out. In fact you can generate a lifetime of work from it :) PG always talks about making something for which you are the customer.
EverNote is of course a company in this space. I used them for awhile; it's nice that they have iPad clients and the like. But I found their products to be a bit complex.
Here are some random things I want to fix:
- mobile access. In the past 10 years, this was a huge change. My Wiki is basically read-only on mobile devices. I have to go back to the desktop or laptop to really write anything. This is OK but I imagine there could be some mobile interface for typing or speaking and hyperlinking. EverNote partially addressed this.
- Data model for bookmarks. Actually I sort of poo-pooed the journal model in favor of a Wiki. But to be honest I've realized that a lot of things are links which I find on HN and the like, which need a date, "read" tag, and free form notes. It's basically delicious / pinboard, but with an emphasis on comprehension and connections to previous ideas.
- Unsurprisingly, I've found the need for a spreadsheet-like data model for finances, and certain kinds of research. Yes, I could just use a spreadsheet, but the hyperlinking and web hosting is huge. I use Google Spreadsheets now but would like something a bit faster and more under my control.
- I like having these notes as my data for all time. Delicious came and went in the last 10 years. A lot of the value for me is that it's personal, and not tied to any cloud service, which conflicts a bit with current business models.
- Search. Right now I have a fast full text scan with sqlite. I've wanted to write a script that would fetch bodies for all the links, put them in a full text indexer, and let me search quickly there. And maybe take screenshots with PhantomJS.
- Information curation. I have over 2000 pages now, and sometimes I probably forget to link back to stuff I should. I create duplicate pages by accident. This should be solveable with some hints.
- Stats on which pages I actually read. With 2000 pages, I need some kind of ranking now. Some are ancient/obsolete.
- Mirroring of my content hosted elsewhere... e.g. it would be nice to suck down my comments from HN and be able to search them later. I had a lot of good conversations on UseNet way back that I wish I still had :)
Not sure if any of these are good business ideas, but those are my thoughts :) A lot of them are partially covered by existing products. But the the thing I am suggesting is for particular programmers to build the particular thing for themselves. Everyone's preferred mode of information organization will be a little different. It's nice to have something you made for yourself. I think it's a good exercise, because you can start very simple and gets you in the feedback loop of product use / product design / implementation.
But it's possible it could lead to a company. For me it to led to some other technology, like a web server container variant of WSGI / CGI (both of these have problems). And backup / deploy stuff.
One problem is that companies are incentivized to go "wide" to acquire new customers. But I want to go "deep" into my own use cases. There are certain problems you only hit after you have 1000 pages. Probably 99.9% of EverNotes customers don't have that volume of content.
update I'd even add perhaps the solution is really to take the best ideas out of org-mode and give it a platform-integrated UI. It does basically what everyone here has been asking for and more... it's just buried in Emacs which is definitely not for everyone.
Document sharing across devices is easy if you're comfortable sharing your content with a third-party like Evernote or Dropbox. I'd be more comfortable with something that's decentralized, encrypted, and versioned like a Git + Freenet backing store.
I hear about org mode a lot when I talk about the Wiki, but I'm not an emacs user so it's not as appealing to me.
Can org mode export HTML? At least for reading, plain HTML is the format that works best across all devices, and gives fast access. Writing is a different story.
Emacs is definitely not the best editor for everyone. It's a shame that org-mode is sort of a secret weapon of Emacs. It's such a great package that I think it could stand on its own.
I wrote a long comment above about future ideas.
I think the main interesting things are speed and and full text search.
The page loads in 150 ms, with the HTML loading in 75 ms (cold cache), using SSL. That's probably 5x slower than it should be, but it's also 5x faster than most products these days. To me, the speed makes a big difference. I decreased latency significantly 4 years ago, and I know my velocity of note taking has gone up, and it has paid off in terms of increased velocity/organization of the projects I plan with the wiki.
I think it took me 6 to 8 years to get to 1000 active pages, and then somehow I have active 2123 active pages now, after 10 years (although I do delete/archive pages, so this number is fuzzy).
Other features:
- I have a JavaScript "jot" button on my browser toolbar that submits the title and URL of the current page in the browser, and then opens up a form for jotting notes. Then this is appended to a selected wiki page.
- The notes are pretty heavily indented, outline-style, so the markup makes that easy.
- You can edit individual sections of a page, like Wikimedia.
Search is indirectly via Windows Search / Indexing. Definitely, there are a couple of bugs with that and I have had to delete and re-build the entire search index in the past.
The problem with hand-rolled solutions is that you may end up working too much on the tool instead of your actual work.
It also reminds me of that MIT paper that gives advice on how to do research. The part it talks about why it is that when your colleague gives you a paper to read and says it's particularly poignant, but when you read it it doesn't seem like anything special. Maybe it's because your colleague had the dependencies in his state of mind that you did not have in yours, so it didn't seem as memorable to you as to him when the code compiled.
There are times in our lives where our state of mind makes us more likely to be moved by a piece of art.
It's why you should revisit your favorite books from your youth - you'll often find the same words mean completely different things later on.
edit: found it
But there's more to it than that. Your unconscious implicit memory includes things you can't even articulate. That's the difference between the date of Christmas and how to ride a bike: the latter is nondeclarative. Learning a different way to ride a bike, or approach programming, is even more difficult than recomputing semantic memory.
You can (and should) read a new books and gain new episodes to base your facts and opinions on. Read diverse material with abandon. But when learning something nondeclarative, like a weight-lifting technique, it can be well worth seeking out an expert and learning it right the first time. With nondeclarative memory, what you don't know can hurt you.
For more on the science and classification of memory, the Wikipedia page is as good a starting place as any.
In general what matters isn't how much you read, but how much you retain and what sorts of connections with past and future insight and information. It's important to have the full experience of having realizations and making connections while you're reading, which is why I just make a dash in the margins as opposed to taking actual notes in real time, but I feel like by not circling back later you're cheating yourself out of the true value of learning.
Especially since you have no idea if the books you're reading are even true or not until you vet the facts with primary sources.
Thought processes like the ones captured in pg's post are fostered by education in critical analysis--the sort of analysis that one learns in the humanities. Art, literature, philosophy, history, etc. are the products of human thought, and learning to critique them is in part an exploration of how humans think. Not the physics or neurology, but how influences can shape each person's mental model.
Part of this is exploring the influences that affected the mental model of the person writing or creating the art. Another is exploring the mental model(s) that the artist or writer sought to create. (This is what we experience when we "get into" a book.)
So, if you're looking for a reason that CS or engineering students should take humanities courses, I think one is illustrated in this post: it teaches you how to read books consciously. It gives you a framework for exploring how the thoughts of others (and therefore yours as well) are influenced and shaped by the information that is consumed during a lifetime.
This was reassuring to hear from someone else, because I've had this exact feeling about books I read, films I've watched, conversations I've had, work projects I've completed, etc. This is true even in cases when I was completely engaged in, for example, reading the book, and the book left a positive impression on me.
I've always felt guilty about this, especially when I see others who don't seem to have the same problem when they talk about the books they've read, etc. I've also found that recall can be greatly improved by repeatedly talking about the specific topic with multiple people.
The strange thing is that I have an excellent memory for certain things - information about people and relationships. In light of our evolutionary history as a social species, perhaps this is not so surprising after all.
I spent several years reading a ton of different books on economics and I can recall very few facts from those books, but it did and has completely altered my world view of many things.
pg's analogy of a program where you've lost the source code doesn't feel quite right, because you can't make modifications to the program without the code. Some sort of machine learning model seems more appropriate, where you've lost the original training data but can still update the model later with fresh data (a new book), and end up with a better/different model, but then lose that training data again.
Using an artificial neural net analogy instead of a compilation analogy: "The same book would optimize your neural net towards a different local minimum at different points in your life."
One of my favorite films that works along these lines is the 1998 Japanese film "After Life," in which a small party of workers attempt to recreate others' memories with very basic film studio equipment. I absolutely treasure the loss of detail in the various recreation scenes, and the way it suggests that there is actually a satisficing point at which we might realize, "yes, I'm actually reliving that memory right now." So I agree with Mr. Graham's conclusion that technology can bring this about.
On an unrelated note, PG's essays always bring to mind the Meyers-Briggs INTJ type. Essays about the annoyance of accumulating "stuff", a focus on abstract / intuitive learning styles, and clever writing which quickly establishes a theoretical framework which is then thrown against the world's (audience's) experience, rather than starting from first principles hoping to eventually reveal a framework as others might do. His seems to me very much a "systems thinker" approach.
(I had no idea this style of thinking was associated with INTJ types.)
So far, I've found it very handy to find something if I at least remember which book it was in. But I need a program that can extract the OCR'd text from .pdf files - anyone know of a simple one?
(I can do it manually, one at a time, by bringing it up in a pdf reader, but that's too tedious and slow.)
Some ideas: - Apache PDFBox https://pdfbox.apache.org/ - command line: https://pdfbox.apache.org/commandline/#extractText - XPDF has a command line tool you can use in Windows - http://www.foolabs.com/xpdf/ - pdftotext - If you're going for accuracy, Tesseract is one of the most accurate https://code.google.com/p/tesseract-ocr/ - Apache Tika is often used the way you suggest: http://tika.apache.org/
If the PDFs already have OCR text, calibre (GUI or CLI, Linux or Windows) can convert to .txt and many other formats. The recoll.org search engine will index PDF files that have OCR text.
(I just tried pdftotext, and it does just what I wanted.)
Am I missing something here?
I personally think this essay is some pretty nice food for thought.
As for "food for thought," have you really never thought about how reading and experience shape your beliefs? I thought this was pretty basic stuff. Maybe I'm wrong.
>Reading and experience train your model of the world. And even if you forget the experience or what you read, its effect on your model of the world persists.
Here, he is pointing out the the relevant information you perceive, your empirical data, is only retained insofar as it effects your deductive model of the world, that is, the model we use to determine truth and falsity. The rest of the data is generally trivial. This is a very sensible insight in my mind, and kudos to him. The dance between empirical data and deductive truth is one of the most difficult things for me to get my head around. This as a model for data retention is something i'd not thought of.
>Eventually we may be able not just to play back experiences but also to index and even edit them. So although not knowing how you know things may seem part of being human, it may not be.
Here, i find this problematic. In Soros's terms, the mind is reflexive. Thus, in reviewing the data, we are experiencing new data. If we edit our thoughts, do we not remember editing them? I don't see away to take away the reflexive nature of self examination, that in creating changes, we create new data about the changes.
That book was hugely influential to me. I graduated college and spent two years teaching. The summer after my second year of teaching, I had no obligations to anyone else for the first time in my life. I remembered Peter Jenkins' story, and decided to bicycle across the US. I knew I wanted to travel under my own power as he had done, but I wanted to go a little faster than he did. Bicycling was perfect for me. I ended up doing two cross-country trips over successive summers, and then I spent a year living on my bicycle, circumnavigating North America.
I reread A Walk Across America some years after doing my own trips. I was amazed at how bad I thought the book was. pg observes that
The same book would get compiled differently at different points in your life.
This is absolutely true. Now that I'm in my 40's, I'm going to go back and reread the most influential books of my 20's. I might even have to change my HN username after doing so, but I hope not.
Where is the knowledge here? That we don't have immediate recollection of retained information? Knowledge is based on a beginning and ending context.
The point is PG wrote it so just like if anyone of note wrote it it would be more of interest than the same thought from anyone else.
After all these are analog thoughts and subjective this isn't science.
I mean, is it not obvious that you can take away new ideas from reading a book multiple times at different stages of your life? As a simplified example, movies with twist endings hinge on exactly that fact -- armed with new information, events you have already experienced take on new meaning. More "important" things will have more significance, but it's the same idea.
Is it not obvious that your own world views are the result of your own experiences and others who you have contact with, even if you cannot precisely remember everything that would lead to that world view?
And if your summary was "boy, new ideas from different stages" you kind of missed it.
Secondly, your summary of pg post doesn't do justice to the its contents. To me it's about the distinction between mere reduction of information, akin to jpg compression, and building a model from it. A model is Turing-complete, a jpg is not. The distinction is important, in that it directs the effort required to correct misconceptions later in life.
If you really can summarize a book in a sentence or two, wouldn't the author have done that already?
Maybe it's time for me to reread Cryptonomicon. There are parts of that book I have absolutely no memory of, flipping through it, yet other parts I remember all too often (bicycle sprockets, comets of pee, bisecting alligators, van eck phreaking).
(also... > seige warfare ?)
Not only is this possible to do, but it's often done. The problem is that it's not necessarily useful or sufficient to hear a mere summary of something.
For example, let's say I tell you that "Idea X is important." That's a simple idea, right? It only too me four words to express it. But do you believe me? Probably not, because I haven't spent any time or effort convincing you that idea X is important. And do you understand what idea X is? Probably not, because I haven't spent any time or effort explaining that. Etc.
Even if you can summarize it, you probably need to write the entire book for people to get the background information necessary for them to find your summary useful, otherwise it will go in one ear and out the other.
I disagree with the parent that you still remember the content but can't summarize it concisely. I believe the opposite: you forget the specifics but retain the ability to summarize them.
Interestingly, if shown a series of hundreds of images, we wouldn't remember many in the list. But if we're shown alternates (was it a goldfish or a watch?), we would instantly recognize the item.
We didn't forget, we just couldn't access the memory on demand. The conclusion is the same: it's there, influencing us and adding to our lives, even if it doesn't feel like it sometimes.
If I had a magic device that recorded all of my experiences, it wouldn't do me much good because I'm too busy collecting new experiences to be remembered. It would be great to be able to search for details and trivia, but I wouldn't have time to peruse the archive to refresh myself about things that I had forgotten completely. Much in the way that google lets us search for and recall anything, except the things we don't remember the name of.
I'm going in the direction of reminding myself about things that I previously read or bookmarked, especially as they tie in to what I'm currently reading. I think one part of the solution is to display existing bookmarks and typed up book notes to myself in a near random fashion. It's not the most sophisticated solution, but at least they won't be lost and I'll have a chance of reconnecting with something and establishing more anchors in my memory. I think a plugin that relates past content to the current page might be a good idea, ie for this page I could see any previous bookmarks that involve memory and retention. And generally reminding myself to review things I've already learned, even if they don't seem relevant at the moment.
I don't have any great ideas yet, but I've been coding like heck for the past few days to try to take small steps toward a solution. I've been on a quest to make my brain work better, and this essay has definitely given me some ideas and helped to push me along.
Go for it; simplicity works. I've pushed a few of my favourite passages to a simple web page which I can flip through randomly (http://www.jasimabasheer.com/amateur_reading/serendipity.htm...). As a bonus I can also link to it when relevant discussions come up.
"However, for the man who studies to gain insight, books and studies are merely rungs of the ladder on which he climbs to the summit of knowledge. As soon as a rung has raised him up one step, he leaves it behind. On the other hand, the many who study in order to fill their memory do not use the rungs of the ladder for climbing, but take them off and load themselves with them to take away, rejoicing at the increasing weight of the burden. They remain below forever, because they bear what should have bourne them." -- Schopenhauer
The book was written as a guide to CIA Analysts to understand the limitations their own filters and mental models place on new information they process.
One important point that I find applies to this essay is that the way we retrieve information is through schema that associate various memories with each other. Creativity is about mapping new pathways through your memory or applying other patterns and schema on top of existing memories.
So, reading a book a second time, or even "forgetting" what you read, can not only give you new patterns and schema to apply to your other mental models and memories and stimulate creativity.
I highly recommend everyone read The Psychology of Intelligence Analysis... or if you want the abridged version you can read my brief recap. http://www.davidmelamed.com/2014/12/05/internet-marketers-ci...
Sometimes I avoid rereading books for the same reason. There was a Summer where Catcher in the Rye felt very important. I'd hate to reread it under my new, adult perspective. I'd prefer to let it linger in nostalgia.
1. The more correct analogy would be training data and machine learned model rather than source code and compiled binary.
2. Lot of people move from book to book, always reading some book at any point in time. This provides great dopamine hit to brain and keeps boredome away. However one need to reflect on what they read to gain any significant "take always". The act of reflecting enforces recall which in turn induces analysis and memory storage. I try to create a book review to formalize my reflection process after I complete a book.
3. These days I also get digital (usually pdf) copies of most books I'm reading. This allows me to use tools like GoodReader to highlight striking statements and make notes of my opinions as I read along. You can sure just use pencil and margin of book :). This habit has rewarded me greatly because it makes me take a pause and think about what I read. I can come back to book anytime and refresh it 10X faster. It's also fun to know what my opinion used to be on some of the things years ago.
I also find that this rings true with movies as well.
I also feel obligated to one-up you. One time in high school English class, the teacher put a sample student essay up on the projector and picked it apart in front of the class. It took me half an hour to realize that I had written the essay in question, and by that point I had concluded that I did not completely agree with it. I learned something that day about writing.
I saw the book at my friends house and thought it looked interesting. After borrowing and reading 50 pages I realized I had it in my shelf.
I suspect this is also a key part of your inability to determine whether or not you have read a given page before. You are reading in an environment that does not lend itself to creating notable memories. This is purely personal conjecture but I suspect that if you went and read somewhere more interesting your memory of what you are reading would magically improve.
Books I read while at home or similar tend to disappear into some sort of memory hole. Meanwhile books I have read while visiting other countries tend to be easier to recall, both in terms of the book's contents and the circumstances I was in when reading it.
Well for one things it gives you pleasure. You could also say "what's the point of listening to music" or "what's the point of watching comedy". Other than pleasure, as a generality, you might read because it makes you feel good to do so.
I find that a key to good mental health (that works for me) is not to question what harmless things make you feel good and why. If I did that it would make me unhappy. Just go with the good feeling.
One thing that I'm sad about is that I don't get the same pleasure that I used to from browsing books at Barnes and Noble. With the internet there is to much to read already. I don't find the same utility that I used to from books that are essentially a single perspective (at least the ones that I used to buy, non-fiction).
Also, in my experience, what sets the smartest people apart from "just" smart people is their ability to retain the how: not only do they have broad and deep models, but they also know how these models are built and can adapt them quickly as they acquire new information.
Most people need to run a disassembler of their compiled thoughts, and after a certain point in life, their binaries are so bloated that they can't decompile them at all.
“If I finish a book a week, I will read only a few thousand books in my lifetime, about a tenth of a percent of the contents of the greatest libraries of our time. The trick is to know which books to read.” ― Carl Sagan, Cosmos
The only reading I ever feel guilty about is my aversion to leaving a book unfinished. I'm pretty good at picking what to read, but about once every year or two I encounter some real stinker that is a literal waste of my time, and I feel a bit annoyed with myself for plowing through to the end even though I have long ceased to expect any literary or intellectual payoff.
"A book is made from a tree. It is an assemblage of flat, flexible parts (still called 'leaves') imprinted with dark pigmented squiggles. One glance at it and you hear the voice of another person - perhaps someone dead for thousands of years. Across the millennia, the author is speaking, clearly and silently, inside your head, directly to you. Writing is perhaps the greatest of human inventions, binding together people, citizens of distant epochs, who never knew one another. Books break the shackles of time, proof that humans can work magic."
Reading and experience train your model of the world.
And even if you forget the experience or what you read,
its effect on your model of the world persists. Your
mind is like a compiled program you've lost the source
of. It works, but you don't know why.
Of all the time I spend with my daughter, of all the activities we do, she probably will not remember much in a few years, but at least I can hope it will have an effect of her model of the world that persists.
I created a simple wristwatch accessory that was scripted to upload a copy of all of your chatbox text to an external service. Later, you could log in to this external service and search through a history of all of the conversations your character ever overheard.
Real-world versions of this technology appear inevitable as digital storage costs trend to zero. A rudimentary digital copy of the physical world is being created in services like Google Maps. The Google self-driving car records a 3D copy of its surroundings with accuracy at the centimeter level. Dropcam uploads video and audio data from within your home to cloud storage.
A world with fully recorded life experiences seems creepy at first blush, but I believe we'll discover a mechanism for trust that will allow everyone to safely record a digital copy of their lives that is inaccessible to third parties. Perhaps in the future we'll each own an open-source private cloud container of CPU and storage resources. Instead of processing your data on external servers, third-party services might provide code that runs in your own container under tight network permission restrictions. Such a system might be able to maintain the benefits of continuous software deployment while allowing consumers to keep their data under their control.
El Aleph - how small we are in this universe, and much much more
Steve Jobs bio - how to focus
Fahrenheit 451 - Ray Bradbury L'Etranger - Albert Camus Frankenstein - Mary Shelley Metamorphosis - Ovid Oryx and Crake - Margaret Atwood The Picadilly Papers - Charles Dickens Permutation City - Greg Egan
Fiction allows us to experience the most intimate thoughts of people we've never met in a way we cannot emulate in reality. We can visit places we've never been to and experience situations we'd try our best to avoid. We sit for hours hallucinating vividly reading these stories as we download these characters, concepts, and ideas into our meat. And if the story resonated with us we walk away a different person: new connections in our synapses, reinforced signals in existing ones. Stories are one of the most powerful tools we have at our disposal; perhaps even more so than mathematics or computation.
[0]http://www.penguin.com/book/frankenstein-by-mary-shelley/978...
The thing that made a lot of history make a lot more sense to me was playing Civilization. I don't know how accurate the details are, but the fundamentals of diplomacy don't change a lot.
Honestly most nonfiction feels like it was telling me what I already knew, or bringing new facts but no new ideas. I can't remember any that actually changed the way I think.
1. Are technological advances required for re-living experiences? Wouldn't (some forms of) meditation achieve similar results? Personally, I have found myself remembering many past events, a few days into the start of meditation.
2. When we re-read books, we often choose to re-read those that we liked. But could there be some benefit in re-reading books that we didn't like (and surpasses a minimum threshold of quality).
I do suspect we will be less likely to record our lives for later playback than have them analysed at or near the time for feedback on how to improve. Twitch TV is (I am told) full of streams of top rated people playing WoW and commenting on their actions (so others can learn, or be entertained). It's probable that there are shows now or soon that have players commentating on other players streams, and a fairly short leap from that to commenting on videos of me training my dog, or performing reps or basically anything in the life coach / therapy repertoire.
Audio and visual analysis already allows therapists to zoom in on the important parts of observed patients (certainly in sleep therapy) and will only get better.
Whilst the unexamined life is not worth living, there is no reason you have to be the only examiner. We shall all have our own life long therapists.
It also recently made it to Netflix. I've been hearing about it all over among my friends the last few days as they've been watching it there (and watched the first couple of episodes myself.)
Because reading is enjoyable?
A nice analogy, as others have pointed out, and that is the thirteen-word summary I'll remember from this essay. Not bad, considering most thousand-page books will be compressed down to a page of take-away memories.
We can see it clearly with the functional paradigm renaissance right now. The same arguments already existed 40 years ago, but _something_ changed recently where the perception of some people toward functional paradigm completely changed.
Another area of interest is "microservices", for the same reason.
I am able to grasp things pretty quickly, I am able to link two different things to get ideas to solve problems, and also, I have grown more confident in approaching challenging problems.
Albeit, with anecdotal evidence, I believe, taking interest in wide variety of fields may not give immediate benefits but it helps you in ways you don't imagine. The very fact about which I used to worry - not focusing on specialising and hoping from one thing to another, is what I think has helped me grow my skills in programming, in general.
They then go on to say: "We will never ask what a book means, as signified or signifier; we will not look for anything to understand in it. We will ask what it functions with..." Books are machines you plug into your understanding of the world and they either have an effect on you or they have no effect at all. What and how a book plugs into your understanding and works on it is more important than the content of the book itself under this view.
Does anyone know what this is referring to? Searching for "stephen fry singing trauma" doesn't return anything useful except pg's essay.
http://www.amazon.com/The-Fry-Chronicles-An-Autobiography/pr...
(search for "singing")
New data is always added to the model, but not in an entirely rational fashion. The updated model is likely to slightly overfit new data ("compiled at the time they happen"), and particularly salient bits and pieces of old data (see https://en.wikipedia.org/wiki/List_of_cognitive_biases) are disproportionately weighted.
EDIT: Found the link http://www.nami.org/Content/NavigationMenu/Top_Story/Using_V...
http://www.nytimes.com/2010/09/19/books/review/Collins-t.htm...
http://www.newyorker.com/books/page-turner/the-curse-of-read...
http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/2000001...
This sounds convincing but then an argument against reading fiction follows since fiction trains your model of the world with fake data.
I would argue that certain types of fiction may actually be beneficiary such as To Kill a Mockingbird or Catcher in the Rye (vs something like 50 Shades of Gray). Those sorts of fictional settings allow the writer to present the experiences and specific feelings/situations to readers. Even though, say Lord of the Rings, isn't exactly relevant to our present day world, the morals and spiritual emotions involved reflect humanity (what makes a human?)
I guess your comment has some parallels in art: why should we draw abstract things when we can recreate what we see? Isn't recreating what we see more important/better art than something that would never end up existing?
While it's not a perfect analogy, I believe reading and looking at the creative arts ultimately benefits your model of the world through bettering your model of humanity.
The scientific method depends on the generation of testable hypotheses.
How do we generate hypotheses?
http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statisti...
By this process I've been able to internalize much of a massively complex system (SAP) in a relatively short period of time.
Like most things this may have unintended consequences. I think our ability to forget is an important "feature" of cognition. What would happen if we were unable to forget even petty squabbles between friends, loved ones, supposed enemies? How far could this escalate? Our ability to forget and put things behind us may be the reason we're still around.
That sounds fascinating, does anyone have a reference for this?
On first read you have a few key points and years later sometimes those end up knitted together forming a greater insight that eluded you previously.
It took me a while to learn that almost everything I have heard or seen has already been stored. The problem of memory is in retrieval.
This also applies to creative work. When you have seen quite a lot of things, it ends up influencing stuff you could swear was original.
― Ralph Waldo Emerson
When you learn about the four aspects of the mind and how each plays a role in your outlook, then you have the key to this "mystery".
Is that simply rubber ducking?
What if our brains are not easily shaped? And maybe our brains are good at forgetting experiences?
There are no facts, only interpretations.
--Friedrich Nietzsche
I've gone through this exact anxiety. Access into past experiences (whether they're books, vacations, or pieces of visual art) isn't random-access. We seem to have a lot of on-demand knowledge but struggle to retain deep knowledge of something in an independent, self-encapsulated state. Depth of knowledge seems to form, in us, emergently and subconsciously. It makes a case for the Buddhist argument that all things are interconnected and interdependent; certainly, in our knowledge bases, that becomes true very quickly.
For me, the solution was to learn to enjoy the process of learning rather than completed act of having acquired knowledge, it being difficult to summarize any acquired knowledge without coming up with a trite, denatured reduction. It took a while because I was secure in the realization that, yes, I had read that 500-page book even if I couldn't reproduce more than a few pages of "raw" and independent knowledge.