We are drowning in information while starving for wisdom (2021)
realizeengineering.blog
realizeengineering.blog
After that it always ends in hard to parse research papers, specifications, jumping around the source code of multiple libraries, debugging or reverse engineering. If I get lucky I might land on a 15 year old blog post that no man has seen since inception and it describes exactly what I wanted to know.
And that's just software. Humans know exactly what parts went into it and how it works. Why must wisdom always be squeezed out of a rock? Why is only the fluff copied thousands of times making research harder each day? Why are there never indicators which way to dig for more details?
T__T
Wisdom comes from an awareness of the greater whole. The insights do not come from analysis, but rather, synthesis. It engages with the intuition rather than the intellect.
Although it can seem mystical, I think there are authors who have been able to express ways to engage in wisdom even if they are not directly talking about it. For example, Christopher Alexander has some interesting things to say about wholes, centers, and unfolding:
Seeing Wholes - https://iamronen.com/blog/2018/01/14/christopher-alexander-s...
Centers - https://iamronen.com/blog/2018/01/08/christopher-alexander-c...
Unfolding - https://www.livingneighborhoods.org/ht-0/whatisanunfolding.h...
Also, many so called prophets can easily exploit this skill, which they might well do if that’s their personality type.
Just typing this out has made me think it’s almost like passing on wisdom and the religious experience are probably inseparable.
As a result these changes can destabilize a persons identity, which causes said person to look for the closest source of stability, often the 'peophet'.
This gives the 'prophet' enourmous power over the person, not just because of the identity destabilization, but because when you change someones fundamental assumptions about the world, those changes don't nessecerily have to be truthful or helpful for the person. The person only has to think the changes do.
This identity change and stability dependence is probably what causes the cult like appearance/ behaviour.
What's amazing to me about Christopher Alexander and his work is that he's able to describe the process of generating such art (in the form of architecture) to ordinary people, using plain, relatable language. The links I posted in the earlier comment are examples.
In other words, you don't need to be someone with a rare skill.
The next step in the process is to undo your abstractions to determine which of the previous solutions (if any) is the best one for your particular situation. It's not entirely identical, but I tend to think of intelligence and wisdom as deductive and inductive parts of the same process.
So I don't think that means that wisdom is so far removed from scientific enquiry that it becomes mystical, far from it: instead, it requires so much more rigour and discipline to codify wisdom into laws that it just doesn't happen that much; and even when it does happen, the most you'll find is fuzzy frameworks on how to deal with certain problems rather than the hard and accurate rules you can find in deductive analysis.
However, wholes are nested. The computer you are using to read these words, wherever you are, are part of a larger whole. Further, there is a paradox in which, while parts make up the whole, it's the whole that makes the parts.
Taking those all the way, "The" Whole in which all wholes are parts of, then, is boundless (no edge), and it is beginningless, (no causal origin).
My understanding of the scientific method is that it is ultimately limited in what it can find. It is not necessarily true that the scientific method is capable of explaining everything, though it is broadly applicable. That method is very good for analysis, but not synthesis, and focused on the origin in causal chains rather than the teleos.
Sure, one also needs understanding to get something out of search. But Google when was in it's sweet-spot, it could get a researcher extra knowledge and insight. After all, a researcher needs both a holistic perspective and information they'd know at the start of an exploration.
I was so tickled that I put it on my profile here on HN and elsewhere -- but all of those things I put down are different ways of saying "Wisdom".
Google has aspirations about organizing the world's knowledge, and making it easy to search for facts and knowledge. What I'm saying here is that people have never been able to find wisdom in a Google search -- and never will. Each person finds it within themselves through awareness of wholes.
So in my view, it is not that it's getting harder to find wisdom from a Google search, but rather, the proliferation of knowledge and facts over the years have increasingly made it even more distracting to find wisdom within themselves.
The Göögle Incompleteness Theorem
As someone who once thought this was true of my experience, yet have since learned they were just common phenomenon, I'm curious what these experiences were. And how do you think they have effected your perspective?
I don't have any experiences I would call "verifiably paranormal", but I've had a few I've not been able to fully account for to my own satisfaction from a purely materialist perspective (despite a total collapse of my childhood evangelical Christian beliefs).
One was sudden, total cessation of a destructive habit I'd struggled to stop for decades, after a single prayer I hadn't realized was related to that struggle ("I don't care what it costs - make me more like You"), a very unexpected turn of improbable events immediately following the prayer, and an intense spiritual experience following the unexpected events.
I have devised pure materialist accountings for it, but they aren't rigorous and haven't persuaded me.
...all that said, I actually found the GP post very understandable and thought it made tremendous amounts of sense - perhaps even was "wise." I certainly wouldn't call it "bafflegab".
I should perhaps clarify that while I had the total collapse and found myself staring nihilism in the face, I currently am drifting towards a very heretical, Bible-is-flawed-and-maybe-not-divine-at-all form of faith in Jesus (in part due to the above-mentioned experience).
I’ve also experienced telepathy, seen a ghost (with multiple corroborating witnesses) and witnessed poltergeist activity. The last of those I can kind of doubt though. The mirror falling off the wall could maybe have been some kind of microquake.
It's good to see this surface up here, though it's the kind of stuff I discuss in other communities.
There are more things in Heaven and Earth, Horatio, than are dreamt of in your philosophy
Email is my profile, if you have the time and inclination to get in touch.
Given your background, I think you might find Christopher Wallis's Tantra Illuminated -- in particular, the chapter on the View of Tantra -- more sensible. Although it does not talk about wholes, centers, and unfoldings in those terms that Christopher Alexander does.
I think the key is in the non-duality of creator and creation. That within all parts are the Whole. I can't tell from the wikipedia page on Leibniz's monadology if monads are distinct from each other, whereas that stuff about wholes and centers point to apparent distinctions on a continuum that is the Whole.
wisdom, 1. in general, the best form of knowledge.
2. specifically, the intellectual virtue
or science concerning the first or
supreme causes of all things.
philosophical wisdom, intuitive knowledge combined
with scientific knowledge of
the objects which are
naturally the highest;
metaphysics, including
natural theology.
practical wisdom, 1. prudence.
2. any excellent form of
practical knowledge, as of
one of the architectonic
arts.
In those days, we would usually read the manual when we needed to dig past a surface-level understanding of how things work. The manuals are dry and dense, but they're much easier to read with modern niceties like "Ctrl+F".
IMO, this is one of the reasons that people recommend using software with a long track record. If you have a question about some parameter in a systemd service script, and the internet doesn't have a ready made answer, the details are all written down in the manual.
https://www.man7.org/linux/man-pages/man5/systemd.service.5....
Plenty of modern tools have comprehensive detailed docs like this. Python, Go, even most widely-used JS frameworks. The primary sources are often downranked in search engines because they don't sell ads, but it's a good idea to find and bookmark them when you start working with a new technology. RTFM!
Then, after that didn't work, we had to go to find the weird bearded guy that didn't like to talk to people, but knew everything. He would usually tell you the answer, but in a condescending way that made you understand that you only came to him in "emergencies" and that you were kind of stupid for asking an "obvious" question. Actually, now that I think of it, that is stackoverflow now!
Thank god message boards started taking off! ... but sadly, then the "real" Internet came and killed it. :-(
"Why would you want to do that?"
"What's this for?"
"Did you try searching before asking such a stupid question?"
OP bemoaned how people on StackOverflow would tell you not to do what you're asking about. Reference material is not the cause of that "weird bearded guy" problem, but it is one possible solution.
We're talking about software developers, not wizards. It's not like you have to offer a dram of blood and draw a pentagram with salt before you approach the ancient grimoires.
On the good threads that become helpful, yeah.
On most of it, no, those are not the questions people ask there. They will focus on unrelated marginal issues, try to refuse to answer the question, just assume the reason the person is doing it (even when explicitly told on the question), and just throw wrong answers on the wall to see if they stick. (Granted, throwing things on the wall is a useful way to answer some questions, but not all.)
It's good that Google rewards the first kind of thread, but the second one is what sends people away from the platform.
Q: "I'm trying to configure Foo to produce Bar, but it's giving me this error!"
A1: "What are you really trying to do?" [Unsaid: I don't know how to produce Bar either]
A2: "You should not be trying to produce Bar. Instead you should produce Baz. I know how to do that--follow the following steps..."
A3: "Producing Bar will not solve what I imagine your goal is. In the general case, you may need to produce many different results, which I would rather answer about..."
Q: "Uhh, thanks everyone, but I'm just trying to configure Foo to produce Bar."
Many such cases!
The Elixir community talked about this a lot on the forums back when I was active on there, because language rankings look at Stack Overflow questions and Stack Overflow questions proliferate in languages with footguns. "Don't use obviously_named_function(), use obscr_wrd_fcn() instead" fits the SO formula quite well. Well designed languages, Elixir included, don't produce many of those kinds of question/answers. That leaves significantly more room for discourse around architecture and other higher-level conceptual stuff, which doesn't really have one correct answer, and thus is ill-suited for SO, being relegated to the lower-discoverability forums (and worse, the un-discoverable Discord).
Back then, it seemed a lot like that!
>"Why would you want to do that?" because it seems like the right thing to do? Is there a better way.
>"What's this for?" the boss wants it
>"Did you try searching before asking such a stupid question?" I read the manual and thought about it a lot (remember this was pre-Internet)
so, there was a lot of "rough" hand-holding to learn the ropes for edge cases.
The thing is... languages were smaller because they hadn't started the accretion of thousands of libraries and frameworks.
In my experience, every language out there is somewhat easy to learn and master. The ecosystem around it is an insane and ever-growing Katamri Damacy of (largely) crap. We all must know it, must use it, and must contribute to it because no one trusts the work of the individual and only the work of an aggregate of individuals... people often don't even trust their own code.
I wouldn't make a blanket statement either way, but there are certainly counterexamples to this:
- The mpv manual[1] is a work of art.
- The Arch Linux wiki[2] is a treasure trove of information for not just Arch-specific topics, but Linux in general.
- MDN[3] is the defacto standard for any web documentation.
- The PostgreSQL[4] documentation is quite thorough and high quality.
What I think explains your point are two things:
1. There's just a vast amount of software since those early days. "Software is eating the world", and it's realistically impossible for most of it to be well documented.
2. A lot of information is spread out and produced by users of the software; in books, on blogs, tutorials, forums, videos, etc. Sure, this might be seen as a failure of the software authors to produce good documentation, but many of these resources wouldn't exist if the web didn't make them accessible. In some ways this is better than having a single source of reference, as you can benefit from the collective wisdom of the hivemind, rather than only from what the author thought relevant to document.
[2]: https://wiki.archlinux.org/
"Back in the day", this problem was addressed by having significant hurdles in order to publish material on a topic. E.g., books used to be the primary way to learn about a topic, and in order to publish a book you must 1) deeply engage with the material for quite a while simply to write the book and 2) become fairly credentialed / convince a publisher you're worth their time, etc. Same thing with academic papers.
The internet has completely removed all barriers to publishing, meaning it's up to us to sort the wheat from the chaff.
I've been feeling this more and more in relation to software. It feels like the more popular a piece of software is and the more people use it, the harder it is to find and talk to users of said software who have engaged with it past an extremely "shallow" level. I constantly encounter significant, easy to reproduce bugs and issues in very popular software used by millions of people, maybe even tens or hundreds of millions, but if I try to look for any sort of discussion about it online, I usually get nothing.
It feels much easier to discuss software with other real people who use it and think about how they use it when it's something really niche used by maybe a few thousand people at most.
My suspicion is that most people who engage with Unity are amateur engineers. Nothing wrong with that, of course.
> meaning it's up to us to sort the wheat from the chaff.
I was thinking about this the other day. How do you sort out conflicting information when your not an expert your self. I don't know who to trust sometimes.
If you don't find an expert in the field and you publish a crappy book you tend to get a number of reviews (hopefully before you send it out to retail) that your book sucks.
When selling books about fake evolution you're not looking for an expert in the field, you're looking to see if you have an audience that would eat it up.
i wouldn't be surprised if a very large percentage of the last decade of tech blog posts are just remnants of someone's promotion, and is now hyperlinked on some resume, or linkedin page
I think this is only true if the commentors are SME's & operating in good faith. We've all seen whole threads hijacked by erroneous groupthink, while the 1 or 2 SME's were buried.
Granted, there's a lot of chaff in books as well, but their quality to junk ratio is wildly better than Google, and they generally go much deeper.
It's the nature of the incentive system that they set up. Their only grand vision for the internet was one where Google hit its growth targets every quarter. If you do the math on that, then Google has to be enormous in 2022. Guess what, Google is enormous in 2022, and the only way they could figure out how to do that was to keep growing their ad business. So here we are.
Free means we the users become the commodity.
Don't pollute the frigging index with SEO garbage. Google didn't tell everyone to SEO. People didn't even start doing SEO.
Then some arsehole went and had a brainwave about "what if we sold people on tweaking their pages to try to get all our customers fighting each other to show up on the front for the front page"?
I still remember my first time overhearing some SEO guys pitching to somebody while eating lunch. Spat out my food when it dawned on me what I was hearing unfold and started to realize that guy was likely not the only doing that.
The librarian's assistant in me had to be held back that day.
Why? Why would you pollute the Index that way? Monsters... Convincing people poisoning the town well was a good idea...
Google gave lots and lots of ad money to websites that did SEO, so they effectively did tell everyone to SEO - crippling their own search product in favor of their ad machine.
In this discussion, the people making the good content don't do it for the money, but the people creating spam do. Kill the money, and the commercial SEO crap goes away, and only the enthusiast content stays - as it was in the 'good old days' when simply getting clicks on your site could not get you any money.
> the people making the good content don't do it for the money
citation needed. there're two driving forces:
- money
- clout
they're interchangeable. if you have one the other becomes much easier to obtain. if the writer don't do it directly for the money, they're most definitely doing it for the clout.
Our imaginations have run out because those sweet ad dollars are too good to pass up.
Only problem is that pesky copyright that makes you buy them
It succeeded in eliminating spam from a lot of small sites and instead resulting in an even bigger spam from the same big sites for every category. Great for people who were doing shoe-shopping at, well, Amazon, probably, but bad for everyone else.
Then they increasingly personalized search results via AI. And that changes your search results so fast that you may not get the same seach results for the same exact keyword in a week's time.
Im increasingly of the opinion that corporate, 'business mentality' should not run any major tech corp. It sees everything in numbers and extreme abstractions, juggling them to make a desirable false reality happen in paper and in metrics, screwing up the real world for every user.
Just before it hit, I had a small site that was on the rise in terms of traffic. A hobbyist site, no ads, all original content, and very much not "thin". An enthusiast site, much like the original internet. Incoming links were organic and spontaneous, from serious sites.
Panda crushed it. Massive pagerank downgrade and traffic decimated to about 5%. In mere hours. A false positive, I suppose. I spent 3 months trying to figure out the reason but nobody, including Google or SEO specialists were able to provide any tangible answer.
That day etched a few important lessons into my long term memory:
1) The size of Google's power. It effectively has the power to decide whether you digitally exist or not. The ultimate traffic controller.
2) Even a good faith move by Google may have 1% of false positives. Which seems totally rational until you realize that at Google's scale this could mean millions of people being totally fucked out of the blue.
3) Worse, when that happens, there's no accountability. You can't go anywhere for help, undo being a false positive, you're just randomly fucked and that's it. You cannot depend on anything ran by Google.
Although I'm fine and it wasn't exactly a pivotal moment in my life, it radically changed my view on Google. Early on, I was a fanboy. Google kicking lame Microsoft's butt with miraculous products like search, maps, gmail. I was cheering them on. As of my "lesson", I've grown increasingly cynical.
That was 10 years ago. They've become infinitely worse.
Yep. I also had various such legitimate sites. They all sank to oblivion. Worse, I had a small web development outfit - we used to put a link back to our site at the footers of the websites of our clients, like 'Designed by', or 'Developed by' etc. As then-advised by Google as good practice.
All of our clients' websites sunk. Small businesses, small ecommerce sites, blogs, open source project sites. All of them became 'low quality'. Not only their livelihood was sunk by Panda, but also our, the developers' websites were sunk because those sites were linked to us per what Google advised. We lost incoming business. Independent web development, small businesses, blogs etc that were thriving before Google murdered them were forced to use the centralized marketplaces - elance, upwork, amazon etc. Causing the consolidation of the internet and the rise of the algorithm.
Its amazing how no class action lawsuit ensued over that. Google literally backstabbing its own users who adhered to the best practices it advocated, and killing their businesses and livelihoods to amplify shopping searches for Amazon and other corporate brands.
> nobody, including Google or SEO specialists were able to provide any tangible answer.
It was because of 'low quality links' that were pointing to websites. Ie, small sites linking to your site. I remember clients fervently going through all the other websites that they asked to link back to them, and begging them to take off those links so their rankings would improve. Some of them succeeded and rose some in ranking, but nothing near enough to keep their small shops alive, leave aside near anything before Panda.
> 3) Worse, when that happens, there's no accountability.
Yep. One sociopathic old-school corporate exec and a host of totally disconnected corporate executives decided the fate and livelihoods of tens of millions of people directly. Maybe up to hundreds of millions of people if you counted in the business ecosystem that those small businesses supported. Literal feudalism, albeit exercised through private tyrannies without any accountability.
That made me conclude that any corporation that holds the livelihoods of people in its hands has to be tightly regulated by the the democratically elected governments.
> it wasn't exactly a pivotal moment in my life
It was a pivotal moment in the lives of countless people and their families. They lost their businesses, their livelihoods. Even worse, the Internet los the thriving grassroots businesses, communities and creative ecosystem that it had. It may be regaining it with the rise of the creator economy, but that will take time. Nothing can make up for the harm that Google did to the Internet and actual people's livelihoods.
You make a great point that as of then, the great centralization of the internet began. It's only heads and the tail is cut off. Social media accelerated it even further. This is why 90% of the internet feels dead.
You see this a lot in recipes, and I've heard tales of military being susceptible to this, large manufacturing processes, etc where the tweaks are susceptible to being lost if large cutbacks/layoffs were to affect the people with that tribal knowledge.
Finding something more complex than few cm deep is becoming a hellish task.
I blame Google for lack of moderation and getting their SOE gamed like a little pipsqueek.
Juniors are dying happy coz of how fast they can find trivial answers.
Seniors get shafted. Its so bad that I started to build my own github repo woth various answers I needed over the years.
Example: it's 2022 and we still somehow upvote blog posts detailing one's stack as if that has any impact on anyone else.
You see this on Reddit all the time. There will be a Q&A sub for a topic dominated by people with scant if any experience who think they're helping people by googling stuff for them and there will be a comedy subreddit for the same topic. Smart people ask questions in the joke sub because they know that having sufficient understanding to be amused by highly contextual stuff weeds out a lot of the casuals.
I think this is due to the fact that the overall concepts are usually quite easy to understand and therefore easy to talk about.
As soon as you have to dig deeper you need to invest significantly more time to talk about it and your audience becomes really small.
This is amplified by people creating newsletters, content (marketing) in general, but the actual knowledge/wisdom is not given away. It's incredibly frustrating but happens in almost all technological field, especially when they are fast moving.
In mechanical engineering for example a lot of wisdom is also put into norms, standards and so on. Books are written about it. But 10 year old books are still completly valid while in software every couple of years a new framewokr is the new hot shit.
The more I look at it, the more it feels like software hasn’t had a fundamentally new idea in decades. Frameworks improve, networks get bigger, compute gets cheaper, and we spend most of our time dealing with domain modeling, cache, and statistical inference.
We’ve made great strides in the ease of building and encoded many lessons in our frameworks, scale is bigger too, but the underlying guts of software engineering have been very stable for decades.
And there are good books about software architecture and so on. Unfortunately, for each good one there are at least 50 that are really really bad. There are a lot of technology-related books as well, that teach you about responsive web design with react or stuff like this.
Encoding lessons in frameworks isn't always helpful either in my opinion. Your mental model of the software needs to be correct, otherwise it's just a really painful experience. What works for large scale software is often just pain for beginners.
Don't worry, it won't be long until we're lamenting that most of ChatGPT's output caps out at about the level a human could deliver.
Partly, because so many developers famously hate writing documentation and hate commenting their code.
If developers followed Donald Knuth's Literate Programming [1] then it would be a big improvement.
People rely on a lot of "tribal knowledge" without ever bothering to write it down, and out of all the fields, software development seems to be particularly notorious for its anti-documentation bias.
I do feel like wisdom is inherently an "internalize for your own mind" process, and on some level, nobody can do that for you. So of course you will be scrabbling around trying to piece things together until it clicks in your own mind.
I've found having a discussion with another human, who understands the topic, is very helpful. And it is one case where an "academic" setting can be useful, assuming you can actually talk to the professor rather than being forced to parse some unfathomable textbook.
Even that example — a textbook — is similar to what you said. It is literally designed to help you understand a topic. And yet, my experience with them mimics yours. A lot of the material is uselessly general, and a lot of it is uselessly specific and technical. And it almost never answers the specific question you actually want to know at a given moment. So, you have to internalize the information (slowly) and eventually get there.
Maybe there is an "event horizon" of me being able to absorb new material, rather than something external.
(It's very impressive how it rewires the logic, but it can't really know anything, by design.)
And yes, that horizon is very real. It's what the internet rewards, so it's what almost everybody delivers.
When you google for some programming story you typically look for a story that describes how to combine concept A, B, C and maybe F. The more concepts you add the likelyhood of somebody having described a good implementation story for the combination of those concepts goes down. But you don't need to add many such concepts before the number of potential stories become very large. So for any non trivial number of concepts a blog article is not likely to exist.
We could only really change the game by going from text representation to intelligent representations. Rules or general AI.
However, my main surprise in all this: why do we need this many combinations of concepts in the first place? Why isn't the total domain of things that are interesting to do for utilitarian reasons more limited in size?
The cure for me has been two fold: 1) Focusing on books from reputable publishing houses (curated long form content) - but even that takes filtering. I noticed O'Riley has been putting out a lot of noise lately. 2) Focusing on open-standards, sound engineering. I'd rather focus on understanding hardware, operating systems, networking etc then myriad software as a service offerings full of vendor lock-in dark patterns.
On the plus side, that means if you're willing to dig through the research papers and other hard but useful sources, you have an opportunity to make a difference by writing that book.
That doesn't happen for trivial topics or problems though! It's a sort of new frontier where we have to re-solve potentially solved problems because the solution isn't pervasive enough to be trivial.
We have to push ourselves to overcome a knowledge limit, which throws a nice fat monkey wrench into the idea of having "All human knowledge in the palm of your hand".
There's undiscovered territory out there and it's hard to see unless you get above the trees.
What you’re describing takes a lot of time and nerve. Those are two of the foremost requisites of the pursuit of knowledge.
The Web is working. It’s enabling you to even access those old manuals via the Internet Archive, papers via Sci-hub, source code on GitHub and of course that old blog post on page 7 of a DuckDuckGo query.
The Web works y’all!
Dictionary says there are 3 components to wisdom: knowledge, experience, and good judgment. Machines can help with the first requirement, knowledge, to an extent but leave you frustrated (as you note). They can even help you dig out wise chestnut buried somewhere (as you did) but then you hit the other two.
This is how it is supposed to work: armed with knowledge, you apply this and gain experience, and after a few close calls the fortunate also develop good judgment, informed by experience of application of knowledge.
You want all that squeezed out how?
Working on a honest straight reality is like drinking water, has no calories in it. Coke is better. Wines are more better, and everyone other than than I these days seems to be always on a magic fissile cocktails, and while I don’t get that, they seem well off doing it.
Comparison of ChatGPT and Google on 500 coding queries: https://news.ycombinator.com/item?id=34089423
I view documentation as a set facts and stack overflow as additional color.
What type of projects require this workflow?
Most responses are there just to get your attention.
This whole thing runs on attention.
The whole idea is that individual papers are supposed to be exploratory, throwing things at the wall and seeing what sticks. They're supposed to be a deluge of information.
But then every decade or so a team of academics take it upon themselves to serve as editors to a handbook, which attempts to survey the field in terms of history, where the most value has been found so far (and what hasn't panned out), and current promising directions. Usually something like 20-50 chapters, each contributed by a different author.
If you want to get into the wisdom of a field, the first thing you do is pull out the most recent 800-page handbook, read the first few chapters, and then drill down in your area of interest on the remaining part.
To say there "are no prizes for wisdom" is absurd, when being selected to publish in a handbook (or being an editor) is prestigious, a mark that you've very much "made it" in the field.
And of course there are plenty of other things that serve similar roles, such as literature review papers or similar. (In philosophy you can write a Stanford Encyclopedia of Philosophy article, for instance.)
If you aren't finding wisdom anywhere, it means you're simply not looking right.
(And this isn't even to mention the fact that at some point somebody will popularize major progress in a field in a general-audience book, e.g. when Daniel Goleman wrote the book "Emotional Intelligence" or Stephen Hawking wrote "A Brief History of Time".)
Unfortunately some of these are absurdly expensive, so if you don't want to go the piracy route the cheapest way to access them is to get a membership to your local public university's library system, which in the US typically costs like $100 a year or something.
[1] https://link.springer.com/referencework/10.1007/978-1-4939-2...
[2] https://link.springer.com/book/10.1007/978-3-662-52844-0
[3] https://link.springer.com/referencework/10.1007/978-3-030-44...
[4] https://link.springer.com/referencework/10.1007/978-1-4939-6...
I've never seen a handbook that led anyone "wildly astray". They're put out by major academic publishers (Oxford, Routledge, etc.) who hire (publishing) editors qualified to select qualified (academic) editors to select qualified chapter contributors. It's not like they're randos self-publishing or something.
The entire point is to be a fairly neutral, comprehensive state of whatever field or subfield the handbook covers. And they generally do a pretty good job. A place like Oxford is never going to publish a handbook that's trying to push some ideological agenda and ignoring half the field.
But if you don't trust the senior editors at major academic presses, then I don't know what to tell you.
And since you've never heard of handbooks, see my peer comment with links so you can see they exist. :)
People on the academic oriented degrees tend to use them a lot. But industry oriented degrees tend to stick with textbooks. (By the way, yes, textbooks are the other kind where you can find wisdom. Normally in an easier to get, more condensed form, but of an older kind.)
About who selects the books, well, who tells you if a book is any good? Some have very radical untested ideas, others stick to older but proven ones. You decide what book to get.
Are you from the EU? Becuase having been in US STEM graduate programs in 2 different fields I have never come across such a handbook or know anyone who has. It's certainly not common in the US.
But more concretely, you can just look up the authors that contributed to the handbook and if you do indeed have a degree in the field, you'll probably recognise them and their affiliations and will be able to know (or at least look up) what tradition they belong to and what this implies for the handbook.
"Cognitive Neuroscience: The Biology of the Mind" by Gazzaniga et al.
I don't own a physical copy of it yet (only digital), but I do have a copy of an equally outstanding book[2] called "Neuroscience: Exploring the Brain" by Bear et al. An excellent thing about this book is that each chapter has a small section called "Path to Discovery" where leading researchers on a given topic (including Nobel winners such as Eric Kandel) briefly share their story of how they arrived at their discoveries. Another excellent aspect of this book is its rich set of visual illustrations of complex topics. It makes learning a joy. I find their high price justified.
[1] https://wwnorton.com/books/9780393603170
[2] https://www.jblearning.com/catalog/productdetails/9781284211...
This often ends up in summary papers, foundational papers and papers with well founded experiments. The fact as a society we pay for millions of researchers to do (mostly) research is a resource many people ignore.
https://www.routledgehandbooks.com/home
https://academic.oup.com/pages/oxford-handbooks
One I've read cover-to-cover, for example, is the OUP Handbook of Political Science:
I know this because I'm subscribed to lots of newsletters in multiple fields. They contain one gem after the other. There's a wealth of top notch content that does contain said wisdom. As in, read just this article and skip the 100 other ones. Because it is so damn good.
But I have to actively store them to notes or bookmarks, because you'll never find a single one in Google.
Google has become completely broken for me. In general it returns low quality verbose content, which seems a variation of keyword spamming from 20 years ago. Gonna be great when those crappy pages can be generated with AI.
Google promotes stale content that is outdated or simply incorrect over anything else. And not only that, this effect strengthens over time.
Google increasingly dismisses your input. You type a few words and it simply drops a few.
Google has no understanding of meaning. You'll type "red flower thailand" and it'll just include flowers from any other random country, including those not even red.
Google indexes everything and still can't figure out an original source. It'll show you spammy Pinterest garbage over the actual high quality source.
So here we are. We do have a treasure chest of wisdom. But wisdom doesn't click ads so fuck us I guess. And if this isn't depressing enough, Google will continue to get away with it. Google only has to work for the masses posing normie surface level questions. It does that. It works.
Which leads people to post what would be their google search to Reddit, after finding any group related to the topic.
Which is leading to Reddit forums being filled with the same beginner questions over and over and over and over...
... from people who don't even bother to join the group.
Can you recommend your favorite ones?
I'm particularly interested in startups/tech/programming/AI topics, but if you know some great examples from other fields, please share them as well!
My before situation: I used to follow dozens of experts on Twitter. Which is a continuous effort, and includes lots of noise. And that's just Twitter, I also followed some Medium authors, Youtubers, and sometimes watch videos of conferences (that I did not attend). I replaced all that with a single newsletter:
It comes weekly. If anything of any importance in the field happened, it's in there. It's such a vast efficiency win. Once per week, I spent some 30-60 mins reading it, and I'm up to speed.
Generally, 80-90% of the contents I can quickly scan through as they're not that new or important for me to dig in to. Yet 10% is impactful and bookmarkable, the gems I referred to earlier.
I'd recommend to just try lots of newsletters from your field and let them compete. Read them for a few weeks and keep unsubscribing until you have the ultimate one. Go for weekly newsletters, not daily ones. Let the information come to you instead of chasing it, reserve fixed reading time in your schedule.
Hope this helps!
If you're not comfortable with sharing those, feel free not to!
I find searching for things more enjoyable now, similar to how I felt in the 90s with Google. However I'm not sure if that's due to a bias in results for sites I prefer. It is significantly less spammy though.
It’s fun to read, informative and insightful.
Take the author's example, 'entropy'. Well, a one-word search is of course going to generate a massive pile of unsorted material. Internet Archive Scholar gives ~873,000 hits. Nice resource by the way:
https://news.ycombinator.com/item?id=33919823
If you knew nothing about entropy, but knew how to research, you might start with, "okay, when is the first appearance of the word 'entropy' in the literature" and you quickly discover it's Ludwig Boltzmann, 1872. Seems to have something to do with energy and information, and if we add in those terms, we're down to 284K hits. At this point, you might think "I bet someone has written a good book on this broad topic, maybe I shouldn't be looking at the primary literature until later."
Notice that adding search terms narrows results? Keep doing that until you get a smaller number of hits. Now, you can grab the top dozen or so papers, and flip to the bibliography, and look to see if they all reference a landmark paper (this is related to searching by citation count). That's probably one you want to look at.
With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. If you find a site with a good article, search that site for more. For example, this gives a lot of good results, for the role of entropy in machine learning concepts at least:
'site:towardsdatascience.com entropy Shannon'
It's basically mining a haystack for needles. There are lots of techniques and strategies one can use to speed up the process.
I really enjoyed Fravia's [2] SearchLores [3] site on how to dig deep into the knowledge of the internet. Most people rarely cross the boundary of the SEO sludge into the really good stuff.
There are a couple easy patterns, starting from a seed to iteratively widen and deepen the context.
Wikipedia, Archive.org, site:edu, filetype:pdf, libgen, and now chat.openai.com
I just asked chatgpt, 'What is a good home experiment to show the concept of "entropy"' and the answer was excellent.
I then tried the same question on DDG but appended "site:edu" and found https://paradigms.oregonstate.edu/activity/325/
Now I am off to do the "Time Dilation Clock Skit" with my kid
https://paradigms.oregonstate.edu/activity/548/
[1] https://www.catb.org/esr/faqs/smart-questions.html (something wrong with his certs)
[2] https://en.wikipedia.org/wiki/Fravia
[3] https://web.archive.org/web/20191201105758/http://search.lor...
-----------------------
The first page of results for
"dan dill entropy site:bu.edu"
Yields
https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondL...
Not anymore. For a decade or so.
Nowadays adding terms on general purpose internet search engines makes your results less focused, not more.
'entropy shannon machine learning differential'
will give more focused results than just 'entropy', certainly. I suppose there is some limit, is that what you mean? Even there, if I add 'literature review' the results get more focused.
Your examples only work because there are plenty of popular sites that talk about this stuff.
Now things boil down to the Chinese room problem. Take a topic with no 'popular' sites that can work as traffic directors. How can Google determine if you're authoritative on the subject, or if you're just a SEO site spitting out spam?
Smart is being able to absorb and process things quickly
Wisdom is knowing and picking the right course of action
Understanding is knowing why, and what you're trading with this choice
More or less, people generally don't seek wisdom because they have poor vision for the future.
learn from history (or even study it)
learn from elders (or even be around them)
read books
etc
I do realize that HN is subset of the population who do enjoy much of these things, but HN self-selects by nature of content.
But of course they all think that they're martyrs for speaking the truth.
Perhaps taking the example of Plato, a starting-point for wisdom is to try to define your terms, in part through recalling what others have said.
Take e.g., the citation "In Chinese thought wisdom is perceived as expertise in the art of living, the ability to grasp what is happening, and to adjust to the imminent future (Simandan, 2018)."
Aristotle and Plato would call this phronesis, practical wisdom, where the root phrein refers to the gut -- where, not co-incidentally, Zeus put/ate his wife and gained by her wisdom (and modern-day scientists eagerly study the GI nervous system's role in anxiety). Aristotle rooted that in understanding politics (vagarities in how people react), economics/incentive systems, and of course the physical world in terms of knowledge-required, but in light of the recent well-educated democratic leaders who became tyrants, he posed it mainly as a question of character, not knowledge. (Remember Aristotle left his home to become the tutor of Alexander the Great.)
Some related terms from that time and place...
Nous: pure thought, thought considering itself, typically as validation for principles and coherence of chains of reasoning. Quite similar to Descartes' notion of the irreducibility of the sense of one's own mental activity, combined with the pureness of its continuity (that must of course be grounded in God).
Dianoia: two-thought, logical and a dialectical thought depending on reasoning chains from point to point. (cf Paranoia, i.e., concurrency dianoia, and Parmenides: "Mortals wander two-headed")
Aisthesis: perception, awareness.
Pistis: belief
Doxa: opinion
Episteme: understanding (standing around, or around the pillar), scientific reasoning, from facts with an account from principles. nb Theaetetus' initial stab at defining Episteme: "as far as I can see at present, episteme is nothing other than aisthesis" - i.e., all knowledge is rooted in perception or all knowledge is a kind of perception, depending on whether you're empiricist.
Most interesting is the term Sophrosyne, which is untranslatable. Sometimes wisdom, sometimes charity, later chastity. It's the basis for the Delphic maxim, "Know thyself". The ability to stand your post, to know what you know and what you don't know. Exemplified by Socrates in the calmness of his retreat at the loss of Potidea, where he saved others by not losing his head (by contrast to the virtue of courage, the ability to move forward notwithstanding danger and fear). If you want to investigate why we don't privilege wisdom, you could start by seeing why Sophrosyne cannot be translated.
Worth mentioning is Parmenides' (much earlier) idea that the philosopher is the person who knows his way through every town. In an era when the Mediterranean world was transformed by openness to trade and other societies, as grounded in the religious obligation to welcome strangers, and when Greeks defined themselves in part through the legend of Odysseus wandering before he returned home after war (another Sophrosyne story), it's somewhat appropriate to our own era.
Theaetetus (who produced a mathematical proof of irrational numbers) gave his definition after feeling completely lost. Socrates replied by saying all philosophy begins in wonder. So, is having a question, or starving for wisdom, is a kind of hunger, "resolved" with knowledge, so all we have to do is define our terms?
After Socrates took the hemlock and lay dying, Plato describes his death rather graphically, as his feet getting firm, then his legs, his body, etc until he was fully fixed. The language used is exactly that used for "defining" terms, suggesting that the process of definition itself is a kind of death. It certainly kills wonder :)
Understanding is far more useful, but (I think) less valued.
Knowledge can be gamed, you can know things that simply aren't true.
Understanding can be used not only to extract new knowlegde from a corpus of data, but also as a means to validate things presented as fact.
Which I tend to politely respond to with "You want insights. Unless you have a means to effectively extract those from all that data we can give you, that data is just a liability for you."
Most data is noise. Finding the needle in the haystacks (or the patterns that are actually the needles more likely) is. where all the value is.
If everything comes down to incentives at the end of the day, it defeats any sense of morality. Perhaps that's the author's point here. The wise scientist/engineer does what they believe is right and the rest is history. In their attempt for doing what's right, they may accidentally create their best work.
This overall reminds me of the search for the philosopher's stone. Where many ended up finding "philosophical gold(wisdom)" through the process of trying to find it and lead to their own individual magnum opus(great work).
We are drowning in data while starving for information!!
We have signal; which only becomes information when you successfully decode it.
Which raises questions: eg, would we be better off sampling less and analyzing more — to extract a better portion of information from the already captured signal?
But the confusion itself also points to a lack of wisdom — lots to think about.
For an interesting perspective the physicist Murray Gell-Mann spoke on similar issues in a 1997 interview:
Apparently they took it offline after seeing that it generates fake information: https://www.vice.com/en/article/3adyw9/facebook-pulls-its-ne...
Second thing, the repetition process. It is very important that we are exposed to powerful ideas, concepts SEVERAL times. And this is something that is not yet properly established in our everyday life. Advertising as a repetition we suffer from should be superseded by proper repetition of more enpowering activities.
There's tons of wisdom out there, but the more widely applicable it is (across different people/experiences), the more abstract or general it must to be and is thus not directly applicable.
Specific wisdom (like for a highly constrained problem) is awesome, but you and your life are quite unique so this is very hard to find or probably doesn't exist. (Should I move away from home for a job? Well.. that depends on a thousand factors AND your unique self)
Trying to optimize for wisdom is a little like optimizing for meaning, I think. Adopting it from others can be counter productive.
This of course does not solve any of the underlying causes (incentives structure..) or the "real" wisdom scarcity itself. How do you solve these in a society where quantity reigns supreme? You can't.
Kong Fu Zi say:
"By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest."
Drowning in information? Youre gaining wisdom the bitter wa ;-)
Starving for wisdom? You may be thinking it is too easy to find wisdom. Try a bit harder. Wisdom might be right behind you when you are focused on how bad something in front of you may seem.
It's intelligent to know how to start a fire and wisdom to know where NOT to start one.
We've got enough intelligence and enough wisdom. What we also have is collective, weaponized purposeful ignorance. People who know better but believe that by banding together with other people, they can force society to abandon both their intelligence and their wisdom.
E.g. lets say 50% of gun owners accidentally shoot themselves. Now lets say Im talking to someone about how I want to buy a gun for home defense. They tell me "well 50% of gun owners accidentally shoot themselves, so you're safer if you dont have a gun"
This is a gross misunderstanding if statistics. There is no way whatsoever to take the observed general trend, and apply it to a future possibioity in a single instance. What this interpretation misses is that there are variables well within your control. A gun owner who follows all safety rules will virtually never shoot themselves accidentally. Turns out 50% of gun owners are stupid as hell, and you are in control of yiur own outcomes.
I see this problem CONSTANTLY in debates online, on the news, etc. It is very frustrating and contributes greatly to propogating dumb ideas.
You go from a stable basin to an astable mountaintop basin where you are responsible for the maintenance of the banks.
And they're both used and useful much of the time. The field of risk management and industry of insurance are based on it.
Your frustrations are not uncommon. They are, however, rather poorly-founded.
They're an excellent illustration failing to get a handle on this safety thing (where "safety" is the inverse of "risk", as one of yesterday's submissions addresses:
<https://news.ycombinator.com/item?id=34036978>
There's also the philosophical question around the "sea battle" question, in which the truth of a statement about a future possibility ("there will be a sea battle tomorrow") is assessed.
<https://en.wikipedia.org/wiki/Problem_of_future_contingents#...>
It is true that fifty years ago many people in Denver (and no doubt not only Denver) regarded New York as the embodiment of The Other and were very quick to pick up on a New York accent. But this was silly.
They have shown us a graph showing more people are using the site.
I have asked for the same chart to include a metric showing that more people are using the site and there are less overdoses or medical interventions or another metric showing more people using the site results in some sort of quantifiable metric of community benefit.
The public health authority who provides the graphs is unable or unwilling to provide the information and charts I want.
I thus worry that data is selectively used to prove a predetermined narrative.
On the other hand, you can use stats for personal gain. For instance, prediction markets. That might have an impact.
Wikipedia is the closest thing to what you describe, and they have to petition for donations yearly.
But the problem is, did you ask the right question (for it to be a useful answer)?
Wisdom: What facts are missing and why do these facts matter?
No matter what we come up with, we still have to seek wisdom from the wise. The internet (AI powered or not) will never be wise.
This gives me hope for humanity, and makes me love God more.
Remember when everyone was all about "the wisdom of crowds," when talking about social media? Yeah, that isn't something anyone says or thinks about now. How quickly did that meme die? And if AI is built from that data, by definition, it cannot be wise.
As Solomon realized, wisdom is a very powerful virtue.
This web site is sort of the epitome of that way of thinking. It's diverse members "vote" on topics and bring them to the front page (or keep them on the front page if they are put there for reasons). And it suffices for a lot of things, but it doesn't really do the job that an editorial staff could do. There is no way to distinguish between votes up or down because the reader agrees or disagrees with the politics of a piece, or the merits. Everyone has their own internal metric for what deserves "up" votes and what deserves "down" votes. The aggregate of those metrics may rarely converge on a consistent notion of "good" or "not good."
At the same time, there is no straight forward way to monetize curation in order to pay the curator(s). Historically that has been via subscriptions to publications, but the resistance to paywalls and intrusive advertising (both mechanisms in use on today's network) have reduced the effectiveness of those techniques tremendously. Worse, the term "blog spam" is a pejorative for a reason, the blog owner, wrapping something they felt was worth reading in some prose and advertisements, do not compensate the author, and for many provide insufficient value to justify their attempts at monetization.
Getting back to the article's main point however, for specialized topics, finding that curation is difficult, getting paid to facilitate it more so. And that means that quality information (wisdom) is drowned in the sea of noise (hot takes).
My guess is that the Overton Window for paying for the work someone has done to distill the wisdom out of the signal has to move from the "radical" to the "acceptable" before we'll see much progress here.
"Is top round tender?
Top round tender is a dish made from the top of a chicken’s head. The name comes from the fact that it is usually served at the peak of a chicken’s life, when its flesh is soft and pink. It is also called an “enormous” or “giardine” bird."
They'll be the new Vice and Vox, but run entirely by AI
https://www.msn.com/en-za/news/other/ai-writing-is-here-and-...
- Reddit (and HN) are bad for "wisdom" because they are deliberately violating netiquette : once in a while, starting a new thread is wrong, and necroposting is right, and reddit prevents this by locking the thread.