upquark

Much more likely just bad data (astronomically more likely)

Hmm, let me see how I can explain this more clearly. "I'm not arrogant enough to do X" implies "Those who do X are sufficiently arrogant". Do you follow this far?

Here's an example relating to your experience. I could claim: "I'm not arrogant enough to use RoR for serious web development".

On the surface, I'm making a claim about myself / my beliefs. You shouldn't feel upset over my statement, it's not about you or any other people...

Just say the safe word

You're making a broad statement about a bunch of people without knowing what they know or don't, and one might say that statement is insulting. Right after that, you're criticizing a guy for making blanket statements about a bunch of people.

Saying "I'm not arrogant enough to have a strong opinion either way" about any statement A is itself a pretty arrogant thing to say (you are making a statement about the collective knowledge of everyone else on the topic of A not being too far from yours).

> The better explanation, of course, is that consciousness somehow is the universe...

What is love? How can an uncaring chunk of bones and flesh feel love? But we do fall in love, so it must exist in some form. Hence, love is the universe itself. Checkmate

> I'm not arrogant enough to have a strong opinion either way > ...you start broadly ascribing motivations to an entire field...

Do you see a contradiction? You're broadly calling a bunch of people "arrogant", who may have a very well informed and valid opinion on a topic.

The space here is the space of probability distributions (each point is a distribution), not the sample space. So, no real connection with PCA.

Possibly naive question:

I'm a married guy in his 30s working at a major investment bank in NYC (lead quantitative role). I have a graduate degree from a non-elite school. No experience at top tech firms (had offers but didn't go).

Now, I have some cool side projects that I believe could become great consumer apps given enough care and resources. Is there any reason for me to apply to YC, and any chance of getting seed funding from you, or am I completely out of your profile for typical candidates / startups?

"Knowing everything" implies far more than our intuitions may initially tell us, and can lead to "a set of all sets" type of paradoxical problems (i.e. the premise of the argument is absurd). This and other philosophical thought experiments like the Chinese room abuse our intuitions about being human, learning things, performing tasks, etc.

In this particular experiment, if Mary's omniscience allows her to invoke the specific pattern of neural activation that is triggered by observing a blue sky, then she won't learn anything new, otherwise she will. It all depends how far we take the absurd claim of her omniscience.

What exactly do you mean by exponentially distributed randomness? LDA is typically applied to textual data, so the application would be to e.g. automatically tag/classify financial news. I don't think the prospects of ML are as grim in this space.

Lots of ignorance in the article and some of the comments regarding data-driven decision making. Does the author realize how much of the civilization around us is built and guided by data-driven decisions? Also, people/companies not being able to effectively use their measurements to their advantage is not really evidence against the idea itself. When your model doesn't work, it's not modeling that's broken, it's just your model.

Gut reactions can take us only so far: they break down as we move away from single human-scale familiar problems (ones that the brain has some built-in, evolved capacity of handling, such as reading other people's facial expressions).

> I have yet to meet a serious AI researcher who worries about AI ending the human race.

As an industry practitioner of machine learning / data science, I believe AGI poses a genuine risk to humanity.

Having said that, what I do for a living is of little relevance. People are pretty terrible at predicting the future (see late 19th / early 20th century predictions of year 2000, with food replaced by pills, etc). Unless someone has put enough thought and research into it, their predictions about the future of civilization are likely to be worthless regardless of their academic credentials.

I wouldn't describe this as a brainteaser. It's a well-defined math problem, not some BS question about manhole covers or barbers in Chicago.

I might be in the minority here, but I think saying "any theory of physics should explain consciousness" is exactly like saying "any theory of physics should explain love" (or replace love with patriotism, friendship, capitalism, any other human concept). I don't see why consciousness is special: in my opinion it is an abstraction several levels removed from fundamental physics. In other words, to a neutrino flying through you, you are only barely distinguishable from empty space, and your consciousness (and love of music, appreciation of art, etc) means nothing.

> I don't have to do any fancy interferometry or statistical inference to be aware of the cat

You do though, you're just not consciously aware of it.

Thank you for putting this together, looks like a great collection of resources.

The list of topics at the end isn't necessarily "very easy to learn", but I do agree that fundamentals are essential.

Depending on the mathematical background of the individual, they may need calculus and general mathematical maturity / ability to solve non-trivial math problems before venturing into the data science field.

Also databases. Absolutely need this in the fundamentals, arguably more so than graph algos etc.

Edit: English

What about this? http://www.cdc.gov/parasites/toxoplasmosis/epi.html

Animal-to-human (zoonotic) transmission section: "Cats play an important role in the spread of toxoplasmosis"

Wolves and dogs are the same species too, AFAIK

You are taking an extreme stance on the topic. There's room for modeling and forecasting even in highly complex dynamical systems, the models are getting better with advances in applied statistics research. They are not perfect, but they are not completely useless either.

For example, here's an article I just found on modeling stochastic nonlinear dynamics in ecological/oceanographic applications: http://arxiv.org/pdf/1211.1717.pdf

As for caterpillars predicting climate patterns, I agree with you that it's unlikely, unless there are some simple environmental indicators the caterpillars learnt/evolved to take into account but we humans haven't paid attention to yet. This doesn't mean the changes are unknowable.

Some academic types have no clue how far behind the rest of humanity is in terms of intellectual and technological readiness to use such tools. It could be an honest mistake and not a deliberate lie.

- Most trivially, your prior distribution can assign equal probabilities to different outcomes, if you have no reason to do otherwise. Beta(1, 1) for modeling a coin's bias, for example, if you have no prior information about its bias.

- There are more advanced tools in Bayesian analysis such as Jeffreys prior (known as uninformative priors, look it up).

- As was mentioned in other responses the same "big problems" exist in every other statistical and mathematical modeling approach, namely that you have to make assumptions and your results are going to be crap if your assumptions are crap.

- Generally, Bayesian stats got a late start due to high computational resource costs, not some theoretical limitations. The issue with priors that gets repeated by philosophers and some statisticians does not stop the huge, monumental progress Bayesian statistics has had in a ton of applied fields, from computer science / machine learning all the way to economics and political science.

Edit: language

> The Greeks, Jews and Armenians left a long time ago

"Left" is an understatement: http://en.wikipedia.org/wiki/List_of_massacres_in_Turkey

> Still I wouldn't qualify Istanbul as a multi-cultural city.

I have this theory that social entities, be it countries, ethnicities, or other types of groups, end up with slogans and myths that are designed to counteract the reality about the group. What with America's "land of freedom" and Russia's "wide and generous Russian soul". Similarly, Turkey's "multiculturalism and tolerance". It's part marketing, part self-delusion therapy on the social entity's part to keep it from disintegrating...

The actual problem I described above (bulk de-duplication of records without any direct clues, such as unique ids, to decide whether two records are duplicates) falls under this category.

The first paper I provided is a survey of commonly used methods in this space, it's the exact opposite of a lot of formulas without the context.

Nope, there's plenty of math and stats involved in solving this problem correctly.

The amount of math you would apply here depends on your familiarity with the problem domain (this is where CS education comes into play), and your willingness and ability to work with higher level math and stats (this is where math proves necessary even in mundane software engineering tasks). For the record, even basic relational algebra can be seen as part of applied math, but the rabbit hole goes way deeper.

The best solutions for de-duplication of records without direct clues like unique IDs involve pretty cool applications of probability theory and machine learning, you'd find things like the Expectation-Maximization algorithm as prerequisites for understanding the current research. The foundational solution in this space is Fellegi-Sunter record linkage [1]. More advanced solutions include things like hierarchical Bayesian models [2]. If you dig deeper, especially if you are dealing with large datasets and basic techniques turn out to be too slow, you'd look into locality sensitive hashing and approximate nearest neighbors, which is a very mathematical subject, see for example [3]. In other words, math is encountered in even the most mundane software engineering problems, and you have to know where to look and what to look for.

[1] http://www.purdue.edu/discoverypark/vaccine/assets/pdfs/publ...

[2] http://arxiv.org/pdf/1207.4180.pdf

[3] http://www.cs.princeton.edu/courses/archive/spring05/cos598E...

>Math is totally unnecessary for everyday software development.

You are writing the simplest possible CRUD app for storing / retrieving some client records. You have a database of these records on the back-end, and users can add/edit/remove these records.

Now, your boss asks you to add a feature of bulk insertion of new client records from some other legacy data source, with proper de-duplication. There are no unique ids across the legacy and new data stores.

I hope this qualifies as "everyday software development". Is math totally unnecessary for solving this problem?

Every now and then there's a discussion on HN about usefulness of math in software engineering / data science / economics / or another practical field. Some people claim it's not useful or relevant, and produce some use cases where elementary math is sufficient to solve the problem at hand.

In reality, math is ubiquitous in computing, at every level from hardware design to algorithms and data structures to coding and compression to UI/UX design to graphics to databases to every other field you can think of. People who are not well versed in more advanced math tend to apply simpler, more accessible techniques, and produce sub-optimal results. Which is why we have layers upon layers of broken, barely working, crappy software everywhere we look.

This guy sounds like a complete joke. Standard deviation is an immensely useful tool in stats. It's not a historical accident we use it everywhere, for example it naturally appears in the very useful Chebyshev inequality, and a million other contexts.

> The reason why people can't do that calculation in their heads is because of our reliance on computers and calculators.

No, we were never well equipped to do such computations, we didn't suddenly get worse at it with the invention of calculators. Consciously aware mental calculation is a very recent thing on an evolutionary time scale. I think even 10-20k years ago no one did any mental calculations beyond the most trivial arithmetic operations.

OTOH we have a pretty cool native physics engine in our brains and can solve pretty challenging problems such as predicting trajectories and modeling fluid flow. These skills were much more useful in our evolutionary past than arithmetic or algebra.

Or, it's illustrative that what used to be truly middle class lifestyle can now be afforded only by people in top income percentiles, whereas the median income probably implies a lifestyle pretty close to poverty.