604 karma · joined June 18, 2020
Do we know that this is due to inherent superiority of DNNs, vs just experiencing a virtuous cycle of success leading to increasing investment leading to more success?
I keep seeing this claim pop up but I've never seen a citation to support it. My understanding is that e.g. statistical learning theory lets you prove theorems of the form "IF it has this number of parameters or fewer, THEN it won't overfit". First year logic is enough to know this theorem tells you nothing in the scenario where the model has lots of parameters. Can anyone provide a solid citation to support the claim that models with many parameters "should" overfit?
BTW, another example of a model with lots of parameters which doesn't overfit is random forests / gradient boosting. I'm wondering if this "should overfit" claim is less of a global property of machine learning models in general, and more just a property of the particular models that this set of researchers have most familiarity with (SVMs).
It's been a while since I read her work, but IIRC one of the positions she argues for, which I find plausible, is that interpretable models can be performance competitive. For example, it could be that the only reason black box methods outperform is because they've been more heavily researched, and if we were to put more research into interpretable methods, we could achieve parity. I also mentioned a few reasons why we might expect interpretable models to perform better a priori in this comment https://news.ycombinator.com/item?id=28838321
"The machine learning model is a two-layer additive risk model, which resembles a two-layer neural network, but is decomposable into subscales. In this model, each node in the first (hidden) layer represents a meaningful subscale model, and all of the nonlinearities are transparent. Our online visualization tool allows exploration of this model, showing precisely how it came to its conclusion. We provide three types of explanations that are simpler than, but consistent with, the global model: case-based reasoning explanations that use neighboring past cases, a set of features that were the most important for the model’s prediction, and summary-explanations that provide a customized sparse explanation for any particular lending decision made by the model."
I was curious about the customized sparse explanation. It looks like there is an illustrative example from later in the paper:
"For all 700 (7.1%) people where:
• ExternalRiskEstimate ≤ 63 , and
• NetFractionRevolvingBurden ≥ 73,
the global model predicts a high risk of default."
"A rule returned by OptConsistentRule is globally-consistent, in the sense that there exists no previous case that satisfies the conditions in the rule but is predicted differently, by the global model, from what is stated in the rule. In contrast, explanations (from other methods) that are not consistent may hold for one customer but not for another, which could eventually jeopardize trust (e.g., “That other person also satisfied the rule but he wasn’t denied a loan, like I was!”)"
You can see the online visualization tool her team built here: http://dukedatasciencefico.cs.duke.edu/models/
In retrospect, it's not all that surprising to me that a model such as this is able to outperform a black box like a neural network. For example, one of the things this model does which black box models don't do is enforce "monotonicity constraints" which ensure that as risk factors increase, the estimated risk should also increase. It makes sense that this would be a useful inductive bias which improves generalization performance -- if a black box model found that an increase in risk factors decreased estimated risk, it seems likely that this would be a result of overfitting (or multicollinearity gone haywire).
Of course another reason to expect simple/interpretable models to generalize better is Occam's Razor.
My big question about this sort of approach would be whether it's able to extend to the sort of unstructured data problems that deep learning has done really well on. It looks like some of her recent papers on Google Scholar address this: https://scholar.google.com/citations?hl=en&user=mezKJyoAAAAJ... (specifically thinking of the BacHMMachine paper and the Interpretable Mammographic Image Classification paper). Maybe someone else can summarize them.
https://www.amazon.com/War-What-Good-Conflict-Civilization/d...
The post-WWII era, which coincides with the era of American dominance, has been a fairly peaceful period by historical standards. It's hard to argue with results.
https://www.youtube.com/watch?v=m6pJd6O_NT0
I don't think Putin is fully honest, but ignoring what he says doesn't seem like a good idea either, and I found the interview fairly enlightening.
I think an important clue here is that all the authoritarian regimes (Russia, Belarus, China, Iran, North Korea, etc.) seem much more likely to form alliances with one another even in the absence of an obvious shared national interest.
An interesting case study may be the recent secular coup attempt in Turkey. Secular values are closer to western values, and according to this editorial in Turkish state news, when given the chance to come out in favor of democracy and political stability (by opposing the coup) or western values (by not opposing it), Europe chose the latter
https://www.aa.com.tr/en/analysis/opinion-the-july-15-coup-h...
Basically I see the modern geopolitical divide as being countries focused on human rights and secular western values on the one hand (US and its allies) and countries focused on stability on the other hand (Russia/China and friends). Both positions have some merit in my view. For example, I think Putin makes a good point that the US screwed up in Libya--we were aiming to achieve human rights but we ended up just destabilizing the region. IMO there has to be a balance between both human rights and stability considerations.
Chomsky is fighting the last war when he talks about US actions in past decades in South America. Back then the global divide was capitalism vs communism. Now it's about human rights vs stability IMO.
Maybe a good metric would be the ratio of primary ballots cast to general election ballots cast. If a small number of extremist primary voters are de facto selecting the candidate that will win the general and get the "safe seat", that suggests an opportunity for disruption by being a high profile 3rd party candidate who's just a bit more moderate than candidates the extremists select (more appealing to the median voter).
Another idea is to reach out to a bunch of independent candidates and try to recruit them for your new party to work towards critical mass.
https://www.politico.com/news/magazine/2021/10/03/andrew-yan...
In any case, he is right about the duopoly. In order to overcome the duopoly, we need the involvement of major public figures. I haven't seen many volunteer--I'll take what I can get. I would prefer he advocate for approval voting though: https://electionscience.org/
In my observation the default state of a student reading a textbook is it goes in one ear and out the other. Most students temporarily acquire a superficial understanding of the concepts which allows them to answer test questions and get a decent grade. To see something in the wild and instantly recognize that it's isomorphic to a concept you studied years ago requires a level of mastery/passion well beyond what it takes to get an A. (I'm talking about school in general here, of course the fact that interviews index so heavily on data structures/algorithms ends up distorting things a lot from the baseline. Still, if you solve this problem in 10 minutes you're one helluva sophomore.)
>Also, working at a "hard startup" has nothing to do with "advancing the state of the art in CS".
I think of "hard technology" like rethinkdb as being exactly equivalent to cutting edge stuff that advances the state of the art in some way... again, maybe there's just been a misunderstanding/miscommunication here
If you don't agree with me that the fluency I described is a significant asset to advancing the state of the art in CS, what do you think a significant asset is?
One idea here is to give candidates an open problem in CS and ask them to solve it. Which is a nice idea, but one of the other things I learned through administering hundreds of interviews is that open-ended questions aren't great interview problems because they depend a lot on creativity/lateral thinking/insight, which highly benefits from being in a relaxed frame of mind. So these questions end up being a test of how relaxed the candidate is.
Stress creates tunnel vision, which is terrible for generating interesting research ideas but it's OK for solving these kind of leetcode problems. So checking for a very high level of programming aptitude, as a proxy for CS research ability, is an approach which is a bit more fair to candidates who are stressed out by interviews. (I also think it is a pretty decent proxy, because doing great CS research requires you to quickly & fluidly generate & evaluate algorithms / data structures which might solve your problem, which is a big part of what a great leetcoder does.)
If you're targeting a very high level of generic programming aptitude, it is arguably most fair to make use of a standard method of measuring it. Leetcode problems are the industry standard for measuring programming aptitude. People know to practice them a lot and they know what to expect. If you came up with your own unique way to measure programming aptitude, that would create an even greater burden on candidates to practice (they'd have to do a whole different sort of practice in order to succeed in your interview), and also create anxiety due to an unexpected interview format.
I think a lot of readers are overreacting because they didn't pay enough attention to this bit: "It's applicable if you're building an extraordinary team at a hard technology startup." The vast majority of companies in SV are not doing hard technology and don't need an extraordinary team. People should not feel inadequate if they aren't capable of improving the state of the art in technical areas of CS such as databases. This is ordinarily the domain of PhDs, and filtering for demonstrated algorithmic aptitude (as opposed to academic credentials) is actually a pretty egalitarian approach.
I've given hundreds of coding interviews. I always timed the candidate (with their knowledge). So I have a general idea of the distribution of programming speed, and my guess for this particular problem is to solve it in 10 minutes, you need both talent and practice... just one isn't enough. So asking people to solve it in 10 minutes ends up filtering out untalented people who have practiced a lot (and talented people who haven't practiced as much)
Serious question... how did these kind of fifth grade playground insults become normalized on HN? It's a blatant violation of the guidelines https://news.ycombinator.com/newsguidelines.html and yet somehow comments like this get upvoted.
Supposing I just left spacex, what funders would I reach out to who would give me money to "search for an idea"?
Anyone know what the ~5 types are?
What does fusion give us that existing nuclear power plant tech doesn't?
My intuition is that a city will be easiest to defend from fire if its area-to-perimeter ratio is low, i.e. it is big. For example, there is no real risk of San Jose burning down in a fire, because its area (square of the radius) means it has lots of water mains and firefighters, while its perimeter (linear in the radius) is comparatively defendable. That suggests the strategy of expanding existing forest cities.
Maybe we could chop the trees down and build homes out of them? Could help w/ California's homelessness problem...