rademacher

Moral of the story, the key to happiness is low expectations.

I haven't read this paper yet so I can't speak to it's quality but it appears to be addressing the same questions in this post. Bengio is a coauthor so maybe that's a good sign . Here's the abstract.

This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning. Based on theoretical observations, we propose new open problems and discuss the limitations of our results.

https://arxiv.org/abs/1710.05468

I suggest this reference [1] for anyone needing to look something up.

[1] https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

The problem is high dimensions knowing the distribution or even characterizing it fully with data is incredibly difficult (curse of dimensionality). I think the real assumption in ML is just that there is some low dimensional space that characterizes the data well and ML algorithms find these directions where the data is constant.

Julia only hit v1.0 in 2018. It's doing pretty well for being so young in my opinion.

All the arguments against Julia are basically that python has a lot of momentum and it takes time and effort to switch to a new language. I think Julia should really seek to displace MATLAB as a near term goal .

Julia has some packages to address this now.

https://github.com/timholy/Revise.jl

SAR has been around since the 70s and works using the same principles as CT scans (projection slice theorem). Think about observing an image by rotating it and projecting it orthogonally to the rotated direction. Take a bunch of these measurements (basically a radon transform) and then you can invert the process using the back projection algorithm.

The resolution of the image is proportional to the bandwidth of the waveform and the distance traversed by the satellite during the collection process.

This article is a bit of an exaggeration and Capella is certainly not the first or only SAR service.

Typically when you downdsample you're going to want to filter than use whatever downdsample kernel you want with the correct stride. Since the filter is lowpass, think just Fourier transform then taking an inner smaller square of the image and inverting, then you can embed the poison image only in that frequency spectrum. Now by playing with the power, if we downdsample by a factor of 4 then just assume that we lose a quarter of the power in the original image while the poison image loses no power. So right off the bat, we are scaling up the poison image power by a factor of the downsampling ratio. For example, we might go from 1/4 power in the poison image relative to the true image then to equivalent power. The other aspect would be if the interpolation kernel and strides are known we can just make sure that the poison image has large values at those specific pixels and further increase the gain.

I think this suffices as a summary, "The other reality is the frustration and drudgery of operating in a world of corporate politics, bureaucracy, envy and greed— a world so depressing, that many people quit in frustration, never to come back."

Some of us just aren't cut out to work in a big corporate environment. From what I've seen, large technical companies are made up of two sets, the technical set and the manager/business set. Unfortunately, it seems that the manager set yields a disproportionate amount of influence and power and therefore is "valued" more. I'm sure there are smaller companies that could make the folks leaving stick around the industry. But, if they've been successful and are mid career they may have priced themselves out of those opportunities.

I was expecting this to be more about writing fast matrix multiplies. Surprisingly, it ends up being a lot more than just a triple loop and there's a reason that everyone uses existing libraries like OpenBLAS and MKL [1].

[1] https://www.cs.utexas.edu/users/flame/laff/pfhp/

They use wavelet transforms for jpeg 2000. Natural images tend to be sparse with respect to wavelet transforms.

I think this is really the tale of two Millennials and is a phenomenon that actually effects all generations currently. There is a concentration of wealth in this country and the wage increases in the bottom 90% have not kept up with the cost of goods limiting the purchasing power of this group. This issue is exacerbated by the increasing costs of goods and education leading to higher levels of debt for Millennials.

Anecdotal evidence shows that a lot of new grads with technical degrees are getting offers around $200k in the bay area and seattle (these are the other Millennials). By 5 years they're making more than many boomers did over a very successful 30+ year career. Clearly this is only a small subset of Millennials, but it is an example of concentration. I'm not sure whether this discrepancy between "classes" of new grads existed previously, does anyone have insight?

This article may have been on here yesterday and seems relevant [1]. For this guy it's probably not about fighting for what he believes is "right", it's all about winning. He bought land, a fight ensued, and now he relishes the opportunity to crush his opponent (average people).

[1] https://www.theatlantic.com/family/archive/2018/12/rich-peop...

The author does make a point of discussing the question of what business does a team like DeepMind have researching the folding problem? The solution is of no apparent value to the parent company Alphabet, and yet they were still funded. Perhaps this has to do with the attitudes or values of "modern" tech companies? Historically, there seems to have been a cyclically nature to the volume of basic research in industry, peaking with Bell Labs, sinking with the rise of Welch, and now coming back with the Googs and Facebooks.

Isn't there utility in accepting the null hypothesis? It's almost as valuable to know that there is no signal in the data as there is in the opposite, i.e., knowing where not to look for information.

I think your example is really justifying a "machine learner" that has some domain expertise and doesn't blindly apply algorithms to some array of numbers.

Here is a reference for the comment above and a brief excerpt,

"Written in the productivity language Julia, the Celeste project—which aims to catalogue all of the telescope data for the stars and galaxies in in the visible universe—demonstrated the first Julia application to exceed 1 PF/s of double-precision floating-point performance (specifically 1.54 PF/s)." [1]

[1] https://www.nextplatform.com/2017/11/28/julia-language-deliv...

I used to only use MATLAB, which for a lot of research code applications is actually nice for getting a prototype running quickly. Now that I have the freedom to choose, I typically use Julia as I'm trying to gain skills in opensource languages that are actually valuable in the job market. The choice of Julia over python is probably due to my nature of going against the grain, which flies in the face of my previous point.

Was there anything specific that turned you off from Julia?

Looks pretty nice. Unfortunately, I never run into this issue because I only get to write research code.

The Julia language was designed to target the two language problem and at least from these benchmarks it looks pretty competitive [1]. I imagine over time, pythran may fix some limitations and beat Julia in most benchmarks.

[1] https://github.com/fluiddyn/BenchmarksPythonJuliaAndCo/tree/...

All good points. It would be nice to see some game theory type assessment of these imbalances.

So he's saying that concentration of "wealth" is bad, and war is bad.

A quick glance at the definition of moral gives, "a person's standards of behavior or beliefs concerning what is and is not acceptable for them to do" which suggests that they may be fluid. Are killer robots necessarily any less moral than killer humans? We seek to replace humans with "robots" in many cases under the assumption that they perform better. I suppose in the case of killer robots this could mean more effective killing, or perhaps it could mean more accurate strikes and less civilian casualties? (I'm not saying I am an advocate for military AI, just posing some questions).

Finally, suggesting that we need to focus less on incremental progress when DL still isn't completely understood seems premature. I'm not sure another great leap in AI is on the horizon until a leap in computational power or a new framework is discovered.

These types of experiments pop up from time to time. You have to compare the performance relative to the single lag error. So the baseline is to use the previous time step to predict the current time step. Ultimately, it's not worthwhile to attempt to predict stock timeseries the SNR is far too low. It's more valuable to attempt to predict a trend, e.g., +1 is there will be upward movement in time window x, -1 if downward, 0 is within some epsilon of the current price.

DRL has definitely shown some excellent results but can someone in the field comment on this paper, Simple random search provides a competitive approach to reinforcement learning [1].

[1] https://arxiv.org/abs/1803.07055

This latest craze of "AI" research seems to be fueled by a sudden glut of computational power (GPUs) that wasn't available previously. I think that most technical people would agree that the mid 2020s is extremely ambitious. I'd also argue that we're actually more likely to experience another AI winter.

The frightening part of the current deep learning research is how susceptible they are to adversarial attacks. Adding small amounts of noise causes misclassification in images, and some papers even explore the inevitability of adversarial examples [1]. This is especially frightening given the amount of autonomous vehicle work being done. I could imagine a situation in which the sensor noise varies just enough to cause such an error. Obviously, the systems will have redundancies built in, but I'm convinced the self-driving cars are still a ways off as well.

EDIT: As others, have stated just adding noise is not enough and it is often used to generalize the model. The paper does discuss that the perturbations can be incredibly small to cause this deviation and that the set of such deviations may be larger than expected especially for complex images.

Regarding the AI winter, I suppose I should have defined it as a reduction in the amount of research and the extent of the progress being made in the area rather than the utility of such research.

[1] https://arxiv.org/abs/1809.02104

I wasn't implying that the interview was easier for candidates with recognizable track records. Perhaps two sigma was not restrictive enough. I was merely suggesting that if a candidate has domain expertise and it is well known, that they may not subjected to the same leetcode hoop jumping as the bulk.

Do you really want to remove bias for experienced candidates with a track record of substantial and nontrivial contributions?

All anecdotal evidence points to arriving at the correct, optimal solution as being key to passing the interview. It also seems like often times the interviewers are not even going to be working with the candidate so their opinion on a 'working relationship' is mostly irrelevant.

The objective of asking these leetcode style questions is to find candidates who are willing to put in the time to study. Success signals that this person is willing to commit to performing well at something that is reasonably challenging. The end result is that they are trying to hire worker bees. This makes sense as the bulk of work at any large company is largely mundane and relatively routine. I'm willing to bet that when a company wants to hire a two sigma candidate they don't go through all this nonsense, although at that point the candidate is already well known in the industry most likely.

As a fan of technical death metal and a guitarist it is one of the few genres that has the depth and complexity to keep me interested. This is also why I enjoy music from Sibelius and other classical greats. In contrast, most pop music is so stock that I can guess the next notes in a sequence and it just becomes dull to listen to.

Also, as someone often described as stoic, metal has a enough energy to actually get me excited and feeling. When I listen to it, I get more of a response than say listening to DMB. I wonder if this is related to being an introvert, a lot of metal heads seem like they like to listen to music alone and get energized from solitude, where as a pop music fan may enjoy listening with friends.

Wow. I'm surprised to see this on here. I used to work Johnson, he's is a super nice guy and always made time to have long conversations with me about optimization.

I believe this algorithm is widely used in the video game industry?

But your estimate isn't very meaningful if you only have enough samples such that the N v (N-1) normalization matters, the variances on the estimator will be too large. The 1/N normalization comes from the MLE BTW, I'm sure most know this already. As an aside, to see the power of a biased estimator look at the James-Stein Estimator [1].

[1] https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator

Seems to me that at some point hiring moved away from hiring based on potential to hiring based on pure skill and how many boxes the candidate has checked. Companies seem to just want people who can turn the crank, which doesn't sound all that appealing.

Why penalize folks who didn't know what they wanted to do in high school. It took me a long time to get a vague idea of what I wanted to do for a career and it's something I have to constantly contend with. In fact, I'm still trying to pinpoint what exactly I'm looking for. I know things have changed in the last few years, to the point where high school kids are in pre-engineering or pre-cs programs and getting special instruction.

As an aside, the advantages of knowing what you want early are huge assuming that you continue down that path for a long while. If we consider something like graduate school, knowing as a freshmen that you want a PhD means growing your network early, cozying up to professors to write recs, doing REUs. You could have someone figure out towards the end of undergrad that they want to continue on and really struggle to put together a compelling package even though they might be just as passionate or have just as much potential.

Sorry that was kind of a rant.